Unpacking the intricacies of building developer tools in a shift left world

At the recent Civo Navigate conference, Emre Baran, co-founder and CEO of Cerbos, delivered a talk on the intricacies of building developer tools in a shift-left world. With a rich background at Google and as a serial entrepreneur, Emre brings a wealth of knowledge and experience to the forefront. In this blog post, we delve into the pivotal moments of his presentation, and explore how Cerbos is pioneering solutions to streamline software authorization and enhance developer productivity.

The shift left approach

Emre began his talk by defining "shift left" — a practice that integrates many operations traditionally done by separate ops teams directly into the hands of developers. This approach not only speeds up the development process but also reduces errors by involving developers early in the lifecycle of software creation.

Challenges in software development

A significant focus of Emre’s presentation was on the "train crash" scenario, a metaphor for the challenges that arise as startups scale. He highlighted how complexity increases with user growth, which often leads to a buildup of technical debt, particularly around user authorization needs.

The importance of developer tools

According to Emre, effective developer tools mitigate the risks associated with scaling and complexity. Cerbos, for instance, is designed to handle the nuances of authorization seamlessly, ensuring that businesses can scale without compromising on security or user access control.

Key principles in building developer tools

Emre outlined six principles essential to the development of effective tools, all of which are at the core of Cerbos:

Simple API: Tools should be straightforward and intuitive, facilitating easy integration and adoption.

Speed: Systems must perform well under various loads, maintaining efficiency as they scale.

Security: Robust security measures are non-negotiable, especially when handling sensitive data.

Extensibility: Tools should be adaptable, allowing developers to customize functionalities to meet specific needs.

Scalability: Tools must grow with the company, handling increased loads without performance degradation.

Reliability: Dependability is key, ensuring tools operate smoothly and consistently.

Embracing open source

Emre advocated for open-source development, which fosters a collaborative environment and enhances tool reliability through community input. Open source not only accelerates innovation but also builds trust and transparency within the developer community.

Conclusion

Emre Baran’s insights at the Civo Navigate conference underscore the critical role that specialized tools like Cerbos play in modern software development. His expert knowledge and firsthand experiences provide valuable lessons in handling the growing pains of software startups and the importance of building robust, scalable, and secure developer tools.

Transcript

Emre: Hello. Thank you for coming. My name is Emre Baran. I'm the co founder and CEO of Cerbos. I'll talk very briefly about Cerbos, but most of what my talk is about, is building developer tools in a shift left world. How and why, like why we built Cerbos, but how we built these developer tools and why do we build these developer tools?

And I would love to walk around, but apparently the microphone is only here, so unfortunately I'm going to have to stay behind this. But so Throughout my career at Google CGI and three of my own startups, developer teams that I worked with have lost countless months building and implementing tools that could have to get, to get a job done.

And those tools directly had nothing to do with the real business problem that we were solving, but they had to be done. And one of those is authorization. So what is authorization? And authorization is the fine grain permissions and access control that we implement in our software. Whenever you have a piece of software that has multiple users in multiple roles, those roles can be need to be implemented.

Those roles need to be enforced. And that's where Cerbos comes in. So it's not authentication. It's not the fact that you're identifying who the user is. It's actually authorization, deciding what a user can do. And this is it. That's all I'll talk about Cerbos. If you want to talk more about how, you know, what we do with Cerbos, we can do that afterwards.

We have a little booth outside. But my, the rest of my talk will be very much so in context of building Cerbos, building an authorization layer so the developers can actually implement. But why we do that and how we went about doing it and what were the different principles that we actually paid attention to.

So can I see a show of hands? How many are software engineers here or software engineer background come from a software? Okay. That's pretty much everyone. Great. So. We are all software developers here, pretty much. What does that mean? That means if there is a problem, our natural reaction is I can solve that.

I can solve that with few lines of codes. However, how often do we actually stop and think about it? And how often is it the right answer to go and write our own code? So always, just because we can, how hard do we actually look around to find a solution to implement and how much do you know, when we're actually building that, how much do we really consider the time and the impact that's going to have on the team, and especially when it comes to maintenance and future maintenance costs.

The answer is, you know, it depends on how mature we are as developers or how well considered we are. Also, how much of a solution actually there is for the actual problem we're trying to solve. But before we go any further in why these, you know, problems usually creep up in a lot of organizations, I want to just take a second on acknowledging the definition of what shift left is.

This is a, I mean, it's off Google. When you type you know, developer lifecycle, it's one of those things that come up. It shows a cycle that developers and ops folks go through in order to build software, deploy it, and run it. And shift left comes from the fact that actually nowadays with all these different tools, a lot of things that can, that were usually done by ops teams can actually be done by developers.

That's the definition of it. And we'll talk about this a little bit more more in a little while, but I first want to talk about a train crash, a slow train crash waiting to happen in every software company and every startup. When growing complexity in software application and architecture meets an increasing requirements of a growing startup, we have a train crash waiting to happen.

And this, you know, this happens because software developers, software companies cannot scale fast enough. So the train number one in that train crash is the users of our software application. Right. These are, you know, they're both the users, the customers users, or it may also be a back end user. Right. And, It always starts very simple when we're building software.

It starts with one or two users. We actually very much so focus on their use case. We actually deliver to their needs and we don't pay much attention to anything else. But as our software succeeds, as that train is actually accelerating. You know, as we get more customers into our products, those customers start getting more complex, our customer support gets more complex, there's no longer a single single prototypical user of our system, they're no longer just, you know, one user role, we need to manage many other stakeholders, and suddenly we have to support a whole company using the software, and All the things that we have done for one or two users are now need to scale up, and we start actually accumulating tech debt when it comes to the user base.

And when the other, when we look at the other train, we can actually take a look at our technology stack. Our technology stack starts very simple. You know, when you're building an MVP application, you just think about what you're going to use as your database, what you're, what's going to be your ORM if you're going to use one, what's going to be the business logic and what's, what are we going to do in the UI?

The goal is building a solution that solves the problem of the users. We don't care about anything else. We have a very simple architecture, a single language it's monolithic application probably. Everything is embedded, libraries can be shared and because The focus is very much so making our secret sauce work for the business case we're trying to solve.

However, this train starts accelerating as we as we get more successful, our secret sauce starts working. Our technology now turns into a product. Our product turns into a business. And the natural thing happens that one monolith system turns into a massive, you know, microservices architecture. This is an example from Uber.

This is it's a representation of Uber's, you know, umpteen thousand microservices and how they actually interact with each other. What happens is, so now we have to, we have a, in this architecture, we have now a backend team, we have a front end team, we have a data science team. They have, we have an AI team, data team, multiple product teams, and.

As the leaders of software companies, software organizations, what we also want to do is empower each one of those teams. We want to actually give them, enable them with the languages that they want to use. If the data team wants to use, you know, very Python and somebody else wants to use JavaScript for the front end, somebody else wants to use Go, we actually empower them and enable them.

But this suddenly starts turning into that second train accelerating. And suddenly when there isn't, there aren't enough resources available in the company, these trains actually crash into each other. Growth dictates that this will happen. And growth dictates that this will have, when it rains, it will actually pour.

And in order to not to do this, as software developers, we try to actually cut corners. It's like, as software developers, as the company is scaling fast, we start cutting corners. Right. We start making sure that things are, are going to be done. Suddenly we start going into writing tools and trying to make things work so that crashes avoided.

And there we go. And that's going back to, I can build that feeling. Actually comes from a good place because we were all trying to avoid that train crash. But, You know, instead of building those things, if we could actually use tools instead of building those things, how life would have been much better.

My life would, it could have been much easier. So why do we build tools? We build tools to avoid that train crash. We, everybody builds tools and especially In, you know, developer tools, companies, we build tools so that we can empower all those users so that they can actually handle their users better, their users growth better.

They can handle their microservices and those microservices growth better. And when building those tools, the main thing to support is making the developer experience as pleasant as possible. Developer experience is the most important thing to satisfy in order to make sure your developer, the build that you're building is going to be used, it's going to be loved, it's going to be adopted, and it's going to be useful, right?

So if we think about this as the roof of what we're trying to build, there are actually six main pillars that we actually paid attention to in order to be able to deliver a great developer experience. So these are basically the pillars that we paid a lot of attention to when we were building Cerbos.

Probably it will apply to 90 percent of other developer tools that are out there that enable shift left. I'm sure we can always argue and talk about what are the different things in different types of developer tools that can be done, but a service when we were building our authorization tooling, we knew that we had to pay attention to every single one of these, because if we missed any of them, that good developer experience.

Was impossible to deliver so let I'm going to go through every single one of them. And I'm going to go through explain why we paid a lot of attention to these things What what what why the each one of these actually relate to delivering a good developer experience? So the very first one up there was a simple API Simple API is win because simplicity in software development always wins Simple.

There are simple, things that are simple to use are adopted easily. Things that are well documented are adopted very easily because it actually is possible. You know, it's a self fulfilling prophecy in a sense that, you know, all the documentation is available right next to the code. The less people who get stuck in using them makes the pull tools that more popular.

And more importantly, predictive and intuitive APIs. get adopted more easily. So I'll give you two examples here, Twilio and Stripe. These two are very successful developer tool, API companies that have very simple APIs. You know, with Twilio, you just, as a developer, you don't have to worry about which mobile network my message needs to be routed over.

You just send, you know, put the phone number, put the message and, you know, Twilio takes care of it, everything in the backend. Stripe, similarly, you don't have to worry about, you know, good old days. We had to deal with a payment gateway, a credit card company, a bank, and had to coordinate all the communication among them.

But, you know, Stripe said, we're going to take care of that. We're going to give you the simplest API you can use, put the credit card number in, and at the end, we're going to tell you whether your payments succeeded or your payment got rejected. And these two tools. Also one because their API on not only was easy to use and simple to understand, but they abstracted that API layer very well.

It was very clear where your business logic ended and where you had to actually send a message. So the Twilio API was very intuitive for people to understand. Okay, now it's time to send a message, I'm going to put my message here, and the only thing this thing will take care of is my message. Back and forth, we're done.

Similarly with Stripe. So when building developer tools, it's not only the tooling to build that we need to pay attention to, but also what's going to be its API that the developers will be interacting with. Where does the handoff happen between one system and the other? Where is that logical line to draw and how simple is going to be that API?

Bye. Second one from that slide is speed. we Need to make sure that our developer tools or whatever we're building needs to be fast. I used to work at Google. Back in the day, Eric Schmidt was a CTO, CEO. And every week, pretty much, I've heard him say speed matters. It does matter. Speed matters. Speed made what Google win quick search results and quality as well.

But in software applications, speed matters. And we live in an expectation economy where all of the user expectations are met. are shrinking when it comes to time to process, time to do anything. So I'll give you this example. This is, you know, a bit dated. I've seen a recent example of this somewhere, but I couldn't find it.

But this is a curve. These are the two different years that Uber did the study. And this is how we behave as humans, as technology improves what we expect. This is our expectation. And these charts, these charts show how long before a car showed up, People actually canceled that ride because they were fed up with it.

They, they, they couldn't wait long enough. And we can see from 2013 to 2014 that, you know, patients threshold shrunk. So what does this mean? It means it shows that as users or as our systems and users. The expectation rules everything. Because if users aren't happy with the speed of your system, they're not using it, and if they're not using it, you're not getting those customers.

You're not getting those users. So I'll leave it at that. Speed matters. The third one is security. Whatever we build for anyone to adopt, it needs to be secure. This is almost a bit of an oxymoron to mention it, but it is a must have, you know, if your tool is not secure is nobody's going to really adopt it.

It's, I mean, it's going to be ditched at some level because we all have some customer data to protect the payment data to protect a medical data to protect. So. If we're not secure, what's the point of having a, a, a tool, a product at all? Like nobody wants a banking application that anybody can log onto and make transactions on your behalf.

Right. And in security, one of the trends that we're seeing zero trust security is becoming the norm. If we go back to, you know, not too far back, back to in 2016, up till 2016, for example I had this conversation with you Kafka didn't even have neither authentication nor encryption, right? Because it was designed to run within the perimeter.

Nobody said, Oh, everybody said Kafka doesn't need that because you're going to put that in your, you know, within your VPN, next to your application. What's the point? It needs to be fast. And that's it. And I believe you mentioned Redis also similarly didn't have anything. So. Again, the end user expectation comes and rules the world here because the expectations are now zero trust security.

And in the world of where we have SOC 2, PCI compliance, HIPAA, FedRAMP. If your tool is not complying with a zero trust security, it's not going to get adopted. Yeah, I'll leave that there.

The fourth one is, so now we're getting into, we've done the fundamentals of speed and and security and simple as a simple API, but as we build developer tools. One of the things that we paid attention to was to make them extensible. What do we mean by extensible? We mean by not just only handling a very simple use case, but also being able to handle much more complex ones.

And if our software cannot out of the box handle it, giving developers ability to be able to extend them so that they can actually take care of their, you know, intricate details of their use case. The examples that come to mind in each one of in this case are, for example, to begin with, are Fivetran or any ETL tool, right?

They cannot know the intricacies and details of every single system. They know the major, you know, data tooling, but sometimes, you know, your data model might be, you know, different. So you need to actually write a custom code to be able to extend the ETL tooling. When we think about earlier, I was talking about Visual Studio, someone we were talking about IDEs.

Yeah, exactly. And Visual Studio is a very good example here as well. It's out of the box works, but then look at the whole ecosystem of plugins and additional things that you can do in customization, you know, way back when. Emacs was a favorite of everyone because it was customizable. It was extensible.

Everybody could write their own, you know, macros. Emacs still rule. I'm more of a VI person myself, but Emacs are okay. And similarly, you know, WordPress. You know, you, you don't think of, think of WordPress as an extensible developer tool, but it is, right? It's an, it's actually a system, it enables, it's open, it enables people to be able to extend it, write, write write additional things into it.

Similarly, SugarCRM with its CRM tooling. And the fifth one. O of these is scalability. What do we mean by scalability? As we build our product, our developer tooling, we want to make sure that there is no performance degradation or bottleneck when many, many of these systems are alive.

Or we, when we are sending millions of QPS through it, we a request through it. We wanna make sure that these are the developer tools that we build. That, you know, are going to perform pretty much the same, whether we have one user on them. Or a million users. We want to make sure that they can be things that, you know, we can actually deploy and forget.

So we want to make sure that whatever we build can very easily run in all these, you know, self scaling systems. I mean, I'm not trying to show these as examples of scalable systems, of course, but all of these systems we have are auto scaling platforms. What does, what does that mean? When we build a developer tool that's going to run on those.

We know that suddenly in a, you know, based on a traffic spike from one instance, they can go to 10, 000 instances and we want to make sure that it's never going to block. It's not going to cause a bottleneck. It's going to be able to perform. And that's why scalability in my mind, as we built was very much so deploy and forget so that it's going to actually perform, do its job.

And sorry, and last but not least, is we want to make sure that when we build a developer tool, it's going to be reliable. And what do we mean by reliable? We mean, we want to make sure that it's going to be up all the time. It's not going to actually cause an issue by going You know, shutting itself down or crashing and blocking everything else.

This is, you know, making sure whether you call it six Sigma or nine, eight, nines, nine, nines, making sure that this tool is going to actually outperform whatever tool it is supporting. And there are lots of, you know, tools out there from um, lots of observability tools out there. We want to make sure that.

As this tool runs, you can actually monitor its performance. You can actually take a look at it. We want to make sure that as we are building a cloud native environment, we can, you know, use tools like Grafana, Jaeger, Prometheus. We can actually trace every single call that goes through it and we can make sure that you know, potentially the tools that we build come out of the box with this tooling in it so that you can actually monitor its performance very easily.

So, clicking the sound. So these were the six main principles that we actually used when we were building Cerbos. We believe it applies to many other developer tooling, but in our case, when we were building a security related, authorization related tool, they were our guiding principles to make sure that every use case, every Every scenario actually made, we, we wanted to make sure that it actually checked the boxes for all these six dimensions and last but not least.

What also helps is making these tools potentially open source making them open source doesn't necessarily mean making them free. There are a lot of commercial options out there that you can actually tack on top of these things, but there's nothing out there like open source that builds trust fast.

And making open source makes these, makes whatever tooling we're building, inspectable, makes building a community around here very easy. makes it more secure as people spot issues. You know, we're all usually are, again, we're all developers pretty much here. Our usual reaction is like, Oh, I can fix that.

I can actually submit a pull request to fix the issue I've seen. And it also distributes its ownership of the tool as it's building a community. So developers love open source. So whenever you're building a developer tool. Feel free to actually share it and share it with the community and get input. So that's my talk.

Very brief. If you have any questions, I'd love to take them now. Otherwise I will be by our booth by the entrance. Thank you.

Any questions? We are 15 right now. We started almost three years ago. Just. Three co founders and we open sourced it after six months actually five months in August. And ever since we've been releasing at least one release every month, I believe we're in release 34, 35 right now.

So thank you very much. If you agree, disagree, I'd love to have a chat after this, but these are the dimensions we pay attention to. Thank you.