Qualifyze 2023 Retro: Tech Edition

Wow, 2023 is almost over and so many things have happened at Qualifyze! I already wrote about the work done and the achievements of the Data Team in a Qualifyze 2023 Retro: Data Edition. This time, it’s the turn for the Tech Team.

Context

When I joined Qualifyze in July 2022, there was a team of 11 developers. The tech product was a B2B product with not so much traffic, and the product was created in 2019. Apriori, it seemed that the technical side was not going to be anything to worry about. However, the reality was a bit different.

Reducing Feedback Loop as the main guideline

The feedback loop is defined as the time between you try something and you know if it works or not. In software development (but also in any other discipline with a high uncertainty level), the shorter the feedback loop, the better. Think about the main development techniques used nowadays: local development, unit testing, CI/CD, etc. It’s all about the feedback loop. As an engineering manager of any level, any decision that affects the feedback loop has to overcome the drawbacks of a longer feedback loop.

Microservices to Modular Monolith

For those who don’t know me, I’m a fan of Domain-Driven Design (DDD). I have written two books related to the subject. Domain-Driven Design is about modeling business complexity to maximize productivity in big-scale systems. You can use some techniques for smaller environments, but using too many or the wrong ones will overcomplicate your landscape.

Microservices is a recommended architectural approach in DDD so every subdomain (or a part of the business problem that you want to solve) can be implemented in isolation, reducing dependencies (tech and team), so the development speed can be maximized. However, supporting microservices requires some extra overhead and tooling: separate infrastructure, separate code repositories, separate deployments, separate local environments, etc. All this overhead pays off within big scales, but in simpler and/or smaller environments, it does not.

So, considering that Qualifyze is not a big-scale/high-complex environment (8/5 app, low traffic, 11 developers, B2B, etc.) then microservices do not make sense.

Qualifyze had approximately 10+ microservices not considering plenty of other single lambdas (I’ll talk about it later). It’s not scientific, but in my experience as CTO and consultant, a good heuristic is h=(#developers/#microservices). If h is ≤ 4 then it’s a smell that you may be in an unnecessary overcomplexity scenario.

From July to December, we managed to move into a Modular Monolith that has the benefits of both Monolith (deployability, local development, logging, error management, simple infrastructure, etc.) and Microservices (module isolation) without the overhead of the Microservices.

It’s important to notice, that nowadays I feel that if you are not doing microservices, you are doing it wrong. Resist the hype! Resist the hype! Especially if you are not Netflix, Google, Amazon, etc. Small anecdote: During a meeting with a new potential investor, they asked me why I was moving away from microservices when everyone was doing microservices, I answered: “Everyone is wrong”.

A quick summary of how we did it is: we chose one of the microservices as the host, added NestJS support to run in parallel, used NestJS as the main entry point forwarded unmatched routes to the previous entry points in the same service, and keep merging other microservices into the host one.

Lambdas to NestJS

As part of the microservices approach, each microservices was running as a set of multiple AWS lambdas. I’m not a big fan of Lamdas, they are useful for specific use cases, but for a regular app, they don’t feel right. In my opinion, they fall short on deployability, testability, vendor lock-in, performance, etc. In general, Lambdas add extra complexity that for a not-complex application is not worthy.

AWS Lamdas are typically used with an API gateway, and I have to reckon that latencies were worse than expected. All in all, from July to December, we managed to remove all AWS Lambdas and now everything is running under a NestJS application with multiple modules.

Microfrontends to Modular Monolith

Microfrontends is another buzzword when working with microservices. Focusing on maximizing productivity in highly complex systems: making every microservice have its frontend/UI may be a good idea. However, microfrontends show similar benefits/drawbacks as microservices.

In a not complex environment, having one single code repository, with all your design system and a unique UI can speed up significantly your development speed and reduce your feedback loop without needing isolation to maximize productivity.

From May to August, we managed to unify all the microfrontends in one single React Application. The process was straightforward: reviewing dependencies, reviewing URLs collision, merging microfrontends into one host, and merging every component separate code repository. The result is less tagging, updating dependencies, faster delivery, easier testability, easier debugability, fewer bugs, and better performance.

Everything is now running now in AWS EKS.

One Backlog, One Product Owner, One Team

The original 11 developers were organized into three teams. The more splits you perform in your overall capacity the higher the risk that you may be working on things that do not pay into the overall company priority. The bigger the number of teams, the more tooling required for alignment. As a recommendation, for small teams (less than 20 people), try to stick with one backlog, one PO, and one team. Hold the feeling of splitting teams “just because the daily stand-up takes too much time”. Improve the daily stand-up before splitting the teams.

Now with 27 people, we have 4 teams working in different verticals of the business but in the same modular monolith. Remember that your team organization is a consequence of the business goals, not the other way around.

As a side note, when I joined Qualifyze, the Product Team was not empowered. The Tech Team was exercising some kind of disabling force over the product team in terms of product definition, processes, business impact reasoning, etc. It was really strange. Don’t get me wrong, the more involved the tech team is in the product and the business the better (ownership, ideation, etc.), however, it’s a collaboration, and it cannot be a Tech Dictatorship.

Changing Culture and Habits

As always, the most difficult part is to change people, not tech. As I explained in my online course for Engineering Managers and CTOs, cto.camp (in Spanish), there are simple techniques you can use.

1. Tech Vision: What is the target picture of where you want to go and why
2. Working Agreements: explicit rules on how the teamwork (written on GitHub in our case)
3. KPIs: tracking of the pulse of the team and the tech
4. Right Team: the team members that are not aligned with Tech Vision and new ways of working are better out of the team (you don’t want to be with someone that does not share the same values and principles)
5. Continuous Improvement: Inspect and adapt. If you define the rules of the game (working agreements), and you measure the result (KPIs), then it’s a matter of keeping improving daily.
6. Start simple, go complex if needed: Think always what is the problem you are solving, start with the minimum viable solution, and add processes/tools when there is a problem that hurts for some time. In summary, don’t overengineer, don’t solve a problem you don’t have, resist any hype, etc. that way you will keep your effort invested low and any effort will solve a clear problem so the wasted effort is minimized.

Head of Engineering

In March 2023, Daniel Funes joined as Head of Engineering. Introducing so many changes, especially on the culture side, it’s a full-time job, so having someone who can take care of all the changes, monitor them, and provide support to the overall team and each team member was key to the success of those changes. He managed to grow the team from 11 developers to 27. Remember that part of your job as CTO is to find a potential candidate to replace you in case anything happens, I have mine.

Tech Due Diligence

Another achievement was the Tech Due Diligence corresponding to the August funding round of 12 MEUR. We scored 20% better than the last Due Diligence with a recognition of the changes that we were applying.

Result

The overall result is an increase in the development speed measured in average lead time (from 4.6 days with a standard deviation of 20.1 days to 2.7 with a standard deviation of 3.4 days) and fewer bugs (from 165 bugs to 11 today passing 0 twice).

Thanks

I want to thank everyone who contributed to the achievements of the tech team during 2023. The overall team did an amazing job carrying on developing the product while introducing the new ways of working. I would like to have a special mention to Daniel Funes, Rafa Moral, and Christian Soronellas for pushing the architectural changes on the backend, frontend, and infrastructure with always a positive, optimistic, and service orientation approach.