Qualifyze 2023 Retro: Data edition

2023 at Qualifyze has been a hell of a ride in the Data area: full of decisions, problems to solve, actions taken, learnings, and achievements that are worthy to share so others can learn from them.

Context

When I joined in July 2022, the Data Team consisted of 5 people with skills in CRM management, Make and Zappier (no-code tools), but no Data Engineering experience. The team was focused on sales reporting. There were no clear priorities or goals. In parallel, some other initiative by the business — outside of the Data Team — was trying to build a new data warehouse to cover the short-term business needs mainly driven by the finance area.

The tech landscape was a mix of some cloud functions running on Google Big Query, Make, Zapier, Amazon Redshift Glue ETLs, and Tableau. Everything was intermingled together, with no data modeling, and a sort of glossary half-written on a Google Doc without real implementation.

2023 business strategy included at that time a funding round to raise (with all the documentation, KPIs, and reporting that would be required), and a need —requested by the leadership where I belong as CTPO — to level up the steering of the business using data and KPIs.

As a summary, the problem to solve: having the right data team, a more effective and simpler Tech Landscape, and better data modeling. Let’s start.

Restructuring Team

It was January. After assessing the team at that time, the skillset gap was too big to solve only by leveraging training. We required not only Data Engineering, DevOps, Data Modelling, and Business knowledge skills, but also a mindset towards maximizing business impact and doing it quickly.

When a team/person is not delivering, I recommend the “don’t know, can’t, don’t want” method to identify the reason for a lack of delivery. Here, it was a clear “don’t know” where just training was not going to solve the problem.

I know that these decisions are always difficult to explain, difficult to understand, and difficult to execute. It’s never personal, but I have decided to restructure the team, keep only one person with a Data Analyst / Science profile, and open two new vacancies with a Data Engineer job description.

We finished the restructuring in February and filled those new positions in June after preparing job descriptions, performing interviews, etc.

Learning: Having the right team is a top priority for a manager to solve any problem. The sooner that you complete this task, the better. So, don’t hesitate to take any action prioritizing: 1. training, 2. blocker removal, 3. motivation. On top, managers have a bias thinking that any know-how kept by a specific person is going to have a huge impact after the person leaves the organization, it’s rarely true. You can suffer a bit in the short term, but by asking for patience from stakeholders and relying on other people in your team to cover, you can minimize the potential impact.

Backtracking from Solution to the Tech

It was March. Together with Simon Rahause, our top VP of Finance, we started working on automatizing the Investor Reporting, the report that we send every month to our Investors. The current method was taking too much time, and other new requirements were popping up such as supporting not only the existing transactional business model but also the new subscription one that was about to be launched. Additionally, we had to include new KPIs.

After the team restructuring, I was a bit alone with another colleague from finance in front of the challenge. We defined what we wanted and we built backward the solution. Following the mantra: “make it work, make it right, make it fast”, I started with a couple of PHP scripts querying the existing data sources are exporting data directly via email. However, the additional information and/or reports that we created the more inconsistencies we started to spot. We quickly realized that if we wanted to have proper data consistency and business modeling, we required a different type of mindset and tooling, especially around data modeling.

After some research, I found DBT and other tooling related and it was a game changer. Simon and I worked a lot of days until 2:00 AM creating data models, reviewing numbers, tuning inconsistencies, etc. but it was worth it to deeply understand the business and model it.

Learning: Without a minimum data modeling, the data inconsistencies and the cost of maintenance quickly increase so it’s good to find solutions. Data modeling is probably something first to solve than the orchestration and other boilerplate.

Fivetran, DBT, and Hightouch to the rescue

I performed multiple analyses on different tooling for Data Ingestion, Data Modelling, Data Exporting, and Reporting. These are some of the final decisions. My main mantra is simplicity: start with the basic solution possible, and iterate when problem complexity increases. This way I avoid overengineering, keep the learning curve low for new joiners, and reduce the bugs generated.

From March to December, we have been working on rolling out the new toolset and simplifying the tech landscape.

1. Data Ingestion / Fivetran: https://www.fivetran.com/

With more than 200 connectors from the typical SaaS solutions, it allows you to copy data from your SaaS tooling into your Data warehouse. It’s a great tool, no need to develop and maintain ETLs (development, versioning, running, bugfixing, etc.), so it’s a cost-effective solution.

With Fivetran, we ingest data from Airtable, Slack, JIRA, Mongo, PostgreSQL, ProductBoard, LinkedIn, Hubspot, and other applications into our Amazon Redshift.

This was fundamental to replacing the existing legacy Google Cloud Functions and the existing AWS Glue ETLs that were difficult to maintain.

The initial Data Engineers hired were a bit reluctant to use Fivetran. They mentioned that the cost was high (700 USD/month), that features-wise didn’t make sense, etc. Now they are big fans, because it’s code that we don’t have to write, or maintain, and it’s cost-effective compared to an in-house developer maintenance that includes the opportunity cost.

Learning: Data, as any other tech team, has to focus on business impact not on tech. Tech is a means to an end.

2. Data Ingestion / AWS Glue: https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html

When Fivetran does not support the application to ingest from, in our case, Hotjar, for example, you need to create a job to do that. A Python script running as a Glue job can do the trick. There is probably a simpler approach like GitHub actions and a plain Python script, but the people joining had experience with that stack, so, it was ok to keep it.

Learning: We have written down this decision as a Working Agreement to speed up the onboarding of new joiners and make a clear statement on how we work.

3. Data Modelling / DBT: https://www.getdbt.com/product/what-is-dbt

I’m in love with DBT. It’s an open-sourced project that just with SQL and some Jinja (Python templating system) allows you to define data entities and models that represent business concepts, combine them to build intermediate entities, and prepare final entities to be used in reporting and other graphical interfaces for data exploring solutions.

It has a really low learning curve, it’s modeling as code, it can be easily integrated into any CI/CD pipeline, and has support with the main data warehouses out there like Redshift, Snowflake, Google Big Query, etc.

Learning: DBT Labs offers for free some interesting courses regardless if you use DBT or not about Agile Data Modelling: https://www.getdbt.com/dbt-learn

4. Scheduling / GitHub Actions

In complex environments, when orchestrating multiple data ingestions, there is the need for orchestrators like Airflow, Luigi, etc. While you don’t have such complexity, you can go with scheduled crons, or in our case, GitHub actions, which trigger periodically and also when the code base changes.

When having multiple and different ETL technologies, it’s typical to need an orchestration mechanism. Airflow is one of the most used. However, there is another approach, to simply the ETL technologies. In our case, with the decision of Fivetran for ingesting raw sources, and DBT that models all the entities in one shot, the orchestration requirements were low. Now, with only a couple of scheduled GitHub actions, the orchestration was solved.

Learning: Before looking to introduce a new technology, remember is there is a way of simplifying the existing stack so the new technology is not required.

5. Hightouch / Data Exporting: https://hightouch.com/

Hightouch is like Fivetran but instead of importing, it does export to 200 different applications. We use it to export simple reports to Google Sheets, Slack notifications, emails, and other applications.

Learning: Create models that are ready to export so that you can use any reporting tool like Tableau, Looker, etc. on them

Head of Data

In July, we opened a vacancy for a Head of Data. In October, Mariela Haya joined us. She worked with me a long time ago at Emagister. Mariela, a part of having experience with our current stack, shares the same pragmatism mindset combined with a huge business knowledge and experience managing stakeholders like business area heads or even CEOs.

Learning: In any Data project, Stakeholder expectation management is fundamental. Stakeholders' trust typically starts really low and improves while delivering consistent and reliable data. It’s a process and it cannot be rushed. So be patient, and stay positive.

No Code: From Zappier and Make to only Make

As no-code solutions, we used Zappier and Make together to integrate some applications between each other like our CRM. We were paying for both and experiencing a lot of issues, especially from Zappier. During December, the team managed to migrate all the jobs from Zappier to Make, improve the reliability of the jobs, and save 500 USD/month.

Learning: No-code solutions are really useful for integrating multiple applications in your landscape. Use them wisely and you can save a lot of time.

Audit Data Extraction

In parallel to the data modeling infrastructure, we have developed and used techniques to extract automatic information from the Audit Reports performed to launch Market Intelligence and Supply Chain Compliance products. But that, it’s a Product story :)

Summary

During 2023, we set the foundations up to be a data-driven company and created a solid data model to build upon it. We have simplified the overall tech landscape making it cheaper, simpler, faster, and more reliable.

Thanks

I want to take the opportunity to thank everyone who has contributed to the 2023 data achievements, especially, the current Qualifyze Data Team. I’m looking forward to the things that we will achieve in 2024.