Improving existing data pipeline used for ingesting, calculating, and distributing music royalty and trends data
Making improvements in existing platform without major refactoring, while keeping the steady pace of delivering new features
business intelligence, data analyst
During the kick-off, the Director of Technology and Business Solutions told us that the ETL Pipeline platform is not an exciting project to work on, has several critical issues left by a previous vendor, and that people were very dissatisfied in the past. His biggest concern was the instability of the platform caused by both technical problems and the churn of the engineers working on it.
As always, we applied a basic analytical approach for onboarding onto the existing platform:
A month in, we created and presented the review document of the platform. The verdict was that the foundations are solid, and the quality of the engineering work that was put in by the previous vendor was admirable. However, the last 20% of the platform that was missing was creating a lot of problems and led to a very bad experience and results.
Senior management was very supportive and gave us the green light and autonomy to implement all the missing pieces. Together with them, we created a plan and a timeline to work on these improvements along with implementing new features and integrations.
After a couple of months we had: