Data pipeline for music royalty and trends data

Data pipeline for music royalty and trends data

17 Aug ยท 1 min read

Challenge: Improving existing data pipeline used for ingesting, calculating, and distributing music royalty and trends data

Solution: Making improvements in existing platform without major refactoring, while keeping the steady pace of delivering new features

User Group: business intelligence, data analyst

During the kick-off, the Director of Technology and Business Solutions told us that the ETL Pipeline platform is not an exciting project to work on, has several critical issues left by a previous vendor, and that people were very dissatisfied in the past. His biggest concern was the instability of the platform caused by both technical problems and the churn of the engineers working on it.

As always, we applied a basic analytical approach for onboarding onto the existing platform:

  1. Learn how the system works
  2. Identify strong and weak points of the system
  3. Identify missing parts
  4. Create a document with the desired state of the platform and development processes around it
  5. Interpolate the steps to get to the desired state from the current state

A month in, we created and presented the review document of the platform. The verdict was that the foundations are solid, and the quality of the engineering work that was put in by the previous vendor was admirable. However, the last 20% of the platform that was missing was creating a lot of problems and led to a very bad experience and results.

Senior management was very supportive and gave us the green light and autonomy to implement all the missing pieces. Together with them, we created a plan and a timeline to work on these improvements along with implementing new features and integrations.

After a couple of months we had:

  1. Serverless development environment per engineer, which improved stability of the product life cycle, and drastically improved the speed of development
  2. Centralized documentation which described the most complex parts of the system, as well as clear deployment and release procedures and schedule
  3. Working automated test suite
  4. Clear separation of responsibilities of the ETL team
data engineeringdata analyticsbusiness intelligence

written by

Kombinat

Kombinat

17 Aug, 2022

0

Helpful

SHARE

Comment as

Login or comment as

Anonymous

0 comments