From Thousands of Files to Just Seven: A Data Lakehouse Optimization Story
At Endowus, data is at the heart of everything we do, from personalizing client experiences to powering our investment insights. As our platform grows, so does our data lakehouse, and with that growth comes the inevitable engineering challenges of managing scale, cost, and performance. Recently, our Data Platform team embarked on a mission to tackle two critical issues that were impacting our data pipelines: the notorious “small file problem” and the crucial challenge of data freshness.
This post shares our journey of diagnosing these challenges, implementing a multi-faceted solution using Delta Lake, and the dramatic improvements we achieved in performance, cost, and reliability. It’s a story about how rethinking the physical layout of our data unlocked efficiencies across the entire platform.