Delhivery, India’s leading fulfillment platform for digital commerce, handles a massive amount of data, over 1.2 TB per day, from its vast network of IoT devices. The company fulfills a million packages a day, 365 days a year, through its extensive network of automated sort centers, fulfillment centers, hubs, direct delivery centers, partner centers, vehicles, and team members. With nearly 60,000 data events and messages per second, data discovery and organization became a significant challenge. The data is organized and processed by hundreds of microservices, which means that ownership over the data is distributed across different teams. As the company grew, the scale and complexity of its data grew even faster. Teams started building their own microservices, motivated by a desire to make data-driven decisions. However, finding and understanding the data became a significant issue. The onboarding process for new team members grew from 1-2 months to 3-4 months due to the growing complexity of the data. By 2019, Delhivery realized it desperately needed a data cataloguing solution.

Chargebee, a leading technology solution for recurring revenue management, faced a significant challenge in early 2021 when its growth led to an increase in data requests. The company's Data Engineering team was responsible for processing these requests, both from internal colleagues and customers. However, the volume of requests was so high that internal data requests were often pushed to the back of the queue, leading to missed SLAs and dissatisfaction among stakeholders. The team attempted to address this issue by hiring new colleagues, but the transactional nature of their Data Engineering function made hiring difficult. They also tried to automate and standardize the process by creating dashboards and workflows, but these were often too bespoke to service with a single view of data. As a result, the data request volumes continued to grow, with the team receiving 350 requests per quarter, 80 of which were repetitive requests. Struggling to meet their SLAs and with growing escalations to subject matter experts, Chargebee needed to find a new way to meet their colleagues’ and customers’ expectations.

When Postman's data team expanded, they faced a significant challenge in managing and understanding their data. The data was scattered across different locations, and often, the same data in different places contradicted each other. As the company grew, the data system, which was initially simple and manageable, became complex and difficult to navigate. The data was stored in tables, and the information about these tables was only known to the early members of the data team. This system was not scalable and could not keep up with the company's exponential growth. The company's goal was to democratize data, making it accessible and understandable to everyone in the company. However, the lack of consistency and context around the data made it difficult for everyone to understand and trust the data. The data team was constantly bombarded with questions about data location and usage, and the loss of any team member would mean the loss of crucial data knowledge.

Snapcommerce, a tech-savvy organization that operates in the travel, fintech, and goods verticals, faced a challenge as it scaled its operations. The company's employees, who are active users of its data platform and assets, needed a reliable source-of-truth documentation in a user-friendly format to support their ongoing requirement for self-serve tools. The company was looking for a way to standardize and share data definitions across the organization. They also wanted a solution that eliminated the need for coding by business stakeholders and provided quick navigational capabilities. The challenge was to find a data catalog that met their specific criteria, including an easy-to-navigate interface, strong search capability, an automated crawler, clear definitions/glossary section, permission handling, a table preview and SQL component, and data lineage visualizations.

Brainly, a rapidly growing organization with a distributed model, faced a significant challenge in data governance. With independent teams each owning their data silos, the discoverability of data became a major issue. The remote setting further complicated the situation as it was difficult for teams to find and access the data they needed. The requirements for addressing this challenge included having metadata of all data assets in one place, making data assets discoverable, enabling collaboration and trust, reducing dependencies between business, analysts, and engineers, and showing where the data comes from.

Elastic, a technology company that powers search solutions and protects against cyber threats, faced several challenges in its data practice. Takashi Ueki, Director of Enterprise Data & Analytics at Elastic, identified multiple sources of truth leading to disconnected reporting, a BI platform strategy that was misaligned with organizational needs, and varying definitions making it difficult to accurately and consistently report. Elastic's distributed and remote nature, along with its diverse team, made it crucial for any solution to be relevant and personalized to a spectrum of needs, expectations, and skill sets. The company's data governance strategy needed to drive transparency, accountability, and engagement, and be embedded seamlessly into the day-to-day experience of its distributed workforce.

Funding Societies | Modalku, a licensed digital peer-to-peer (P2P) lending platform in South East Asia, faced several regulatory and compliance requirements that factored into its data strategy. As data becomes an increasingly valuable asset in the FinTech world, the company needed to ensure high-quality data and meaningful management information to identify and monitor risks and understand the performance of various business functions. The challenge was to implement a solid understanding of data governance to equip the organization with better decision-making capabilities, uniform data across the organization, increased data literacy, and improved regulatory compliance.

Funding Societies | Modalku, a licensed digital peer-to-peer (P2P) lending platform in South East Asia, faced several challenges related to data governance due to its regulatory and compliance requirements. The company recognized the increasing value of data and its potential to provide significant competitive advantages in the FinTech world. However, without high-quality data and upward reporting of meaningful management information, the company was unable to identify and monitor risks or understand the performance of various business functions. The company faced daily operational and regulatory challenges, including understanding and classifying data, applying flexible governance and security policies, and integrating across different applications. The company also needed to ensure data was organized, accessible, and compliant.

Nasdaq, the world's second-largest exchange, has been a data-driven company for over five decades. Despite having a decade of experience operating in AWS and moving the bulk of its critical workloads to the cloud, Nasdaq faced significant challenges. The trading system data was complex in size and structure, with as many as 140 billion events processed per day in the U.S. alone. The data was optimized for operational performance, not for analytics, making it difficult to manage. Additionally, Nasdaq's process for preparing and presenting data was outdated, with their legacy ETL tools unable to keep up with the scaling types of data and demand. The rigidity of these tools did not align with Nasdaq's ambitions. The data team was overwhelmed with maintaining the technical landscape and struggled to support their business partners effectively. This led to the emergence of parallel teams, each with a unique approach to creating data solutions, causing inefficiencies and confusion.

Postman, an API collaboration platform, experienced rapid growth, with its valuation reaching $5.6 billion and its user base expanding to over 17 million people from 500,000 companies globally. However, the company's data team was not growing at the same pace. In April 2020, the data team consisted of only six or seven people. Over the next year, the team expanded by 4-5x to 25 people. This rapid growth presented challenges in terms of onboarding new hires, handling requests from the rest of the company, and planning their work. The data team was also grappling with the decision between a centralized and decentralized team structure, with the former leading to conflicting data systems and metrics. Additionally, the team faced difficulties in prioritizing work and allocating projects fairly among team members.

TechStyle, a fashion retailer with a portfolio of five brands, faced a significant challenge in early 2020. The company, which has built its business model around embedding data across its operations, decided to overhaul its common systems and roll out a new data warehouse. This was a daunting task due to legacy backends, a relatively new team, and a sudden shift to remote work due to the COVID-19 pandemic. TechStyle uses a 'hub-and-spoke analytics model', where each brand has its own embedded Analytics Team, and the Data Platforms Team creates and manages common data systems. However, the company had been struggling with making data discoverable and understandable to everyone, not just long-time team members. The documentation for their systems was often limited or non-existent, and the growth of data sources that weren’t owned by TechStyle’s central data team added to the confusion and complexity. The challenge was further compounded when the company had to shift to remote work, disrupting the informal information flow that worked naturally in the office.

Tide, a UK-based digital bank with nearly 500,000 small business customers, faced a significant challenge in improving their compliance with GDPR’s Right to Erasure, also known as the “Right to be forgotten”. The bank's data and legal teams needed to define personally identifiable information and propagate those definitions across their data estate. The process of compliance was difficult and time-consuming, involving manual effort to find and delete data that persisted in secondary systems. Complicating this challenge was a lack of shared definitions of personal data, with differing opinions across organizations from Legal to IT. As Tide's technology stack and architecture grew more complicated, new products and services were introduced, and customers increased over time, the compliance process took only more time and effort. Automating this process became a priority.