Rockset

Rumble, an Israeli company, developed an application that converts user's steps into reward coins, which can be used to purchase unique products or services. The company initially used PostgreSQL to handle data comprising users’ step counts, with three different tables tracking daily, weekly, and monthly steps. However, as user growth started to increase, PostgreSQL performance began declining, especially during peak times. The application's responsiveness declined with around 20+ requests per second, as PostgreSQL was unable to maintain the latency required to serve the leaderboards and eventually ran out of CPU and memory. Rumble's users are goal-oriented and being able to instantaneously see their steps and purchase coupons encourages them to maintain their active lifestyles. Therefore, Rumble needed to deliver real-time, data-driven applications to meet those needs. They needed a database that could handle complex queries, scale easily as their number of users grew, handle high concurrency, maintain low-latency queries, and require low ops.

Dairy farmers in the United States face challenges in efficiently managing large herds and maximizing milk production. With the average farm keeping 234 cows and larger operations managing herds of 5,000 to 100,000 cows, traditional methods of data tracking and management have become insufficient. Farmers have historically used PC-based applications and manual data entry to track key metrics such as herd information, breeding history, feed intake, and milk production. However, these methods are time-consuming and result in siloed data that cannot be easily analyzed for historical trends or combined to make informed decisions. As a result, farmers spend a significant amount of time gathering data, which limits their ability to optimize operations and improve profitability. The need for better connectivity and digitalization in the dairy industry has become apparent, as farmers seek more efficient and powerful ways to utilize their data.

SkyHive faced significant challenges with MongoDB for analytical queries due to its slow performance in handling complex analytics involving data across jobs, resumes, courses, and different geographics. The query latency was high, and the system struggled with multidimensional queries and joins, making it impossible to provide the interactive performance required by users. Additionally, there were limitations on payload sizes and other hardcoded quirks, such as the inability to query certain countries like Great Britain. These issues hindered SkyHive's ability to deliver immediate results to customers, especially when expanding searches to non-English speaking countries, as data normalization across different languages was problematic.

Providing real-time visibility into ticket data proved challenging for Command Alkon. Initially, it was not possible to retrieve data quickly enough to enable the real-time analytics that Command Alkon wanted and their users expected. As strong proponents of serverless architecture, the Command Alkon team had chosen Amazon DynamoDB as the transactional database for their application. While it was ideal for storing incoming ticket data, a different solution was required for analytics. Building secondary indexes would speed up specific queries, but given the multitude of ways users could query the ticket data, anticipating access patterns and optimizing for all of them was not a practical solution. Command Alkon had a product objective to support any type of data access at any time. However, the performance challenges associated with unanticipated user queries meant that the application could not deliver real-time views of ticket data. They needed to find a way past these performance issues to meet the demands of their largest customers and scale CONNEX usage.

Dimona, a leading apparel company in Latin America, faced significant challenges with its inventory management system due to its vertically-integrated business model and extensive supply chain. The company needed to track raw fabric, t-shirts in various production stages, and finished products across multiple locations. The existing off-the-shelf ERP systems were inadequate, as they only allowed for end-of-day inventory counts, leading to errors and mistrust in the data. Additionally, the company's use of Amazon Aurora for its custom-built inventory management system resulted in performance issues as the volume of data increased, especially with the expansion of warehouses and stores. The COVID-19 pandemic further exacerbated the situation, as international demand for Dimona's drop shipment services increased, leading to an unusable inventory management system and high costs for simple data aggregations.

1NCE faced challenges in providing real-time analytics for their IoT data using Amazon DynamoDB. While DynamoDB was effective for storing monitoring and management data, it was limited in its ability to perform real-time analytics. The company attempted to use BI and dashboard solutions compatible with DynamoDB, but these were not granular or real-time enough. They also tried building Lambda functions and step-function logic to enable customer queries, but this approach stretched DynamoDB's indexes too thin and resulted in slow query times. The need for real-time business observability was becoming increasingly critical as more customers relied on IoT devices for mission-critical operations.

DataBrain, a fast-growing SaaS company, faced several challenges with its data stack as it scaled. Initially using PostgreSQL through Amazon RDS for landing and querying customer data, DataBrain encountered high query latency and inefficiencies in handling large volumes of data. The dynamic schema of incoming customer data further complicated the process, requiring significant effort to manage schema changes. Additionally, DataBrain needed to accelerate customer time-to-value by providing fast, actionable insights without requiring extensive setup or engineering support.

The challenge faced by Press & Say was the need to create interactive dashboards for speech therapists to monitor patient KPIs effectively. Initially, they used AWS DynamoDB, a NoSQL database, to handle the anticipated massive flow of data. However, they encountered a barrier in making the data visually alive in dashboards. The startup, being in its seed stage and based in Latin America, found the cost of custom dashboard analytics platforms prohibitive. They needed a solution that was easy to implement and manage, especially given the co-founder's limited experience in raw-coding development on a NoSQL database.

PCH International faced challenges with its existing data infrastructure, which was built on MongoDB and DynamoDB. These systems could not support real-time querying of data, leading to slow ingestion and query times. The company needed faster, more complex queries to make its supply chain fully visible to its analysts and customers. Existing solutions like Snowflake and Redshift were considered but found to be too costly and slow for real-time analytics. PCH required a solution that could handle large datasets with low latency and was easy to deploy and manage for its small data engineering team.

Windward faced challenges with its existing data stack, which was not ideal for the search and analytical features needed to transform the maritime industry. The company was using a batch-based data stack with multiple databases, including S3, MongoDB, PostgreSQL, Cassandra, and Elasticsearch, which made it difficult to support new use cases and achieve performant contextual queries. The existing data stack also limited the ability to generate significant new revenue from product requests that were hard to support, such as geo queries, vessel search, and partial word search. Additionally, the AIS transmission data was flaky, making it challenging to associate a transmission with the right vessel, and the enrichment data had changing schemas, making it difficult to support using relational databases with strict schemas.

Own the Moment faced significant challenges in building a robust data infrastructure to support their NFT marketplace and fantasy sports platform. The initial setup using Amazon's DynamoDB revealed several limitations, particularly in handling complex, large-scale queries required for their diverse contests. DynamoDB's NoSQL nature meant it lacked support for SQL commands like JOIN, necessitating data denormalization, which led to difficulties in maintaining data accuracy and increased storage needs. Additionally, the use of Dynamoose, an ORM tool, introduced latency issues, resulting in slow query performance. With the NFL season approaching, the company needed a solution that could handle real-time data ingestion, concurrent usage spikes, and efficient data exchange with the Ethereum blockchain.

To maintain and increase growth, Whatnot needed to enhance its home feed by ranking show suggestions based on the most interesting and relevant content in real time. This required an increase in the amount and variety of data to be ingested and analyzed in real time. The existing data pipeline, built on AWS-hosted Elasticsearch, was not scalable enough to handle the anticipated growth of 5-10x in the next year. The high operational overhead of Elasticsearch was draining productivity and limiting the ability to improve the recommendation engine. Adding new user signals to the analytics pipeline was a time-consuming process, taking weeks to implement changes. Maintaining existing queries was also labor-intensive, requiring frequent updates to Elasticsearch indexes and manual testing of the data pipeline components.

As Ritual expanded into new product lines, a key to effective monetization was personalized experiences. Personalization is no new concept to the e-commerce industry. 80% of shoppers are more likely to buy from brands that offer personalized experiences. And, personalization is shown to increase sales by 20% (Bloomreach). The team at Ritual looked to personalized offers and bundles to increase the average order value and the lifetime value of the subscriber. Ritual piloted personalized banner ads in their online portal, a spot where subscribers could add new products to their existing subscriptions. Prior to personalization, there was a generic promotion for the different product lines in the portal. The team at Ritual believed that they could make more relevant offers and increase the conversion rate with personalization. The seamless integration of personalization in the online portal prompted Ritual to expand to personalize the cart checkout experience and email campaigns. As customers went to checkout, they would receive personalized offers to add product lines to their subscription. Post checkout emails and delivery notifications would also include personalized offers.

Zembula faced significant challenges with their legacy analytics engine, which was based on Elasticsearch. As their business grew, the demand for real-time email personalization increased, leading to scalability and cost issues. The existing infrastructure was unable to handle the spiky nature of their email campaigns, resulting in underutilized resources during low-traffic periods and overwhelmed systems during high-traffic periods. The cost of maintaining numerous Elasticsearch servers on Amazon EC2 was high, and the engineering effort required to optimize Elasticsearch for efficient data storage was a significant drain on resources. The situation worsened when Zembula expanded their Smart Banners™ to all promotional emails, causing a 10x increase in traffic within three months. They needed a more scalable, cost-effective, and low-ops solution to support their growing business needs.

ConveYour faced significant challenges in scaling its SaaS sales training platform due to the increasing volume of data and the need for real-time analytics. The original data infrastructure, built around an on-premises MongoDB database and a MySQL database running in Google Cloud, was unable to keep up with the demands of real-time data ingestion and query performance. The CRM dashboard, which provided real-time aggregated performance results for thousands of sales reps, was becoming slow and unresponsive as the data volume and number of users grew. Additionally, the seasonal nature of the business meant that deploying permanent infrastructure to accommodate spiky demand would have been expensive and wasteful. ConveYour needed a data platform that could scale up and down as needed, while also providing real-time analytics capabilities.

Seesaw Learning Inc. faced a significant challenge when the COVID-19 pandemic forced schools to switch to remote learning, causing a massive increase in demand for their platform. This surge in usage led to a wealth of data being generated, but Seesaw struggled with observability and analytics. Their existing data infrastructure, primarily based on Amazon DynamoDB, was not optimized for real-time analytics, making it difficult to provide timely insights to educators and internal teams. The batch-oriented analytical tools they were using, such as Amazon Athena, were insufficient for their needs. Seesaw needed a solution that could provide 360-degree real-time observability and allow both internal and external users to access fresh data for decision-making.