Firebolt

The client, a market research tech company, provides interactive analytics to marketing teams, enabling them to research their competitors’ brand and marketing activities as well as their own. The platform relies on massive volumes of web-traffic data and other sources, which are continuously streamed into S3 and used for various use cases and features within the customer-facing analytics platform. A key component of delivering a great user experience for their customers is ensuring that users don’t have to wait long to get results for their questions. However, waiting more than a couple of seconds was considered unacceptable. This led the company to make several painful tradeoffs. They had to limit their analysis to a month of data (~10TB), instead of a quarter (~40TB), due to the decrease in performance when trying to analyze a larger data set. They also had to aggregate data for full-year analysis features to maintain performance, while sacrificing the ability to drill into the granular data. Furthermore, they had to limit the types of data that could be analyzed, as many semi-structured sources could not be analyzed quickly enough compared to structured data.

F&B franchisees face challenges in automating financial control processes, consolidating financial data, and driving sales and profit growth. They interact with disparate proprietary applications for operations, including point-of-sale, financials, inventory, and workforce planning systems. Building a unified business view requires manual processes for bookkeeping, reconciliation, and financial reporting, leading to inefficiencies and increased error risks. Complex data integrations are needed to generate KPIs on sales, inventory, labor, fraud risk, customer experience, and operational costs. The advent of services like Uber Eats and DoorDash adds complexity, requiring analytics KPIs to track delivery and financial performance. These challenges, coupled with the need for skilled IT staff, make the status quo difficult for F&B franchisees.

Similarweb faced significant challenges in delivering fast and efficient analytics over massive data volumes. The company needed to enable users to analyze segments within larger websites, such as comparing FootLocker.com traffic with Amazon.com shoe searches. The dynamic input from users required handling an exponential number of combinations, making pre-processing unfeasible. Additionally, the existing solutions like Athena and DynamoDB were not fast enough or lacked SQL support for dynamic grouping. The challenge was further compounded by the need to scan large data volumes, such as 150GB of data generated daily by Amazon, and the requirement to analyze up to two years' worth of data.

As a rapidly growing startup in the ad tech space, Infy.TV had been hard-pressed to build out a scalable architecture to handle frequent data ingestion and low-latency queries for their bespoke, customer-facing financial metrics dashboard. Infy.TV currently generates over 200 million records per day. Efficiently ingesting and analyzing these kinds of data volumes maxed out the capabilities of a scaled-out instance of Postgres on AWS’ RDS (Relational Database Services) offering. Ingestion from Infy.TV’s in-memory Aerospike document store into an aggregated form in RDS could only be processed once a day. The architecture for the RDS cluster simply couldn’t scale for high-performance analytical queries, and the cost of the entire setup was becoming prohibitive.

As data volumes and access requirements grew, DTN faced challenges in analyzing data beyond a 30-day window and comparing inventory snapshots over multiple time frames. These comparisons were crucial for identifying potential trades within the DTN network. The effectiveness of DTN's digital products depended on providing timely trend reports covering a 12-month data retention period. To address the need for fast response times while analyzing large data volumes, Firebolt was evaluated at the recommendation of consulting partner, Untitled Firm.

The company faced limitations with its homegrown analytics solution, which restricted its offering and market growth. The existing system was fast for specific queries but limited in the types of analytics and data size it could support. Onboarding new customers was challenging and time-consuming due to the need for extensive customization. Adding new analytics features required significant custom work, and scalability issues limited data sizes and types of analytics. The company evaluated various solutions, including Elasticsearch and BigQuery, but found them either too expensive or not suitable for their needs.

Bigabid's analytical databases, originally based on MySQL, were not meeting the performance requirements needed for their operations. The company faced significant delays in generating data insights, with some processes taking days to complete. Additionally, they struggled to access data older than three months due to heavy data aggregations, which hindered their ability to perform seasonal or year-to-year comparisons. Bigabid aimed to analyze data for a million ad auctions every second and manage data lakes containing hundreds of terabytes in near real-time. This required accessing tables with billions of rows to create hundreds of live dashboards. To address these challenges, Bigabid needed to build a high-performance big data infrastructure and merge its internal BI and data analysis platforms into a central data platform.

Audiohook, an adtech company, faced challenges with their existing cloud data warehouse, which was unable to meet their performance needs for dynamic data models and speed of analysis. The company needed a solution that could handle complex joins, windows, and aggregations across large event-based datasets to provide insights into marketing campaign effectiveness. Their existing solution was slow, taking up to an hour to compute impressions and conversions, which was not sustainable for their business needs.

Explorium faced significant performance challenges as their data and customer requests grew. Their existing setup, which involved using a Presto cluster on AWS for processing time series data, was unable to handle high loads efficiently. The shared nature of the Presto cluster meant that large jobs could impact the performance of other requests, leading to slowdowns and customer dissatisfaction. Explorium's data volumes and requests were expected to triple, necessitating a new solution to handle customer requests for time series data enrichment.

OmniPanel faced the challenge of scaling their data layer to meet the growing needs of their customers. As they built out their technology stack, they realized that their existing data infrastructure would not be able to handle the increasing volume of customer data and the complexity of analytics required. They needed a solution that could scale efficiently without requiring significant overhead for deployment. The challenge was to find a database solution that could support their expanding data needs while maintaining high performance and low query times.