DeepL’s Transformation Journey with ClickHouse: A Case Study
Technology Category
- Application Infrastructure & Middleware - Middleware, SDKs & Libraries
- Platform as a Service (PaaS) - Application Development Platforms
Applicable Industries
- Buildings
- E-Commerce
Applicable Functions
- Maintenance
- Product Research & Development
Use Cases
- Building Automation & Control
- Experimentation Automation
Services
- System Integration
- Training
About The Customer
DeepL is a language translation service that uses artificial intelligence to provide translations that are more accurate and natural-sounding than other services. The company was founded in 2017 and is based in Cologne, Germany. DeepL supports translation between several languages, including English, German, French, Spanish, Italian, Dutch, and Polish. The company is committed to protecting user privacy and does not share personal data with third parties. DeepL's mission is to break down language barriers and bring cultures closer together. The company's services are used by millions of people worldwide, including individuals, businesses, and organizations.
The Challenge
DeepL, a language translation service, was looking to enhance its analytics capabilities in a privacy-friendly manner in 2020. The company wanted to self-host a solution that could handle large amounts of data and provide quick query times. They evaluated several options, including the Hadoop world, but found it too maintenance-intensive and time-consuming to set up. DeepL also wanted to automate the process of changing table schemas when frontend developers created new events, which would have otherwise overwhelmed the team. The company needed a system that could handle complex events and queries to understand user interactions, something that traditional tools like Google Analytics couldn't provide. Additionally, DeepL wanted to maintain full control over the data while keeping user privacy in mind.
The Solution
DeepL chose ClickHouse as their central data warehouse due to its single binary deployment from an apt-repository, which made it easy and quick to set up a Minimum Viable Product (MVP). The MVP consisted of an API where the user’s browser would send events to, Kafka as a message broker, a sink writing from Kafka to ClickHouse, ClickHouse itself, and Metabase to visualize the results. The company heavily invested in automation and decided to have a combined source of truth for all events and the table schema. When frontend developers wanted to create a new event, they would need to define this event in protobuf. This protobuf schema file was used for three purposes: validating events, computing ClickHouse table schemas, and creating documentation about all events. Over time, DeepL expanded from a single node setup to a cluster of 3 shards with 3 replicas, ingesting about half a billion raw rows per day. ClickHouse also played a crucial role in DeepL's experimentation framework and ML-Infrastructure of Personalization.
Operational Impact
Quantitative Benefit
Case Study missing?
Start adding your own!
Register with your work email and create a new case study profile for your business.
Related Case Studies.
Case Study
Energy Saving & Power Monitoring System
Recently a university in Taiwan was experiencing dramatic power usage increases due to its growing number of campus buildings and students. Aiming to analyze their power consumption and increase their power efficiency across 52 buildings, the university wanted to build a power management system utilizing web-based hardware and software. With these goals in mind, they contacted Advantech to help them develop their system and provide them with the means to save energy in the years to come.
Case Study
Intelligent Building Automation System and Energy Saving Solution
One of the most difficult problems facing the world is conserving energy in buildings. However, it is not easy to have a cost-effective solution to reduce energy usage in a building. One solution for saving energy is to implement an intelligent building automation system (BAS) which can be controlled according to its schedule. In Indonesia a large university with a five floor building and 22 classrooms wanted to save the amount of energy being used.
Case Study
Powering Smart Home Automation solutions with IoT for Energy conservation
Many industry leaders that offer Smart Energy Management products & solutions face challenges including:How to build a scalable platform that can automatically scale-up to on-board ‘n’ number of Smart home devicesData security, solution availability, and reliability are the other critical factors to deal withHow to create a robust common IoT platform that handles any kind of smart devicesHow to enable data management capabilities that would help in intelligent decision-making
Case Study
Protecting a Stadium from Hazardous Materials Using IoT2cell's Mobility Platform
There was a need for higher security at the AT&T Stadium during the NFL draft. There was a need to ensure that nuclear radiation material was not smuggled inside the stadium. Hazmat materials could often be missed in a standard checkpoint when gaining entry into a stadium.
Case Study
Commercial Building Automation Boosts Energy Efficiency
One of the challenges to building automation is the multitude of non-interoperable communications protocols that have evolved over the years. Buildings have several islands of automation. Bridging the islands of different automation without losing the considerable investment in each specialized control network is the main focus in this solution.
