CloverDX

The company, a major financial and leasing provider, was grappling with the challenge of managing over 27 million records across two legacy systems - Microsoft Dynamics NAV and Target. The presence of a large number of duplicate records was affecting business operations and increasing overhead costs. The simultaneous running of both systems was causing slow processing speeds and necessitating an increase in staff. Furthermore, the company was facing compatibility issues among the legacy systems, platforms, and databases. The daily management of critical data assets was often met with these compatibility issues, further complicating the situation.

The consulting firm was facing challenges with the quality of legacy data during the migration process to Workday. The poor quality of data was slowing down project delivery and taking valuable time away from the firm’s more critical objectives. The firm required a framework that would quickly identify and report data issues. Reconciliation of data in Human Resources was crucial for compliance with legal and regulatory concerns. The firm was looking for a solution that would simplify the data migration process and allow consultants to recoup time they were spending on the manual work.

The online retailer had achieved rapid growth in just a few years since being founded. Having built the business as very much a technology-first enterprise, they were looking for a data solution to better help all areas of the organization, and one that could scale with and support the company’s expansion. The decision to build a data warehouse was made to provide rapid access to the value held within their operational data. By deploying a flexible data lake they had direct access to source systems via a replicated database. One of the key requirements from the IT team was to handle changes in the business more easily, and to be able to get quick answers to business questions from the data. While a traditional data warehouse is built to answer specific questions, the ability to access source data meant that the company could be much more flexible and responsive in its approach, and get the valuable insights they needed, fast.

The financial enterprise uses Oracle Identity Manager (OIM) to track and monitor user access permissions associated with employee transitions and reassignments. However, with more than 6,000 different systems, updates to user permissions result in large volumes of constantly changing data. The disparity in data formats and sources coming from the systems makes feeding OIM a challenge because it’s strict about the data format it ingests. The IT security team had a choice to make: to work with a similar manual approach in-house or utilize specialist data integration software that would automate the process.

The company had been growing through competitor acquisition and had developed an in-house solution to run daily data feeds into Salesforce. However, with tens of millions of records being processed daily, the time needed to load them was getting longer, peaking at 12 hours. This meant the necessary daily updates could no longer be guaranteed. In addition, errors in loading the data into Salesforce, sometimes by as much as 50%, resulted in another day’s (lost) productivity dedicated to fixing the problems. As a result, the financial advisors relying on the data were potentially working with information that was up-to two days old. The company needed a solution that could process data quickly and deliver it, error free, into Salesforce.

The publishing house was facing a significant challenge due to the poor quality of their customer data. The company communicates with its clients by phone, email, and direct mail. However, customer contacts had been stored in different systems and often appeared more than once. Additionally, only 30% of addresses were in the acceptable format. Because of inconsistent and duplicate data records, marketing materials were not being delivered, and some customers could not be reached by phone. Furthermore, multiple marketing packages were sometimes sent to the same address, or the same person received marketing materials at different addresses. On top of the direct costs incurred due to these data errors, the company was also experiencing indirect damages such as loss of credibility and missed opportunities.

The New Democratic Party of Canada was facing a dilemma. The amount of available data was growing and their homemade data integration solution wasn’t able to handle this data anymore. They needed to find a way to process this data, or lose the chance to effectively work with their constituents. The NDP quest for quality data started with over one hundred datasets from organizations like Elections Canada, Canada411, NDP regional counterparts and more. They needed a solution that could match those records, cleanse, duplicate and delete faulty records, validate them with external and internal validators and migrate them into the NDP system.

The company was facing a challenge with the processing of phone invoices that were sent from providers in three incompatible formats. This made electronic merging and processing impossible, and values had to be manually entered into spreadsheets. The spreadsheets were updated every few months, but this caused inconsistencies in historical data. The company was spending 6 man-days per month handling this burdensome task manually. Additionally, the company’s phone bill—$44,000 monthly—had been growing by 24% per year, but it was difficult to gather adequate insight as to why. The company required a solution that would enable them to automate phone bill processing, develop an efficient calls monitoring system through bill analysis, and monitor private and company phone usage to determine the best possible call rates.

BTC Solutions, a UK-based company, was facing a demand for increased performance of their autoVHC platform due to growing amounts of data and rising customer expectations for a quicker turnaround. Dealers were having difficulties accessing fresh data, leading BTC to search for an ETL tool that could handle fast-changing data. BTC had three main requirements for the data warehouse solution: Data History, Robustness, and Flexibility. The new data warehouse needed to keep track of changes and record all modifications done to the database. It also needed to be able to handle ten times more data than their current database. Lastly, the solution needed to be user-friendly, so that BTC could define and modify the transformation procedures.

The case study revolves around the challenge of managing identity and access in a banking environment with over 6,000 systems. The bank needed to ensure that the right people had access to the right systems, a task that was critical given the sensitive nature of the data involved. The bank was already using Oracle Identity Manager (OIM), but transforming data into a format that OIM could understand required a large development team and a cumbersome process of getting scripts into production. The bank needed a solution that was repeatable, configurable, and could be implemented quickly. Another challenge was the over-reliance on one team member for validating user permissions, which led to business risk and a lack of transparency.

ProcessGold AG is a company that specializes in process mining, a Business Intelligence (BI) discipline that allows consultants to recreate, visualize, and analyze processes within a customer’s organization based on digital footprints. These footprints can be anything from records in SAP to financial transaction logs or any other participating source of information. The success of process mining heavily relies on the completeness, accuracy, and quality of the data that is fed to the analysis. Before the implementation of CloverETL, ProcessGold would consume up to 80% of their project time extracting the required data and preparing it for analysis. This was a significant challenge as it reduced the time available for core analytical business and ensuring heightened transparency for their customers.

The logistics company was facing a challenge with the expansion to new territories which meant more address data that was notoriously difficult to validate and interpret. Particularly in Emerging Markets, the company was facing vast regional differences in both address structure and rules, and had to contend with the availability of tools and databases, or lack thereof, to process them. The company had assembled a data quality team tasked with manually verifying and cleaning the data, working shifts to meet overnight delivery deadlines. However, the ever-growing need for manpower to process more data with a quick turnaround became a bottleneck that blocked the company’s ability to expand further.

GoodData, a cloud-based BI tool provider, was looking for a best-of-breed technology to build their CloudConnect platform. They needed a solution that would allow them to access data and get it into their platform efficiently. The company was previously using a REST API-based platform, which worked well for developers familiar with these processes or who had integration technologies in-house. However, this system was not user-friendly for a larger customer base. GoodData needed a solution that would allow anyone to design integrations and do data modeling work visually.

Allant Group, a marketing service provider, was dealing with an increasing amount of diverse data formats due to the surge of international data. They were using a multitude of homegrown applications for data manipulation, which was becoming increasingly difficult to manage with the growing data complexity. The expectation of faster turnaround, higher quality data, and more quality control steps in the process run on data led Allant to search for an ETL to support this. They needed a tool that could integrate with their existing software packages and legacy software, and was easy to use so that the data processing workload could shift from the programmer-side to an operational staff.

MyPublisher, a leading online retailer of personalized Photo Applications, was facing a challenge in managing its retail and marketing data. The company wanted to tie this data closely with web traffic and email data to capture customer activity. However, the primary challenge for the data warehouse team was to bring together disparate sources of data in the company. Before the introduction of a data warehouse, MyPublisher was not using a data warehouse at all – ad hoc reports and a number of systems were used instead. This made it difficult for the company to gain a deeper insight into business activity, and do so on a recurring basis.

The digital insights team at EE, a UK telecommunications company, faced the challenge of integrating their business intelligence from various data sources into one place and visualizing them for a non-technical audience. The data sources included Adobe Analytics, VOC databases, clickstream records, and personalization tools. The team needed to pull data from these sources into Amazon RedShift and then push it all together into Tableau to see the story behind the data. The challenge was to handle this colossal amount of digital data with only a small team and without outsourcing the project to IT services, which would take months to set up and weeks for change requests.

Slovenska sporiteľňa, Slovakia’s biggest bank, required a simple and reliable solution for reconciliation of data from multiple Identity and Access Management systems. They needed an overarching ‘safety net’ that would pick up any violations and omissions. With such a large workforce user access requirements are constantly changing, so the task to keep the user privileges under control was a challenge. The team had given careful consideration to the all the options available to them. One route they pursued was the development of a ‘home-grown’ scripted solution, but it quickly became apparent that this approach had significant flaws. For one, it relied too heavily on one team member creating and managing the script. There was also the potential for the script to become complex and unwieldy over time.

Customology, a division of the GJI group, was facing challenges in managing and utilizing massive amounts of raw transactional and client data in myriad types and formats. As the business grew, it became increasingly difficult to manage the data without a comprehensive data management plan. The data they worked with was not standardized, which led to the development of a single common data model with different presentation layers to have adequate control over the data coming in. Their previous homegrown ETL solution comprised a mixture of off-the-shelf stacks—MSSQL with an ever-growing pool of stored procedures, shell scripts watching hot folders, and the beginnings of microservices written in PHP. While this setup could extract meaningful insights from data, it was slow, cumbersome, and difficult to scale.