Case study

Data Operations pipeline

Valor developed a cloud-native data pipeline and warehouse for an e-commerce retailer, reducing inefficiencies, improving communication, and creating a custom ML solution to fit the company’s unique needs.

Background

A mid-sized e-commerce retail company (the name can ) was struggling with scaling its data pipeline and warehouse. The previous vendor had implemented an AWS-based solution using Lambda functions and Redshift for ETL processes. However, performance was inconsistent, and latency was becoming an issue. Meetings to resolve these problems consumed valuable management time, leading to delays in decision-making.

The company reached out to Valor, a software development company with niche expertise in e-commerce and retail, to design and implement a more agile and scalable solution using GCP services, leveraging Valor’s extensive domain knowledge.

Specifics

The client’s e-commerce platform was heavily reliant on real-time customer data and analytics. The existing solution had limitations, including:

High operational costs due to inefficient usage of cloud resources.

A rigid data pipeline that did not scale well during peak loads.

A generic machine learning model that often produced inaccurate results.

Frequent, lengthy meetings that delayed decision-making and implementation.

A need for better communication due to global time zone differences between internal teams.

Technologies & services

Deliverables

Custom Data Pipeline Implementation

Valor replaced the previous vendor’s AWS-based pipeline with a GCP-native solution tailored to the client’s real-time analytics needs. The solution included Dataflow for stream processing, Pub/Sub for message queuing, and BigQuery as the central data warehouse.

Challenge 1

Inefficient Data Processing

The client’s original solution utilized AWS Lambda functions and Redshift for ETL tasks. This led to high costs and performance issues, especially during peak load times. Processing delays were common, causing slow reporting and customer dissatisfaction.

Solution

Valor implemented GCP’s Dataflow service for real-time ETL processing, reducing latency and enabling the system to handle high loads more efficiently. BigQuery replaced Redshift, significantly lowering costs and improving query performance for real-time analytics.

Challenge 2

Unreliable Machine Learning Model

The client's existing solution used a generic, black-box machine learning model from AWS SageMaker that produced inconsistent results, especially in personalized recommendations. Debugging issues was difficult, and the business impact was unclear.

Solution

Valor developed a custom ML pipeline using TensorFlow on GCP’s AI Platform. This allowed for more accurate predictions tailored to the client’s specific data, which improved customer experience and resulted in better conversion rates.

Future

Valor continues to support the e-commerce client by providing regular updates to the custom data pipeline and ML models. There are plans to expand the solution to include predictive analytics for inventory management and more personalized customer experiences using advanced AI models.

Next Case

The booking app for non-profits

Valor’s client, being a volunteer organization and the provider of different kinds of help to the countries in need, has hundreds of staff representatives in numerous countries around the world.

Check it out