From GCP to AWS: Cyble’s Path to Better Performance, Cost Control, and Scalability

Project Overview:  

Cyble, a leading cybersecurity company that provides threat intelligence and cyber risk monitoring services. Cyble Hawk the core product of Cyble, relies on heavy data operations and it involves running multiple queries for anomaly detection in almost real time frequency. Cyble was leveraging GCP’s Big Query, which was well suited for spiky workloads and running rapid queries few times a day, making it an expensive option and was not meeting the expected Cost to performance index as BigQuery’s pricing model based on the volume of processed data.  

Cyble required a more cost-effective and efficient alternative for handling their data warehouse (DW) workloads, looking at AWS for a potential solution.  

They also faced the challenge of migrating its infrastructure from Google Cloud Platform (GCP) to Amazon Web Services (AWS) within a tight two-month deadline, before their service agreement with GCP expires. Their existing setup involved interdependent Google projects, Kubernetes clusters hosting ElasticSearch, and GCP’s native services like Cloud Compose and BigQuery. Cyble engaged Minfy to lead their migration, emphasizing the need to optimize their core services and implement efficient solutions in line with best practices.

‍

Challenges:

Optimization and Migration Deadline: Cyble's infrastructure included multiple Google projects and Kubernetes clusters hosting ElasticSearch workloads, as well as GCP-native services like BigQuery and Cloud Compose. The challenge was to migrate all workloads to AWS within two months, before their GCP contract expired, while ensuring optimized performance and compliance with industry standards.

Performance Optimization and Cost Control: Cyble needed a simplified architecture with optimized performance, reduced operational overhead, and tighter cost control. Additionally, their extensive use of BigQuery for multiple daily data queries made it an expensive option for their DW workloads. Since BigQuery charges based on the volume of data processed, it was not cost-effective for their frequent queries. Moving to AWS Redshift provided a better alternative by allowing them to run DW workloads within a set budget, capped by the number of nodes required.

‍

Solutions:  

Migration: Minfy migrated Cyble’s ElasticSearch workloads from GCP Kubernetes clusters to self-managed EC2 clusters on AWS, utilizing Application Load Balancers (ALBs) for scalability and control. Additionally, a dedicated AWS Landing Zone was established to enhance security and governance, implementing robust measures aligned with the AWS Well-Architected Framework and the AWS Cloud Adoption Framework. BigQuery was refactored to AWS EMR for big data processing, adopting a pay-as-you-go pricing model that improved the Price to Performance Index. AWS DataSync was employed for seamless bulk data transfer.

Infrastructure Optimization: Cloud Compose was replaced with Managed Workflows for Apache Airflow (MWAA) to streamline operations further. The overall architecture was optimized for performance and compliance, ensuring that Cyble's workloads ran efficiently while adhering to industry best practices.

Performance Optimization: By leveraging Graviton3 instances, Cyble gained a 60% improvement in power efficiency. The transition from Kubernetes to EC2 clusters eliminated the overhead of managing Kubernetes control planes, simplifying operations and improving cost efficiency. The architecture was optimized through proper sizing to meet SLA and uptime requirements, while the team also took advantage of AWS’s 3-year savings plan with a 1-year commitment for cost optimization.

‍

Business Outcome:  

Improved Efficiency and Cost Savings: Migrating to AWS and utilizing Graviton3 instances resulted in a 60% improvement in power efficiency. The transition from Kubernetes to EC2 clusters reduced the operational overhead and improved control over infrastructure, contributing to long-term cost savings through AWS savings plans.  

Simplified Architecture and Operations: By migrating workloads from GCP to AWS and eliminating Kubernetes control planes, Cyble's infrastructure became more streamlined, reducing operational complexity and ensuring better management of cloud resources.  

Performance and Scalability: The optimized setup improved performance, with the infrastructure being sized to meet SLA and uptime requirements, allowing Cyble to deliver on its business objectives with greater reliability.