Migration Project

Authored by Ameex Technologies on 04 Sep 2019

Migrated 350 million records (700+ GB) from salesforce to AWS MySQL

Large data migration, including daily ETL processing of 10+GB of data, for an American multi-industry publishing conglomerate.

Current State

  • There were two source systems primarily – one to hold subscription details and another to hold customer details. There were 20+ tables in these two systems
  • There were few other systems such as Demand Tools which was used for cleansing and data deduping
  • Millions of rows of data were generated daily between these two systems and this was pushed into Salesforce through a nightly batch process after applying different transformations
  • Domo was the platform used to meet the reporting needs of the various stakeholders

Complications

  • Salesforce license was expiring within 3 months of the project initiation
  • Over 350GB of data from Salesforce had to be migrated to the new environment
  • Salesforce had complex transformation rules (50+ Apex Triggers) which had to be understood and replicated
  • There was no prior documentation available, making the understanding of business rules complex

Desired State

  • All the historical data (350+GB) is migrated from Salesforce into AWS
  • All the data transformation rules used in Salesforce have been replicated in AWS
  • Data from the two primary source systems and other systems (as appropriate) are now flowing seamlessly into AWS daily

Case Studies