Data migration to AWS

In this article, I’ll tell you how we migrated our databases from our previous provider to AWS RDS.
Our previous DB infrastructure was already in the cloud and the entire db was sharded, so we had about 350 GB of data split in about 1000 databases. Want to know more about how we achieved database sharding? Read here!

So, how did we migrate to AWS?

We could stop our apps for about an hour (I know, we were very fortunate!)  and so we designed the migration process to fit in this interval. We first designed the new infrastructure entirely as code using Ansible as automation tool. Hence, running some Ansible scripts, we created the entire new infrastructure (web, db, services, DNS, …) on AWS (we gave a speech at AWS Rome Meetup about it!).

Then, to move all the relational data, we had two possibilities: using DMS or make a custom Ansible playbook. We had about 15/20 minutes for moving data so, despite the fact DMS is a great solution, we choose to stick with custom migration scripts, being simpler than setting up DMS for our case!

Thanks for having everything as scripts, we could simulate the entire migration many times and so we were quite confident about the entire process.

Here is the main playbook for moving data.

Thanks to ansible_hostname we could declared the new hostname:

Then we read the databases list on the source server, made a dump and restored. Please note these three steps were done using “pipe”.

This is an example of the the inventory file:

Finally we created a bash script to run the playbook:

We also prepared a python script to compare migrated databases:

Here is an extract: for each server we compared the number of databases, the number of tables per database and the number of rows for each table.

We migrated about 350GB of data in 20 minutes, using Ansible parallelism (20 databases in parallel).

In the next article we’ll show you another chapter of our migration to AWS.

Stay tuned!

Loreno Edelmondo
Loreno Edelmondo
Articoli: 24

Lascia una risposta

Il tuo indirizzo email non sarà pubblicato. I campi obbligatori sono contrassegnati *

Questo sito usa Akismet per ridurre lo spam. Scopri come i tuoi dati vengono elaborati.