AWS DMS

Posted on Edited on In , , , , Symbols count in article: 11k Reading time ≈ 10 mins.

Introduction

Migrate your databases to AWS with minimal downtime

AWS Database Migration Service
AWS Database Migration Service Documentation

AWS Database Migration Service helps you migrate databases to AWS quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. The AWS Database Migration Service can migrate your data to and from most widely used commercial and open-source databases.

AWS Database Migration Service supports homogeneous migrations such as Oracle to Oracle, as well as heterogeneous migrations between different database platforms, such as Oracle or Microsoft SQL Server to Amazon Aurora. With AWS Database Migration Service, you can continuously replicate your data with high availability and consolidate databases into a petabyte-scale data warehouse by streaming data to Amazon Redshift and Amazon S3. Learn more about the supported source and target databases.

When migrating databases to Amazon Aurora, Amazon Redshift, Amazon DynamoDB or Amazon DocumentDB (with MongoDB compatibility) you can use DMS free for six months

AWS Database Migration Service (AWS DMS) is a cloud service that makes it easy to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud, between on-premises instances (through an AWS Cloud setup), or between combinations of cloud and on-premises setups.

With AWS DMS, you can perform one-time migrations, and you can replicate ongoing changes to keep sources and targets in sync. If you want to change database engines, you can use the AWS Schema Conversion Tool (AWS SCT) to translate your database schema to the new platform. You then use AWS DMS to migrate the data. Because AWS DMS is a part of the AWS Cloud, you get the cost efficiency, speed to market, security, and flexibility that AWS services offer.

Using AWS DMS with other AWS services

Using AWS DMS with other AWS services

You can use AWS DMS with several other AWS services:

  • You can use an Amazon EC2 instance or Amazon RDS DB instance as a target for a data migration.
  • You can use the AWS Schema Conversion Tool (AWS SCT) to convert your source schema and SQL code into an equivalent target schema and SQL code.
  • You can use Amazon S3 as a storage site for your data, or you can use it as an intermediate step when migrating large amounts of data.
  • You can use AWS CloudFormation to set up your AWS resources for infrastructure management or deployment. For example, you can provision AWS DMS resources such as replication instances, tasks, certificates, and endpoints. You create a template that describes all the AWS resources that you want, and AWS CloudFormation provisions and configures those resources for you.

Working with an AWS DMS replication instance

Working with an AWS DMS replication instance

DMS Replication Instances’ VPC

Step 5: Create an AWS DMS Replication Instance

The migration workload is a huge burden. For reducing the influence on the source DB instance that needs to be queried by applications, we should reduce the cross-region or cross-AZ traffic from the source DB to DMS replication instance.

Configuration with two VPCs
If your source endpoint and target endpoints are in different VPCs, you can create your replication instance in one of the VPCs. You can then link the two VPCs by using VPC peering.

A VPC peering connection is a networking connection between two VPCs that enables routing using each VPC’s private IP addresses as if they were in the same network. You can create a VPC peering connection between your own VPCs, with a VPC in another AWS account, or with a VPC in a different AWS Region. For more information about VPC peering, see VPC peering in the Amazon VPC User Guide.

The following illustration shows an example configuration using VPC peering. Here, the source database on an Amazon EC2 instance in a VPC connects by VPC peering to a VPC. This VPC contains the replication instance and the target database on an Amazon RDS DB instance.

Decrease Provisioned Storage Size

How can I decrease the total provisioned storage size of my Amazon RDS DB instance?

Short description
After you create an Amazon RDS DB instance, you can’t modify the allocated storage size of the DB instance to decrease the total storage space it uses. To decrease the storage size of your DB instance, create a new DB instance that has less provisioned storage size. Then, migrate your data into the new DB instance using one of the following methods:

  • Use the database engine’s native dump and restore method.
    Note: This method causes some downtime.
  • Use AWS Database Migration Service (AWS DMS) for minimal downtime.

Data Validation

AWS Database Migration Service FAQs
AWS DMS data validation
Using AWS SCT with AWS DMS

Q: How are AWS Database Migration Service (DMS) and AWS Schema Conversion Tool (SCT) related?
DMS and SCT work in conjunction to both migrate databases and support ongoing replication for a variety of uses such as populating datamarts, synchronizing systems, etc. SCT can copy database schemas for homogeneous migrations and convert them for heterogeneous migrations. The schemas can be between databases e.g.) Oracle to PostgreSQL or between data warehouses e.g.) Netezza to Amazon Redshift.

Once a schema has been created on an empty target, depending on the volume of data and/or supported engines, either DMS or SCT are then used to move the data. DMS traditionally moves smaller relational workloads (<10 TB) and MongoDB, whereas SCT is primarily used to migrate large data warehouse workloads. DMS supports ongoing replication to keep the target in sync with the source; SCT does not.

Using an AWS SCT replication agent with AWS DMS
For very large database migrations, you can use an AWS SCT replication agent (aws-schema-conversion-tool-dms-agent) to copy data from your on-premises database to Amazon S3 or an Amazon Snowball Edge device. The replication agent works in conjunction with AWS DMS and can work in the background while AWS SCT is closed.

When working with Amazon Snowball Edge, the AWS SCT agent replicates data to the Amazon Snowball device. The device is then sent to AWS and the data is loaded to an Amazon S3 bucket. During this time, the AWS SCT agent continues to run. The agent then takes the data on Amazon S3 and copies the data to the target endpoint.

Tasks

Working with AWS DMS tasks

Creating a task

Creating a task

To create an AWS DMS migration task, you do the following:

  • Create a source endpoint, a target endpoint, and a replication instance before you create a migration task.
  • Choose a migration method:
    • Migrating data to the target database – This process creates files or tables in the target database and automatically defines the metadata that is required at the target. It also populates the tables with data from the source. The data from the tables is loaded in parallel for improved efficiency. This process is the Migrate existing data option in the AWS Management Console and is called Full Load in the API.
    • Capturing changes during migration – This process captures changes to the source database that occur while the data is being migrated from the source to the target. When the migration of the originally requested data has completed, the change data capture (CDC) process then applies the captured changes to the target database. Changes are captured and applied as units of single committed transactions, and you can update several different target tables as a single source commit. This approach guarantees transactional integrity in the target database. This process is the Migrate existing data and replicate ongoing changes option in the console and is called full-load-and-cdc in the API.
    • Replicating only data changes on the source database – This process reads the recovery log file of the source database management system (DBMS) and groups together the entries for each transaction. In some cases, AWS DMS can’t apply changes to the target within a reasonable time (for example, if the target isn’t accessible). In these cases, AWS DMS buffers the changes on the replication server for as long as necessary. It doesn’t reread the source DBMS logs, which can take a large amount of time. This process is the Replicate data changes only option in the AWS DMS console.

Task settings

Specifying task settings for AWS Database Migration Service tasks

Setting LOB support for source databases in an AWS DMS task

Setting LOB support for source databases in an AWS DMS task
How can I improve the speed of an AWS DMS task that has LOB data?

Large binary objects (LOBs) can sometimes be difficult to migrate between systems. AWS DMS offers a number of options to help with the tuning of LOB columns. To see which and when data types are considered LOBs by AWS DMS, see the AWS DMS documentation.

When you migrate data from one database to another, you might take the opportunity to rethink how your LOBs are stored, especially for heterogeneous migrations. If you want to do so, there’s no need to migrate the LOB data.

If you decide to include LOBs, you can then decide the other LOB settings:

  • The LOB mode determines how LOBs are handled:
    • Full LOB mode – In full LOB mode AWS DMS migrates all LOBs from source to target regardless of size. In this configuration, AWS DMS has no information about the maximum size of LOBs to expect. Thus, LOBs are migrated one at a time, piece by piece. Full LOB mode can be quite slow.
    • Limited LOB mode – In limited LOB mode, you set a maximum size LOB that AWS DMS should accept. Doing so allows AWS DMS to pre-allocate memory and load the LOB data in bulk. LOBs that exceed the maximum LOB size are truncated and a warning is issued to the log file. In limited LOB mode, you get significant performance gains over full LOB mode. We recommend that you use limited LOB mode whenever possible.

Creating multiple tasks

Creating multiple tasks

Creating tasks for ongoing replication using AWS DMS

Creating tasks for ongoing replication using AWS DMS

You can create an AWS DMS task that captures ongoing changes to the source data store. You can do this capture while you are migrating your data. You can also create a task that captures ongoing changes after you complete your initial (full-load) migration to a supported target data store. This process is called ongoing replication or change data capture (CDC). AWS DMS uses this process when replicating ongoing changes from a source data store. This process works by collecting changes to the database logs using the database engine’s native API.

Note
You can migrate views using full-load tasks only. If your task is either a CDC-only task or a full-load task that starts CDC after it completes, the migration includes only tables from the source. Using a full-load-only task, you can migrate views or a combination of tables and views. For more information, see Specifying table selection and transformations rules using JSON.

Setting up for AWS Database Migration Service

Setting up for AWS Database Migration Service

Getting started with AWS Database Migration Service

Getting started with AWS Database Migration Service

AWS Database Migration Service (AWS DMS) helps you migrate databases easily and securely. You can migrate your data to and from most widely used commercial and open-source databases. AWS DMS supports nearly all of today’s most popular database engines as data sources, and provides a broad coverage of target engines. For more information about source and target endpoints, see Working with AWS DMS endpoints.

At the basic level, AWS DMS is a server in the AWS Cloud that runs replication software. You create a source and target connection to tell AWS DMS where to extract from and load to. Then you schedule a task that runs on this server to move your data. AWS DMS creates the tables and associated primary keys if they don’t exist on the target. You can precreate the target tables yourself if you prefer. Or you can use AWS Schema Conversion Tool (AWS SCT) to create some or all of the target tables, indexes, views, triggers, and so on.

The following diagram illustrates the AWS DMS replication process.

To perform a database migration, you do the following:

  • Complete the steps in Setting up for AWS Database Migration Service.
  • Create a replication instance to perform all the processes for the migration.
  • Specify source and target database endpoints.
  • Create a task or set of tasks to define what tables and replication processes you want to use.

AWS DMS can then create your replication instance and perform the tasks on the data being migrated.

You begin a database migration in one of two ways:

  1. You can choose the AWS DMS console and perform each step there.
  2. You can use the AWS Command Line Interface (AWS CLI). For more information about using the CLI with AWS DMS, see AWS CLI for AWS DMS.

Best practices

Best practices

AWS Database Migration Service Step-by-Step Walkthroughs

AWS Database Migration Service Step-by-Step Walkthroughs

You cannot use DMS to migrate a on-premises database or a RDS database to Aurora cluster directly. For the former, you can use DMS to migrate a on-premises database to S3, then create an Aurora cluster by S3 files; for the latter, you can create a Aurora read replica then promote Aurora cluster.