Migrating Node.js and MongoDB from On-Premises to AWS Cloud with latest database changes

Step-by-Step Guide for Synchronizing MongoDB Data

1. Initial Data Migration

• Use AWS Database Migration Service (DMS):
  • Set up an AWS DMS replication instance.
  • Create a target endpoint for Amazon DocumentDB (or MongoDB on EC2).
  • Create a source endpoint for your on-premises MongoDB.
  • Configure the migration task to perform a full load of your existing data.

2. Continuous Data Replication

• Enable Change Data Capture (CDC):
  • Configure DMS to capture ongoing changes from your on-premises MongoDB.
  • This ensures that any changes (inserts, updates, deletes) in the on-premises database are continuously replicated to the cloud.

3. Data Consistency Check

• Verify Data Consistency:
  • After the initial full load, perform a data consistency check to ensure all data has been correctly migrated.
  • Use tools like mongodiff to compare data between the on-premises MongoDB and the cloud MongoDB.

Detailed Steps

Initial Data Load

1. Set Up DMS Replication Instance:
   • Go to the AWS DMS console and create a replication instance with enough capacity to handle your data migration workload.
2. Create Source and Target Endpoints:
   • Source Endpoint (On-premises MongoDB):
     • Ensure your on-premises MongoDB is accessible from the replication instance. This may involve setting up a VPN or Direct Connect.
     • In the DMS console, create a source endpoint with the connection details of your on-premises MongoDB.
   • Target Endpoint (Amazon DocumentDB or MongoDB on EC2):
     • Set up Amazon DocumentDB or a MongoDB instance on an EC2 instance in your AWS environment.
     • Create a target endpoint in the DMS console with the connection details of your cloud MongoDB.
3. Create and Run Migration Task:
   • In the DMS console, create a migration task.
   • Select your source and target endpoints.
   • Configure the task to perform a full load followed by CDC.
   • Start the migration task. DMS will initially perform a full data load and then switch to capturing ongoing changes.

Continuous Data Replication

1. Enable Change Data Capture (CDC):
   • Ensure that the DMS task is configured to capture ongoing changes.
   • Monitor the task to ensure it is running and capturing changes without issues.

Data Consistency Check

1. Verify Initial Load Consistency:
   • Use tools like mongodiff to compare data between the on-premises MongoDB and the cloud MongoDB after the initial load.
   • Resolve any discrepancies found during the comparison.
2. Monitor Ongoing Replication:
   • Continuously monitor the DMS task to ensure it is capturing and replicating changes correctly.
   • Set up alerts and notifications for any issues or errors in the replication process.

Example Configuration for AWS DMS

Source Endpoint JSON Configuration

{
  "EndpointIdentifier": "source-mongodb",
  "EndpointType": "source",
  "EngineName": "mongodb",
  "Username": "your-mongodb-username",
  "Password": "your-mongodb-password",
  "ServerName": "your-onprem-mongodb-server",
  "Port": 27017,
  "DatabaseName": "your-database-name",
  "ExtraConnectionAttributes": "authSource=admin"
}

Target Endpoint JSON Configuration

{
  "EndpointIdentifier": "target-documentdb",
  "EndpointType": "target",
  "EngineName": "docdb",
  "Username": "your-docdb-username",
  "Password": "your-docdb-password",
  "ServerName": "your-docdb-endpoint",
  "Port": 27017,
  "DatabaseName": "your-database-name",
  "ExtraConnectionAttributes": "authSource=admin"
}

Migration Task JSON Configuration

{
"ReplicationTaskIdentifier": "mongodb-to-docdb-migration",
"SourceEndpointArn": "arn:aws:dms:region:account:endpoint/source-mongodb",
"TargetEndpointArn": "arn:aws:dms:region:account:endpoint/target-documentdb",
"MigrationType": "full-load-and-cdc",
"TableMappings": {
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "your-database-name",
"table-name": "%"
},
"rule-action": "include"
}
]
},
"ReplicationTaskSettings": {
"FullLoadSettings": {
"TargetTablePrepMode": "DO_NOTHING"
},
"Logging": {
"EnableLogging": true
}
}
}

Conclusion

By following these steps, you can synchronize your latest data from on-premises MongoDB to AWS Cloud services with minimal downtime and data loss. Using AWS DMS ensures a reliable and efficient migration process, continuously replicating data changes to keep your cloud database in sync with your on-premises MongoDB.