If you’re working with DynamoDB and CloudFormation, there’s little chance you haven’t been eagerly awaiting CloudFormation support for Global Tables. If you were like me, you were feverishly refreshing the GitHub support issue for this for a number of years, and getting frustrated every time a maintainer said “it’s coming in a few months” only for that to pass with radio silence.
Then finally! AWS launches CloudFormation support for Global Tables! But wait, it required a whole new resource from an existing DynamoDB table. If you read through the post launch comments, the new resource made it impossible to migrate existing DynamoDB tables to Global Tables without a whole lot of migration headache.
Either by complete chance, or maybe as a hidden feature, I did find a way around the migration from an AWS::DynamoDB::Table to an AWS::DynamoDB::GlobalTable without a single shred of data loss, recreating tables, or any kind of exhausting and expensive ETL jobs in AWS.
Turns out, you can adopt an existing AWS::DynamoDB::Table as an AWS::DynamoDB::GlobalTable with a few little tweaks to your CloudFormation template, and utilizing the CloudFormation resource adoption process.
Full disclosure, I have not discussed this solution with the DynamoDB service or development team, it’s completely undocumented anywhere, which means this process could be unintended and something the team will button up in the future, but as of the writing of this post we’ve migrated dozens of production DynamoDB tables in this way. Also, please test this outside of your production environment first, and be clear you’re happy with the results – I’d hate for something to get missed, and you accidentally delete your table full of data.
Step 1: Add a “Retain” DeletionPolicy to your AWS::DynamoDB::Table resources you want to migrate. Make sure you deploy this change to your template in AWS before continuing.
"MyTable": {
"Type": "AWS::DynamoDB::Table",
"DeletionPolicy": "Retain",
"Properties": {
"TableName": "MyDynamoTable",
"SSESpecification": {
"SSEEnabled": true
},
"StreamSpecification": {
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"BillingMode": "PAY_PER_REQUEST",
"PointInTimeRecoverySpecification": {
"PointInTimeRecoveryEnabled": true
},
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "id",
"KeyType": "HASH"
}
]
}
}Step 2: Add your AWS::DynamoDB::GlobalTable with the same settings as your original table.
Note that the SSESpecification is different in the AWS::DynamoDB::GlobalTable, and that the PointInTimeRecoverySpecification has moved into the Replicas property.
You can either hard code the initial Replicas region to the region you deploy to, or use {“Ref”:”AWS::Region”}.
"MyGlobalTable": {
"Type": "AWS::DynamoDB::GlobalTable",
"DeletionPolicy": "Retain",
"Properties": {
"TableName": "MyDynamoTable",
"StreamSpecification": {
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"BillingMode": "PAY_PER_REQUEST",
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "id",
"KeyType": "HASH"
}
],
"SSESpecification": {
"SSEEnabled": true,
"SSEType": "KMS"
},
"Replicas": [{
"Region": { "Ref":"AWS::Region" },
"PointInTimeRecoverySpecification": {
"PointInTimeRecoveryEnabled": true
}
}]
}
}Make sure you don’t ADD any additional Replicas here yet. We need to make this look identical to our deployed AWS::DynamoDB::GlobalTable so CloudFormation doesn’t think this is a new resource to deploy. My example is very light on additional properties like autoscaling, and GSI’s, but just ensure your configuration matches the existing table for the import and it’ll work. We have production tables using GSIs that adopted just fine. I have not tested this with tables utilizing autoscaling, or with tables that utilize Customer Managed Keys for encryption at rest, so your mileage may vary on those configurations.
Step 3. Adopt your table to CloudFormation using the resource adoption process in AWS. When you’re asked for the table name for the AWS::DynamoDB::GlobalTable, specify the TableName of your AWS::DynamoDB::Table.
Once the adoption process is complete, you can safely remove the AWS::DynamoDB::Table from your CloudFormation template. CloudFormation will skip deleting the physical resource because of the DeletionPolicy we set in step 1.
And voila! You have now migrated an existing table from the old to the new global table resource! You’re free to add additional replicas to your configuration, though be aware that CloudFormation only allows you to add or remove one replica per change.
…you can only add or remove a single replica in each update.
AWS Documentaion
I’d love to hear how you made out with the same process, this worked extremely well for us. Drop me a message on Twitter to let me know if you ran into an issue, or if this worked for you as well!