QNAP Backups With Glacier

I’ve been experimenting with glacier from aws to backup some of our content. The glacier service sounds very attractive (great durability, uptime etc.) but the pricing can be quite difficult to pick apart. This post documents an experiment to see how it might work for us and to get an idea of any potential costs. I think everything I want to do will fit in the free tier, but time will tell!

AWS CLI

First, we need to set up the aws-cli. We assume you already have accounts set up ready to use, but if not the IAM documentation can help you get started. Next, grab awscli from homebrew and run the configure task. This will create a config for you under ~/.aws in your home directory.

  $ brew upgrade awscli
  $ aws configure
  ...

AWS IAM Users

In addition to your AWS CLI access, we will need a user with programmatic access to glacier for qnap. In this configuration we are trusting the glacier app to manage our vault, and we are not using glacier for any other purpose, so we grant it full access to the glacier for the user. If you are uncomfortable with this you can tune the permissions more aggressive, and set vault lock policies on the vault.

  $ aws iam create-user --user-name qnap-backup

  $ aws iam list-policies \
    --query 'Policies[?PolicyName==`AmazonGlacierFullAccess`].{ARN:Arn}' \
    --output text
    arn:aws:iam::aws:policy/AmazonGlacierFullAccess

  $ aws iam attach-user-policy --user-name qnap-backup \
    --policy-arn arn:aws:iam::aws:policy/AmazonGlacierFullAccess

  $ aws iam list-attached-user-policies --user-name qnap-backup
{
    "AttachedPolicies": [
        {
            "PolicyName": "AmazonGlacierFullAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonGlacierFullAccess"
        }
    ]
}

  $ aws iam create-access-key --user-name qnap-backup
{
    "AccessKey": {
        "UserName": "qnap-backup",
        "AccessKeyId": "ACCESS_ID",
        "Status": "Active",
        "SecretAccessKey": "SECRET_ACCESS_KEY",
        "CreateDate": "2019-01-20T17:46:50Z"
    }
}

We will need the values of ACCESS_ID and SECRET_ACCESS_KEY for the next section where we configure glacier backup on qnap.

QNAP Glacier Backup

QNAP have instructions on setting up backups, for more details, but here is the process.

  1. Log in to your qnap as an administrator and install the Glacier backup app.
  2. Open and lick on “Backup” in the sidebar and “Create Job”.
  3. Select a backup folder (or folders)
  4. Choose “Start Manually/immediate” schedule. Once things are working this can be tweaked.
  5. For the policy, choose your options. I choose * Only back up updated files * Client side encryption We are using client side encryption here and there is server side encryption performed within glacier (both are AES-256). Make absolutely sure you make a record of the client encryption key somewhere secure!
  6. Apply any filters you need (probably none).
  7. Set your account and region * Account: qnap-backup * Region name: us-west-2 * Vault name: backup-qnap -> Test

N.B. It is best to set a new vault name and let the qnap create it, you might run into errors otherwise. There is discussion of this issue here, and the implication is it might be difficult or impossible to reconfigure an existing backup if you reinstall your qnap.

Verifying and Restoring data

The Glacier qnap app includes a restore task, but I want to walk through the manual process with glacier for completeness. These instructions are adapted from this blog.

Notifications

Retrieval tasks in glacier are asynchronous. Tasks will typically take a few hours to complete, and once they do, you usually have 24 hours to collect the results. With this in mind it is sensible to set up an SNS topic to receive notifications when the tasks complete.

  $ aws sns create-topic --name qnap-backup-sns
{
    "TopicArn": "arn:aws:sns:us-west-2:{YOUARACCOUNTID}:qnap-backup-sn"
}

  $ aws sns subscribe --topic-arn arn:aws:sns:us-west-2:{YOURACCOUNTID}:qnap-backup-sn \
    --protocol email --notification-endpoint "my-email@example.com"
{
    "SubscriptionArn": "pending confirmation"
}

Go to your email and confirm the subscription email which has been generated.

For most requests we can manually add the SNS endpoint to receive notifications individually, but for our case it is simpler to adjust the vault to always receive notifications,

  $ vi qnap-notifications.json
{
  "SNSTopic": "arn:aws:sns:us-west-2:{YOURACCOUNTID}:qnap-backup-sn",
  "Events": ["ArchiveRetrievalCompleted", "InventoryRetrievalCompleted"]
}

  $ aws glacier set-vault-notifications --account-id - --vault-name backup-qnap --vault-notification-config file://qnap-notifications.json

It will also be necessary to adjust the access settings for the topic to allow the glacier service to publish to your new topic. The general idea is discussed in this post. But you will need to set the service name to glacier.amazonaws.com and update your accountID and the name of the resource. Knowing the service name to use seems to be a bit of a black art, Amazon just tell you to use the “long name of the service”, but some kind people have compiled a list as a gist

{
"Sid": "__console_pub_0",
"Effect": "Allow",
"Principal": {
"Service": "glacier.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-west-2:{accountId}:backup-qnap"
}

We should also set a data retrieval policy to try to keep costs down.

  $ aws glacier set-data-retrieval-policy --account-id - --policy '{"Rules":[{"Strategy":"FreeTier"}]}'

Inventory

Get an inventory of the contents of your vault.

  $ aws glacier initiate-job --account-id - --vault backup-qnap \
  --job-parameters '{ "Type": "inventory-retrieval" }'

You can query the job status immediately, but it will take several hours to complete

  $ aws glacier list-jobs --account-id - --vault-name backup-qnap
{
    "JobList": [
        {
            "JobId": "jmplABaFs8l39U4VTuXUIcRfTBd_UbELq-xW3BpA59Kry_jkwDkoMITC9OSUYWoEGE8na77MV421FHFg15x9YotmhQ0m"
            "Action": "InventoryRetrieval",
            "VaultARN": "arn:aws:glacier:us-west-2:{YOURACCOUNTID}:vaults/backup-qnap",
            "CreationDate": "2019-01-20T02:17:53.793Z",
            "Completed": false,
            "StatusCode": "InProgress",
            "InventoryRetrievalParameters": {
                "Format": "JSON"
            }
        }
    ]
}

You will receive an email from the notification service when the action has completed (or failed) and you can then look at the job output to decide what should be restored (using the JobID).

  $ aws glacier get-job-output --account-id - --vault-name backup-qnap \
  --job-id "jmplABaFs8l39U4VTuXUIcRfTBd_UbELq-xW3BpA59Kry_jkwDkoMITC9OSUYWoEGE8na77MV421FHFg15x9YotmhQ0m" \
  backup-qnap-out

The content of backup-qnap-out (in the current directory) should be a json file with an ArchiveList of all of the available archives.

  $ python -m json.tooll backup-qnap-out
...
        {
            "ArchiveId": "mF5N2CA8fY2YyVE356l32-pEVOZteVkabZAhwXFjLi2taD53q1R0m0konWl-iV9KRZYfAOtl09PQV36Oz8U0J9_84zYsPLIScgHrdiF2ApG99ldeGDtTBaK_569rY4Y3iHpe7PJDyQ",
            "ArchiveDescription": "{\"path\": \"/container-vols/hello/backups/1542656186hello.tar\", \"type\": \"file\"}",
            "CreationDate": "2019-01-19T11:26:28Z",
            "Size": 100055072,
            "SHA256TreeHash": "9d5a353fb2dee5e73efb1f204762a565c365fc2bd7e1fb7af9940779595e8acf"
        },
...

We can use the associated ArchiveId to retrieve the data.

Retrieving Data

Retrieving data is also asynchronous. The request below uses the Bulk tier to make the request. Leaving out this parameter or setting it to another value failed with an error message that the request would violate the retrieval policy set above.

  $ vi backup-qnap-retrieval.json
{
    "Type": "archive-retrieval",
    "ArchiveId": "mF5N2CA8fY2YyVE356l32-pEVOZteVkabZAhwXFjLi2taD53q1R0m0konWl-iV9KRZYfAOtl09PQV36Oz8U0J9_84zYsPLIScgHrdiF2ApG99ldeGDtTBaK_569rY4Y3iHpe7PJDyQ",
    "Description": "Retreive tar file to confirm backup status",
    "Tier": "Bulk"
}

 $ aws glacier initiate-job \
  --account-id - \
  --vault-name backup-qnap \
  --job-parameters file://backup-qnap-retrieval.json
{
    "location": "/{YOURACCOUNTID}/vaults/qnap-backup/jobs/Hd5dF5p50owvy8uM8apKcMLIEJC9yeO1StbhGJxxNEL_32lnIIuY8edfOSHfKQvgEqj9pMMzKFBbnfV5fEmitYXWgJy9",
    "jobId": "Hd5dF5p50owvy8uM8apKcMLIEJC9yeO1StbhGJxxNEL_32lnIIuY8edfOSHfKQvgEqj9pMMzKFBbnfV5fEmitYXWgJy9"
}

Again we will be notified (by Email) when the job completes, but we can also check the job status manually.

  $ aws glacier list-jobs --account-id - --vault-name backup-qnap
{
    "JobList": [
        {
            "JobId": "Hd5dF5p50owvy8uM8apKcMLIEJC9yeO1StbhGJxxNEL_32lnIIuY8edfOSHfKQvgEqj9pMMzKFBbnfV5fEmitYXWgJy9",
            "JobDescription": "Retreive tar file to confirm backup status",
            "Action": "ArchiveRetrieval",
            "ArchiveId": "mF5N2CA8fY2YyVE356l32-pEVOZteVkabZAhwXFjLi2taD53q1R0m0konWl-iV9KRZYfAOtl09PQV36Oz8U0J9_84zYsPLIScgHrdiF2ApG99ldeGDtTBaK_569rY4Y3iHpe7PJDyQ",
            "VaultARN": "arn:aws:glacier:us-west-2:{YOURACCOUNTID}:vaults/backup-qnap",
            "CreationDate": "2019-01-20T19:00:39.881Z",
            "Completed": false,
            "StatusCode": "InProgress",
            "ArchiveSizeInBytes": 100055072,
            "SHA256TreeHash": "9d5a353fb2dee5e73efb1f204762a565c365fc2bd7e1fb7af9940779595e8acf",
            "ArchiveSHA256TreeHash": "9d5a353fb2dee5e73efb1f204762a565c365fc2bd7e1fb7af9940779595e8acf",
            "RetrievalByteRange": "0-100055071",
            "Tier": "Bulk"
        },
...

When the job completes we can actually download the archive

  $ aws glacier get-job-output \
  --account-id - \
  --vault-name backup-qnap \
  --job-id "Hd5dF5p50owvy8uM8apKcMLIEJC9yeO1StbhGJxxNEL_32lnIIuY8edfOSHfKQvgEqj9pMMzKFBbnfV5fEmitYXWgJy9"
  backup.tar

Our output will be a file called backup.tar. This file will be in the same format as whatever was originally uploaded, but will still be AES-256 encrypted. To decrypt it we can use OpenSSL along with the passphrase we specified when creating the qnap backup job.

  $ openssl enc -d -aes-256-cbc -k PASSPHRASE-HERE -in backup.tar \
  -out backup-plain.tar

The resulting tar file should be identical to the original file in the folder we selected to backup from the qnap. If you hit an error when decrypting the archive e.g.

bad decrypt
140266290147392:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../crypto/evp/evp_enc.c:536:

check the version of openssl being used. The qnap encryption is done with OpenSSL 1.0.2s 28 May 2019. Decrypting with OpenSSL 1.1.0l 10 Sep 2019 throws the error above. The root cause is that the default digest was changed from MD5 to SHA256 in OpenSSL 1.1.0. The simplest thing might be to do the decryption on the qnap or you can manually specify the digest e.g.

openssl enc -md md5 -d -aes-256-cbc -in backup.tar > backup-plain.tar