Photo backup and sync using Google Cloud Bucket Storage

This morning I spotted a tweet by Greg Wilson whom I don't know and have never met. However it appears he's some kind of director at Google Cloud, so that might explain why I'm following him and why he tweeted this:

https://twitter.com/gregsramblings/status/1284743960955510787?s=20

Archiving my newly organized 238k+ photo library (2.2TB) to Google Cloud Storage with: gsutil -m rsync -r -d . gs://{mybucketname} I'm using crazy-cheap 'archive' storage class in single regionStorage price: $0.0012/GB/Month (!) @gcpcloud

This really interested me as I've been trying to upload photos to an AWS (Amazon Web Services) bucket recently but it's been a tedious process.

I am aware that various cloud services like dropbox and pcloud exist, but I want the following:

1. Encryption
2. Ease of use
3. Cheap

I really only want this as a backup in case of fire or failure of one of my crummy old USB HDDs fails. I genuinely don't understand why so many of these services make such a bad job of encryption or offer it as some kind of bizarre add on. Come on people this is 2020, it's not hard and it shouldn't be expensive.

So I realised with some basic AWS skills I can spin up a bucket running glacier for pennies a month. This worked really really well, but the upload tools are slow and not terribly reliable for bulk uploads. Which I guess is understandable because the whole point of AWS is to build your own right?

Back to my original point, I stumbled across this tweet and thought hmm, that sounds nice and easy. Let's give it a go on a dull lockdown Sunday.

The following I followed from here: https://cloud.google.com/storage/docs/quickstart-gsutil

1. Install Python 3.8

Did someone say snakes on a plane?

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.8

2. Download & Extract Google Cloud SDK
https://cloud.google.com/sdk/docs

3. Create a bucket
Then I created a new Google Cloud Project on their cloud console and a new Cloud Storage Bucket. This was actually really easy and the bucket creation process easily walked me through the access policies and encryption setup. Although to be fair the AWS process is pretty good now as well. Don't forget to select your preferred storage type but bare in mind different types have different latency, pricing and minimum storage commitments.
https://cloud.google.com/storage/pricing#operations-pricing

4. Initialise the sdk
Run gcloud init this will initialise your sdk and link it to your google account. You can then of course select the project you're working on.

5. Copy a file
That's pretty much it, from there you can copy individual files using gloud cp or as Greg's tweet suggest you can use the sync option for bulk upload.

This is such a simple process I was really impressed.

One thing it's important to note is that Google Storage buckets may have a minimum storage time period:
https://cloud.google.com/storage/pricing#archival-pricing

Amazon AWS Key Management Service with Android

I recently came across Amazon AWS's new Key Management Service (KMS). This seemed like a pretty cool idea so I thought I'd give it a go and see if I could get Amazon to manage my keys on Android.

The idea is pretty straightforward, Amazon host your secure keys for you, so you can encrypt and decrypt without having to worry about key management and storage. As far as I know KMS currently supports only symmetric encryption.

Here's how to use the Amazon Web Services console to create a Key Management Service (KMS) key:

1. Goto AWS management console and click services, under Security and Compliance click IAM
2. Goto groups on the left menu and click create new group
1. Create a group and name and click next
2. Search for “AWSKeyManagementServicePowerUser” and check it, click next
3. Goto users on the left menu and click add user
1. Give the user a name “kms_user” or something
2. Click programatic access
3. You should see your group in the add user to group section, check this
4. Click next
4. Click Encryption Keys at the bottom
1. Give the key a name
2. Do not give the key any administrators, for this tutorial the account owner will be the only one who can administer this key. Click next.
3. Give your kms_user user account use permissions for the key
4. Click next and finish adding the key
5. Setup Android Studio
   compile 'com.amazonaws:aws-android-sdk-kms:2.2.+'

You should now have an encryption key and be ready to start coding. For the Android part of this I used two Async tasks. You could do this in a service but it needs to be off the main UI thread as it's a network call.

public class AsyncEncrypt extends AsyncTask<String, String, ByteBuffer>{

    private static final String TAG = AsyncEncrypt.class.getSimpleName();

    public interface AsyncEncryptListener {
        void processFinish(ByteBuffer cipherText);
    }

    private AsyncEncryptListener listener;

    @Override
    protected ByteBuffer doInBackground(String... strings) {

        final AWSCredentials creds = new AWSCredentials() {
            @Override
            public String getAWSAccessKeyId() {
                return "xxx";
            }

            @Override
            public String getAWSSecretKey() {
                return "yyy";
            }
        };

        AWSKMSClient kms = new AWSKMSClient(creds);

        String keyId = "zzzzz";
        ByteBuffer bytePlainText = ByteBuffer.wrap(strings[0].getBytes());

        EncryptRequest req = new EncryptRequest().withKeyId(keyId).withPlaintext(bytePlainText);
        ByteBuffer ciphertext = kms.encrypt(req).getCiphertextBlob();

        Log.d(TAG, "onCreate: " + ciphertext.toString());

        return ciphertext;
    }

    @Override
    protected void onPostExecute(ByteBuffer cipherText) {
        super.onPostExecute(cipherText);
        if(listener != null){
            listener.processFinish(cipherText);
        }
    }

    public void setListener(AsyncEncryptListener listener){
        this.listener = listener;
    }
}

Decrypt:

public class AsyncDecrypt extends AsyncTask<ByteBuffer, String, String>{

     private static final String TAG = AsyncDecrypt.class.getSimpleName();

     public interface AsyncDecryptListener {
        void processFinish(String plainText);
    }

     private AsyncDecryptListener listener;

     @Override
    protected String doInBackground(ByteBuffer... ciphertextBlob) {

         final AWSCredentials creds = new AWSCredentials() {
            @Override
            public String getAWSAccessKeyId() {
                return "xxx";
            }

             @Override
            public String getAWSSecretKey() {
                return "yyy";
            }
        };

         AWSKMSClient kms = new AWSKMSClient(creds);

         DecryptRequest req = new DecryptRequest().withCiphertextBlob(ciphertextBlob[0]);
        ByteBuffer plainText = kms.decrypt(req).getPlaintext();

         String decoded = new String(plainText.array());
        Log.d(TAG, "onCreate: " + decoded);

         return decoded;
    }

     @Override
    protected void onPostExecute(String plainText) {
        super.onPostExecute(plainText);
        if(listener != null){
            listener.processFinish(plainText);
        }
    }

     public void setListener(AsyncDecryptListener listener){
        this.listener = listener;
    }
}

That's pretty much it. You create a set of AWS credentials supplying the security data given to you in the console for your user, then pass those credentials to the KMS client and make an encryption request.

The only other thing you might want to consider is whether it's worth it or not, in order to use KMS on Android you've got to store your secret key and access key somewhere. If an attacker can get those, they can access your encryption key. It's the classic chicken and egg scenario that distributed systems suffer from again and again. Oh well, it was a neat experiment.

At least it means it's much easier to rotate keys without having to re-release a new app!

Security isn't a dirty word Blackadder

Hopefully we all understand the concepts behind asymmetric (Public / Private key) encryption. It’s something we use all the time (https, SSL etc) but I've never actually put code to screen and tried to implement it. I've always relied on standard symmetric-key algorithm. So I thought I’d give it a go in Java / Android and along the way I learned a lot. In this blog article I thought I’d outline a few things I've learned. I’ll put some code up as I go, but I’d like to try and focus a bit more on the lessons.

All security is equal, except some more equal than others

We all know in symmetric encryption DES is bad, right?

DES is now considered to be insecure for many applications
http://en.wikipedia.org/wiki/Data_Encryption_Standard#Brute_force_attack

Well in theory public / private key encryption is infinitely better as long as you never give out your private key. Easy peasy, so we choose one of these and away we go:

• RSA
• EIGamal
• Diffie-Helman

So I want to take a basic string “All your base are belong to me” and I want to encrypt it using a public key and decrypt it using a private key.

Rush to code

Impatient as I am I rush to get some code in and find this code to try out:

byte[] data = readBytesFromfile(“mypublickey”);
PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(data); 
KeyFactory kf = KeyFactory.getInstance("RSA");
pubKey = kf.generatePublic(keySpec);

Great, now I need a public and private key to test my amazing new code.

I learn a few bits about encryption

I’d heard of PGP and apparently this has no known weaknesses, perfect, I want this! So away I go and find a tool to generate my PGP public and private key. I throw these into my java code and straight away get an error. Hmmm. Something doesn't feel right here. Plus I'm not too sure about PGP and OpenPGP. OK back to wikipedia. Apparently PGP is a protocol, not an algorithm and for its asymmetric files it uses RSA anyway! Doh! So I resort to RSA and now I need to generate an RSA key.

So I've already got puttyGen installed and it I don’t know why it says SSH but it definitely says RSA right there with a big button saying “Generate”. Boom, surely this will work, nothing can stop me now!

Now I get a java error InvalidKeySpecException

Hmm, after some serious time googling I ask stack overflow

http://stackoverflow.com/questions/28218636/invalidkeyspecexception-using-public-key

Although it doesn't solve my problem, Maarten alerts me to the fact that in java I need to use the X509 format.

Hang on SSH is a protocol isn't it? (Back to wikipedia). Right ok so PGP and SSH are protocols that implement asymmetric encryption using the RSA algorithm. I don’t want a protocol, I want to just generate an encrypted string.

To do that I need a public key which java can read. So all I need is to generate a key in the right format….That makes sense.

Using KeyTool and OpenSSL to generate a 509 certificate

This doesn't take me long using java keytool and openssl to generate an x509 certificate.

keytool -genkey -keyalg RSA -keysize 1024 -keystore C:\temp\hello.keystore

keytool -importkeystore -srckeystore C:/temp/hello.keystore -destkeystore C:/temp/hello.p12 -deststoretype PKCS12

openssl pkcs12 -in C:/temp/hello.p12 -out C:/temp/hellonew.pem -nodes

openssl x509 -in C:/temp/hellonew.pem -inform PEM -out C:/temp/hellodernew.der -outform DER

Reading an X509 certificate

Now we need to adapt our code to read the key in the new format.

public static PublicKey getPublicKey(byte[] keyBytes){
    PublicKey publicKey = null;

    if(keyBytes != null) {

        X509EncodedKeySpec spec = new X509EncodedKeySpec(keyBytes);
        KeyFactory kf = null;
        try {
            kf = KeyFactory.getInstance("RSA");
            publicKey = kf.generatePublic(spec);
        } catch (NoSuchAlgorithmException e) {
            Log.e(TAG, "NoSuchAlgorithmException");
            e.printStackTrace();
        } catch (InvalidKeySpecException e) {
            Log.e(TAG, "InvalidKeySpecException " + e.getMessage());
            e.printStackTrace();
        }
    }

    return publicKey;
}

This means we can now use our PublicKey to encrypt our message! Wohoo

Decryption

To decrypt we use a similar function

    public static PrivateKey getPrivateKey(byte[] keyBytes){
        PrivateKey privatekey = null;

         if(keyBytes != null) {
            PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(keyBytes);
            KeyFactory kf = null;
            try {
                kf = KeyFactory.getInstance("RSA");
                privatekey = kf.generatePrivate(spec);
            } catch (NoSuchAlgorithmException e) {
                Log.e(TAG, "NoSuchAlgorithmException");
                e.printStackTrace();
            } catch (InvalidKeySpecException e) {
                Log.e(TAG, "InvalidKeySpecException");
                e.printStackTrace();
            }
        }

         return privatekey;
    }

Encrypting a large file

For various reasons I've also learned you can’t just encrypt a file with your public key. This guy gives an excellent description of what you should be doing:

http://stackoverflow.com/a/16547038

The basic idea is as such:

1. Create a session key using symmetric encryption, AES for example.
2. Then encrypt the file using this session key
3. Using your public key encrypt the session key
4. Send the AES encrypted file and the public key encrypted session key together
5. Your server / recipient uses the private key they have and decrypts the session key then using that decrypts the file.

This method allows for fast encryption and secure transmission. If anyone brute forces the session key, they've only ever got one file. They need to start again for every single new file!

Anyway, that’s about what I've learned on this particular journey. I hope its of some help.