Photo backup and sync using Google Cloud Bucket Storage

This morning I spotted a tweet by Greg Wilson whom I don't know and have never met. However it appears he's some kind of director at Google Cloud, so that might explain why I'm following him and why he tweeted this:

https://twitter.com/gregsramblings/status/1284743960955510787?s=20

Archiving my newly organized 238k+ photo library (2.2TB) to Google Cloud Storage with: gsutil -m rsync -r -d . gs://{mybucketname} I'm using crazy-cheap 'archive' storage class in single regionStorage price: $0.0012/GB/Month (!) @gcpcloud

This really interested me as I've been trying to upload photos to an AWS (Amazon Web Services) bucket recently but it's been a tedious process.

I am aware that various cloud services like dropbox and pcloud exist, but I want the following:

1. Encryption
2. Ease of use
3. Cheap

I really only want this as a backup in case of fire or failure of one of my crummy old USB HDDs fails. I genuinely don't understand why so many of these services make such a bad job of encryption or offer it as some kind of bizarre add on. Come on people this is 2020, it's not hard and it shouldn't be expensive.

So I realised with some basic AWS skills I can spin up a bucket running glacier for pennies a month. This worked really really well, but the upload tools are slow and not terribly reliable for bulk uploads. Which I guess is understandable because the whole point of AWS is to build your own right?

Back to my original point, I stumbled across this tweet and thought hmm, that sounds nice and easy. Let's give it a go on a dull lockdown Sunday.

The following I followed from here: https://cloud.google.com/storage/docs/quickstart-gsutil

1. Install Python 3.8

Did someone say snakes on a plane?

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.8

2. Download & Extract Google Cloud SDK
https://cloud.google.com/sdk/docs

3. Create a bucket
Then I created a new Google Cloud Project on their cloud console and a new Cloud Storage Bucket. This was actually really easy and the bucket creation process easily walked me through the access policies and encryption setup. Although to be fair the AWS process is pretty good now as well. Don't forget to select your preferred storage type but bare in mind different types have different latency, pricing and minimum storage commitments.
https://cloud.google.com/storage/pricing#operations-pricing

4. Initialise the sdk
Run gcloud init this will initialise your sdk and link it to your google account. You can then of course select the project you're working on.

5. Copy a file
That's pretty much it, from there you can copy individual files using gloud cp or as Greg's tweet suggest you can use the sync option for bulk upload.

This is such a simple process I was really impressed.

One thing it's important to note is that Google Storage buckets may have a minimum storage time period:
https://cloud.google.com/storage/pricing#archival-pricing

Firebase Tutorial For Android Developers

I've sadly never had the chance to use Firebase in a production app. After Google talked about it so much at this year's IO I though it was about time I had a play and created a tutorial.

Firebase is primarily a real time database that makes data storage simple and easy. In the last few years it's been expanded to be an incredible fleet of tools that Android, iOS and Web developers can make use of.

The Firebase database is an inspired solution to data storage and syncing. The Firebase db is a NoSQL, cloud hosted database that allows for fast and easy sync across different devices. No more clumsy sync adapters or troublesome push notifications, Firebase handles all this for you with a simple API.

Also included in the suite is analytics, crash reporting, file storage, authentication, remote config and more.

I know I'm beginning to sound a bit like a marketing rep, but I really must say I'm flawed by how awesome Firebase is. The cloud sync database is really simple to set-up and it works incredibly well.

OK let's get into a tutorial. I wanted to link my app to Firebase and create a cloud db. Sounds easy! First you will be required to have a Google account, then we goto Firebase and create a new project:

https://console.firebase.google.com/

Now you have a Firebase app. Next we need to add it to Android, the wizard walks you through this process with a lovely little Material designed guide. You'll need the package name of your app and then to make some gradle and config changes.

Now we need to disable authentication. If you goto the Firebase console and click database on the left hand side you should see an empty data set. In the tabs above click on rules and change the json to match this:

{
  "rules": {
    ".read": true,
    ".write": true
  }
}

This basically disables all authentication giving anyone write and read access. For a real application you'd want to set-up proper user role based access, but for this tutorial will just turn authentication off.

Next you'll need to setup your Android project. You should already have the json config file and the gradle additions. Now you need to add the database gradle dependancy:

compile 'com.google.firebase:firebase-database:9.0.0'

That's it, you're pretty much ready to go. However there are a few important db constraints to get your head around before you launch your new app.

Data in Firebase is all JSON but it does support saving objects, lists, maps as well as simple data types such as strings, booleans, doubles etc. It seems to me largely based around a key value pair type of system. You put a value with it's corresponding key to Firebase and it syncs it. Then use the same key to retrieve that data.

SetValue

SetValue is an assignment method. At first I thought you could use it to create new db entries, but that is not the case. Instead you call

FirebaseDatabase database = FirebaseDatabase.getInstance();
DatabaseReference myRef = database.getReference("message");

myRef.setValue("Hello, World!");

This creates a key value pair with the key "message" and the value "Hello, World!". If you call setValue again on that key, the value is overwritten, a new pair is not created in this way.

ValueEventListener

The ValueEventListener gives you the ability to receive a callback when your data changes. This is useful when you create your UI so you can receive updates whenever the data changes. No more db listeners or broadcasts.

public void onDataChange(DataSnapshot dataSnapshot)

All you need to do is attatch the listener

myRef.addValueEventListener(this);

Push

Push allows create a list of data. Using this method generates a unique key instead of forcing you to create a new key for every object.

Save Data

Here's how I added a list of simple objects. Player is just a simple pojo. We use push to create a new child of the DB_KEY_PLAYERS type. We then get the key of that new child.
Using that key we can now update it's data. The second param to setValue is a callback so we can update the UI when the data has been updated successfully.

FirebaseDatabase database = FirebaseDatabase.getInstance();
DatabaseReference myplayers = database.getReference(DB_KEY_PLAYERS);

//Get key for a new player
String key = myplayers.push().getKey();
Player player = new Player(playername);
//set value of new player
myplayers.child(key).setValue(player, ActivityAddPlayer.this);

That's it, you should now have a working cloud synch'd database. You can now add analytics and crash reporting as well, super simple and very well designed. I really hope I get to put it into practice in a live app sometime soon.

This content is copyright and owned by https://webdeveloperpadawan.blogspot.com/