Git on Windows with Putty

This is something that's super simple on Linux but every couple of years I have to setup a new Windows machine with GIT and Putty (pageant to be exact). Every time I do it I seem to forget one of the steps and spend a few hours trying to remember what I've missed. So here is the note to myself so I can remember how to do it.

Components to install:

1. Putty including of course puttygen and pageant

2. Git Bash

1) Generate a ssh key 

I won’t go into details here as there are a million tutorials already on the internet but you want to use Puttygen to create a new private/public security key pair. You can save this as a .ppk file on your local machine.

2) Upload the public key to GitLab / Github

Copy the public key from inside Puttygen to GitLab or GitHub. You'll need to find the page on your account where you can add an SSH key and paste it in there. Be sure this is the public key and not the private one.

3) Run pageant 

Run pageant and it should appear in your system tray. Now open it and add the .ppk to your keys.

4) Add a GIT_SSH environment variable

Using this for advice:

https://stackoverflow.com/a/43313491

Create an environment variable that points to plink. This is probably the step I forget the most.

5) Configure GIT

You now need to ensure you have the correct url added to your git repo. A few times I've used the http one or the wrong url or something. Carefully copy the SSH (git@gitlab.com...) url from Gitlab or Github and add this to your git project as a remote repo url. You can check you've done this correctly by typing git remote -v

That's it. You should definitely restart Git bash and restarting your PC wouldn't hurt either, just make sure you start up pageant again. Another thing I forget is how to get this to run on startup. Oh it's a complicated life!

Publishing Android Jacoco test results to SonarQube

This is something I really though would be pretty straightforward but I wanted my Team City build server to push my unit test coverage to SonarQube. 

SonarQube is a really cool application that shows you all sorts of really interesting insights and problems with your code. Inefficiencies, security problems and tech debt are all highlighted by SonarQube in a nicely presented dashboard. It works with all sorts of languages but of course with Android we're focusing on Java and Kotlin.

I'd managed to get SonarQube setup and the basic code details pushed over fairly easily, but I couldn't get the test coverage over.

There are of course other ways around, I could have installed a better plugin on Team City which would have helped I'm sure, or I could have installed a Sonar Gradle plugin, but I didn't want to do that.

The trick ended up being to get xml test coverage. Once I had that sorted I could push that xml report to Sonar and it started showing my test coverage.

Nothing I did in the standard build.gradle file seemed to work, the standard jacoco plugin seemed to ignore my pleas for xml coverage reports. Eventually I found a really great little script that helped me get there.

https://medium.com/wandera-engineering/android-kotlin-code-coverage-with-jacoco-sonar-and-gradle-plugin-6-x-3933ed503a6e

Big thanks go to the Auther of this piece for figuring this out.

RxJava filter an Observable list

I had an observable with data type list, and I wanted to filter the objects in that list and maintain the observable. Sounds simple enough right, but it took a while and I wanted to record it as I've used this concept a few times now.

So let's start with our object of which we will create the list.

data class Muppet(var name: String, var age: int)

Now let's create a list of those objects

val myList = Observable.just(arrayListOf<Muppet>())

Simple stuff so far. Now let's get to the interesting Rx goodness. 

What we need is to extract the individual objects so we can filter them, so we need an observable of objects rather than an observable of a list. 

.flatMap { iterable: List<Muppet> -> Observable.from(iterable) }

So here we use flatmap to change the data type and we use the very clever Observable.from to change the data to a single result.

Now we can filter as our heart desire

.filter { muppet: Muppet -> muppet.age > 30 }

and put it back together using toList.

Here's the complete example

return getMuppets()
       .flatMap { iterable: List<Muppet> -> Observable.from(iterable) }
       .filter { muppet: Muppet -> muppet.age > 30 }
       .toList()

Android Hilt example and a unit test (with Robolectric)

Android Hilt has been released and I must say I'm quite impressed. I absolutely hate Dagger which I think comprises of far too much boiler plate code, its badly documented and incredibly confusing. Hilt is of course built on top of dagger and I think this is a huge win for the Android community. Hilt decreases confusion, increases code generation and is nice and simple.

I don't want to go into what Dependency Injection (DI) is, that's been done to death.

There are two resources I found really useful to get started with Hilt:

The Hilt documentation - https://dagger.dev/hilt/

Coding with Mitch videos: https://www.youtube.com/watch?v=zTpM2olXCok

There are two things I want to talk about in this blog post:

1. A really quick intro and setup of a Hilt module.
2. A unit test using Robolectic.

Before I go much further I would discourage you from using Robolectric. It's become a real pain to work with, requiring Java 9 for its latest version and regularly deprecating things without clear documentation on how to migrate. That said I've been using Robolectic for a while so I'm stuck with it until I can remove it from my code. I'm hoping Hilt and ViewModels will help me do this.

The first thing I wanted to look at is (I think) a fairly standard use case where a class has some dependencies and we want to use that class in our activity without creating it every time.

Step 1 - The application

@HiltAndroidApp
class MyApplication : Application()

Step 2 - The Class

class Petrol(val input: String) {

    fun getTheInfo(): String {
        return input
    }
}

Step 3 - The Module

Now this is the interesting bit, here we define a "factory" where we list how the class is to be created. Obviously this is a very simple example, but I think that's a good way to start.

@InstallIn(ApplicationComponent::class)
@Module
class PetrolFactory {

    @Provides
    fun getPetrol(): Petrol {
        return Petrol("Petroll")
    }
}

Step 4 - The Activity

Now we setup the activity and you'll note how little code we need to write to create and start using the class

@AndroidEntryPoint
class MainActivity : AppCompatActivity() {

    @Inject
    lateinit var petrol: Petrol

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        setSupportActionBar(findViewById(R.id.toolbar))
        
        findViewById<TextView>(R.id.main_text).text = petrol.getTheInfo()
    }
}

I think this is fantastic and hopefully it's clear to see how easy this is and how quick you can make complex interactions really simple.

Step 5 - The Tests

With the tests I want to show how you would pass in something different for the purpose of testing. 

@UninstallModules(PetrolFactory::class)
@HiltAndroidTest
class MainActivityTest : RobolectricTestConfig() {

    @get:Rule
    var hiltRule = HiltAndroidRule(this)

    @Module
    @InstallIn(ApplicationComponent::class)
    class TestEngine {
        @Provides
        fun getPetrol(): Petrol {
            return Petrol("TestPetrol")
        }
    }


    @Before
    fun init() {
        hiltRule.inject()
    }

    @Test
    fun testActivityMainText() {
        val scenario = ActivityScenario.launch(MainActivity::class.java)

        scenario.onActivity {
            assertEquals("TestPetrol", it.findViewById<TextView>(R.id.main_text).text)
        }
    }
}

One thing that's really important to remember is you can't run the test with the green arrow in Android Studio. You need to use gradle. So just run the gradle task instead. That's it I hope that helps demonstrate how you would create a DI app with Hilt in a few easy steps. I also hope you can see how easy that makes testing.

Photo backup and sync using Google Cloud Bucket Storage

This morning I spotted a tweet by Greg Wilson whom I don't know and have never met. However it appears he's some kind of director at Google Cloud, so that might explain why I'm following him and why he tweeted this:

https://twitter.com/gregsramblings/status/1284743960955510787?s=20

Archiving my newly organized 238k+ photo library (2.2TB) to Google Cloud Storage with: gsutil -m rsync -r -d . gs://{mybucketname} I'm using crazy-cheap 'archive' storage class in single regionStorage price: $0.0012/GB/Month (!) @gcpcloud

This really interested me as I've been trying to upload photos to an AWS (Amazon Web Services) bucket recently but it's been a tedious process.

I am aware that various cloud services like dropbox and pcloud exist, but I want the following:

1. Encryption
2. Ease of use
3. Cheap

I really only want this as a backup in case of fire or failure of one of my crummy old USB HDDs fails. I genuinely don't understand why so many of these services make such a bad job of encryption or offer it as some kind of bizarre add on. Come on people this is 2020, it's not hard and it shouldn't be expensive.

So I realised with some basic AWS skills I can spin up a bucket running glacier for pennies a month. This worked really really well, but the upload tools are slow and not terribly reliable for bulk uploads. Which I guess is understandable because the whole point of AWS is to build your own right?

Back to my original point, I stumbled across this tweet and thought hmm, that sounds nice and easy. Let's give it a go on a dull lockdown Sunday.

The following I followed from here: https://cloud.google.com/storage/docs/quickstart-gsutil

1. Install Python 3.8

Did someone say snakes on a plane?

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.8

2. Download & Extract Google Cloud SDK
https://cloud.google.com/sdk/docs

3. Create a bucket
Then I created a new Google Cloud Project on their cloud console and a new Cloud Storage Bucket. This was actually really easy and the bucket creation process easily walked me through the access policies and encryption setup. Although to be fair the AWS process is pretty good now as well. Don't forget to select your preferred storage type but bare in mind different types have different latency, pricing and minimum storage commitments.
https://cloud.google.com/storage/pricing#operations-pricing

4. Initialise the sdk
Run gcloud init this will initialise your sdk and link it to your google account. You can then of course select the project you're working on.

5. Copy a file
That's pretty much it, from there you can copy individual files using gloud cp or as Greg's tweet suggest you can use the sync option for bulk upload.

This is such a simple process I was really impressed.

One thing it's important to note is that Google Storage buckets may have a minimum storage time period:
https://cloud.google.com/storage/pricing#archival-pricing

Android library using GitLab as an aar repository

As your Android project begins to grow you will likely start to accumulate a few libraries. Not the official ones, like rxjava, androidx and so on, but I'm talking about ones you've developed yourself. These could be your entire network layer, your really really useful utils folder, or whatever. If you're a good developer you'll have these in separate modules or possibly even completely separate projects. To start with this is fine, you may be copying your aar files around or just being vocal about changes to your modules so your teammates can keep up to date.

This article gives a fantastic explanation of why it's a good idea to stop doing that:
https://inthecheesefactory.com/blog/how-to-setup-private-maven-repository/en

The author makes some excellent points why modules and copying aars are a bad idea. The ones I most agree with are:

Library distributed to other developers should not be modifiable. In case there is some problem, issue should be reported to developer involved to let them fix. Letting other developers directly access the source code might cause them accidentally doing a quick fix by themselves which will cause a big problem afterwards. 

Most of the problem listed above are solved except one: it's still hard to update and roll back over the version.

The solution is to use a repository. Basically give your library a specific version and deploy it to a repository so it can be referenced simply using nothing more than a gradle import. Luckily for us there are a few repositories out there and they can be easily made private. So we only allow permitted developers access to our libraries.

Jfrog is a great way to go, but this means you've probably got to manage a server yourself. What I was hoping for was a cloud method where I had no server admin and no setup process. So I turned to my old friend GitLab.

Now the first and most important thing to note is that the GitLab repository pattern (named packages) is not free. GitLab Packages is only currently available on the Silver package, which does incur a monthly cost. Although they do have a trial period, if you just want to give it a go. There is a project level setting to enable packages, but for me this was already turned on.

So that said, let's get to the code. First step is to create a library.

I created an Android library named "WorstLibEver" this is the extent of it:

public class HodorUtils {
    public String askHodor(String question){
        return "Hodor";
    }
}

I know, you're blown away aren't you?

Now we need to edit our gradle file:

First add this line to the top:

apply plugin: 'maven-publish'

Next you'll need to add the following to the bottom:

task sourceJar(type: Jar) {
    from android.sourceSets.main.java.srcDirs
    classifier "sources"
}

publishing {
    publications {
        bar(MavenPublication) {
            groupId 'com.test'
            artifactId 'worstlibever'
            version '1.7'
            artifact(sourceJar)
            artifact("$buildDir/outputs/aar/app-release.aar")
        }
    }
    repositories {
        maven {
            url "https://gitlab.com/api/v4/projects/zzz/packages/maven"
            credentials(HttpHeaderCredentials) {
                name = "Private-Token"
                value = GITLAB_PERSONAL_TOKEN
            }
            authentication {
                header(HttpHeaderAuthentication)
            }
        }
    }
}

Note that zzz refers to the project ID. This is the numeric ID from your gitlab main page.

One other thing I needed to do was upgrade to gradle version 5.5 in the gradle-wrapper.properties. Not doing this meant gradle couldn't find HttpHeaderCredentials.

You now need to go into GitLab and generate an access token. You do this is the user settings (not the project settings) and ensure the API access level is ticked. Copy the access token and add it to your gradle.properties file.

GITLAB_PERSONAL_TOKEN=xxx

That should be all you need. Clean, build and assemble your library and then run the gradle publish command. Your aar should be deployed to GitLab packages and it should now be ready for downloading. You can check in your GitLab packages section and you should see the details displayed there.

Now we goto our client, the app that is going to download and read our worst ever library. We open up our gradle file and add the following:

buildscript {
    repositories {
        google()
        jcenter()
        maven { url 'https://maven.google.com'}
        maven {
            url 'https://gitlab.com/api/v4/projects/zzz/packages/maven'
            credentials(HttpHeaderCredentials) {
                name = "Private-Token"
                value = GITLAB_PERSONAL_TOKEN
            }
            authentication {
                header(HttpHeaderAuthentication)
            }
        }
    }
}

allprojects {
    repositories {
        maven { url 'https://maven.google.com' }
        maven {
            url 'https://gitlab.com/api/v4/projects/zzz/packages/maven'
            credentials(HttpHeaderCredentials) {
                name = "Private-Token"
                value = GITLAB_PERSONAL_TOKEN
            }
            authentication {
                header(HttpHeaderAuthentication)
            }
        }
    }
}

Lastly we add in our dependency and we should be away:

implementation 'com.test:worstlibever:1.7'

That's it, happy coding!

Enrol on App Signing in the Google Play Store for existing Android App

App Signing by the Google Play Store is on the face of it an incredibly brilliant idea. I absolutely love it. However I found it impossibly difficult to follow Google’s rather short documentation on how to get setup, especially for existing app users. So I thought I’d write this little helper to try and spell it out to anyone who tries to follow the same process I have.

App signing is the process of creating a keystore with which you sign your Android app and verify it is in fact you who is releasing an update to your Android app. A keystore remember is basically just a special store for a private and a public key.

If you lose the original master keystore you signed your app with, bad news. You need to create a new app and a new Google Play Store listing. Not fun.

The concept of App Signing for Google Play Store is basically that you surrender the master keystore that you used to sign your initial app to Google (let’s call this the release key). Google then securely stores and manages your release key. You then create a brand new keystore called an upload key. You do this as if you were creating a brand new app, through Android Studio or if you’re a keyboard warrior, using java keytool. You then tell Google about this upload key and you’re done. All future releases can be signed with your upload key and Google will re-sign with the original, now super secret and secure release key. The benefit of this is Google keeps everything secure, and if you lose or compromise your upload key, Google will let you discard it and create a new one. Your users are none the wiser and everyone is happy all of the time.

The process for implementing all this is a little more complicated, first you need to head the play store and goto “App signing” under “Release management”.




Now we need to get all our keystores, private keys, certificates and koala bears in some sort of order. I’m doing this in Android Studio 3.5, older versions may need an upgrade.

First step we need to extract the private key from your original master release key.

Step 1 - Get an App Signing Private Key

Open your app in Android Studio and ensure it builds

1. Goto Build -> Generate Signed Bundle / APK (Don’t worry you don’t need to actually release this build)
2. Even if you’re not wanting to use App Bundle, select Android App Bundle and click next.
3. Enter the details for your master, original release key, including store password, alias and key password
4. IMPORTANT -> check “Export encrypted key for enrolling published apps in Google Play App Signing”
5. Finish off the process and make a build. You should now have a pepk file which is the private key for your original master release key.

Now we need to generate a brand new upload key to use to sign our future apps.

Step 2 - Create an upload key

1. Again goto Build -> Generate Signed Bundle / APK (Again not actually going to release an app)
2. Select APK or App Bundle, whichever you would normally do.
3. Click “Create new…”, we’re going to create a new keystore.
4. Fill in all the details and make the build.
5. You should now have a brand new keystore with brand new passwords and an alias. KEEP THIS CAREFULLY.
6. This keystore will be used forevermore as your main key to release your apps.

Now we need to get a public key from our new upload key, this will be how we tell Google about our new upload key.

Step 3 - Generate a Public Certificate

1. Use keytool and run a command like this
   keytool -export -rfc -keystore upload-keystore.jks -alias upload -file upload_certificate.pem
2. Use your NEW UPLOAD KEY for “upload-keystore.jks” and your new alias 
3. Keytool is usually somewhere in jdk/bin
4. After running this command you should have a .pem file.

Now go back to the Google Play Store and click “Upload a key exported from Android Studio” now you can add your private key and your public certificate. This gives Google the details of the release key and the upload key and you should be all set.

Click finish and you should now be signed up to App Signing. Well done, keep a close eye on that upload key and it’s passwords.