Android docker compose image with Gitlab for CI/CD

This was a right pain and I do love to blog about things that cause me trouble and strife! I created a new Gitlab project and noticed that you can create from template. How exciting! So I selected Android and was delighted to spot a docker file and a yml file had been generated for CI/CD so it integrates nicely with Gitlab pipelines. Amazing! I can run my unit tests and do a build on every commit, this is fantastic.

Gitlab pipelines are a great little tool that builds and runs your project and can be configured as needed to do all sorts of clever things. What's more there's no hardware config involved, it uses docker so you just commit the docker image and let some hardware somewhere in the world build it for you. So simple.

Naturally this builds nicely from the template but the first thing I did with this file was upgrade the compile and target versions in Android's gradle file to the latest version so my app would be compatible with the Google Play Store.

Fail!

Warning: License for package Android SDK Platform 33 not accepted.

FAILURE: Build failed with an exception.

What went wrong:

A problem occurred configuring project ':app'.

Failed to install the following Android SDK packages as some licences have not been accepted.

platforms;android-33 Android SDK Platform 33

To build this project, accept the SDK license agreements and install the missing components using the Android Studio SDK 
Manager.

What's happening here is Gitlab pipeline is creating a docker image from our new dockerfile and running the Android build using this docker image. However the docker image is out of date so when it looks for Android SDK 33 it can't find it and tries to install it, but this fails because it can't find an accepted license for that SDK. Not a big deal, I've played with sdkmanager before we can easily update this docker image.

So looking at our newly generated docker file I can see this is badly out of date. It's using the old android-sdk tool which has been replaced by sdkmanager and it's using SDK 28 and some old build tools. Also it's running off jdk 8 which I'm sure we can improve on.

I won't go through every step I took as this would be a really long post, but I'll post some of the things I learnt along the way.

Firstly playing with Docker was really fun, but it was also quite frustrating. Constantly building and re-building took a while. What I found faster was to install Docker desktop and run it locally. This also means you can mount your docker image and run bash to debug your environment.

Also I learnt that a docker file is not docker compose. A docker file is just a dockerfile and you use this to create an image using the docker build command. Docker compose is the yml or yaml file. This allows you to create multi-container applications. Think of this a bit like config for the docker container that is running your docker image.

Another thing I learnt was that Android sdkmanager is fussy and you need to be quite precise with paths and directories or the toys and the pram part ways. The license problem mentioned above ended up being really frustrating. What I eventually found was I had to make sure the sdk wasn't in the root folder and just giving /sdk write permissions wasn't enough. I had to explicitly grant the /sdl/licenses folder write permissions. After many many runs that straightened all that out.

Below is my new dockerfile for Gitlab. You'll notice the following big differences from the stock file I got from Gitlab:

• It now uses JDK11
• It uses SDK 33
• It uses sdkmanager 
• It updates sdkmanager as it goes
• It should (hopefully) be able to install the latest SDK and accept the license at build time

I'm wondering if I should pass in the SDK and Android build tools versions from the yaml file. I'm also wondering about installing gradle in the docker file as this takes a few seconds each time.

# This Dockerfile creates a static build image for CI
#!/bin/sh

FROM openjdk:11-jdk

# Just matched `app/build.gradle`
ENV ANDROID_COMPILE_SDK "33"
# Just matched `app/build.gradle`
ENV ANDROID_BUILD_TOOLS "30.0.0"
# Version from https://developer.android.com/studio/releases/sdk-tools
ENV ANDROID_SDK_TOOLS "8512546_latest"
ENV ANDROID_HOME /opt/sdk
ENV ANDROID_SDK_PATH /opt/sdk

# install OS packages
RUN apt-get --quiet update --yes
RUN apt-get --quiet install --yes wget apt-utils tar unzip lib32stdc++6 lib32z1 build-essential ruby ruby-dev

# We use this for xxd hex->binary
RUN apt-get --quiet install --yes vim-common
# create sdk directory, install Android SDK
# https://dl.google.com/android/repository/commandlinetools-linux-8512546_latest.zip
RUN mkdir -p /opt/sdk
RUN wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_TOOLS}.zip && \
mv android-sdk.zip /opt/sdk && \
cd /opt/sdk/ && \
unzip android-sdk.zip

#Sort out the mess - https://stackoverflow.com/a/65262939
RUN mkdir /opt/sdk/cmdline-tools/tools && \
mv /opt/sdk/cmdline-tools/bin /opt/sdk/cmdline-tools/tools/ && \
mv /opt/sdk/cmdline-tools/lib /opt/sdk/cmdline-tools/tools/ && \
mv /opt/sdk/cmdline-tools/NOTICE.txt /opt/sdk/cmdline-tools/tools/ && \
mv /opt/sdk/cmdline-tools/source.properties /opt/sdk/cmdline-tools/tools/ && \
export PATH="${PATH}:/opt/sdk/cmdline-tools/tools/bin:${ANDROID_HOME}"

RUN chmod -R 777 /opt/sdk

RUN /opt/sdk/cmdline-tools/tools/bin/sdkmanager --licenses
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager --update
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "platforms;android-${ANDROID_COMPILE_SDK}"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "platform-tools"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "build-tools;${ANDROID_BUILD_TOOLS}"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "extras;m2repository;com;android;support;constraint;constraint-layout;1.0.2"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "extras;android;m2repository"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "extras;google;google_play_services"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "patcher;v4" "tools"
RUN chmod -R 777 /opt/sdk
RUN chmod -R 777 /opt/sdk/licenses
RUN yes | /opt/sdk/cmdline-tools/tools/bin/sdkmanager --licenses
RUN chmod -R 777 /opt/sdk

# install FastLane
COPY Gemfile.lock .
COPY Gemfile .
RUN gem install bundler -v 1.16.6
RUN bundle install

Anyway, I really hope this post is of use. I've certainly enjoyed learning about Docker.

Git on Windows with Putty

This is something that's super simple on Linux but every couple of years I have to setup a new Windows machine with GIT and Putty (pageant to be exact). Every time I do it I seem to forget one of the steps and spend a few hours trying to remember what I've missed. So here is the note to myself so I can remember how to do it.

Components to install:

1. Putty including of course puttygen and pageant

2. Git Bash

1) Generate a ssh key 

I won’t go into details here as there are a million tutorials already on the internet but you want to use Puttygen to create a new private/public security key pair. You can save this as a .ppk file on your local machine.

2) Upload the public key to GitLab / Github

Copy the public key from inside Puttygen to GitLab or GitHub. You'll need to find the page on your account where you can add an SSH key and paste it in there. Be sure this is the public key and not the private one.

3) Run pageant 

Run pageant and it should appear in your system tray. Now open it and add the .ppk to your keys.

4) Add a GIT_SSH environment variable

Using this for advice:

https://stackoverflow.com/a/43313491

Create an environment variable that points to plink. This is probably the step I forget the most.

5) Configure GIT

You now need to ensure you have the correct url added to your git repo. A few times I've used the http one or the wrong url or something. Carefully copy the SSH (git@gitlab.com...) url from Gitlab or Github and add this to your git project as a remote repo url. You can check you've done this correctly by typing git remote -v

That's it. You should definitely restart Git bash and restarting your PC wouldn't hurt either, just make sure you start up pageant again. Another thing I forget is how to get this to run on startup. Oh it's a complicated life!

Publishing Android Jacoco test results to SonarQube

This is something I really though would be pretty straightforward but I wanted my Team City build server to push my unit test coverage to SonarQube. 

SonarQube is a really cool application that shows you all sorts of really interesting insights and problems with your code. Inefficiencies, security problems and tech debt are all highlighted by SonarQube in a nicely presented dashboard. It works with all sorts of languages but of course with Android we're focusing on Java and Kotlin.

I'd managed to get SonarQube setup and the basic code details pushed over fairly easily, but I couldn't get the test coverage over.

There are of course other ways around, I could have installed a better plugin on Team City which would have helped I'm sure, or I could have installed a Sonar Gradle plugin, but I didn't want to do that.

The trick ended up being to get xml test coverage. Once I had that sorted I could push that xml report to Sonar and it started showing my test coverage.

Nothing I did in the standard build.gradle file seemed to work, the standard jacoco plugin seemed to ignore my pleas for xml coverage reports. Eventually I found a really great little script that helped me get there.

https://medium.com/wandera-engineering/android-kotlin-code-coverage-with-jacoco-sonar-and-gradle-plugin-6-x-3933ed503a6e

Big thanks go to the Auther of this piece for figuring this out.

RxJava filter an Observable list

I had an observable with data type list, and I wanted to filter the objects in that list and maintain the observable. Sounds simple enough right, but it took a while and I wanted to record it as I've used this concept a few times now.

So let's start with our object of which we will create the list.

data class Muppet(var name: String, var age: int)

Now let's create a list of those objects

val myList = Observable.just(arrayListOf<Muppet>())

Simple stuff so far. Now let's get to the interesting Rx goodness. 

What we need is to extract the individual objects so we can filter them, so we need an observable of objects rather than an observable of a list. 

.flatMap { iterable: List<Muppet> -> Observable.from(iterable) }

So here we use flatmap to change the data type and we use the very clever Observable.from to change the data to a single result.

Now we can filter as our heart desire

.filter { muppet: Muppet -> muppet.age > 30 }

and put it back together using toList.

Here's the complete example

return getMuppets()
       .flatMap { iterable: List<Muppet> -> Observable.from(iterable) }
       .filter { muppet: Muppet -> muppet.age > 30 }
       .toList()

Android Hilt example and a unit test (with Robolectric)

Android Hilt has been released and I must say I'm quite impressed. I absolutely hate Dagger which I think comprises of far too much boiler plate code, its badly documented and incredibly confusing. Hilt is of course built on top of dagger and I think this is a huge win for the Android community. Hilt decreases confusion, increases code generation and is nice and simple.

I don't want to go into what Dependency Injection (DI) is, that's been done to death.

There are two resources I found really useful to get started with Hilt:

The Hilt documentation - https://dagger.dev/hilt/

Coding with Mitch videos: https://www.youtube.com/watch?v=zTpM2olXCok

There are two things I want to talk about in this blog post:

1. A really quick intro and setup of a Hilt module.
2. A unit test using Robolectic.

Before I go much further I would discourage you from using Robolectric. It's become a real pain to work with, requiring Java 9 for its latest version and regularly deprecating things without clear documentation on how to migrate. That said I've been using Robolectic for a while so I'm stuck with it until I can remove it from my code. I'm hoping Hilt and ViewModels will help me do this.

The first thing I wanted to look at is (I think) a fairly standard use case where a class has some dependencies and we want to use that class in our activity without creating it every time.

Step 1 - The application

@HiltAndroidApp
class MyApplication : Application()

Step 2 - The Class

class Petrol(val input: String) {

    fun getTheInfo(): String {
        return input
    }
}

Step 3 - The Module

Now this is the interesting bit, here we define a "factory" where we list how the class is to be created. Obviously this is a very simple example, but I think that's a good way to start.

@InstallIn(ApplicationComponent::class)
@Module
class PetrolFactory {

    @Provides
    fun getPetrol(): Petrol {
        return Petrol("Petroll")
    }
}

Step 4 - The Activity

Now we setup the activity and you'll note how little code we need to write to create and start using the class

@AndroidEntryPoint
class MainActivity : AppCompatActivity() {

    @Inject
    lateinit var petrol: Petrol

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        setSupportActionBar(findViewById(R.id.toolbar))
        
        findViewById<TextView>(R.id.main_text).text = petrol.getTheInfo()
    }
}

I think this is fantastic and hopefully it's clear to see how easy this is and how quick you can make complex interactions really simple.

Step 5 - The Tests

With the tests I want to show how you would pass in something different for the purpose of testing. 

@UninstallModules(PetrolFactory::class)
@HiltAndroidTest
class MainActivityTest : RobolectricTestConfig() {

    @get:Rule
    var hiltRule = HiltAndroidRule(this)

    @Module
    @InstallIn(ApplicationComponent::class)
    class TestEngine {
        @Provides
        fun getPetrol(): Petrol {
            return Petrol("TestPetrol")
        }
    }


    @Before
    fun init() {
        hiltRule.inject()
    }

    @Test
    fun testActivityMainText() {
        val scenario = ActivityScenario.launch(MainActivity::class.java)

        scenario.onActivity {
            assertEquals("TestPetrol", it.findViewById<TextView>(R.id.main_text).text)
        }
    }
}

One thing that's really important to remember is you can't run the test with the green arrow in Android Studio. You need to use gradle. So just run the gradle task instead. That's it I hope that helps demonstrate how you would create a DI app with Hilt in a few easy steps. I also hope you can see how easy that makes testing.

Photo backup and sync using Google Cloud Bucket Storage

This morning I spotted a tweet by Greg Wilson whom I don't know and have never met. However it appears he's some kind of director at Google Cloud, so that might explain why I'm following him and why he tweeted this:

https://twitter.com/gregsramblings/status/1284743960955510787?s=20

Archiving my newly organized 238k+ photo library (2.2TB) to Google Cloud Storage with: gsutil -m rsync -r -d . gs://{mybucketname} I'm using crazy-cheap 'archive' storage class in single regionStorage price: $0.0012/GB/Month (!) @gcpcloud

This really interested me as I've been trying to upload photos to an AWS (Amazon Web Services) bucket recently but it's been a tedious process.

I am aware that various cloud services like dropbox and pcloud exist, but I want the following:

1. Encryption
2. Ease of use
3. Cheap

I really only want this as a backup in case of fire or failure of one of my crummy old USB HDDs fails. I genuinely don't understand why so many of these services make such a bad job of encryption or offer it as some kind of bizarre add on. Come on people this is 2020, it's not hard and it shouldn't be expensive.

So I realised with some basic AWS skills I can spin up a bucket running glacier for pennies a month. This worked really really well, but the upload tools are slow and not terribly reliable for bulk uploads. Which I guess is understandable because the whole point of AWS is to build your own right?

Back to my original point, I stumbled across this tweet and thought hmm, that sounds nice and easy. Let's give it a go on a dull lockdown Sunday.

The following I followed from here: https://cloud.google.com/storage/docs/quickstart-gsutil

1. Install Python 3.8

Did someone say snakes on a plane?

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.8

2. Download & Extract Google Cloud SDK
https://cloud.google.com/sdk/docs

3. Create a bucket
Then I created a new Google Cloud Project on their cloud console and a new Cloud Storage Bucket. This was actually really easy and the bucket creation process easily walked me through the access policies and encryption setup. Although to be fair the AWS process is pretty good now as well. Don't forget to select your preferred storage type but bare in mind different types have different latency, pricing and minimum storage commitments.
https://cloud.google.com/storage/pricing#operations-pricing

4. Initialise the sdk
Run gcloud init this will initialise your sdk and link it to your google account. You can then of course select the project you're working on.

5. Copy a file
That's pretty much it, from there you can copy individual files using gloud cp or as Greg's tweet suggest you can use the sync option for bulk upload.

This is such a simple process I was really impressed.

One thing it's important to note is that Google Storage buckets may have a minimum storage time period:
https://cloud.google.com/storage/pricing#archival-pricing

Android library using GitLab as an aar repository

As your Android project begins to grow you will likely start to accumulate a few libraries. Not the official ones, like rxjava, androidx and so on, but I'm talking about ones you've developed yourself. These could be your entire network layer, your really really useful utils folder, or whatever. If you're a good developer you'll have these in separate modules or possibly even completely separate projects. To start with this is fine, you may be copying your aar files around or just being vocal about changes to your modules so your teammates can keep up to date.

This article gives a fantastic explanation of why it's a good idea to stop doing that:
https://inthecheesefactory.com/blog/how-to-setup-private-maven-repository/en

The author makes some excellent points why modules and copying aars are a bad idea. The ones I most agree with are:

Library distributed to other developers should not be modifiable. In case there is some problem, issue should be reported to developer involved to let them fix. Letting other developers directly access the source code might cause them accidentally doing a quick fix by themselves which will cause a big problem afterwards. 

Most of the problem listed above are solved except one: it's still hard to update and roll back over the version.

The solution is to use a repository. Basically give your library a specific version and deploy it to a repository so it can be referenced simply using nothing more than a gradle import. Luckily for us there are a few repositories out there and they can be easily made private. So we only allow permitted developers access to our libraries.

Jfrog is a great way to go, but this means you've probably got to manage a server yourself. What I was hoping for was a cloud method where I had no server admin and no setup process. So I turned to my old friend GitLab.

Now the first and most important thing to note is that the GitLab repository pattern (named packages) is not free. GitLab Packages is only currently available on the Silver package, which does incur a monthly cost. Although they do have a trial period, if you just want to give it a go. There is a project level setting to enable packages, but for me this was already turned on.

So that said, let's get to the code. First step is to create a library.

I created an Android library named "WorstLibEver" this is the extent of it:

public class HodorUtils {
    public String askHodor(String question){
        return "Hodor";
    }
}

I know, you're blown away aren't you?

Now we need to edit our gradle file:

First add this line to the top:

apply plugin: 'maven-publish'

Next you'll need to add the following to the bottom:

task sourceJar(type: Jar) {
    from android.sourceSets.main.java.srcDirs
    classifier "sources"
}

publishing {
    publications {
        bar(MavenPublication) {
            groupId 'com.test'
            artifactId 'worstlibever'
            version '1.7'
            artifact(sourceJar)
            artifact("$buildDir/outputs/aar/app-release.aar")
        }
    }
    repositories {
        maven {
            url "https://gitlab.com/api/v4/projects/zzz/packages/maven"
            credentials(HttpHeaderCredentials) {
                name = "Private-Token"
                value = GITLAB_PERSONAL_TOKEN
            }
            authentication {
                header(HttpHeaderAuthentication)
            }
        }
    }
}

Note that zzz refers to the project ID. This is the numeric ID from your gitlab main page.

One other thing I needed to do was upgrade to gradle version 5.5 in the gradle-wrapper.properties. Not doing this meant gradle couldn't find HttpHeaderCredentials.

You now need to go into GitLab and generate an access token. You do this is the user settings (not the project settings) and ensure the API access level is ticked. Copy the access token and add it to your gradle.properties file.

GITLAB_PERSONAL_TOKEN=xxx

That should be all you need. Clean, build and assemble your library and then run the gradle publish command. Your aar should be deployed to GitLab packages and it should now be ready for downloading. You can check in your GitLab packages section and you should see the details displayed there.

Now we goto our client, the app that is going to download and read our worst ever library. We open up our gradle file and add the following:

buildscript {
    repositories {
        google()
        jcenter()
        maven { url 'https://maven.google.com'}
        maven {
            url 'https://gitlab.com/api/v4/projects/zzz/packages/maven'
            credentials(HttpHeaderCredentials) {
                name = "Private-Token"
                value = GITLAB_PERSONAL_TOKEN
            }
            authentication {
                header(HttpHeaderAuthentication)
            }
        }
    }
}

allprojects {
    repositories {
        maven { url 'https://maven.google.com' }
        maven {
            url 'https://gitlab.com/api/v4/projects/zzz/packages/maven'
            credentials(HttpHeaderCredentials) {
                name = "Private-Token"
                value = GITLAB_PERSONAL_TOKEN
            }
            authentication {
                header(HttpHeaderAuthentication)
            }
        }
    }
}

Lastly we add in our dependency and we should be away:

implementation 'com.test:worstlibever:1.7'

That's it, happy coding!