GitLab pipeline - Container registry configuration

This one has taken me a few hours and is pretty stupid. If you try running your GitLab pipeline and you get this error:

$ docker pull --quiet $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG || true
invalid reference format
$ docker build --cache-from $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG .
invalid argument ":appname" for "-t, --tag" flag: invalid reference format

Then I believe the problem is your container registry. This is a random setting your need to enable buried deep in the GitLab settings.

Goto:

Settings -> General -> Visibility, project features, permissions

The random setting you're looking for is 

Container registry - Every project can have its own space to store its Docker images

https://docs.gitlab.com/ee/user/packages/container_registry/index.html

https://stackoverflow.com/a/57729959

AWS S3 API call using babbel to check incomplete multi part uploads

Sometimes AWS is hard work!

For a long time I've had some photos backed up in an S3 Glacier bucket. Glacier is just long term storage, very cheap and slow to access. Meaning you shouldn't really rely on getting regular access to these files. In my case they're for worst case scenarios use only.

The problem is every month my bill looks a bit strange. I have a Glacier entry and I also have an S3 entry on my bill. We aren't talking much money here, just a few pennies but it bothers me, there shouldn't really be anything on S3, it should all be under Glacier. So I gave every bucket a tag so I could see which bucket was the culprit. Tags are really useful because I can group by tag spending on the AWS cost explorer. So this should identify what's going on. Unfortunately no such luck, the S3 spending is always under "No tag key". After what was probably longer than I should admit curiosity got the better of me and I emailed support. Support came back to me saying this is probably multi part uploads that have failed.

The plot thickens! I can't confirm this is indeed the cause yet, but I learnt a few things along the way, so here are some general AWS related findings.

Incomplete Multi Part Uploads.

There's been a fair amount written about this, and if you're reading this blog you probably get it already so I'll be short. When you try to upload a large file it can't be sent in one chunk across the internet. So we break it down and send it bit by bit. So what support is implying is that I started uploading files and some parts succeeded and some failed. So that left incomplete "chunks" or packets probably more accurately on my S3 account. 

AWS's solution

They have written a blog post about this, but it's not very up to date and personally I couldn't get Storage Lens to show incomplete multi part uploads.

https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/

Listing incomplete multi part uploads

It's absurd that this is not simple from the console. However I really wanted to see if I did in fact have any incomplete MPU's before I applied a rule that deleted them. So I set about dusting off my old REST API skills to see if I could create a script that would list incomplete MPUs. I discovered I could use an API method for this:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html

I really wanted to use Kotlin for this but I wasn't setup to run Kotlin stand alone apps, so it was faster for me to make this an Android "app". I used an awesome tool called babbel okhttp-aws-signer to sort out the AWS Signature Version 4 stuff as I know how frustrating that can be:

https://github.com/babbel/okhttp-aws-signer

This library is fantastic. So much easier than trying to workout the AWS Signature Version 4 nonsense yourself.

Here is the script I used to generate and make an AWS S3 REST API request:

val url = "http://" + bucketname + "." + serviceName + "." + region + ".amazonaws.com/?uploads=1"
val dateused = SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'", Locale.US).format(Date())

val signer = OkHttpAwsV4Signer(region, serviceName)

val request = Request.Builder()
    .url(url)
    .build()

val newRequest = request.newBuilder()
    .addHeader("host", request.url.host)
    .addHeader("x-amz-date", dateused)
    .addHeader("x-amz-content-sha256", "".hash())
    .build()

val signed = signer.sign(newRequest, accessKeyId, secretAccessKey)

You'll note that I had to use ?uploads=1 to get the babel library to work. This is because it requires query params to be pairs and you get a NPE if you don't do so. You'll also notice I had to add the new (to me) header x-amz-content-sha256 which apparently is just an empty hash if the body is empty. This is a GET request so of course it is.

You'll need your access key and your access key secret which you can generate in the security credentials area of your account.

So after all that and quite a lot of work I was able to prove that there were in fact a few incomplete multi part uploads on my S3 account. So I've created a policy to delete them, let's see if that works.

I really recommend babbel okhttp-aws-signer as it makes things like this quite simple and helps you answer a few AWS questions without too much drama.

Hope this has helped you with something AWS related!

Android docker compose image with Gitlab for CI/CD

This was a right pain and I do love to blog about things that cause me trouble and strife! I created a new Gitlab project and noticed that you can create from template. How exciting! So I selected Android and was delighted to spot a docker file and a yml file had been generated for CI/CD so it integrates nicely with Gitlab pipelines. Amazing! I can run my unit tests and do a build on every commit, this is fantastic.

Gitlab pipelines are a great little tool that builds and runs your project and can be configured as needed to do all sorts of clever things. What's more there's no hardware config involved, it uses docker so you just commit the docker image and let some hardware somewhere in the world build it for you. So simple.

Naturally this builds nicely from the template but the first thing I did with this file was upgrade the compile and target versions in Android's gradle file to the latest version so my app would be compatible with the Google Play Store.

Fail!

Warning: License for package Android SDK Platform 33 not accepted.

FAILURE: Build failed with an exception.

What went wrong:

A problem occurred configuring project ':app'.

Failed to install the following Android SDK packages as some licences have not been accepted.

platforms;android-33 Android SDK Platform 33

To build this project, accept the SDK license agreements and install the missing components using the Android Studio SDK 
Manager.

What's happening here is Gitlab pipeline is creating a docker image from our new dockerfile and running the Android build using this docker image. However the docker image is out of date so when it looks for Android SDK 33 it can't find it and tries to install it, but this fails because it can't find an accepted license for that SDK. Not a big deal, I've played with sdkmanager before we can easily update this docker image.

So looking at our newly generated docker file I can see this is badly out of date. It's using the old android-sdk tool which has been replaced by sdkmanager and it's using SDK 28 and some old build tools. Also it's running off jdk 8 which I'm sure we can improve on.

I won't go through every step I took as this would be a really long post, but I'll post some of the things I learnt along the way.

Firstly playing with Docker was really fun, but it was also quite frustrating. Constantly building and re-building took a while. What I found faster was to install Docker desktop and run it locally. This also means you can mount your docker image and run bash to debug your environment.

Also I learnt that a docker file is not docker compose. A docker file is just a dockerfile and you use this to create an image using the docker build command. Docker compose is the yml or yaml file. This allows you to create multi-container applications. Think of this a bit like config for the docker container that is running your docker image.

Another thing I learnt was that Android sdkmanager is fussy and you need to be quite precise with paths and directories or the toys and the pram part ways. The license problem mentioned above ended up being really frustrating. What I eventually found was I had to make sure the sdk wasn't in the root folder and just giving /sdk write permissions wasn't enough. I had to explicitly grant the /sdl/licenses folder write permissions. After many many runs that straightened all that out.

Below is my new dockerfile for Gitlab. You'll notice the following big differences from the stock file I got from Gitlab:

• It now uses JDK11
• It uses SDK 33
• It uses sdkmanager 
• It updates sdkmanager as it goes
• It should (hopefully) be able to install the latest SDK and accept the license at build time

I'm wondering if I should pass in the SDK and Android build tools versions from the yaml file. I'm also wondering about installing gradle in the docker file as this takes a few seconds each time.

# This Dockerfile creates a static build image for CI
#!/bin/sh

FROM openjdk:11-jdk

# Just matched `app/build.gradle`
ENV ANDROID_COMPILE_SDK "33"
# Just matched `app/build.gradle`
ENV ANDROID_BUILD_TOOLS "30.0.0"
# Version from https://developer.android.com/studio/releases/sdk-tools
ENV ANDROID_SDK_TOOLS "8512546_latest"
ENV ANDROID_HOME /opt/sdk
ENV ANDROID_SDK_PATH /opt/sdk

# install OS packages
RUN apt-get --quiet update --yes
RUN apt-get --quiet install --yes wget apt-utils tar unzip lib32stdc++6 lib32z1 build-essential ruby ruby-dev

# We use this for xxd hex->binary
RUN apt-get --quiet install --yes vim-common
# create sdk directory, install Android SDK
# https://dl.google.com/android/repository/commandlinetools-linux-8512546_latest.zip
RUN mkdir -p /opt/sdk
RUN wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_TOOLS}.zip && \
mv android-sdk.zip /opt/sdk && \
cd /opt/sdk/ && \
unzip android-sdk.zip

#Sort out the mess - https://stackoverflow.com/a/65262939
RUN mkdir /opt/sdk/cmdline-tools/tools && \
mv /opt/sdk/cmdline-tools/bin /opt/sdk/cmdline-tools/tools/ && \
mv /opt/sdk/cmdline-tools/lib /opt/sdk/cmdline-tools/tools/ && \
mv /opt/sdk/cmdline-tools/NOTICE.txt /opt/sdk/cmdline-tools/tools/ && \
mv /opt/sdk/cmdline-tools/source.properties /opt/sdk/cmdline-tools/tools/ && \
export PATH="${PATH}:/opt/sdk/cmdline-tools/tools/bin:${ANDROID_HOME}"

RUN chmod -R 777 /opt/sdk

RUN /opt/sdk/cmdline-tools/tools/bin/sdkmanager --licenses
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager --update
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "platforms;android-${ANDROID_COMPILE_SDK}"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "platform-tools"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "build-tools;${ANDROID_BUILD_TOOLS}"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "extras;m2repository;com;android;support;constraint;constraint-layout;1.0.2"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "extras;android;m2repository"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "extras;google;google_play_services"
RUN echo y | /opt/sdk/cmdline-tools/tools/bin/sdkmanager "patcher;v4" "tools"
RUN chmod -R 777 /opt/sdk
RUN chmod -R 777 /opt/sdk/licenses
RUN yes | /opt/sdk/cmdline-tools/tools/bin/sdkmanager --licenses
RUN chmod -R 777 /opt/sdk

# install FastLane
COPY Gemfile.lock .
COPY Gemfile .
RUN gem install bundler -v 1.16.6
RUN bundle install

Anyway, I really hope this post is of use. I've certainly enjoyed learning about Docker.

Git on Windows with Putty

This is something that's super simple on Linux but every couple of years I have to setup a new Windows machine with GIT and Putty (pageant to be exact). Every time I do it I seem to forget one of the steps and spend a few hours trying to remember what I've missed. So here is the note to myself so I can remember how to do it.

Components to install:

1. Putty including of course puttygen and pageant

2. Git Bash

1) Generate a ssh key 

I won’t go into details here as there are a million tutorials already on the internet but you want to use Puttygen to create a new private/public security key pair. You can save this as a .ppk file on your local machine.

2) Upload the public key to GitLab / Github

Copy the public key from inside Puttygen to GitLab or GitHub. You'll need to find the page on your account where you can add an SSH key and paste it in there. Be sure this is the public key and not the private one.

3) Run pageant 

Run pageant and it should appear in your system tray. Now open it and add the .ppk to your keys.

4) Add a GIT_SSH environment variable

Using this for advice:

https://stackoverflow.com/a/43313491

Create an environment variable that points to plink. This is probably the step I forget the most.

5) Configure GIT

You now need to ensure you have the correct url added to your git repo. A few times I've used the http one or the wrong url or something. Carefully copy the SSH (git@gitlab.com...) url from Gitlab or Github and add this to your git project as a remote repo url. You can check you've done this correctly by typing git remote -v

That's it. You should definitely restart Git bash and restarting your PC wouldn't hurt either, just make sure you start up pageant again. Another thing I forget is how to get this to run on startup. Oh it's a complicated life!

Publishing Android Jacoco test results to SonarQube

This is something I really though would be pretty straightforward but I wanted my Team City build server to push my unit test coverage to SonarQube. 

SonarQube is a really cool application that shows you all sorts of really interesting insights and problems with your code. Inefficiencies, security problems and tech debt are all highlighted by SonarQube in a nicely presented dashboard. It works with all sorts of languages but of course with Android we're focusing on Java and Kotlin.

I'd managed to get SonarQube setup and the basic code details pushed over fairly easily, but I couldn't get the test coverage over.

There are of course other ways around, I could have installed a better plugin on Team City which would have helped I'm sure, or I could have installed a Sonar Gradle plugin, but I didn't want to do that.

The trick ended up being to get xml test coverage. Once I had that sorted I could push that xml report to Sonar and it started showing my test coverage.

Nothing I did in the standard build.gradle file seemed to work, the standard jacoco plugin seemed to ignore my pleas for xml coverage reports. Eventually I found a really great little script that helped me get there.

https://medium.com/wandera-engineering/android-kotlin-code-coverage-with-jacoco-sonar-and-gradle-plugin-6-x-3933ed503a6e

Big thanks go to the Auther of this piece for figuring this out.

RxJava filter an Observable list

I had an observable with data type list, and I wanted to filter the objects in that list and maintain the observable. Sounds simple enough right, but it took a while and I wanted to record it as I've used this concept a few times now.

So let's start with our object of which we will create the list.

data class Muppet(var name: String, var age: int)

Now let's create a list of those objects

val myList = Observable.just(arrayListOf<Muppet>())

Simple stuff so far. Now let's get to the interesting Rx goodness. 

What we need is to extract the individual objects so we can filter them, so we need an observable of objects rather than an observable of a list. 

.flatMap { iterable: List<Muppet> -> Observable.from(iterable) }

So here we use flatmap to change the data type and we use the very clever Observable.from to change the data to a single result.

Now we can filter as our heart desire

.filter { muppet: Muppet -> muppet.age > 30 }

and put it back together using toList.

Here's the complete example

return getMuppets()
       .flatMap { iterable: List<Muppet> -> Observable.from(iterable) }
       .filter { muppet: Muppet -> muppet.age > 30 }
       .toList()

Android Hilt example and a unit test (with Robolectric)

Android Hilt has been released and I must say I'm quite impressed. I absolutely hate Dagger which I think comprises of far too much boiler plate code, its badly documented and incredibly confusing. Hilt is of course built on top of dagger and I think this is a huge win for the Android community. Hilt decreases confusion, increases code generation and is nice and simple.

I don't want to go into what Dependency Injection (DI) is, that's been done to death.

There are two resources I found really useful to get started with Hilt:

The Hilt documentation - https://dagger.dev/hilt/

Coding with Mitch videos: https://www.youtube.com/watch?v=zTpM2olXCok

There are two things I want to talk about in this blog post:

1. A really quick intro and setup of a Hilt module.
2. A unit test using Robolectic.

Before I go much further I would discourage you from using Robolectric. It's become a real pain to work with, requiring Java 9 for its latest version and regularly deprecating things without clear documentation on how to migrate. That said I've been using Robolectic for a while so I'm stuck with it until I can remove it from my code. I'm hoping Hilt and ViewModels will help me do this.

The first thing I wanted to look at is (I think) a fairly standard use case where a class has some dependencies and we want to use that class in our activity without creating it every time.

Step 1 - The application

@HiltAndroidApp
class MyApplication : Application()

Step 2 - The Class

class Petrol(val input: String) {

    fun getTheInfo(): String {
        return input
    }
}

Step 3 - The Module

Now this is the interesting bit, here we define a "factory" where we list how the class is to be created. Obviously this is a very simple example, but I think that's a good way to start.

@InstallIn(ApplicationComponent::class)
@Module
class PetrolFactory {

    @Provides
    fun getPetrol(): Petrol {
        return Petrol("Petroll")
    }
}

Step 4 - The Activity

Now we setup the activity and you'll note how little code we need to write to create and start using the class

@AndroidEntryPoint
class MainActivity : AppCompatActivity() {

    @Inject
    lateinit var petrol: Petrol

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        setSupportActionBar(findViewById(R.id.toolbar))
        
        findViewById<TextView>(R.id.main_text).text = petrol.getTheInfo()
    }
}

I think this is fantastic and hopefully it's clear to see how easy this is and how quick you can make complex interactions really simple.

Step 5 - The Tests

With the tests I want to show how you would pass in something different for the purpose of testing. 

@UninstallModules(PetrolFactory::class)
@HiltAndroidTest
class MainActivityTest : RobolectricTestConfig() {

    @get:Rule
    var hiltRule = HiltAndroidRule(this)

    @Module
    @InstallIn(ApplicationComponent::class)
    class TestEngine {
        @Provides
        fun getPetrol(): Petrol {
            return Petrol("TestPetrol")
        }
    }


    @Before
    fun init() {
        hiltRule.inject()
    }

    @Test
    fun testActivityMainText() {
        val scenario = ActivityScenario.launch(MainActivity::class.java)

        scenario.onActivity {
            assertEquals("TestPetrol", it.findViewById<TextView>(R.id.main_text).text)
        }
    }
}

One thing that's really important to remember is you can't run the test with the green arrow in Android Studio. You need to use gradle. So just run the gradle task instead. That's it I hope that helps demonstrate how you would create a DI app with Hilt in a few easy steps. I also hope you can see how easy that makes testing.