Kotlin script running on GitHub Actions

I wanted to run a script regularly and didn't really want to run it on my local machine each time. So it turns out you can run scripts regularly on GitHub Actions. What's even better is its super simple to setup. Of course there are some limits to how many minutes you can run per GitHub's billing page. Currently this is 2000 minutes.

First I created a fresh repository with one single file in it. I named that file Test.main.kts and added the following contents:

val helloName = "Bob"

println("Hello $helloName from the kotlin script")

I commit that to my repo and then create a GitHub Action, this can be named whatever you want, mine is named main.yml here are the contents:

name: Just a test

# configure manual trigger
on:
  workflow_dispatch:

jobs:
  just-a-test:
    name: Run a Kotlin script 
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v2
      - name: Run Kotlin script
        run: kotlinc -script ./Test.main.kts

This checks out your repo onto a ubuntu runner and runs your script. Super simple and this will run whenever you manually execute it using the web control in GitHub. Of course you can schedule this if needed.

The slight complication comes when you want imports, this is where you need to balance local setup and GitHub with scripts. I have been using IntelliJ and INtelliJ with scripts don't seem to play super well yet, maybe I'm missing something. You can add imports into the script file using @file:DependsOn() but how you get this into your project locally is still a bit confusing to me. It might be that IntelliJ doesn't really support this lightweight type of script based project yet.

AWS S3 API call using babbel to check incomplete multi part uploads

Sometimes AWS is hard work!

For a long time I've had some photos backed up in an S3 Glacier bucket. Glacier is just long term storage, very cheap and slow to access. Meaning you shouldn't really rely on getting regular access to these files. In my case they're for worst case scenarios use only.

The problem is every month my bill looks a bit strange. I have a Glacier entry and I also have an S3 entry on my bill. We aren't talking much money here, just a few pennies but it bothers me, there shouldn't really be anything on S3, it should all be under Glacier. So I gave every bucket a tag so I could see which bucket was the culprit. Tags are really useful because I can group by tag spending on the AWS cost explorer. So this should identify what's going on. Unfortunately no such luck, the S3 spending is always under "No tag key". After what was probably longer than I should admit curiosity got the better of me and I emailed support. Support came back to me saying this is probably multi part uploads that have failed.

The plot thickens! I can't confirm this is indeed the cause yet, but I learnt a few things along the way, so here are some general AWS related findings.

Incomplete Multi Part Uploads.

There's been a fair amount written about this, and if you're reading this blog you probably get it already so I'll be short. When you try to upload a large file it can't be sent in one chunk across the internet. So we break it down and send it bit by bit. So what support is implying is that I started uploading files and some parts succeeded and some failed. So that left incomplete "chunks" or packets probably more accurately on my S3 account. 

AWS's solution

They have written a blog post about this, but it's not very up to date and personally I couldn't get Storage Lens to show incomplete multi part uploads.

https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/

Listing incomplete multi part uploads

It's absurd that this is not simple from the console. However I really wanted to see if I did in fact have any incomplete MPU's before I applied a rule that deleted them. So I set about dusting off my old REST API skills to see if I could create a script that would list incomplete MPUs. I discovered I could use an API method for this:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html

I really wanted to use Kotlin for this but I wasn't setup to run Kotlin stand alone apps, so it was faster for me to make this an Android "app". I used an awesome tool called babbel okhttp-aws-signer to sort out the AWS Signature Version 4 stuff as I know how frustrating that can be:

https://github.com/babbel/okhttp-aws-signer

This library is fantastic. So much easier than trying to workout the AWS Signature Version 4 nonsense yourself.

Here is the script I used to generate and make an AWS S3 REST API request:

val url = "http://" + bucketname + "." + serviceName + "." + region + ".amazonaws.com/?uploads=1"
val dateused = SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'", Locale.US).format(Date())

val signer = OkHttpAwsV4Signer(region, serviceName)

val request = Request.Builder()
    .url(url)
    .build()

val newRequest = request.newBuilder()
    .addHeader("host", request.url.host)
    .addHeader("x-amz-date", dateused)
    .addHeader("x-amz-content-sha256", "".hash())
    .build()

val signed = signer.sign(newRequest, accessKeyId, secretAccessKey)

You'll note that I had to use ?uploads=1 to get the babel library to work. This is because it requires query params to be pairs and you get a NPE if you don't do so. You'll also notice I had to add the new (to me) header x-amz-content-sha256 which apparently is just an empty hash if the body is empty. This is a GET request so of course it is.

You'll need your access key and your access key secret which you can generate in the security credentials area of your account.

So after all that and quite a lot of work I was able to prove that there were in fact a few incomplete multi part uploads on my S3 account. So I've created a policy to delete them, let's see if that works.

I really recommend babbel okhttp-aws-signer as it makes things like this quite simple and helps you answer a few AWS questions without too much drama.

Hope this has helped you with something AWS related!

RxJava filter an Observable list

I had an observable with data type list, and I wanted to filter the objects in that list and maintain the observable. Sounds simple enough right, but it took a while and I wanted to record it as I've used this concept a few times now.

So let's start with our object of which we will create the list.

data class Muppet(var name: String, var age: int)

Now let's create a list of those objects

val myList = Observable.just(arrayListOf<Muppet>())

Simple stuff so far. Now let's get to the interesting Rx goodness. 

What we need is to extract the individual objects so we can filter them, so we need an observable of objects rather than an observable of a list. 

.flatMap { iterable: List<Muppet> -> Observable.from(iterable) }

So here we use flatmap to change the data type and we use the very clever Observable.from to change the data to a single result.

Now we can filter as our heart desire

.filter { muppet: Muppet -> muppet.age > 30 }

and put it back together using toList.

Here's the complete example

return getMuppets()
       .flatMap { iterable: List<Muppet> -> Observable.from(iterable) }
       .filter { muppet: Muppet -> muppet.age > 30 }
       .toList()

Kotlin Coroutines

I don't by any means propose to be a master on this topic, but there's been so much chat on this topic on the blog world, I thought I'd give it a try. As is my usual I always like to start with the most insanely simple scenario I can think of.

I started off reading this article which actually gives a really nice overview of what coroutines are and why they're important:
https://medium.com/androiddevelopers/coroutines-on-android-part-i-getting-the-background-3e0e54d20bb

I started by creating a new Kotlin project, I'm on Android Studio 3.3.2 and Kotlin version 1.3.21 and added the following library:

implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.0.1'

Maybe this will be added into standard Kotlin in the future. For me this was quite a surprise
that it needed a separate library.

Now let's jump straight into some reckless coding, I added a TextView to my activity with the ID 

android:id="@+id/hello_world"

Now let's create a function that will run in the background. I'm thinking of this like the main thread on an AsyncTask. For my function, I'm going to pause for ten seconds and then return a string.

suspend fun get(): String {
    //Delay for ten seconds
    delay(10000)
    return "All your base are belong to me"
}

Now a plain vanilla function to show the results

fun show(result: String) {
    val tv = findViewById<TextView>(R.id.hello_world)
    tv.setText(result)
    println("Done!")
}

Now we need to setup the Activity to allow for Coroutines. This in my opinion is a bit of a mess. I've no idea why I need all this boilerplate nonsense. Oh well

class MainActivity : AppCompatActivity(), CoroutineScope {

    private var job: Job = Job()

    override val coroutineContext: CoroutineContext
        get() = Dispatchers.Main + job

    override fun onDestroy() {
        super.onDestroy()
        job.cancel()
    }

You'll see we need to implement the CoroutineScope and add some other fluff just to use Coroutines. Lastly we can call our function in the background from our onCreate method

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    launch {
        // Dispatchers.IO
        val result = get()
        // Dispatchers.Main
        show(result)
    }

    println("Start!")
}

So as you can see onCreate launches get in the background which waits ten seconds then returns a string. The launch method then, (when complete) calls the show method with that result. I know that's an incredibly basic example but I hope it helps. I found the concept easy to grasp, but the implementation was a bit fiddly. Hence the blog post.