How to cache go mod for faster builds in Github Task?

[alok87]

Since we mount the code everytime in a fresh container, the fresh container in github task has no cache of go mod. make push becomes slow sometimes due to the network getting choked downloading the same thing.

What can we add support to fix this?

We do not vendor check in our org at present so this is a major concern.

[thockin]

The way we do this in Kubernetes is with a script which wraps go commands and sets the GOMODCACHE and GOCACHE variables to a local directory.

I suppose this repo could volume-mount those to the in-container equivalent place.

I don't have a lot of free time in the near future to try it, but if you want to try it, that's how I would go.

docker -v $(go env GOMODCACHE):/go/pkg/mod:rw -v $(go env GOCACHE):/.cache/go-build ...

[alok87]

@thockin does your solution prevent the download of go modules in Github task/action as well without committing the vendor?

[thockin]

I took a quick look at this, and I am confused:

It already does a volume-mount for these, or it thinks it does. The go-build rule calls docker run with these args:

            -v $$(pwd)/.go/cache:/.cache                            \
            --env GOCACHE="/.cache/gocache"                         \
            --env GOMODCACHE="/.cache/gomodcache"                   \

That SHOULD cause the Go module cache to be reused across invocations, as long as you don't make clean in between. It seems to work:

$ make clean

$ ls -l .go
ls: cannot access '.go': No such file or directory

$ git diff
diff --git a/cmd/myapp-1/main.go b/cmd/myapp-1/main.go
index d5d2cc7..38f5aab 100644
--- a/cmd/myapp-1/main.go
+++ b/cmd/myapp-1/main.go
@@ -20,6 +20,7 @@ package main
 import (
        "log"
 
+       _ "github.com/spf13/pflag"
        "github.com/thockin/go-build-template/pkg/version"
 )
 
diff --git a/go.mod b/go.mod
index 8cda077..010a866 100644
--- a/go.mod
+++ b/go.mod
@@ -1,3 +1,5 @@
 module github.com/thockin/go-build-template
 
 go 1.21
+
+require github.com/spf13/pflag v1.0.5 // indirect
diff --git a/go.sum b/go.sum
index e69de29..287f6fa 100644
--- a/go.sum
+++ b/go.sum
@@ -0,0 +1,2 @@
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=

$ make
# building for linux/amd64
go: downloading github.com/spf13/pflag v1.0.5
binary: bin/linux_amd64/myapp-1  
binary: bin/linux_amd64/myapp-2  

$ ls -l .go/cache/gomodcache/
total 8
drwxr-xr-x 3 thockin primarygroup 4096 Aug 16 16:20 cache
drwxr-xr-x 3 thockin primarygroup 4096 Aug 16 16:20 github.com

$ make
# building for linux/amd64
binary: bin/linux_amd64/myapp-1  (cached)
binary: bin/linux_amd64/myapp-2  (cached)

$ rm -rf .go/cache/gomodcache/

$ make
# building for linux/amd64
go: downloading github.com/spf13/pflag v1.0.5
binary: bin/linux_amd64/myapp-1  (cached)
binary: bin/linux_amd64/myapp-2  (cached)

So .,..what's not working?

[alok87]

Yes @thockin it works in local setup. But it was needed in CI - github task. The same command with the cache wont work, as the github task does not have a stateful disk option between multiple invocations

[thockin]

Do GitHub actions have a module cache which spans runs? That would surprise me a little bit. Seems like an opportunity for one user to poison another.

In theory we could change this to look for these variables explicitly, and if it finds them bind them out those instead. But I'm I'm a little skeptical about this. What about permissions?