<!--
.. title: GitLab and go get
.. slug: gitlab-and-go-get
.. date: 2022-06-02 01:30:23 UTC
.. updated: 2023-05-04 01:13:22 UTC
.. tags: gitlab, go
.. category:
.. link:
.. description:
.. type: text
-->

I was trying to run `go get` on a git repository hosted on a private GitLab
instance. This repository was stored under a sub-group. I was working from a
non-default branch. Since it is a monorepo the module/package was in a
subdirectory. So a combination of four factors complicated matters for me.

<!-- TEASER_END: Read more -->

For example, the repository URL was gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.
The go.mod in that repo was

    module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage
    go 1.18


I was trying `go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage`
in a separate package, essentially using mygopackage as a dependency. I got
this error,

    go: module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage: git ls-remote -q origin in /Users/aikchar/go/pkg/mod/cache/vcs/27f0a3bd58674de4db4e2a3e38601eecc893ec233ba6addcf6eeb2b43f240ef9: exit status 128:
        fatal: could not read Username for 'https://gitlab.my-internal-domain.com': terminal prompts disabled
    Confirm the import path was entered correctly.
    If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

I figured I made a mistake. I needed to add _.git_ to my go.mod in _myrepo_ so
gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage becomes
gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo*.git*/src/mygopackage.

After fixing go.mod in gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage,
it was,

    module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage
    go 1.18

I ran `go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage`
and got this error,

    go: downloading gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git v0.0.0-20220303202727-f334bf068bdb
    go: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: verifying module: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: reading https://sum.golang.org/lookup/gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: 410 Gone
        server response:
        not found: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b9eaa09fa03cd0f3139730aa61579fcd85cd6fd748674c4aa228cd6a57e99b31: exit status 128:
            fatal: unable to look up gitlab.my-internal-domain.com (port 9418) (Name or service not known)

I added `@commit-id` to `go get` like so: `go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@e091a5ea01d7`
and got this error,

    gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: verifying module: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: reading https://sum.golang.org/lookup/gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: 410 Gone
        server response:
        not found: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b9eaa09fa03cd0f3139730aa61579fcd85cd6fd748674c4aa228cd6a57e99b31: exit status 128:
            fatal: unable to look up gitlab.my-internal-domain.com (port 9418) (Name or service not known)

I added environment variable _GOPRIVATE_ and ran this command: `GOPRIVATE=gitlab.my-internal-domain.com go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@e091a5ea01d7`
and IT WORKED!

    go: added gitlab.my-internal-domain.commygroup/mysubgroup/myrepo.gitsrc/mygopackage v0.0.0-20220531210353-e091a5ea01d7

Basically, to counter the four factors I mentioned at the top, I needed to do
this,

- Make sure _.git_ is part of the module name and during import or `go get`
- Add commit-id (NOT branch name) at the end of `go get` so when getting from non-default branch
- Since GitLab is private and not publically accessible, set the _GOPRIVATE_ environment variable to the right hostname

Let's add one more factor: the GitLab server is not reachable over port 22 for
ssh but instead uses a custom port. How about one more factor i.e. anonymous,
non-logged in read access is not allowed; you need to login to even read a
repository.

The symptom of the second new factor, no anonymous read allowed, is an error
message like
_fatal: could not read Username for 'https://gitlab.my-internal-domain.com': terminal prompts disabled_.

The fix for this factor is to use ssh instead of http or https to connect to
GitLab. This way we can setup ssh keys to work seamlessly without needing a
human to enter username and password to login. We will use git config's
[insteadOf](https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf)
option (thanks to [Set GOPRIVATE to match your github organization](https://gist.github.com/StevenACoffman/866b06ed943394fbacb60a45db5982f2)).

    $ git config --global url."ssh://git@gitlab.my-internal-domain.com".insteadOf "https://gitlab.my-internal-domain.com"

Remember the other factor we wanted to consider i.e. ssh is accessible on a
non-standard port e.g. 2222? Modify the config to include the port,

    $ git config --global url."ssh://git@gitlab.my-internal-domain.com:2222".insteadOf "https://gitlab.my-internal-domain.com"

One more thing to possibly cause issues is that regardless of the configrations
discussed above, `go get` will use http(s) instead of ssh. I couldn't find a
solution for it. A workaround is to _always_ use `go mod tidy` instead.

But it's not just that. You may have to use both _require_ and _replace_ in
_go.mod_. I had a situation where the module I wanted to pull in was named
gitlab.my-internal-domain.com/group/subgroup/module and the git repository was
in a sub-group like
gitlab.my-internal-domain.com:2222/group/subgroup/module.git. When I tried
`go get` or even `go mod tidy` (with debug flag `-x`) it showed that instead of
pulling module.git it was pulling subgroup.git. That was a real surprise. The
fix for it was to add the following lines in _go.mod_ (_v0.4.7_ is the
version/tag of the module I needed, change it to suit your situation),

    require gitlab.my-internal-domain.com/group/subgroup/module v0.4.7
    replace gitlab.my-internal-domain.com/group/subgroup/module => gitlab.my-internal-domain.com/group/subgroup/module.git v0.4.7

What the above does is when you run `go mod tidy` it will treat module.git as
the git respository instead of assuming subgroup was the git repository. I
can't find the Stack Overflow post anymore but it said that GitLab doesn't give
visibility to repositories in subgroups to `go get` so when you have
`require gitlab.my-internal-domain.com/group/subgroup/module` in _go.mod_ then
`go get` assumes submodule.git is the repository and module/ is the
directory/path to the _go.mod_ in it. But with _replace_ and `go mod tidy` it
all works like it should. To repeat, in this situation, always use
`go mod tidy` because `go get` will not work (it didn't for me with Go 1.20.3).