<!--
.. title: GitLab and go get
.. slug: gitlab-and-go-get
.. date: 2022-06-02 01:30:23 UTC
.. updated: 2024-02-13 23:46:58 UTC
.. tags: gitlab, go
.. category:
.. link:
.. description:
.. type: text
-->

I was trying to run `go get` on a git repository hosted on a private GitLab
instance. This repository was stored under a sub-group. I was working from a
non-default branch. Since it is a monorepo the module/package was in a
subdirectory. So a combination of four factors complicated matters for me.

<!-- TEASER_END: Read more -->

[TOC]

# Easiest Solution

These are the first few things we must clarify,

- `go get`, `go mod tidy`, and `go mod download` can use https or ssh to fetch
  Go modules from a git server (like GitLab or GitHub)
- `go` will by default fetch modules without first authenticating
- `go` needs to fetch subgroups depending on the Go module's name. For example,
  if the module is called
  _gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo_, `go` will need to
  fetch all subgroups of _mygroup_.
- `go` needs to fetch tags and commits from the git server
- GitLab has [designed private subgroups](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37832#note_53668438)
  to not be accessible by unauthenticated users. This includes repositories,
  tags, commits, etc.

The easiest way to deal with this situation is a combination of,

- `go` **must** use https not ssh to work with GitLab
- `go` **must** use
  [.netrc](https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html)
  stored in the user's home directory
- **Always** use either tag or commit to specify version; **never** use branch name
- **Always** use `go get` because it also updates _go.mod_ and _go.sum_ files
- Environment variables GOPROXY and GOPRIVATE **must** be configured
  appropriately

Big thanks to [Jonathan Hall](https://stackoverflow.com/a/57099566) for this.

## Setup

`go` will prefer https over ssh (from what I have observed). The next step is
to create a GitLab API *read-only* (because `go` just needs to read)
[personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html).
Create ~/.netrc as below (change values per your environment),

    $ export GITLAB_HOSTNAME='<gitlab.my-internal-domain.com>'
    $ export GITLAB_USERNAME='<your_username>'
    $ export GITLAB_API_TOKEN='<your_read_only_token>'
    $ printf 'machine %s\n    login %s\n    password %s\n\n' $GITLAB_HOSTNAME $GITLAB_USERNAME $GITLAB_API_TOKEN >> ~/.netrc

Read more about environment variables
[GOPROXY and GOPRIVATE](link://slug/goproxy-and-goprivate).

## go get

With the above setup complete, you can start using various ways
to get modules,

    $ go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo

The above will get the latest from the default branch. If you want to specify a
different tag or commit ID, append _@ID_ (where ID is one of
tag or commit),

    $ go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo@tag
    $ go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo@commit-id

I have not had success with using branch names. I always get error message,
_invalid version: disallowed version string_.

# Alternative But Finicky Solution

This solution tries to get you going with ssh instead of https.

Let's say the repository URL I was working with was
gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.
The go.mod in that repo was

    module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage
    go 1.20

I was trying `go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage`
in a separate package, essentially using mygopackage as a dependency. I got
this error,

    go: module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage: git ls-remote -q origin in /Users/aikchar/go/pkg/mod/cache/vcs/27f0a3bd58674de4db4e2a3e38601eecc893ec233ba6addcf6eeb2b43f240ef9: exit status 128:
        fatal: could not read Username for 'https://gitlab.my-internal-domain.com': terminal prompts disabled
    Confirm the import path was entered correctly.
    If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

I figured I made a mistake. I needed to add _.git_ to my go.mod in _myrepo_ so
gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage becomes
gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo*.git*/src/mygopackage.

After fixing go.mod in gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage,
it was,

    module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage
    go 1.20

I ran `go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage`
and got this error,

    go: downloading gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git v0.0.0-20220303202727-f334bf068bdb
    go: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: verifying module: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: reading https://sum.golang.org/lookup/gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: 410 Gone
        server response:
        not found: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b9eaa09fa03cd0f3139730aa61579fcd85cd6fd748674c4aa228cd6a57e99b31: exit status 128:
            fatal: unable to look up gitlab.my-internal-domain.com (port 9418) (Name or service not known)

I added `@commit-id` to `go get` like so: `go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@e091a5ea01d7`
and got this error,

    gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: verifying module: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: reading https://sum.golang.org/lookup/gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: 410 Gone
        server response:
        not found: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b9eaa09fa03cd0f3139730aa61579fcd85cd6fd748674c4aa228cd6a57e99b31: exit status 128:
            fatal: unable to look up gitlab.my-internal-domain.com (port 9418) (Name or service not known)

I added environment variable _GOPRIVATE_ and ran this command: `GOPRIVATE=gitlab.my-internal-domain.com go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@e091a5ea01d7`
and IT WORKED!

    go: added gitlab.my-internal-domain.commygroup/mysubgroup/myrepo.git/src/mygopackage v0.0.0-20220531210353-e091a5ea01d7

Basically, to counter the four factors I mentioned at the top, I needed to do
this,

- Make sure _.git_ is part of the module name and during import or `go get`.
  This is not always possible to do.
- Add commit-id (NOT branch name) at the end of `go get` when getting from non-default branch
- Since GitLab is private and not publically accessible, set the _GOPRIVATE_ environment variable to the right hostname

Let's add one more factor: the GitLab server is not reachable over port 22 for
ssh but instead uses a custom port. How about one more factor i.e. anonymous,
non-logged in read access is not allowed; you need to login to even read a
repository.

The symptom of the second new factor, no anonymous read allowed, is an error
message like
_fatal: could not read Username for 'https://gitlab.my-internal-domain.com': terminal prompts disabled_.

The fix for this factor is to use ssh instead of http or https to connect to
GitLab. This way we can setup ssh keys to work seamlessly without needing a
human to enter username and password to login. We will use git config's
[insteadOf](https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf)
option (thanks to [Set GOPRIVATE to match your github organization](https://gist.github.com/StevenACoffman/866b06ed943394fbacb60a45db5982f2)).

    $ git config --global url."ssh://git@gitlab.my-internal-domain.com".insteadOf "https://gitlab.my-internal-domain.com"

Remember the other factor we wanted to consider i.e. ssh is accessible on a
non-standard port e.g. 2222? Modify the config to include the port,

    $ git config --global url."ssh://git@gitlab.my-internal-domain.com:2222".insteadOf "https://gitlab.my-internal-domain.com"

One more thing to possibly cause issues is that regardless of the configrations
discussed above, `go get` will use http(s) instead of ssh. I couldn't find a
good solution for it. A workaround is to _always_ use `go mod tidy` instead.
Another less-than-optimial workaround is to modify the module name by appending
_.git_. This has additional effects which may make this impossible in most (in
my opinion) cases. This ties into the next point as well.

You may need to use both _require_ and _replace_ in _go.mod_. I had a situation
where the module I wanted to pull in was named
gitlab.my-internal-domain.com/group/subgroup/module and the git repository was
in a sub-group like
gitlab.my-internal-domain.com:2222/group/subgroup/module.git. When I tried
`go get` or even `go mod tidy` (with debug flag `-x`) it showed that instead of
pulling module.git it was pulling subgroup.git. That was a real surprise. The
fix for it was to add the following lines in _go.mod_ (_v0.4.7_ is the
version/tag of the module I needed, change it to suit your situation),

    require gitlab.my-internal-domain.com/group/subgroup/module v0.4.7
    replace gitlab.my-internal-domain.com/group/subgroup/module => gitlab.my-internal-domain.com/group/subgroup/module.git v0.4.7

What the above does is when you run `go mod tidy` it will treat module.git as
the git respository instead of assuming subgroup was the git repository. I
can't find the Stack Overflow post anymore but it said that GitLab doesn't give
visibility to repositories in subgroups to `go get` so when you have
`require gitlab.my-internal-domain.com/group/subgroup/module` in _go.mod_ then
`go get` assumes submodule.git is the repository and module/ is the
directory/path to the _go.mod_ in it. But with _replace_ and `go mod tidy` it
all works like it should. To repeat, in this situation, always use
`go mod tidy` because `go get` will not work (it didn't for me with Go 1.20.3).