GitLab and go get
I was trying to run go get
on a git repository hosted on a private GitLab
instance. This repository was stored under a sub-group. I was working from a
non-default branch. Since it is a monorepo the module/package was in a
subdirectory. So a combination of four factors complicated matters for me.
For example, the repository URL was gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo. The go.mod in that repo was
module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage go 1.18
I was trying go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage
in a separate package, essentially using mygopackage as a dependency. I got
this error,
go: module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage: git ls-remote -q origin in /Users/aikchar/go/pkg/mod/cache/vcs/27f0a3bd58674de4db4e2a3e38601eecc893ec233ba6addcf6eeb2b43f240ef9: exit status 128: fatal: could not read Username for 'https://gitlab.my-internal-domain.com': terminal prompts disabled Confirm the import path was entered correctly. If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
I figured I made a mistake. I needed to add .git to my go.mod in myrepo so gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo/src/mygopackage becomes gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage.
After fixing go.mod in gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage, it was,
module gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage go 1.18
I ran go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage
and got this error,
go: downloading gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git v0.0.0-20220303202727-f334bf068bdb go: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: verifying module: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: reading https://sum.golang.org/lookup/gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: 410 Gone server response: not found: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git@v0.0.0-20220303202727-f334bf068bdb: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b9eaa09fa03cd0f3139730aa61579fcd85cd6fd748674c4aa228cd6a57e99b31: exit status 128: fatal: unable to look up gitlab.my-internal-domain.com (port 9418) (Name or service not known)
I added @commit-id
to go get
like so: go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@e091a5ea01d7
and got this error,
gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: verifying module: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: reading https://sum.golang.org/lookup/gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: 410 Gone server response: not found: gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@v0.0.0-20220531210353-e091a5ea01d7: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b9eaa09fa03cd0f3139730aa61579fcd85cd6fd748674c4aa228cd6a57e99b31: exit status 128: fatal: unable to look up gitlab.my-internal-domain.com (port 9418) (Name or service not known)
I added environment variable GOPRIVATE and ran this command: GOPRIVATE=gitlab.my-internal-domain.com go get gitlab.my-internal-domain.com/mygroup/mysubgroup/myrepo.git/src/mygopackage@e091a5ea01d7
and IT WORKED!
go: added gitlab.my-internal-domain.commygroup/mysubgroup/myrepo.gitsrc/mygopackage v0.0.0-20220531210353-e091a5ea01d7
Basically, to counter the four factors I mentioned at the top, I needed to do this,
- Make sure .git is part of the module name and during import or
go get
- Add commit-id (NOT branch name) at the end of
go get
so when getting from non-default branch - Since GitLab is private and not publically accessible, set the GOPRIVATE environment variable to the right hostname
Let's add one more factor: the GitLab server is not reachable over port 22 for ssh but instead uses a custom port. How about one more factor i.e. anonymous, non-logged in read access is not allowed; you need to login to even read a repository.
The symptom of the second new factor, no anonymous read allowed, is an error message like fatal: could not read Username for 'https://gitlab.my-internal-domain.com': terminal prompts disabled.
The fix for this factor is to use ssh instead of http or https to connect to GitLab. This way we can setup ssh keys to work seamlessly without needing a human to enter username and password to login. We will use git config's insteadOf option (thanks to Set GOPRIVATE to match your github organization).
$ git config --global url."ssh://git@gitlab.my-internal-domain.com".insteadOf "https://gitlab.my-internal-domain.com"
Remember the other factor we wanted to consider i.e. ssh is accessible on a non-standard port e.g. 2222? Modify the config to include the port,
$ git config --global url."ssh://git@gitlab.my-internal-domain.com:2222".insteadOf "https://gitlab.my-internal-domain.com"
One more thing to possibly cause issues is that regardless of the configrations
discussed above, go get
will use http(s) instead of ssh. I couldn't find a
solution for it. A workaround is to always use go mod tidy
instead.
But it's not just that. You may have to use both require and replace in
go.mod. I had a situation where the module I wanted to pull in was named
gitlab.my-internal-domain.com/group/subgroup/module and the git repository was
in a sub-group like
gitlab.my-internal-domain.com:2222/group/subgroup/module.git. When I tried
go get
or even go mod tidy
(with debug flag -x
) it showed that instead of
pulling module.git it was pulling subgroup.git. That was a real surprise. The
fix for it was to add the following lines in go.mod (v0.4.7 is the
version/tag of the module I needed, change it to suit your situation),
require gitlab.my-internal-domain.com/group/subgroup/module v0.4.7 replace gitlab.my-internal-domain.com/group/subgroup/module => gitlab.my-internal-domain.com/group/subgroup/module.git v0.4.7
What the above does is when you run go mod tidy
it will treat module.git as
the git respository instead of assuming subgroup was the git repository. I
can't find the Stack Overflow post anymore but it said that GitLab doesn't give
visibility to repositories in subgroups to go get
so when you have
require gitlab.my-internal-domain.com/group/subgroup/module
in go.mod then
go get
assumes submodule.git is the repository and module/ is the
directory/path to the go.mod in it. But with replace and go mod tidy
it
all works like it should. To repeat, in this situation, always use
go mod tidy
because go get
will not work (it didn't for me with Go 1.20.3).