A few weeks ago I updated our gitlab-installation to the glorious, shiny and new 8.12.6! Awesome new stuff! It had only one drawback. We couldn’t clone repos via HTTP(S) anymore.

It wasn’t that much of a drawback at the time but then we wanted to use the included CI-stuff. And that – you guessed it already – only works with cloning or fetching the repo via HTTP(S).

Setup

We have a rather awkward setup with GitLab running on a MacOS X 10.9 machine with all the necessary packages installed via homebrew. So all the bits and pieces like ruby, go, postgres, git and nginx are maintained via homebrew, no MacOS-Stuff involved there.

And I’m also running GitLab in a subdirectory. So I reach my gitlab via https://example.com/gitlab. That was messy right from the start, got better, but we’ll come back to it later.

Finally the interaction between nginx and gitlab-workhorse as well as between gitlab-workhorse and gitlab is done via sockets. There’s no tcp involved in the internal communication between the three parts.

Problem

So after that update checkouts via HTTPS didn’t work. We didn’t notice at first as all interaction we are doing with gitlab is SSH-based. It arose once where we wanted to checkout via HTTPS due the SSH-Port being blocked in a firewall but it was easier to open that port then to debug the issue.

So it all stroke when I wanted to use CI a week ago. A git checkout didn’t work and returned an Error 500.

Google revealed I should double- and triple-check my nginx-config which I did. No issues. Everything as it should be. Perhaps a missing / at the end of my proxy-pass directive? Tested: Now nothing works at all. Even though before I could use the webinterface, that was now broken. So that’s not the solution.

So I had a look into the production.log:

Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"[namespace]", "project_id"=>"[project].git"}
Completed 500 Internal Server Error in 148ms (ActiveRecord: 7.3ms)

JWT::DecodeError (Nil JSON web token):
  lib/gitlab/workhorse.rb:120:in `verify_api_request!'
  app/controllers/projects/git_http_client_controller.rb:154:in `verify_workhorse_api!'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

Hey! Yeah! Something to go with!

So I dug deep into the ruby-code to find (around line 120 in lib/gitlab/workhorse.rb) that an expected HTTP-Header isn’t sent from the workhorse. Tests in overriding that check failed.

So I went to the workhorse and dug into Go. Yeah, I found the place where something went wrong but I couldn’t find a way to get hold of the error. Looks like I need to dig deeper into Go at some point.

But it looked like some config param was strange. So I again rechecked the configuration of the Workhorse. And there was something that struck me: Relative URL Support! Yes, we are running on a relative URL. But that’s irrelevant here as all the communication is handled via sockets! That was the reason why I originally removed that parameter completely from the parameter-list.

As it turns out, that parameter is the only way to tell gitlab-workhorse that we are running on a relative URL. Even though the communication runs via sockets (did I point out that the communication runs via sockets? (-: ) and there’s a tomcat listening on localhost:8080 the /gitlab part is necessary to get the communication into the right channels.

I started a workhorse with ./gitlab-workhorse -listenUmask 0 -secretPath /Users/git/gitlab/.gitlab_workhorse_secret -listenNetwork unix -listenAddr /Users/git/gitlab/tmp/sockets/gitlab-workhorse.socket -authSocket /Users/git/gitlab/tmp/sockets/gitlab.socket -documentRoot /Users/git/gitlab/public -proxyHeadersTimeout 1m0s -authBackend http://localhost:8080/gitlab and again tried a checkout via HTTPS: And it worked!

So even though I have a completely unrelated server running on localhost:8080 the parameter is important for the path-component…

What did I learn from it? A bit of Go, Ruby and more GoogleFu. But most of all: Getting Gitlab to run on a relative Path is a pain. But when it works, it’s awesome!