Monday, August 6, 2012

Setting up a dedicated git server


This tutorial explains how to set up a git server on linux. My distribution of choice is Gentoo, but the setup will be similar for other distributions.

Git allows to connect using several protocols:
  • git://
    Generally used to allow for anonymous access to a repository. As it does not support user authentication it is generally used for read access only.
  • ssh://
    The default protocol to use for git access. SSH logon will be done with the same username for all users. Git users will be diversified by the key they are using for authentication. So no password authentication shall be used here.
  • http://
    Access over a web frontend. Again for anonymous read access.
There are even more protocols available, but in this tutorial we will focus on these connection types.

As some of you might not want to install this on their own, I provide a ready to use VM image at the end of this post.

Step 1: Setup Gitolite

Gitolite is currently the git server of choice for the gentoo community. Set it up with
emerge gitolite

[ebuild  N     ] dev-vcs/gitolite-2.3.1  USE="-contrib -vim-syntax"
Portage will automatically create a user 'git' and set its homedir to /var/lib/gitolite, which is where we will host our git repositories.

Step 2: Configure Gitolite

As we need to login with our git user during setup provide some password for the user:
passwd git
For administration of repositories, git users and access rights we need an administration user. This can be any user on any system. Further on I will call this user GitAdmin.

So login with GitAdmin and create a public/private key pair if you don't have one yet.
ssh-keygen
Copy the public key to the git account
scp ~/.ssh/id_rsa.pub git@localhost:admin.pub
Login with your git account and initialize the git repositories:
su git
cd ~
gl-setup admin.pub
The setup will fire up an editor with the default settings. Change the repositories mask to
$REPO_UMASK = 0027;
This will allow group access to the repositories. It is important to set this during gl-setup as otherwise some generated folders would have too strict access conditions.

Now we can delete admin.pub and leave the git account.
rm ~/admin.pub
exit
Step 3: Managing users & repositories

Git keeps its own repository for administration. Switch back to your GitAdmin account and clone the gitolite-admin repository:
git clone git@localhost:gitolite-admin.git
Now you can change your master configuration file ~/gitolite-admin/conf/gitolite.conf
repo    gitolite-admin
        RW+     =   admin

repo    testing
        RW+     =   @all
You can add/remove repositories and users by changing this file. Gitolite tracks changes on that file and triggers all the required actions from that. See the documentation for details on the syntax.

For each username used in the gitolite.conf file there should exist an adequate public key file under ~/gitolite-admin/keydir. Eg for a user 'testuser' there should exist a file testuser.pub. Again you can find more info in the documentation.

After changing the configuration file we need to commit and push those changes.
cd ~/gitolite-admin
git add keydir conf
git commit -m 'adding some new users/repos'
git push origin master
The push triggers creation of new repositories or changing access rights.

ssh setup is complete and you can connect to a repository by using ssh://git@localhost/repositoryName.git.

The private key used for authentication will be used by git to grant correct repository access rights.

Optional: Setup git:// access

For git:// access we need to run the git daemon. First change some settings in /etc/conf.d/git-daemon
GITDAEMON_OPTS="--syslog --export-all --base-path=/var/lib/gitolite/repositories"

GIT_USER="gitdaemon"
GIT_GROUP="git"

Add the gitdaemon user.
useradd -g git -G git -s /sbin/nologin -d /dev/null gitdaemon
Start the service now and at boot time:
rc-update add git-daemon default 
/etc/init.d/git-daemon start

Hint: Please read the git-daemon documentation (man git-daemon) as you might not want to export all repositories found under --base-path. Eg. you normally would not export gitolite-admin.git this way.

Hint: If you run on a trusted network you might add --enable=receive-pack to GITDAEMON_OPTS which would allow for anonymous write access over git://.

Optional: Setup CGit web interface (http:// access)

CGit offers access to repositories over http. You need a webserver to run this cgi script. For this tutorial we will use a default apache installation.

At the time of writing cgit is masked by the ~x86 keyword, so you need to unmask it before emerging.
emerge cgit apache

[ebuild  N     ] www-servers/apache-2.2.22-r1  USE="ssl -debug -doc -ldap (-selinux) -static -suexec -threads" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias -asis -auth_digest -authn_dbd -cern_meta -charset_lite -dbd -dumpio -ident -imagemap -log_forensic -proxy -proxy_ajp -proxy_balancer -proxy_connect -proxy_ftp -proxy_http -proxy_scgi -reqtimeout -substitute -version" APACHE2_MPMS="-event -itk -peruser -prefork -worker" 0 kB
[ebuild  N     ] app-admin/webapp-config-1.50.16-r4  102 kB
[ebuild  N     ] virtual/httpd-cgi-0  0 kB
[ebuild  N    ~] www-apps/cgit-0.9.0.2-r1  USE="vhosts -doc -highlight" 2,704 kB
Install it using webapp-config.
webapp-config -I -h localhost -d git cgit 0.9.0.2-r1
Edit the cgit configuration file /etc/cgitrc
# Enable caching of up to 1000 output entries
cache-size=1000 

# Specify the css url 
css=/git/cgit.css

# Use a custom logo
logo=/git/cgit.png

## List of repositories.
include=/etc/cgit-repos
As mentioned by webapp-config we will create our own file to manage available repositories: Execute following command on demand or run it using cron.
/usr/share/webapps/cgit/0.9.0.2-r1/hostroot/cgi-bin/cgit.cgi --scan-tree=/var/lib/gitolite/repositories >/etc/cgit-repos
Now add the apache user to the git group, start apache and add it to your default runlevel.
usermod -a -G git apache
/etc/init.d/apache2 start
rc-update add apache2 default
Point your favorite browser to http://<your-server>/cgi-bin/cgit.cgi and watch cgit in action.

Hint: You might want to hide the cgi-bin path within your urls. Therefore you can change /etc/apache2/vhosts.d/default_vhost.include and append some aliases at the end of the alias_module section
<IfModule alias_module>
        # Redirect: Allows you to tell clients about documents that used to
        # ... snip ...

        ScriptAlias /cgi-bin/ "/var/www/localhost/cgi-bin/"

        Alias /git/cgit.css "/var/www/localhost/htdocs/git/cgit.css"
        Alias /git/cgit.png "/var/www/localhost/htdocs/git/cgit.png"
        ScriptAlias /git/ "/var/www/localhost/cgi-bin/cgit.cgi/"
</IfModule>
After restarting apache you can use http://<your-server>/git/ as a starting point.

Further reading:

  1. Install gitolite on Debian and Gentoo 
  2. How gitolite uses ssh
  3. Configuring repositories
  4. Configuring users
  5. What your users need to know
VM image:

For your convenience you can download an OVA image, created with VirtualBox. I guess it will run with other virtualizers too.

Root password is "gentoo", I used the root user as GitAdmin.

In case your network is missing refer to the gentoo wiki on vmware and delete your udev persistent net rules file.

If you want to use portage you need to emerge --sync first as I deleted the tree for a smaller image size.

Thursday, August 2, 2012

Mirroring a p2 repository

For builds or for resolving a target platform eclipse often needs to connect to remote update sites. If you are behind a proxy or on a slow connection this can be annoying. A solution to that could be a local mirror of an update site.

Eclipse provides two applications to do exactly that (executed on a shell):
<eclipse install dir>/eclipse -nosplash -verbose -consoleLog -application org.eclipse.equinox.p2.artifact.repository.mirrorApplication -source <update site> -destination<local folder>
<eclipse install dir>/eclipse -nosplash -verbose -consoleLog -application org.eclipse.equinox.p2.metadata.repository.mirrorApplication -source <update site> -destination<local folder>
The first one mirrors artifacts like, plugins, features, additional libraries and so on. The second one mirrors metadata like bundle dependencies or repository content overview.

You can add proxy settings by providing http.proxyHost / http.proxyPort properties to the command. To mirror the Juno repository to C:\p2\Juno you would use:
<eclipse install dir>/eclipse -Dhttp.proxyHost=proxy.nowhere.com -Dhttp.proxyPort=80 -nosplash -verbose -consoleLog -application org.eclipse.equinox.p2.artifact.repository.mirrorApplication -source http://download.eclipse.org/releases/juno -destination file:/C:/p2/Juno/
<eclipse install dir>/eclipse -Dhttp.proxyHost=proxy.nowhere.com -Dhttp.proxyPort=80 -nosplash -verbose -consoleLog -application org.eclipse.equinox.p2.metadata.repository.mirrorApplication -source http://download.eclipse.org/releases/juno -destination file:/C:/p2/Juno/

For details see the online documentation or the wiki entry.