From git to pip – Continuous Integration with Strider and Python

My last blog post only a few hours ago showed how to install StriderCD –  a node.js-based continuous integration/deployment server.

I’ve been rapidly learning the ins and outs of Strider over the past couple of hours and I now have it hooked into a bitbucket private repo, checking out every new commit pushed there, running the tests, creating the pypi packages and moving them to a web server so I can publish them.

This is how I did it…

Step 1: Make sure you have a git repository to work on.

In  the last blog post, we added github and bitbucket credentials to the upstart script so that we could view all of our repositories once we logged in.

Once you have added a project, go to the configure screen by clicking the “Spanner” icon next to the project.

Step 2: Configure the custom scripts

Go into the “Plugins” section and drag the “Custom Scripts” plugin into the “Active Plugins” then click on the newly-arrived “Custom Scripts” link on the left of the screen.

The “Custom Scripts” screen is much like the “Execute Bash” plugin in Jenkins however it is clearly divided into five sections:

1. Environment
   This is where you configure any environment variables that are required for the other four sections.
2. Prepare
   This is where you can install or run any commands that are needed in advance of the following stages
3. Test
   Reasonably self-explanatory, scripts and commands placed here will be executed as tests and the job will pass or fail based upon their output
4. Deploy
   These commands are only run if all the tests pass and the job was either triggered by a code-change or the “test and deploy” button was pressed
5. Cleanup
   These commands are run regardless of the outcome of the “test” section above

Assuming that our python repo has a setup.py file in it then we can use sdist and pip2pi to generate our pypi packages and store them in a known location by adding the following content to the Custom Script sections:

Prepare:

pip install pip2pi

Deploy:

python setup.py sdist
cp dist/*.gz /var/www/pypi/packages
dir2pi /var/www/pypi/packages

Cleanup:

rm -rf *.egg-info build dist

Step 3: Configure the pypi server

There are a huge number of tutorials out there on how to create a pypi server that will mirror the official servers inside your network, and a number of massively complex options for hosting your PyPi mirror with your private packages, however it turns out you can do this with just pip2pi and nginx.

Install nginx and create a config file as follows:

server {
 listen 80;
 listen [::]:80 ipv6only=on;
 root /var/www/pypi/packages;
 index index.html index.htm;
 server_name pypi.your.domain;
 location / {
   try_files $uri $uri/ =404;
 }
}

now create the root file system:

mkdir -p /var/www/pypi/packages
chown -Rvf strider: /var/www/pypi/packages

and restart nginx

service nginx restart

Creating the packages is now easy.  Simply push a commit to your repo and watch Strider build the packages and copy them into the correct directory. Pip2pi will create the pypi site from that directory and you will be able to see your packages at http://pypi.your.domain/simple/

Step 4: Using the packages

Possibly one of the most straight-forward things to do and yet it took me ages to find out how to do it is to use these packages that have just been built as dependencies for other code repos.

The answer is simple.  Just add the following line at the top of each of your requirements.txt file:

--extra-index-url=http://pypi.your.domain/simple/

and add your privately packaged dependencies as you usually would if you were getting them from the pypi.python.org mirrors.

Now each time that your code is built and passes all the tests, the pypi server will be updated and you can use these packages in your requirements.txt files.