Travis/Local
From charlesreid1
This page covers notes on running local Travis tests (replicating the Travis environment on your local machine).
Guide here: https://github.com/erdc/proteus/wiki/Replicating-the-TravisCI-Environment-on-your-Local-Machine
Setup
Docker Pull
The Travis environment can be replicated with a Docker image. The docker image is travisci/ci-garnet and there is no latest tag.
Go to this page: https://hub.docker.com/r/travisci/ci-garnet/tags/
Click the "Tags" tab
Do a docker pull like this:
docker pull travisci/ci-garnet:packer-1510529493-6aa37a4
where the packer part comes from the latest tag.
Run the Travis Container
Run the container and have it execute the bash shell:
docker run -it travisci/ci-garnet:packer-1510529493-6aa37a4 /bin/bash
Use Travis.yml
To get the container set up before you run your tests, run the commands contained in the travis.yml script in the repository you're trying to run tests for.
Let's look at a sample travis.yml: https://github.com/soft-matter/trackpy/blob/master/.travis.yml
We have to run these commands to set up the conda environment:
Before we install anything:
wget http://repo.continuum.io/miniconda/Miniconda-3.5.5-Linux-x86_64.sh -O miniconda.sh; chmod +x miniconda.sh ./miniconda.sh -b -p /home/travis/mc export PATH=/home/travis/mc/bin:$PATH
To install stuff:
DEPS="numpy scipy matplotlib pillow pandas!=0.18.0 scikit-image pyyaml pytables numba scikit-learn" BUILD_DOCS=false conda update --yes conda conda config --append channels conda-forge export TRAVIS_PYTHON_VERSION="3.6" conda create -n testenv --yes $DEPS pip nose setuptools python=$TRAVIS_PYTHON_VERSION source activate testenv # This is not necessary, it just outputs useful info to log for Travis CI: echo $PATH which python conda info conda list # clone the required material git clone https://github.com/soft-matter/trackpy cd trackpy/ # Back to required stuff: python setup.py install
Now for the actual tests:
nosetests --nologcapture -a '!slow'
Output
root@025e1ca5221b:/# wget http://repo.continuum.io/miniconda/Miniconda-3.5.5-Linux-x86_64.sh -O miniconda.sh
[...]
root@025e1ca5221b:/# ./miniconda.sh -b -p /home/travis/mc
PREFIX=/home/travis/mc
installing: python-2.7.7-0 ...
installing: openssl-1.0.1h-0 ...
installing: pycosat-0.6.1-py27_0 ...
installing: pyyaml-3.11-py27_0 ...
installing: readline-6.2-2 ...
installing: requests-2.3.0-py27_0 ...
installing: sqlite-3.8.4.1-0 ...
installing: system-5.8-1 ...
installing: tk-8.5.15-0 ...
installing: yaml-0.1.4-0 ...
installing: zlib-1.2.7-0 ...
installing: conda-3.5.5-py27_0 ...
Python 2.7.7 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
root@025e1ca5221b:/# conda update --yes conda
root@025e1ca5221b:/# conda config --append channels conda-forge
root@025e1ca5221b:/# export PATH=/home/travis/mc/bin:$PATH
root@025e1ca5221b:/# export TRAVIS_PYTHON_VERSION="3.6"
root@025e1ca5221b:/# conda create -n testenv --yes $DEPS pip nose setuptools python=$TRAVIS_PYTHON_VERSION
root@025e1ca5221b:/# source activate testenv
(testenv) root@025e1ca5221b:/# which python
/home/travis/mc/envs/testenv/bin/python
(testenv) root@025e1ca5221b:/# conda info
Current conda install:
platform : linux-64
conda version : 4.3.30
conda is private : False
conda-env version : 4.3.30
conda-build version : not installed
python version : 2.7.13.final.0
requests version : 2.14.2
root environment : /home/travis/mc (writable)
default environment : /home/travis/mc/envs/testenv
envs directories : /home/travis/mc/envs
/root/.conda/envs
package cache : /home/travis/mc/pkgs
/root/.conda/pkgs
channel URLs : https://repo.continuum.io/pkgs/main/linux-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/linux-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/linux-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/linux-64
https://repo.continuum.io/pkgs/pro/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
config file : /root/.condarc
netrc file : None
offline mode : False
user-agent : conda/4.3.30 requests/2.14.2 CPython/2.7.13 Linux/4.9.49-moby debian/jessie/sid glibc/2.19
UID:GID : 0:0
(testenv) root@025e1ca5221b:/# conda list
# packages in environment at /home/travis/mc/envs/testenv:
#
bokeh 0.12.10 py36hbb0e44a_0
bzip2 1.0.6 h0376d23_1
ca-certificates 2017.08.26 h1d4fec5_0
certifi 2017.7.27.1 py36h8b7b77e_0
click 6.7 py36h5253387_0
cloudpickle 0.4.0 py36h30f8c20_0
cycler 0.10.0 py36h93f1223_0
dask 0.15.4 py36h31fc154_0
dask-core 0.15.4 py36h7045e13_0
dbus 1.10.22 h3b5a359_0
decorator 4.1.2 py36hd076ac8_0
distributed 1.19.1 py36h25f3894_0
expat 2.2.4 h6ea4f2b_2
fontconfig 2.12.4 h88586e7_1
freetype 2.8 h52ed37b_0
glib 2.53.6 h5d9569c_2
gst-plugins-base 1.12.2 he3457e5_0
gstreamer 1.12.2 h4f93127_0
hdf5 1.8.18 h525d4c3_0
heapdict 1.0.0 py36h79797d7_0
icu 58.2 h211956c_0
imageio 2.2.0 py36he555465_0
intel-openmp 2018.0.0 h15fc484_7
jinja2 2.9.6 py36h489bce4_1
jpeg 9b h024ee3a_2
libedit 3.1 heed3624_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 7.2.0 h7cc24e2_2
libgfortran-ng 7.2.0 h9f7466a_2
libpng 1.6.32 hda9c8bc_2
libstdcxx-ng 7.2.0 h7a57d05_2
libtiff 4.0.8 h90200ff_9
libxcb 1.12 h84ff03f_3
libxml2 2.9.4 h6b072ca_5
llvmlite 0.20.0 py36_0
locket 0.2.0 py36h787c0ad_1
lzo 2.10 h1bfc0ba_1
markupsafe 1.0 py36hd9260cd_1
matplotlib 2.1.0 py36hba5de38_0
mkl 2018.0.0 hb491cac_4
msgpack-python 0.4.8 py36hec4c5d1_0
ncurses 6.0 h9df7e31_2
networkx 2.0 py36h7e96fb8_0
nose 1.3.7 py36hcdf7029_2
numba 0.35.0 np113py36_10
numexpr 2.6.2 py36hc561933_2
numpy 1.13.3 py36ha12f23b_0
olefile 0.44 py36h79f9f78_0
openssl 1.0.2m h8cfc7e7_0
pandas 0.21.0 py36h78bd809_1
partd 0.3.8 py36h36fd896_0
pcre 8.41 hc71a17e_0
pillow 4.2.1 py36h9119f52_0
pip 9.0.1 py36h6c6f9ce_4
psutil 5.4.0 py36h84c53db_0
pyparsing 2.2.0 py36hee85983_1
pyqt 5.6.0 py36h0386399_5
pytables 3.4.2 py36hc5f0ea4_2
python 3.6.3 h1284df2_4
python-dateutil 2.6.1 py36h88d3b88_1
pytz 2017.2 py36hc2ccc2a_1
pywavelets 0.5.2 py36he602eb0_0
pyyaml 3.12 py36hafb9ca4_1
qt 5.6.2 h974d657_12
readline 7.0 ha6073c6_4
scikit-image 0.13.0 py36had3c07a_1
scikit-learn 0.19.1 py36h7aa7ec6_0
scipy 1.0.0 py36hbf646e7_0
setuptools 36.5.0 py36he42e2e1_0
sip 4.18.1 py36h51ed4ed_2
six 1.11.0 py36h372c433_1
sortedcontainers 1.5.7 py36hdf89491_0
sqlite 3.20.1 hb898158_2
tblib 1.3.2 py36h34cf8b6_0
tk 8.6.7 hc745277_3
toolz 0.8.2 py36h81f2dff_0
tornado 4.5.2 py36h1283b2a_0
wheel 0.29.0 py36he7f4e38_1
xz 5.2.3 h55aa19d_2
yaml 0.1.7 h014fa73_2
zict 0.1.3 py36h3a3bf81_0
zlib 1.2.11 ha838bed_2
Alternative Approach
Quay.io images
A more convenient approach would look like this:
- Docker pull one of the travisci images at quay.io - see https://quay.io/organization/travisci
- Download the repository of interest
- Use travis-convert as a travis.yml to build.sh converter: https://github.com/travis-ci/travis-build
First, get the image:
$ docker pull quay.io/travisci/travis-python
Now run the image:
$ docker run -ti quay.io/travisci/travis-python /bin/bash
and...............???
Use travis-build to convert travis.yml to travis.sh
To use travis-build you need ruby stuff.
OMG I F****KG HAAAAAAAATE RUBY.
Attempt... 3? 4?
All of this happens in a Docker image (e.g., travis-garnet):
$ git clone https://github.com/travis-ci/travis-build $ cd travis-build
Now install ruby 2:
$ apt-add-repository ppa:brightbox/ruby-ng $ apt-get update $ apt-get install ruby2.2 ruby2.2-dev
Now install the rake file to build this crap:
$ bundle install $ bundle exec rake
Nope. At this point I have no clue how to install travis-build.
Attempt 5
Via https://stackoverflow.com/questions/34052460/how-to-build-install-travis-build
git clone the repo into some persistent directory (~/GitHub)
from ~/GitHub/travis-build run ln -s $(pwd) ~/.travis/travis-build (taken almost verbatim from the README)
After running travis (no arguments) - oh wait, there IS no travis command.
Attempt 6
Okay, let's just try and install Travis.
root@01a7a68152ad:/# gem install travis -v 1.8.8 --no-rdoc --no-ri
Fetching: multipart-post-2.0.0.gem (100%)
Fetching: faraday-0.13.1.gem (100%)
Fetching: faraday_middleware-0.12.2.gem (100%)
Fetching: highline-1.7.8.gem (100%)
Fetching: backports-3.10.3.gem (100%)
Fetching: multi_json-1.12.2.gem (100%)
Fetching: addressable-2.4.0.gem (100%)
Fetching: net-http-persistent-2.9.4.gem (100%)
Fetching: net-http-pipeline-1.0.1.gem (100%)
Fetching: gh-0.15.1.gem (100%)
Fetching: launchy-2.4.3.gem (100%)
Fetching: ffi-1.9.18.gem (100%)
Building native extensions. This could take a while...
ERROR: Error installing travis:
ERROR: Failed to build gem native extension.
/usr/bin/ruby1.9.1 extconf.rb
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- mkmf (LoadError)
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from extconf.rb:4:in `<main>'
Gem files will remain installed in /var/lib/gems/1.9.1/gems/ffi-1.9.18 for inspection.
Results logged to /var/lib/gems/1.9.1/gems/ffi-1.9.18/ext/ffi_c/gem_make.out
Nope.
Attempt 7
Tried this:
$ git clone https://github.com/travis-ci/travis-build $ cd travis-build $ docker-compose up
Should have been:
$ git clone https://github.com/travis-ci/travis-build $ cd travis-build $ docker-compose build $ docker run -ti travisbuild_web /bin/bash
aaaaand no way to use travis-build:
root@294c95445521:/usr/src/app# travis convert unknown command convert
More tries:
root@294c95445521:/usr/src/app# bundle exec rake
Attempt 8
Start over - build the travis-build Docker image:
$ git clone https://github.com/travis-ci/travis-build $ cd travis-build $ docker-compose up
Now follow the instructions on the README:
root@294c95445521:/usr/src/app# ln -s ${PWD} ~/.travis/travis-build
root@294c95445521:/usr/src/app# gem install bundler
root@294c95445521:/usr/src/app# bundle install --gemfile ~/.travis/travis-build/Gemfile
Bingo, now we have a convert command:
root@294c95445521:/usr/src/app# travis wno_proxy is unsupported hichUsage: travis COMMAND ... Available commands: t accounts displays accounts and their subscription status branches displays the most recent build for each branch cache lists or deletes repository caches cancel cancels a job or build compile compiles a build script from .travis.yml console interactive shell disable disables a project enable enables a project encrypt encrypts values for the .travis.yml encrypt-file encrypts a file and adds decryption steps to .travis.yml endpoint displays or changes the API endpoint env show or modify build environment variables help helps you out when in dire need of information history displays a projects build history init generates a .travis.yml and enables the project lint display warnings for a .travis.yml login authenticates against the API and stores the token logout deletes the stored API token logs streams test logs monitor live monitor for what's going on open opens a build or job in the browser pubkey prints out a repository's public key raw makes an (authenticated) API call and prints out the result report generates a report useful for filing issues repos lists repositories the user has certain permissions on requests lists recent requests restart restarts a build or job settings access repository settings setup sets up an addon or deploy target show displays a build or job sshkey checks, updates or deletes an SSH key status checks status of the latest build sync triggers a new sync with GitHub token outputs the secret API token version outputs the client version whatsup lists most recent builds whoami outputs the current user run `/usr/local/bundle/bin/travis help COMMAND` for more infos
Would it kill them to improve their documentation?