Amebis/wxWidgets
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revamped automated build system to use a Python script for the master

Browse Source 
control script, make parallel builds the only way to do it
(facilitated by the taskrunner module) split out the guts of build-all
into separate scripts, etc.


git-svn-id: https://svn.wxwidgets.org/svn/wx/wxWidgets/trunk@30377 c3d73ce0-8a6f-49c7-b76d-6d57e0e08775
This commit is contained in:
Robin Dunn
2004-11-08 23:55:25 +00:00
parent 17dc7ddd7c
commit e4bb599887
14 changed files with 2144 additions and 646 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
wxPython/distrib/all/README.txt
View File

@@ -1,8 +1,8 @@
The collection of scripts in this directory are an attempt to fully The collection of scripts in this directory are an attempt to fully
automate the build of the wxPython source and binary packages on all automate the build of the wxPython source and binary packages on all
build platforms. It does this through creative use of shared folders build platforms. It does this through creative use of ssh and scp
on network drives, and ssh commands to the remote machines. So this commands to the remote build machines, so this will likely only work
will likly only work in my somewhat unique environment. in my somewhat unique environment.
The goal here is to be able to start a build on one machine and have The goal here is to be able to start a build on one machine and have
it take care of all the steps, including moving the source tarball to it take care of all the steps, including moving the source tarball to
@@ -12,7 +12,7 @@ may be copied to a public server for others to play with.
Types of builds: Types of builds:
dry-run dryrun
Nothing extra is done with the build, this is just for Nothing extra is done with the build, this is just for
my own testing. my own testing.
@@ -21,14 +21,17 @@ Types of builds:
datestamp, and if the build is successful the results datestamp, and if the build is successful the results
are copied to a daily build folder on starship. are copied to a daily build folder on starship.
release-cantidate release
The results are uploaded to the previews foler on The results are uploaded to the previews foler on
starship if the build is successful. starship if the build is successful.
The master script in this folder is "make-all" which will setup and The master script in this folder is build-all (written in Python)
control the whole process. The other scripts are what are run on each which will setup and control the whole process. The other scripts
build machine, most of which will also call out to other scripts that (using bash) are launched from build-all either to do specific tasks
already exist, etc. locally, or to run on each individual build machine to manage the
build process there, usually by calling out to other scripts that
already exist. The build-all script uses the taskrunner.py and
subprocess Python modules.

659
wxPython/distrib/all/build-all
View File

@@ -1,501 +1,256 @@
#!/bin/bash #!/usr/bin/python
# --------------------------------------------------------------------------- #----------------------------------------------------------------------
# Master build script for building all the installers and such on all the # Name: build-all.py
# build machines in my lab, and then distributing the results as needed. # Purpose: Master build script for building all the installers and
# --------------------------------------------------------------------------- # such on all the build machines in my lab, and then
# distributing the results as needed.
#
# This will replace the build-all bash script and is
# needed because the needs of the build have outgrown
# what I can do with bash.
#
# Author: Robin Dunn
#
# Created: 05-Nov-2004
# RCS-ID: $Id$
# Copyright: (c) 2004 by Total Control Software
# Licence: wxWindows license
#----------------------------------------------------------------------
set -o errexit import sys
#set -o xtrace import os
import time
from taskrunner import Job, Task, TaskRunner
# --------------------------------------------------------------------------- #----------------------------------------------------------------------
# Some control variables... # Configuration items
class Config:
def write(self, filename="config", outfile=None):
if outfile is None:
f = file(filename, "w")
else:
f = outfile
for k, v in self.__dict__.items():
f.write('%s="%s"\n' % (k, v))
config = Config()
# the local spot that we put everything when done, before possibly copying # the local spot that we put everything when done, before possibly copying
# to remote hosts # to remote hosts
STAGING_DIR=./BUILD config.STAGING_DIR = "./BUILD"
# host name of the machine to use for windows builds # host name of the machine to use for windows builds
WIN_HOST=beast config.WIN_HOST = "beast"
# Where is the build dir from the remote machine's perspective? # Where is the build dir from the remote machine's perspective?
WIN_BUILD=/c/BUILD config.WIN_BUILD = "/c/BUILD"
# Just like the above # Just like the above
OSX_HOST_panther=bigmac config.OSX_HOST_panther = "bigmac"
OSX_HOST_jaguar=whopper config.OSX_HOST_jaguar = "whopper"
OSX_BUILD=/BUILD config.OSX_BUILD = "/BUILD"
# Alsmost the same... See below for hosts and other info # Alsmost the same... See below for hosts and other info
LINUX_BUILD=/tmp/BUILD config.LINUX_BUILD = "/tmp/BUILD"
# Upload server locations # Upload server locations
UPLOAD_HOST=starship.python.net config.UPLOAD_HOST = "starship.python.net"
UPLOAD_DAILY_ROOT=/home/crew/robind/public_html/wxPython/daily config.UPLOAD_DAILY_ROOT = "/home/crew/robind/public_html/wxPython/daily"
UPLOAD_PREVIEW_ROOT=/home/crew/robind/public_html/wxPython/preview config.UPLOAD_PREVIEW_ROOT = "/home/crew/robind/public_html/wxPython/preview"
# defaults for build options
config.KIND = "dryrun"
config.PYVER = "2.3"
config.skipsource = "no"
config.onlysource = "no"
config.skipdocs = "no"
config.skipwin = "no"
config.skiposx = "no"
config.skiplinux = "no"
config.skipclean = "no"
config.skipupload = "no"
config.skipnewdocs = "no"
#----------------------------------------------------------------------
# Define all the build tasks
# --------------------------------------------------------------------------- class Job(Job):
# functions LOGBASE = "./tmp"
function usage { CFGFILE = "./tmp/config"
echo ""
echo "Usage: $0 [command flags...]"
echo ""
echo "build types:"
echo " dryrun Do the build, but don't copy anywhere (default)"
echo " daily Do a daily build, copy to starship"
echo " release Do a normal release (cantidate) build, copy to starship"
echo ""
echo "optional command flags:"
echo " 2.2 Build for Python 2.2 (default=off)"
echo " 2.3 Build for Python 2.3 (default=on)"
echo " all Build for all supported Python versions"
echo ""
echo " skipsource Don't build the source archives, use the ones"
echo " already in the staging dir."
echo " onlysource Exit after building the source and docs archives"
echo " skipdocs Don't rebuild the docs"
echo " skipwin Don't do the remote Windows build"
echo " skiposx Don't do the remote OSX build"
echo " skiplinux Don't do the remote Linux build"
echo " skipclean Don't do the cleanup step on the remote builds"
echo " skipupload Don't upload the builds to starship"
echo " parallel parallelize the builds where possible"
echo ""
}
# Things that need to be done before any of the builds
initialTask = Task([ Job("", ["distrib/all/build-setup", CFGFILE]),
Job("", ["distrib/all/build-docs", CFGFILE]),
Job("", ["distrib/all/build-sources", CFGFILE]),
])
function PrefixLines { # Build tasks. Anything that can be done in parallel (depends greatly
label=$1 # on the nature of the build machine configurations...) is a separate
tee tmp/$label.log | awk "{ print \"** $label: \" \$0; fflush(); }" # task.
} windowsTask = Task( Job("beast", ["distrib/all/build-windows", CFGFILE]) )
# --------------------------------------------------------------------------- jaguarTask = Task( Job(config.OSX_HOST_jaguar,
["distrib/all/build-osx", CFGFILE, config.OSX_HOST_jaguar, "jaguar"]) )
# Make sure we are running in the right directory. TODO: make this pantherTask = Task( Job(config.OSX_HOST_panther,
# test more robust. Currenly we just test for the presence of ["distrib/all/build-osx", CFGFILE, config.OSX_HOST_panther, "panther"]) )
# 'wxPython' and 'wx' subdirs.
if [ ! -d wxPython -o ! -d wx ]; then rpmTask = Task([ Job("co-rh9", ["distrib/all/build-rpm", CFGFILE, "none", "co-rh9", "rh9", config.PYVER]),
echo "Please run this script from the root wxPython directory." Job("co-fc2", ["distrib/all/build-rpm", CFGFILE, "beast", "co-fc2", "fc2", "2.3"]),
exit 1 Job("co-mdk92", ["distrib/all/build-rpm", CFGFILE, "none", "co-mdk92", "mdk92", "2.3"]),
fi Job("co-mdk101", ["distrib/all/build-rpm", CFGFILE, "none", "co-mdk101","mdk101","2.3"]),
])
buildTasks = [ windowsTask,
jaguarTask,
pantherTask,
rpmTask,
]
# Finalization. This is for things that must wait until all the
# builds are done, such as copying the isntallers someplace, sending
# emails, etc.
finalizationTask = Task( Job("", ["distrib/all/build-finalize", CFGFILE]) )
#----------------------------------------------------------------------
# Set defaults and check the command line options def usage():
KIND=dryrun print ""
PYVER=2.3 print "Usage: build-all [command flags...]"
skipsource=no print ""
onlysource=no print "build types:"
skipdocs=no print " dryrun Do the build, but don't copy anywhere (default)"
skipwin=no print " daily Do a daily build, copy to starship"
skiposx=no print " release Do a normal release (cantidate) build, copy to starship"
skiplinux=no print ""
skipclean=no print "optional command flags:"
skipupload=no print " 2.2 Build for Python 2.2 (default=off)"
parallel=no print " 2.3 Build for Python 2.3 (default=on)"
print " all Build for all supported Python versions"
for flag in $*; do print ""
case $flag in print " skipsource Don't build the source archives, use the ones"
dryrun) KIND=dryrun ;; print " already in the staging dir."
daily) KIND=daily ;; print " onlysource Exit after building the source and docs archives"
release) KIND=release ;; print " skipdocs Don't rebuild the docs"
print " skipwin Don't do the remote Windows build"
2.2) PYVER=2.2 ;; print " skiposx Don't do the remote OSX build"
2.3) PYVER=2.3 ;; print " skiplinux Don't do the remote Linux build"
all) PYVER="2.2 2.3" ;; print " skipclean Don't do the cleanup step on the remote builds"
print " skipupload Don't upload the builds to starship"
skipsource) skipsource=yes ;; print ""
onlysource) onlysource=yes ;;
skipdocs) skipdocs=yes ;;
skipwin) skipwin=yes ;;
skiposx) skiposx=yes ;;
skiplinux) skiplinux=yes ;;
skipclean) skipclean=yes ;;
skipupload) skipupload=yes ;;
parallel) parallel=yes ;;
noparallel) parallel=no ;;
help) usage; exit 1 ;;
*) echo "Unknown flag \"$flag\""
usage
exit 1
esac
done
# ensure the staging area exists
if [ ! -d $STAGING_DIR ]; then
mkdir -p $STAGING_DIR
fi
# Figure out the wxPython version number, possibly adjusted for being a daily build
if [ $KIND = daily ]; then
DAILY=`date +%Y%m%d` # should it include the hour too? 2-digit year?
echo $DAILY > DAILY_BUILD
fi
VERSION=`python -c "import setup;print setup.VERSION"`
#echo VERSION=$VERSION
#exit 0
echo "Getting started at " `date`
# ---------------------------------------------------------------------------
# Make the sources and other basic stuff.
if [ $skipsource != yes -o $onlysource = yes ]; then
# clean out the local dist dir
rm -f dist/*
if [ $skipdocs != yes ]; then
# Regenerate the reST docs
echo "Regenerating the reST docs..."
cd docs
for x in *.txt; do
docutils-html $x `basename $x .txt`.html
done
cd -
# build the doc and demo tarballs
distrib/makedemo
distrib/makedocs
# build the new docs too
docs/bin/everything
fi
# make the source tarball and srpm
distrib/makerpm 2.3 srpm
# Copy everything to the staging dir
echo "Moving stuff to $STAGING_DIR..."
mv dist/* $STAGING_DIR
if [ $skipdocs != yes ]; then
for doc in CHANGES BUILD INSTALL MigrationGuide default; do
cp docs/$doc.* $STAGING_DIR
done
fi
# cleanup
echo "Cleaning up..."
rm -f dist/*
fi
if [ $KIND = daily ]; then
rm DAILY_BUILD
fi
if [ $onlysource = yes ]; then
exit 0
fi
# ---------------------------------------------------------------------------
# Windows build
function DoWindowsBuild {
set -o errexit
# test if the target machine is online
if ping -q -c1 -w1 $WIN_HOST > /dev/null; then
echo "-----------------------------------------------------------------"
echo " The $WIN_HOST machine is online, Windows build continuing..."
echo "-----------------------------------------------------------------"
else
echo "-----------------------------------------------------------------"
echo "The $WIN_HOST machine is offline, skipping the Windows build."
echo "-----------------------------------------------------------------"
return 0
fi
echo "-=-=- Starting Windows build..."
echo "Copying source file and build script..."
scp $STAGING_DIR/wxPython-src-$VERSION.tar.gz \
distrib/all/build-windows \
$WIN_HOST:$WIN_BUILD
echo "Running build script on $WIN_HOST..."
wxdir=$WIN_BUILD/wxPython-src-$VERSION
cmd=./build-windows
ssh $WIN_HOST "cd $WIN_BUILD && $cmd $wxdir $WIN_BUILD $skipclean $VERSION $PYVER && rm $cmd"
echo "Fetching the results..." #----------------------------------------------------------------------
scp "$WIN_HOST:$WIN_BUILD/wxPython*-win32*" $STAGING_DIR
ssh $WIN_HOST "rm $WIN_BUILD/wxPython*-win32*"
}
if [ $skipwin != yes ]; then def main(args):
if [ $parallel = no ]; then # Make sure we are running in the right directory. TODO: make
DoWindowsBuild # this test more robust. Currenly we just test for the presence
else # of 'wxPython' and 'wx' subdirs.
DoWindowsBuild 2>&1 | PrefixLines $WIN_HOST & if not os.path.isdir("wxPython") or not os.path.isdir("wx"):
winPID=$! print "Please run this script from the root wxPython directory."
fi sys.exit(1)
fi
# Check command line flags
for flag in args:
if flag in ["dryrun", "daily", "release"]:
config.KIND = flag
elif flag in ["2.2", "2.3"]:
config.PYVER = flag
elif flag == "all":
config.PYVER = "2.2 2.3"
elif flag == "skipsource":
config.skipsource = "yes"
elif flag == "onlysource":
config.onlysource = "yes"
elif flag == "skipdocs":
config.skipdocs = "yes"
elif flag == "skipnewdocs":
config.skipnewdocs = "yes"
elif flag == "skipwin":
config.skipwin = "yes"
elif flag == "skiposx":
config.skiposx = "yes"
elif flag == "skiplinux":
config.skiplinux = "yes"
elif flag == "skipclean":
config.skipclean = "yes"
elif flag == "skipupload":
config.skipupload = "yes"
else:
print 'Unknown flag: "%s"' % flag
usage()
sys.exit(2)
# --------------------------------------------------------------------------- # ensure the staging area exists
# OSX build if not os.path.exists(config.STAGING_DIR):
os.makedirs(config.STAGING_DIR)
function DoOSXBuild { # Figure out the wxPython version number, possibly adjusted for being a daily build
local host=$1 if config.KIND == "daily":
local flavor=$2 t = time.localtime()
config.DAILY = time.strftime("%Y%m%d") # should it include the hour too? 2-digit year?
file("DAILY_BUILD", "w").write(config.DAILY)
sys.path.append('.')
import setup
config.VERSION = setup.VERSION
set -o errexit # write the config file where the build scripts can find it
config.write(CFGFILE)
# test if the target machine is online print "Build getting started at: ", time.ctime()
if ping -q -c1 -w1 $host > /dev/null; then
echo "-----------------------------------------------------------------"
echo " The $host machine is online, OSX-$flavor build continuing..."
echo "-----------------------------------------------------------------"
else
echo "-----------------------------------------------------------------"
echo "The $host machine is offline, skipping the OSX-$flavor build."
echo "-----------------------------------------------------------------"
return 0
fi
echo "-=-=- Starting OSX-$flavor build on $host..."
echo "Copying source files and build script..."
ssh root@$host "mkdir -p $OSX_BUILD && rm -rf $OSX_BUILD/* || true"
#ssh root@$host "mkdir -p $OSX_BUILD || true"
scp $STAGING_DIR/wxPython-src-$VERSION.tar.gz \
$STAGING_DIR/wxPython-docs-$VERSION.tar.gz \
$STAGING_DIR/wxPython-demo-$VERSION.tar.gz \
distrib/all/build-osx \
root@$host:$OSX_BUILD
echo "Running build script on $host..."
wxdir=$OSX_BUILD/wxPython-src-$VERSION
cmd=./build-osx
ssh root@$host "cd $OSX_BUILD && $cmd $wxdir $OSX_BUILD $skipclean $VERSION $flavor $PYVER && rm $cmd"
echo "Fetching the results..."
scp "root@$host:$OSX_BUILD/wxPython*-osx*" $STAGING_DIR
ssh root@$host "rm $OSX_BUILD/wxPython*-osx*"
}
if [ $skiposx != yes ]; then # Run the first task, which will create the docs and sources tarballs
if [ $parallel = no ]; then tr = TaskRunner(initialTask)
DoOSXBuild $OSX_HOST_jaguar jaguar rc = tr.run()
DoOSXBuild $OSX_HOST_panther panther
else # cleanup the DAILY_BUILD file
DoOSXBuild $OSX_HOST_jaguar jaguar 2>&1 | PrefixLines $OSX_HOST_jaguar & if config.KIND == "daily":
DoOSXBuild $OSX_HOST_panther panther 2>&1 | PrefixLines $OSX_HOST_panther & os.unlink("DAILY_BUILD")
fi
fi # Quit now?
if rc != 0 or config.onlysource == "yes":
sys.exit(rc)
# Run the main build tasks
# --------------------------------------------------------------------------- tr = TaskRunner(buildTasks)
# Linux build rc = tr.run()
if rc != 0:
# The remote Linux builds are different than those above. The source sys.exit(rc)
# RPMs were already built in the source step, and so building the
# binary RPMs is a very simple followup step. But then add to that
# the fact that we need to build on more than one distro...
# when all the builds are done, run the finalization task
tr = TaskRunner(finalizationTask)
rc = tr.run()
if rc != 0:
sys.exit(rc)
function DoLinuxBuild {
local host=$1
local reltag=$2
shift;shift
local pyver=$@
set -o errexit
# test if the target machine is online
if ping -q -c1 -w1 $host > /dev/null; then
echo "-----------------------------------------------------------------"
echo " The $host machine is online, build continuing..."
echo "-----------------------------------------------------------------"
else
echo "-----------------------------------------------------------------"
echo "The $host machine is offline, skipping the binary RPM build."
echo "-----------------------------------------------------------------"
return 0
fi
echo "Copying source files and build script..."
ssh root@$host "mkdir -p $LINUX_BUILD && rm -rf $LINUX_BUILD/*"
scp $STAGING_DIR/wxPython-src* $STAGING_DIR/wxPython.spec\
distrib/all/build-linux \
root@$host:$LINUX_BUILD
echo "Running build script on $host..."
cmd=./build-linux
ssh root@$host "cd $LINUX_BUILD && ./build-linux $reltag $skipclean $VERSION $pyver"
echo "Fetching the results..." print "Build finished at: ", time.ctime()
scp "root@$host:$LINUX_BUILD/wxPython*.i[0-9]86.rpm" $STAGING_DIR sys.exit(0)
ssh root@$host "rm $LINUX_BUILD/wxPython*.i[0-9]86.rpm"
}
if [ $skiplinux != yes ]; then if __name__ == "__main__":
main(sys.argv[1:])
if [ $parallel = no ]; then
DoLinuxBuild co-rh9 rh9 $PYVER
DoLinuxBuild co-fc2 fc2 2.3
DoLinuxBuild co-mdk92 mdk92 2.3
DoLinuxBuild co-mdk101 mdk101 2.3
else
# Since the linux builds are currently done in coLinux
# 'machines' running on the WIN_HOST let's wait for the
# windows build to be done before launching them
#if [ -n $winPID ]; then
# wait $winPID
#fi
DoLinuxBuild co-rh9 rh9 $PYVER 2>&1 | PrefixLines co-rh9 &
DoLinuxBuild co-fc2 fc2 2.3 2>&1 | PrefixLines co-fc2 &
wait $! # wait for the previous two to complete before starting the next two
DoLinuxBuild co-mdk92 mdk92 2.3 2>&1 | PrefixLines co-mdk92 &
DoLinuxBuild co-mdk101 mdk101 2.3 2>&1 | PrefixLines co-mdk101 &
fi
fi
# ---------------------------------------------------------------------------
if [ $parallel = yes ]; then
# TODO: Figure out how to test if all the builds were successful
echo "***********************************"
echo " Waiting for builds to complete... "
echo "***********************************"
wait
fi
# ---------------------------------------------------------------------------
# Final disposition of build results...
chmod a+r $STAGING_DIR/*
if [ $KIND = dryrun ]; then
# we're done
echo "Finished at " `date`
exit 0
fi
if [ $KIND = daily ]; then
echo "Copying to the local file server..."
destdir=/stuff/temp/$VERSION
mkdir -p $destdir
cp $STAGING_DIR/* $destdir
if [ $skipupload != yes ]; then
destdir=$UPLOAD_DAILY_ROOT/$DAILY
echo "Copying to the starship at $destdir..."
ssh $UPLOAD_HOST "mkdir -p $destdir"
scp $STAGING_DIR/* $UPLOAD_HOST:/$destdir
ssh $UPLOAD_HOST "cd $destdir && ls -al"
# TODO: something to remove old builds from starship, keeping
# only N days worth
# Send email to wxPython-dev
DATE=`date`
TO=wxPython-dev@lists.wxwidgets.org
cat <<EOF | /usr/sbin/sendmail $TO
From: R'bot <rbot@wxpython.org>
To: $TO
Subject: $DAILY test build uploaded
Date: $DATE
Hi,
A new test build of wxPython has been uploaded to starship.
Version: $VERSION
URL: http://starship.python.net/crew/robind/wxPython/daily/$DAILY
Changes: http://starship.python.net/crew/robind/wxPython/daily/$DAILY/CHANGES.html
Have fun!
R'bot
EOF
fi
echo "Cleaning up staging dir..."
rm $STAGING_DIR/*
rmdir $STAGING_DIR
echo "Finished at " `date`
exit 0
fi
if [ $KIND = release ]; then
echo "Copying to the local file server..."
destdir=/stuff/Development/wxPython/dist/$VERSION
mkdir -p $destdir
cp $STAGING_DIR/* $destdir
if [ $skipupload != yes ]; then
echo "Copying to the starship..."
destdir=$UPLOAD_PREVIEW_ROOT/$VERSION
ssh $UPLOAD_HOST "mkdir -p $destdir"
scp $STAGING_DIR/* $UPLOAD_HOST:/$destdir
# Send email to wxPython-dev
DATE=`date`
TO=wxPython-dev@lists.wxwidgets.org
cat <<EOF | /usr/sbin/sendmail $TO
From: R'bot <rbot@wxpython.org>
To: $TO
Subject: $VERSION release candidate build uploaded
Date: $DATE
Hi,
A new RC build of wxPython has been uploaded to starship.
Version: $VERSION
URL: http://starship.python.net/crew/robind/wxPython/preview/$VERSION
Changes: http://starship.python.net/crew/robind/wxPython/preview/$VERSION/CHANGES.html
Have fun!
R'bot
EOF
fi
echo "Cleaning up staging dir..."
rm $STAGING_DIR/*
rmdir $STAGING_DIR
echo "Finished at " `date`
exit 0
fi
# ---------------------------------------------------------------------------

36
wxPython/distrib/all/build-docs Executable file
View File

@@ -0,0 +1,36 @@
#!/bin/bash
#----------------------------------------------------------------------
set -o errexit
# read the config variables from the file given on the command line
. $1
if [ $skipdocs != yes ]; then
# Regenerate the reST docs
echo "Regenerating the reST docs..."
cd docs
for x in *.txt; do
docutils-html $x `basename $x .txt`.html
done
cd -
for doc in CHANGES BUILD INSTALL MigrationGuide default; do
cp docs/$doc.* $STAGING_DIR
done
# build the doc and demo tarballs
distrib/makedemo
distrib/makedocs
mv dist/wxPython-docs-$VERSION.tar.gz $STAGING_DIR
mv dist/wxPython-demo-$VERSION.tar.gz $STAGING_DIR
# build the new docs too
if [ $skipnewdocs != yes ]; then
docs/bin/everything
mv dist/wxPython-newdocs-$VERSION.tar.gz $STAGING_DIR
fi
fi
#----------------------------------------------------------------------

115
wxPython/distrib/all/build-finalize Executable file
View File

@@ -0,0 +1,115 @@
#!/bin/bash
#----------------------------------------------------------------------
set -o errexit
# read the config variables from the file given on the command line
. $1
chmod a+r $STAGING_DIR/*
if [ $KIND = dryrun ]; then
# we're done leave the files in the staging dir and quit
echo "Not uploading dryrun."
exit 0
fi
if [ $KIND = daily ]; then
echo "Copying to the local file server..."
destdir=/stuff/temp/$VERSION
mkdir -p $destdir
cp $STAGING_DIR/* $destdir
if [ $skipupload != yes ]; then
destdir=$UPLOAD_DAILY_ROOT/$DAILY
echo "Copying to the starship at $destdir..."
ssh $UPLOAD_HOST "mkdir -p $destdir"
scp $STAGING_DIR/* $UPLOAD_HOST:/$destdir
ssh $UPLOAD_HOST "cd $destdir && ls -al"
# TODO: something to remove old builds from starship, keeping
# only N days worth
# Send email to wxPython-dev
DATE=`date`
TO=wxPython-dev@lists.wxwidgets.org
cat <<EOF | /usr/sbin/sendmail $TO
From: R'bot <rbot@wxpython.org>
To: $TO
Subject: $DAILY test build uploaded
Date: $DATE
Hi,
A new test build of wxPython has been uploaded to starship.
Version: $VERSION
URL: http://starship.python.net/crew/robind/wxPython/daily/$DAILY
Changes: http://starship.python.net/crew/robind/wxPython/daily/$DAILY/CHANGES.html
Have fun!
R'bot
EOF
fi
echo "Cleaning up staging dir..."
rm $STAGING_DIR/*
rmdir $STAGING_DIR
exit 0
fi
if [ $KIND = release ]; then
echo "Copying to the local file server..."
destdir=/stuff/Development/wxPython/dist/$VERSION
mkdir -p $destdir
cp $STAGING_DIR/* $destdir
if [ $skipupload != yes ]; then
echo "Copying to the starship..."
destdir=$UPLOAD_PREVIEW_ROOT/$VERSION
ssh $UPLOAD_HOST "mkdir -p $destdir"
scp $STAGING_DIR/* $UPLOAD_HOST:/$destdir
# Send email to wxPython-dev
DATE=`date`
TO=wxPython-dev@lists.wxwidgets.org
cat <<EOF | /usr/sbin/sendmail $TO
From: R'bot <rbot@wxpython.org>
To: $TO
Subject: $VERSION release candidate build uploaded
Date: $DATE
Hi,
A new RC build of wxPython has been uploaded to starship.
Version: $VERSION
URL: http://starship.python.net/crew/robind/wxPython/preview/$VERSION
Changes: http://starship.python.net/crew/robind/wxPython/preview/$VERSION/CHANGES.html
Have fun!
R'bot
EOF
fi
echo "Cleaning up staging dir..."
rm $STAGING_DIR/*
rmdir $STAGING_DIR
exit 0
fi

97
wxPython/distrib/all/build-osx
View File

@@ -1,71 +1,40 @@
#!/bin/bash #!/bin/bash
# --------------------------------------------------------------------------- #----------------------------------------------------------------------
# Build wxWidgets and wxPython on a OSX box. This is normally
# called from build-all but it should be able to be used standalone too...
#
# The command line must have the following parameters:
#
# 1. the path to the base of the wx source tree
# 2. the path of where to put the resulting installers
# 3. skipclean flag (yes|no)
# 4. the VERSION
# 5. the KIND (panther or jaguar)
# *. the remaining args are the versions of Python to build for
#
# ---------------------------------------------------------------------------
set -o errexit set -o errexit
#set -o xtrace
echo "-=-=-=- Hello from $HOSTNAME -=-=-=-" # read the config variables from the file given on the command line
. $1
if [ $# -lt 6 ]; then
echo "Usage: $0 WXDIR DESTDIR SKIPCLEAN VERSION KIND PYVER..." host=$2
exit 1 flavor=$3
if [ $skiposx != yes ]; then
# test if the target machine is online
if ping -q -c1 -w1 $host > /dev/null; then
echo " The $host machine is online, OSX-$flavor build continuing..."
else
echo "The $host machine is **OFFLINE**, skipping the OSX-$flavor build."
exit 0
fi
echo "Copying source files and build script..."
ssh root@$host "mkdir -p $OSX_BUILD && rm -rf $OSX_BUILD/* || true"
scp $STAGING_DIR/wxPython-src-$VERSION.tar.gz \
$STAGING_DIR/wxPython-docs-$VERSION.tar.gz \
$STAGING_DIR/wxPython-demo-$VERSION.tar.gz \
distrib/all/do-build-osx \
root@$host:$OSX_BUILD
echo "Running build script on $host..."
wxdir=$OSX_BUILD/wxPython-src-$VERSION
cmd=./do-build-osx
ssh root@$host "cd $OSX_BUILD && $cmd $wxdir $OSX_BUILD $skipclean $VERSION $flavor $PYVER && rm $cmd"
echo "Fetching the results..."
scp "root@$host:$OSX_BUILD/wxPython*-osx*" $STAGING_DIR
ssh root@$host "rm $OSX_BUILD/wxPython*-osx*"
fi fi
WXDIR=$1
DESTDIR=$2
SKIPCLEAN=$3
VERSION=$4
KIND=$5
shift;shift;shift;shift;shift
PYVER=$@
#export PATH=/sw/bin:/usr/local/bin:$PATH
export PATH=/sw/bin:/sw/sbin:/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin:.:/usr/X11R6/bin
echo "PATH =" $PATH
echo "which gcc = " `which gcc`
#exit 0
# untar the source
echo "Unarchiving wxPython-src-$VERSION.tar.gz"
cd $DESTDIR
tar xzf wxPython-src-$VERSION.tar.gz
rm wxPython-src-$VERSION.tar.gz
echo "Invoking wxPythonOSX build script..."
cd $WXDIR/wxPython
export TARBALLDIR=$DESTDIR
mkdir -p dist
if [ $KIND = panther ]; then
distrib/mac/wxPythonOSX/build $KIND inplace unicode
fi
distrib/mac/wxPythonOSX/build $KIND inplace
echo "Copying installers to $DESTDIR..."
cp dist/*.dmg $DESTDIR
cd $DESTDIR
if [ $SKIPCLEAN != yes ]; then
echo "Cleaning up..."
rm -r $WXDIR || true
rm wxPython-docs-$VERSION.tar.gz
rm wxPython-demo-$VERSION.tar.gz
fi
echo "-=-=-=- Goodbye! -=-=-=-"

94
wxPython/distrib/all/build-rpm Executable file
View File

@@ -0,0 +1,94 @@
#!/bin/bash
#----------------------------------------------------------------------
set -o errexit
# read the config variables from the file given on the command line
. $1
coHost=$2
host=$3
reltag=$4
shift;shift;shift;shift
pyver=$@
function TestOnline {
local host=$1
local message=$2
if ping -q -c1 -w1 $host > /dev/null; then
return 0
else
return 1
fi
}
if [ $skiplinux != yes ]; then
startedCoHost=no
hostAvailable=no
# test if the target machine is online
if TestOnline $host; then
hostAvailable=yes
else
# Attempt to start the host via it's coLinux host, if there is one
if [ $coHost != none ]; then
if TestOnline $coHost; then
echo "Attempting to start $host via coLinux on $coHost..."
ssh $coHost "/c/coLinux/VMs/$host.bat -d > /dev/null 2>&1 &"
# Give it time to boot and be ready for conenctions,
# and then test with ssh, limiting retries.
for x in `seq 12`; do
sleep 5
echo "checking..."
if ssh root@$host "true" >/dev/null 2>&1; then
# success! the host is ready so we can break out of the loop
break;
fi
done
# test if the host is ready
if TestOnline $host; then
echo "coLinux start of $host on $coHost successful."
startedCoHost=yes
hostAvailable=yes
fi
else
echo "The $coHost machine is offline, unable to start coLinux for $host"
fi
fi
fi
if [ $hostAvailable = yes ]; then
echo "The $host machine is online, build continuing..."
else
echo "The $host machine is **OFFLINE**, skipping the binary RPM build."
exit 0
fi
echo "Copying source files and build script..."
ssh root@$host "mkdir -p $LINUX_BUILD && rm -rf $LINUX_BUILD/*"
scp $STAGING_DIR/wxPython-src* $STAGING_DIR/wxPython.spec\
distrib/all/do-build-rpm \
root@$host:$LINUX_BUILD
echo "Running build script on $host..."
cmd=./do-build-rpm
ssh root@$host "cd $LINUX_BUILD && $cmd $reltag $skipclean $VERSION $pyver"
echo "Fetching the results..."
scp "root@$host:$LINUX_BUILD/wxPython*.i[0-9]86.rpm" $STAGING_DIR
ssh root@$host "rm $LINUX_BUILD/wxPython*.i[0-9]86.rpm"
if [ $startedCoHost = yes ]; then
echo "Halting $host on $coHost..."
ssh root@$host "/sbin/halt"
sleep 10
fi
fi

12
wxPython/distrib/all/build-setup Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/bash
#----------------------------------------------------------------------
set -o errexit
# read the config variables from the file given on the command line
. $1
# clean out the local dist dir
rm -f dist/*

18
wxPython/distrib/all/build-sources Executable file
View File

@@ -0,0 +1,18 @@
#!/bin/bash
#----------------------------------------------------------------------
set -o errexit
# read the config variables from the file given on the command line
. $1
if [ $skipsource != yes -o $onlysource = yes ]; then
# make the source tarball and srpm
distrib/makerpm 2.3 srpm
# Copy everything to the staging dir
echo "Moving stuff to $STAGING_DIR..."
mv dist/* $STAGING_DIR
fi

144
wxPython/distrib/all/build-windows
View File

@@ -1,128 +1,32 @@
#!/bin/bash #!/bin/bash
# --------------------------------------------------------------------------- #----------------------------------------------------------------------
# Build wxWidgets and wxPython on a Windows box. This is normally called
# from build-all but it should be able to be used standalone too...
#
# The command line must have the following parameters:
#
# 1. the path to the base of the wx source tree
# 2. the path of where to put the resulting installers
# 3. skipclean flag (yes|no)
# 4. the VERSION
# 5. the remaining args are the versions of Python to build for
#
# ---------------------------------------------------------------------------
set -o errexit set -o errexit
#set -o xtrace
echo "-=-=-=- Hello from $HOSTNAME -=-=-=-" # read the config variables from the file given on the command line
. $1
if [ $# -lt 5 ]; then
echo "Usage: $0 WXDIR DESTDIR SKIPCLEAN VERSION PYVER..."
exit 1
fi
WXDIR=$1
DESTDIR=$2
SKIPCLEAN=$3
VERSION=$4
shift;shift;shift;shift
PYVER=$@
# WXDIR is the cygwin path, WXWIN is the DOS path if [ $skipwin != yes ]; then
WXWIN_OLD=$WXWIN # test if the target machine is online
WXWIN=`cygpath -w $WXDIR` if ping -q -c1 -w1 $WIN_HOST > /dev/null; then
export WXWIN echo " The $WIN_HOST machine is online, Windows build continuing..."
else
echo "The $WIN_HOST machine is **OFFLINE**, skipping the Windows build."
return 0
fi
echo "Copying source file and build script..."
scp $STAGING_DIR/wxPython-src-$VERSION.tar.gz \
distrib/all/do-build-windows \
$WIN_HOST:$WIN_BUILD
echo "Running build script on $WIN_HOST..."
wxdir=$WIN_BUILD/wxPython-src-$VERSION
cmd=./do-build-windows
ssh $WIN_HOST "cd $WIN_BUILD && $cmd $wxdir $WIN_BUILD $skipclean $VERSION $PYVER && rm $cmd"
# # Fix the PATH. (Why is this needed??) echo "Fetching the results..."
# PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/robind/bin:.:$WXDIR/lib/vc_dll:$PATH scp "$WIN_HOST:$WIN_BUILD/wxPython*-win32*" $STAGING_DIR
# export PATH ssh $WIN_HOST "rm $WIN_BUILD/wxPython*-win32*"
# echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" fi
# echo $PATH
# echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="
# exit 0
# untar the source
echo "Unarchiving wxPython-src-$VERSION.tar.gz"
cd $DESTDIR
tar xzf wxPython-src-$VERSION.tar.gz
rm wxPython-src-$VERSION.tar.gz
# Fix line endings
echo "Converting wxPython line endings to CRLF..."
cd $WXDIR
names=`find wxPython -name "*.py" -o -name "*.txt" -o -name "*.htm*" -o -name "*.css" -o -name "*.xml" `
unix2dos -D $names
# change to the right spot in the source tree and copy our build scripts
echo "Setting up for the build..."
cd $WXDIR/build/msw
cp $WXDIR/wxPython/distrib/msw/.m* .
# replace some settings in setup0.h and write to setup.h
cat > .my.sedexpr <<EOF
s/wxDIALOG_UNIT_COMPATIBILITY *1/wxDIALOG_UNIT_COMPATIBILITY 0/g
s/wxUSE_DEBUG_CONTEXT *0/wxUSE_DEBUG_CONTEXT 1/g
s/wxUSE_MEMORY_TRACING *0/wxUSE_MEMORY_TRACING 1/g
s/wxUSE_DIALUP_MANAGER *1/wxUSE_DIALUP_MANAGER 0/g
s/wxUSE_GLCANVAS *0/wxUSE_GLCANVAS 1/g
s/wxUSE_POSTSCRIPT *0/wxUSE_POSTSCRIPT 1/g
s/wxUSE_AFM_FOR_POSTSCRIPT *1/wxUSE_AFM_FOR_POSTSCRIPT 0/g
s/wxUSE_DISPLAY *0/wxUSE_DISPLAY 1/g
EOF
cat $WXDIR/include/wx/msw/setup0.h | sed -f .my.sedexpr > $WXDIR/include/wx/msw/setup.h
rm .my.sedexpr
echo "Building the wx DLLs..."
.make hybrid
.make hybrid-uni
#echo "Building the wx tools..."
#.make_tools
# cheat and just copy the .CHM files from the regular project dir
# TODO: Copy over the wxPython-docs fle and run hhc on the contents of that.
mkdir -p $WXDIR/docs/htmlhelp
cp `cygpath $WXWIN_OLD/docs/htmlhelp`/*.chm $WXDIR/docs/htmlhelp
echo "Building wxPython and installers..."
cd $WXDIR/wxPython
mkdir -p dist
for ver in $PYVER; do
echo $ver
b $ver d USE_SWIG=0
b $ver h USE_SWIG=0 EP_ADD_OPTS=1
b $ver r USE_SWIG=0
b $ver d UNICODE=1 USE_SWIG=0
b $ver h UNICODE=1 USE_SWIG=0 EP_ADD_OPTS=1
b $ver r UNICODE=1 USE_SWIG=0
done
echo "Building the developer package..."
WXWIN=`cygpath -w $WXDIR`
export WXWIN
4nt /c distrib/makedev.bat $VERSION
echo "Copying installers to $DESTDIR..."
mv dist/wxPython* $DESTDIR
cd $DESTDIR
if [ $SKIPCLEAN != yes ]; then
echo "Cleaning up..."
rm -r $WXDIR || true
fi
echo "-=-=-=- Goodbye! -=-=-=-"

71
wxPython/distrib/all/do-build-osx Executable file
View File

@@ -0,0 +1,71 @@
#!/bin/bash
# ---------------------------------------------------------------------------
# Build wxWidgets and wxPython on a OSX box. This is normally
# called from build-all but it should be able to be used standalone too...
#
# The command line must have the following parameters:
#
# 1. the path to the base of the wx source tree
# 2. the path of where to put the resulting installers
# 3. skipclean flag (yes|no)
# 4. the VERSION
# 5. the KIND (panther or jaguar)
# *. the remaining args are the versions of Python to build for
#
# ---------------------------------------------------------------------------
set -o errexit
#set -o xtrace
echo "-=-=-=- Hello from $HOSTNAME -=-=-=-"
if [ $# -lt 6 ]; then
echo "Usage: $0 WXDIR DESTDIR SKIPCLEAN VERSION KIND PYVER..."
exit 1
fi
WXDIR=$1
DESTDIR=$2
SKIPCLEAN=$3
VERSION=$4
KIND=$5
shift;shift;shift;shift;shift
PYVER=$@
#export PATH=/sw/bin:/usr/local/bin:$PATH
export PATH=/sw/bin:/sw/sbin:/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin:.:/usr/X11R6/bin
echo "PATH =" $PATH
echo "which gcc = " `which gcc`
#exit 0
# untar the source
echo "Unarchiving wxPython-src-$VERSION.tar.gz"
cd $DESTDIR
tar xzf wxPython-src-$VERSION.tar.gz
rm wxPython-src-$VERSION.tar.gz
echo "Invoking wxPythonOSX build script..."
cd $WXDIR/wxPython
export TARBALLDIR=$DESTDIR
mkdir -p dist
if [ $KIND = panther ]; then
distrib/mac/wxPythonOSX/build $KIND inplace unicode
fi
distrib/mac/wxPythonOSX/build $KIND inplace
echo "Copying installers to $DESTDIR..."
cp dist/*.dmg $DESTDIR
cd $DESTDIR
if [ $SKIPCLEAN != yes ]; then
echo "Cleaning up..."
rm -r $WXDIR || true
rm wxPython-docs-$VERSION.tar.gz
rm wxPython-demo-$VERSION.tar.gz
fi
echo "-=-=-=- Goodbye! -=-=-=-"

6
wxPython/distrib/all/build-linux → wxPython/distrib/all/do-build-rpm
View File

@@ -42,7 +42,11 @@ function DoRPMBuild {
# $1 : python version # $1 : python version
# $2 : port # $2 : port
# $3 : unicode # $3 : unicode
echo "-=-=-=-=-=-=-=-=-=-=-"
echo $1 $2 $3
echo "-=-=-=-=-=-=-=-=-=-=-"
$RPMBUILD --define "_topdir $PWD/$rpmtop" \ $RPMBUILD --define "_topdir $PWD/$rpmtop" \
--define "_tmppath $PWD/$rpmtop/tmp" \ --define "_tmppath $PWD/$rpmtop/tmp" \
--define "release ${RELEASE}_py$1" \ --define "release ${RELEASE}_py$1" \

128
wxPython/distrib/all/do-build-windows Executable file
View File

@@ -0,0 +1,128 @@
#!/bin/bash
# ---------------------------------------------------------------------------
# Build wxWidgets and wxPython on a Windows box. This is normally called
# from build-all but it should be able to be used standalone too...
#
# The command line must have the following parameters:
#
# 1. the path to the base of the wx source tree
# 2. the path of where to put the resulting installers
# 3. skipclean flag (yes|no)
# 4. the VERSION
# 5. the remaining args are the versions of Python to build for
#
# ---------------------------------------------------------------------------
set -o errexit
#set -o xtrace
echo "-=-=-=- Hello from $HOSTNAME -=-=-=-"
if [ $# -lt 5 ]; then
echo "Usage: $0 WXDIR DESTDIR SKIPCLEAN VERSION PYVER..."
exit 1
fi
WXDIR=$1
DESTDIR=$2
SKIPCLEAN=$3
VERSION=$4
shift;shift;shift;shift
PYVER=$@
# WXDIR is the cygwin path, WXWIN is the DOS path
WXWIN_OLD=$WXWIN
WXWIN=`cygpath -w $WXDIR`
export WXWIN
# # Fix the PATH. (Why is this needed??)
# PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/robind/bin:.:$WXDIR/lib/vc_dll:$PATH
# export PATH
# echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="
# echo $PATH
# echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="
# exit 0
# untar the source
echo "Unarchiving wxPython-src-$VERSION.tar.gz"
cd $DESTDIR
tar xzf wxPython-src-$VERSION.tar.gz
rm wxPython-src-$VERSION.tar.gz
# Fix line endings
echo "Converting wxPython line endings to CRLF..."
cd $WXDIR
names=`find wxPython -name "*.py" -o -name "*.txt" -o -name "*.htm*" -o -name "*.css" -o -name "*.xml" `
unix2dos -D $names
# change to the right spot in the source tree and copy our build scripts
echo "Setting up for the build..."
cd $WXDIR/build/msw
cp $WXDIR/wxPython/distrib/msw/.m* .
# replace some settings in setup0.h and write to setup.h
cat > .my.sedexpr <<EOF
s/wxDIALOG_UNIT_COMPATIBILITY *1/wxDIALOG_UNIT_COMPATIBILITY 0/g
s/wxUSE_DEBUG_CONTEXT *0/wxUSE_DEBUG_CONTEXT 1/g
s/wxUSE_MEMORY_TRACING *0/wxUSE_MEMORY_TRACING 1/g
s/wxUSE_DIALUP_MANAGER *1/wxUSE_DIALUP_MANAGER 0/g
s/wxUSE_GLCANVAS *0/wxUSE_GLCANVAS 1/g
s/wxUSE_POSTSCRIPT *0/wxUSE_POSTSCRIPT 1/g
s/wxUSE_AFM_FOR_POSTSCRIPT *1/wxUSE_AFM_FOR_POSTSCRIPT 0/g
s/wxUSE_DISPLAY *0/wxUSE_DISPLAY 1/g
EOF
cat $WXDIR/include/wx/msw/setup0.h | sed -f .my.sedexpr > $WXDIR/include/wx/msw/setup.h
rm .my.sedexpr
echo "Building the wx DLLs..."
.make hybrid
.make hybrid-uni
#echo "Building the wx tools..."
#.make_tools
# cheat and just copy the .CHM files from the regular project dir
# TODO: Copy over the wxPython-docs fle and run hhc on the contents of that.
mkdir -p $WXDIR/docs/htmlhelp
cp `cygpath $WXWIN_OLD/docs/htmlhelp`/*.chm $WXDIR/docs/htmlhelp
echo "Building wxPython and installers..."
cd $WXDIR/wxPython
mkdir -p dist
for ver in $PYVER; do
echo $ver
b $ver d USE_SWIG=0
b $ver h USE_SWIG=0 EP_ADD_OPTS=1
b $ver r USE_SWIG=0
b $ver d UNICODE=1 USE_SWIG=0
b $ver h UNICODE=1 USE_SWIG=0 EP_ADD_OPTS=1
b $ver r UNICODE=1 USE_SWIG=0
done
echo "Building the developer package..."
WXWIN=`cygpath -w $WXDIR`
export WXWIN
4nt /c distrib/makedev.bat $VERSION
echo "Copying installers to $DESTDIR..."
mv dist/wxPython* $DESTDIR
cd $DESTDIR
if [ $SKIPCLEAN != yes ]; then
echo "Cleaning up..."
rm -r $WXDIR || true
fi
echo "-=-=-=- Goodbye! -=-=-=-"

1158
wxPython/distrib/all/subprocess.py Normal file
View File

File diff suppressed because it is too large Load Diff

231
wxPython/distrib/all/taskrunner.py Normal file
View File

@@ -0,0 +1,231 @@
#----------------------------------------------------------------------
# Name: taskrunner.py
# Purpose: Classes that can manage running of external processes,
# either consecutively, simultaneously, or both, and can
# log the output of those jobs
#
# Author: Robin Dunn
#
# Created: 05-Nov-2004
# RCS-ID: $Id$
# Copyright: (c) 2004 by Total Control Software
# Licence: wxWindows license
#----------------------------------------------------------------------
import sys
import os
import signal
import select
import fcntl
from subprocess import Popen, PIPE, STDOUT
__all__ = ["Job", "Task", "TaskRunner"]
#----------------------------------------------------------------------
class Job(object):
"""
Each Job is a monitor wrapped around an externally executing
process. It handles starting the process, polling if it is still
running, reading and logging it's output, and killing it if
needed.
"""
LOGBASE="."
def __init__(self, label, args):
self.label = label
self.args = args
self.proc = None
if self.label:
self.log = file("%s/%s.log" % (self.LOGBASE, label), "w", 0)
def start(self):
self.proc = Popen(self.args, # the command and args to execute
stdout=PIPE, stderr=STDOUT,
bufsize=0, # line-buffered
)
# put the file in non-blocking mode
#flags = fcntl.fcntl (self.proc.stdout, fcntl.F_GETFL, 0)
#flags = flags | os.O_NONBLOCK
#fcntl.fcntl (self.proc.stdout, fcntl.F_SETFL, flags)
def stop(self):
if self.proc is not None and self.proc.returncode is None:
os.kill(self.proc.pid, signal.SIGTERM)
self.logLines()
def fileno(self):
if self.proc is not None:
return self.proc.stdout.fileno()
else:
return -1
def logLines(self):
if self.proc is not None:
while self.linesAvailable():
line = self.proc.stdout.readline()
if not line: break
if self.label:
self.log.write(line)
line = "** %s: %s" % (self.label, line)
sys.stdout.write(line)
def linesAvailable(self):
if self.proc is None:
return False
ind, outd, err = select.select([self], [], [], 0)
if ind:
return True
else:
return False
def finished(self):
if self.proc is None:# or self.linesAvailable():
return False
return self.proc.poll() is not None
def wait(self):
if self.proc is None: return None
return self.proc.wait()
def poll(self):
if self.proc is None: return None
return self.proc.poll()
def returnCode(self):
if self.proc is None: return None
return self.proc.returncode
#----------------------------------------------------------------------
class Task(object):
"""
This class helps manage the running of a Task, which is a simply a
sequence of one or more Jobs, where subesquent jobs are not
started until prior ones are completed.
"""
def __init__(self, jobs=[]):
if type(jobs) != list:
jobs = [jobs]
self.jobs = jobs[:]
self.active = 0
def append(self, job):
self.jobs.append(job)
def activeJob(self):
if self.active > len(self.jobs)-1:
return None
else:
return self.jobs[self.active]
def next(self):
self.active += 1
if self.active < len(self.jobs):
self.jobs[self.active].start()
#----------------------------------------------------------------------
class TaskRunner(object):
"""
Manages the running of multiple tasks.
"""
def __init__(self, tasks=[]):
if type(tasks) != list:
tasks = [tasks]
self.tasks = tasks[:]
def append(self, task):
self.tasks.append(task)
def run(self):
# start all the active jobs
for task in self.tasks:
task.activeJob().start()
try:
# loop, getting output from the jobs, etc.
while True:
# get all active Jobs
jobs = [t.activeJob() for t in self.tasks if t.activeJob()]
if not jobs:
break
# wait for a job to have output ready, then log it
input, output, err = select.select(jobs, [], [], 1)
for job in input:
job.logLines()
# check for finished jobs
for task in self.tasks:
job = task.activeJob()
if job and job.finished():
if job.returnCode() != 0:
rc = job.returnCode()
print "JOB RETURNED FAILURE CODE! (%d)" % rc
self.stopAllJobs()
return rc
else:
task.next()
except KeyboardInterrupt:
print "STOPPING JOBS..."
self.stopAllJobs()
except:
print "Unknown exception..."
self.stopAllJobs()
raise
return 0
def stopAllJobs(self):
for task in self.tasks:
job = task.activeJob()
if job:
job.stop()
#----------------------------------------------------------------------
if __name__ == "__main__":
j1 = Job("label1", ["./tmp/job-1.py", "TEST-1"])
j2 = Job("label2", ["./tmp/job-2.sh", "TEST-2"])
t1 = Task()
t1.append(j1)
t1.append(j2)
j3 = Job("task2a", ["./tmp/job-1.py", "TASK-2a"])
j4 = Job("task2b", ["./tmp/job-2.sh", "TASK-2b"])
t2 = Task()
t2.append(j4)
t2.append(j3)
t3 = Task([Job("error", ["./tmp/job-3.sh", "TASK-3"])])
tr = TaskRunner()
tr.append(t1)
tr.append(t2)
tr.append(t3)
for task in tr.tasks:
for job in task.jobs:
print job.label
print tr.run()