Automation – saving 15 mins/day at a time

Posted: October 25th, 2011 | Filed under: Programming, Python
Author: Tom
1 Comment →

So my wife laughs at me quite a bit, but I’m a little obsessed with efficiency and automation. I hate wasting my time on non-engaging tasks. Which is, as it happens, most of life. So I try to find ways to automate things. The less I have to invest my time in tasks with a clearly defined process the more time I can spend working on things I actually care about. At work I put this to the test constantly. Working as a software engineer there is just a pile of repetitive tasks that are ripe for automation. Fortunately a number of tools have entered the battlefield against wasting our time on processes that can be automated.

I love the growth and ever increasing adoption of continuous integration. At my new job I was thrilled to see how much they had already begun to move in that direction. It’s not perfect, that’s for sure, but we’ve got a solid start and we always keep our eyes out for ways to improve that process. That said, there’s still a lot of room for grabbing the low hanging fruit and Python/IronPython (two tools I’ve mentioned a number of times) come in to save the day for me.

One place I decided to tackle recently was our revision control. We use Hg which is a Python based tool itself, but they expressly discourage people from importing their libraries and using them, simply because they don’t guarantee they won’t change and therefore break everything. I had a short internal debate about if I thought it was worth it and eventually I opted against it, simply because it was just too easy to work around that issue (for my needs at least).

The problem came up that we have a pile of repositories that we all work in frequently. Keeping these repositories up to date was kind of a pain. That said the more familiar I get with the command line for Hg the easier, however I opted to write something with python to handle the task at hand. I came across this script which really accomplished what I needed, but I just wasn’t in love with the implementation. I wanted something a bit more flexible, I had in mind that I could use some of this for some tasks with our automated build too.

I started off by writing a simple “repo” class.

import os
PENDING_STATUS = ['A','M']
ERRORS = ['access denied', 'abort']

PULL = 'hg pull'
UPDATE = 'hg update'
STATUS = 'hg status'
TIP = 'hg tip'
NUDGE = 'hg push --rev .'

class repo:
    def __init__(self, path=''):
        self.path = path
        self.validdir = False

        if path:
            self.__validate_dir__()

    def __validate_dir__(self):
        if not os.path.exists(os.path.join(self.path,'.hg')):
            print self.path, 'is not an hg directory.'
            self.validdir = False
            return
        
	self.validdir = True

    def __haspending__(self, show=False):
        if not self.__prep__(): return False
        out = os.popen(STATUS)
        changes = [ln for ln in out if ln[0] in PENDING_STATUS]
        if show:
            for chg in changes: print chg

        return len(changes) > 0

    def __indir__(self):
        return os.getcwd() == self.path

    def __gotorepo__(self):
        if not self.validdir: return False
        if not self.__indir__():
            os.chdir(self.path)

        return True

    def __prep__(self):
        if not self.validdir: return False
        return self.__gotorepo__()

    def __checkerror__(self, output):
        for l in output:
            err = [e for e in ERRORS if e in l]
            if err:
                print err
                return True

        return False

    def setpath(self, path):
        self.path = path
        self.__validate_dir__()
    
    def issafe(self):
        if not self.__prep__(): return False
        if self.__haspending__(True):
            print 'Will not update. Pending changes found.'
            return False
        
        print 'No pending changes found... safe to update.'
        return True

    def getchg(self):
        if not self.validdir: return False
        changeset = [ln for ln in os.popen(TIP) if 'changeset:' in ln]
        return ln.split(':')[2]

    def anypending(self):
        return __safe_to_update__()

    def pull(self):
        if not self.__prep__(): return
	out = [ln for ln in os.popen(PULL)]
	if self.__checkerror__(out): return
	
	for ln in out:            
            if 'no changes found' in ln:
                print ln
                return False

        print 'pulled latest'
        return True
			
    def update(self, show=True):
	if not self.__prep__(): return
	out = [ln for ln in os.popen(UPDATE)]
	if self.__checkerror__(out): return
        for ln in out:
            if show: print ln
            
    def sync(self, update_if_safe = True):
        if not self.__prep__(): return
        
        print 'Getting latest for ' + os.getcwd()
        #only try to update if there were changes...
        if self.pull():
            if update_if_safe and self.issafe():
                print 'Updating...'
                self.update(False)
                
        print 'done'

    def nudge(self, show=True):
        if not self.__prep__(): return
        out = [ln for ln in os.popen(NUDGE)]
	if self.__checkerror__(out): return
        for ln in out:
            if show: print ln

It’s not the prettiest, in fact I’ll change it soon to have a ‘verbose’ property or something so I can ditch the “show=True” params from a number of those functions. The idea for the “nudge” function came from an hgtip.com post where they address what is honestly an annoying part of mercurial to me.

Once I had this repo class then I just needed a simple script to wrap around it. This is a bit closer to the script I mentioned earlier, but it’s got some options for command line params.

import os, hgutil, config, sys

def main(argv=None):
    if argv == None:
        argv = sys.argv

    arg_path = None
    # start at 1, 0 is the command
    for i in range(1, len(argv)):
        if argv[i] == '-p':
            arg_path = argv[i+1]
        if argv[i] == '-h':
            show_help()
            return True
        
    path = get_path(arg_path)

    if has_connection():
        for f in os.listdir(path):
            p = os.path.join(path,f)
            if os.path.isdir(p):
                repo = hgutil.repo(p)
                repo.sync()
    else:
        print 'No connection to hg found. Please ensure pageant is running.'
        return False

    return True

def has_connection():
    return len([ln for ln in os.popen('tasklist') if 'pageant' in ln]) > 0

def get_path(path=None):
    if path != None and os.path.exists(path):
        print 'starting from path:', path
        return path

    ##default
    print 'No valid path found... using default.'
    return config.get_home()

def show_help():
    print '--- sync.py ---'
    print 'Attempts to sync all hg repositories under a path.'
    print 'Returns errors if pageant is not running.'
    print 'Args:'
    print '-p : path to work from'
    print 'Example: sync.py -p c:\dev'
    print 'Done.'

if __name__ == "__main__":
    sys.exit(main())

Since we use pageant for our connections to mercurial it’s nice to have this be able to check and let me know in case I’ve forgotten to enable it.

Now that I had this full script, I create a windows scheduled task to run it behind the scenes for me every 10 mins. Now if I’m not actually working with a specific repository it just gets updated for me, and if I am, I pull the changes, but don’t update until I’m ready. This needs more work before it’s terribly useful on my automated build server, but the ground work is there and I’ve been really happy with it so far. It might not seem like it but this script saves me a solid 10-15 mins a day manually doing pulls and updates on my repos and a lot of help where I run into issues from running an older version of some code.

Now, it is possible that I’ll be working on something I don’t want automatically updated. Then I’ll just change it over to default to not update. The automated pull is still useful.

Tags: automation, continuous integration, hg, mercurial, Python, source control

---