Automation – saving 15 mins/day at a time

So my wife laughs at me quite a bit, but I’m a little obsessed with efficiency and automation. I hate wasting my time on non-engaging tasks. Which is, as it happens, most of life. So I try to find ways to automate things. The less I have to invest my time in tasks with a clearly defined process the more time I can spend working on things I actually care about. At work I put this to the test constantly. Working as a software engineer there is just a pile of repetitive tasks that are ripe for automation. Fortunately a number of tools have entered the battlefield against wasting our time on processes that can be automated.

I love the growth and ever increasing adoption of continuous integration. At my new job I was thrilled to see how much they had already begun to move in that direction. It’s not perfect, that’s for sure, but we’ve got a solid start and we always keep our eyes out for ways to improve that process. That said, there’s still a lot of room for grabbing the low hanging fruit and Python/IronPython (two tools I’ve mentioned a number of times) come in to save the day for me.

One place I decided to tackle recently was our revision control. We use Hg which is a Python based tool itself, but they expressly discourage people from importing their libraries and using them, simply because they don’t guarantee they won’t change and therefore break everything. I had a short internal debate about if I thought it was worth it and eventually I opted against it, simply because it was just too easy to work around that issue (for my needs at least).

The problem came up that we have a pile of repositories that we all work in frequently. Keeping these repositories up to date was kind of a pain. That said the more familiar I get with the command line for Hg the easier, however I opted to write something with python to handle the task at hand. I came across this script which really accomplished what I needed, but I just wasn’t in love with the implementation. I wanted something a bit more flexible, I had in mind that I could use some of this for some tasks with our automated build too.

I started off by writing a simple “repo” class.
Read the rest of this entry »

Python and socket programming: a quick chat server/client

For some reason tonight I thought, I don’t know enough about lower level networking or threading. What better way to make it all happen than with Python? So I started looking over the Python docs for the socket module and the recommended BSD socket lib documentation. At the bottom of the page the Python docs have a basic server-client setup to test out the basic functionality.

I worked over that and found it easy enough but wanted more. Then I wanted more after that… Once I got a bit of a handle on what was actually happening with the socket.accept() method and send() and recv(). I had a much better idea of what needed to be done and I set off to write my own little server module. I quickly found that to be just not quite enough, so I wrote up a quick Server class and an accompanying ChatClient.

I was pretty happy how easily I could just offload to a thread. In a few places I got a little ambitious about how I implemented things, the truth is, I don’t suspect it would be that hard for me to expand what I’m doing and I could have server commands and chat alias’ etc. The thing is that I’m amazed and thrilled that I was able to go from effectively no knowledge of socket programming at all to a little knowledge and a mini socket communication server in just a couple of hours. Python is pretty awesome. I almost wish I had a purpose for this idea, but I just don’t really need it.

Read the rest of this entry »

Copying files in IronPython

I’ve been talking a lot about IronPython and Python. So this is just continuing the theme there. A little more about a script I wrote today. Because of the way I’m using things, I needed to be able to copy a big collection of assemblies from one my project folders. I’m sharing this just to show simple it was to write. I added it to my new utils module.

import os
import shutil
from System.IO import Directory

## wrapper to clear screen
clear = lambda: os.system('cls')
## wrapper to know the current directory
curdir = lambda: Directory.GetCurrentDirectory()

## update - turns out I don't need the .NET API for the current directory
## this does the job
os.getcwd()


FROM_PATH = '..\\....\\bin'
DEST_PATH = curdir()

def copylatest(dest = DEST_PATH, from_path=FROM_PATH):
    for s, d, files in os.walk(from_path):
        for f in files:
            shutil.copy(from_path + '\\' + f, dest)

I added that wrapper for the CurrentDirectory because the built-in os function wasn’t working for me in the IronPython console, and the clear screen wrapper was nice too. Essentially 3 lines of code to find all the files and copy them out and yet, I have a feeling there is an even more concise way to do this.

Extending my IronPython ToolSet: Truth Values

If you haven’t seen what I’ve already done with my config module, then you should probably look over my last couple of posts about IronPython before you read on since I’m kind of building on that existing information/knowledge. My IronPython console config module, How I override the ConfigurationManager for the IronPython console, and to a lesser extent Talking to Google’s Custom Search API.

After reading over my last post about some extensions to the small config module I’d built a friend and coder extraordinaire (Corey Tabaka) raised a couple of points about how truth values in Python are evaluated. Which was nice because it cleaned up a few lines of code and helped me with my next addition to my config module. One of the key things he pointed out was this check I was using a lot in changing and loading my environments:

if _envs == 0:
    ## do stuff

This isn’t wrong (although slightly wordy), but it made it obvious I wasn’t aware of really how Python evaluates for truth. So he pointed me to the Python Docs on truth values. Basically, a collection that’s empty will evaluate to false, as well as a None (null for your .NET peeps) value. That essentially turned all those calls like above into this:

if not _envs: # do stuff

Read the rest of this entry »

Extending my IronPython ToolSet

I had a recent post about some issues I was having in the ConfigurationManager when I was using the IronPython Console. So I wrote funky hack to get around everything there and today I extended that config module I have, just a little farther.

In our config files to make local testing as easy as possible we actually have a big collection of connections strings. Essentially four connection strings for each environment that we use. We store them like this:

<connectionStrings>
    <add name="connection1" connectionString="..." providerName="..." />
    <add name="connection1.QA" connectionString="..." providerName="..." />
    <add name="connection1.Training" connectionString="..." providerName="..." />
    <add name="connection1.Production" connectionString="..." providerName="..." />
    <add name="connection2" connectionString="..." providerName="..." />
    <add name="connection2.QA" connectionString="..." providerName="..." />
    ....
</connectionStrings>

Sometimes we may need to use another environment for debugging or testing and this is purely a convenience measure so we can just add and remove the extensions to the connection string to change our environment without the need to remember all the details.

Knowing that all of this information is there, I thought it would be really handy if I could simply change environments from my IronPython console. So… now I can. Accomplishing this change was surprisingly easy. It took me a little bit to familiarize myself with Python dictionaries and find the places where the IronPython implementation didn’t quite work the same as normal Python, but once I did, piece of cake. And the code is readable too!

Read the rest of this entry »