As it would happen, a post I never thought would really matter has actually gotten a fair amount of traffic. A couple of years back I was heavily into IronPython and I found a way to use config files in the IronPython shell. It turns out that for one, I made some copy paste errors and my posted code didn’t compile – that’s a bit embarrassing – but, despite my mistakes, a couple of people have found some use for it. Moshe Basanchig fixed my compilation errors, cleaned things up a bit, and took the next step that in retrospect I really should have done: He posted the code, a compiled version and some examples.

I was pretty excited to see this, not least because I was happy to see more people using IronPython which I really love. But the story doesn’t end here, it turns out that someone took the step to implement it in Python. Now its a little easier. One less assembly to load in python.

I’m always happy to see that some slightly obscure work I do helps someone else. Curious that it seems to be the posts I didn’t imagine anyone besides me would find interesting are the ones seem to attract the most traffic.

Resume.Github.com is an app that takes a github account name and creates a resume from the activity. Some news outlets even reported on the concept as the future. Brandon Weiss of Anti-Pattern.com, posted an article in which he declared that my github account should be seen as my resume. I’m not writing to argue against his article. I think he has some valid points and importantly, I think the concept has a place in the world. But I don’t think it applies as universally as these articles imply.

The real problem I want to address is that I was nervous about releasing and open source project or making contributions because I didn’t want to be judged by it. Perhaps unlike some, the open source world is never likely to get my best work. Not because of some resistance to the concept, but rather because I simply invest a large majority of my effort at my job. I work 10-12 hours days typically and when I get home I’m just not likely to invest heavily in a project that isn’t crucial to my job.  When I’m coding at home, I’m experimenting with ideas and trying things out. At best I’m hacking together a tool to make my life easier. I’m not developing elegant solutions.

I realized that this hesitation meant that despite the fact that I had some utilities that could actually improve the life and efficiency of other developers around the world, they simply sat rotting on my PC not being fully exploited. I was essentially forcing someone to redo work I had already done, because I was afraid that someone would use my hack of a tool as ammunition in a job interview at some point. Not only that, but I was losing the potential insight and improvements of other engineers that could increase my own efficiency because I wasn’t publishing this code.

I honestly feel that the idea that my open source work which (I freely admit) is often a little hacked together should be the judgement of the quality of my work on the whole goes against the very heart of the open source movement. I’m publishing my code, not because it’s so amazing, but rather because it solves a problem and I want others to be able to take advantage of the work I have done so far. I want to encourage people to improve upon the idea, perhaps make it a more elegant solution. Or maybe due to circumstances it is useful exactly as is, even without being made more elegant. I don’t see why that should be a problem.

So now I resist the idea that my open source code is my resume. My open source code is nothing more than me making available work I’ve done to solve a problem. If you don’t like my solution, don’t use it. With my new found liberating mindset on open source, I’ve released not one but two projects in recent months. Feel free to use them or contribute, maybe they will solve a problem you experienced as well. The code isn’t great and certainly not even close to my best (or most interesting work) but they solve a problem, which is all that concerned me.

• Visual Studio Reference Swapper – a utility for programmatically updating Visual Studio References
• NAnt Assistant - a simple GUI geared around simplicity and keyboard shortcuts for running nant builds

I’ll go ahead and spoil the fun now. It took me approximately 3-4 hours to read about and implement the ETW APIs in my application. However, I lost another ~16 hours trying to get windows configured to support my new event provider. Once I had gotten through that there was another round of trying to turn that data into useful information that I could act on, but that wasn’t so bad. Most of the time was lost in sorting out the configuration then finding a way to make the process easily repeatable for the rest of my team.

The basic process you need is not so bad:

1. Create your Instrumentation Manifest
2. Compile the Manifest
3. Add compiled data to your application and implement your events – rebuild the application
4. Register the Manifest

I’ll go over each step and to gotcha’s that I encountered. Perhaps I can save someone else some time.

Create an Instrumentation Manifest

First things first, Microsoft it seems has adjusted the internal Windows event apis/methodology for pretty much each major release of windows since XP. It has been evolving, and that’s generally a good thing. They (at least for now) seem to have settled on something called the Instrumentation Manifest. This is an xml file that defines the details of the provider you are developing. It includes all of the event information you’re accustomed to seeing in the EventLog, and the Windows SDK even comes with a handy little tool (ECManGen.exe) that builds and edits your manifest file for you.

I could go into the schema but the Microsoft documentation surrounding this aspect is pretty solid. I’ll just cover one very important thing that I didn’t fully understand when I was getting started. When setting up your provider in ECManGen, there is a set of properties that were not clear to me, the Decoding File Locations. The decoding file locations are an absolute path to your assembly that has been compiled with the Manifest details. Once you’ve written a manifest, you have to compile it (we’ll cover this more later). The output of the compilation needs to be added to your application. When you go to register the your manifest Windows will need to know where to find the Event Provider. That’s the purpose of these properties. In my case, they are both ( exactly the same. There must be a situation where they would be different, but I don’t know what the situation would be.

Compile the Instrumenation Manifest

There isn’t much I’m going to cover here. It is generally pretty simple. Microsoft has provided a tool called mc.exe that is also included as part of the Windows SDK. The output is a series of files generated based on your manifest, a header file, a message .bin, a TEMP.bin and a .rc (Resource) file. You need to add the Resource and header files to your application so it is compiled into the application. The Header file just makes your life very easy when writing the code.

I didn’t run into any difficulty getting this to work correctly. Microsoft has good docs on the Message Compiler and it worked reliably.

Writing the code

I already mentioned that you’ll need to add the Resource and Header files from your Message Compiler output to your application. Once this is done, it is just a matter of following the MSDN docs on providing events. In fact, this code example was very easy to start from.

A couple of simple notes, make sure you initialize correctly with EventRegister and that you correctly close out your provider’s process with EventUnregister. I had a bug that meant the Unregister wasn’t being correctly called and it led to some pretty odd behavior.

Here’s a quick example of some of my code – because of my implementation, I actually support enabling and disabling the eventing as you’ll noticed is referenced. Also, my application can run through a pretty substantial amount of functionality without ever entering code that does any EventLogging so you’ll notice that my code doesn’t assume the provider has been registered at the time of writing an event.

unsigned char eventLogging_RegisterHandle()
{
	ULONG status;
	// this one check will stop all events from being written
	if(eventLogging_IsEnabled != TRUE)
		return FALSE;

	status = EventRegister(&EXAMPLE_INSTRUMENTATION_PROVIDER, NULL, NULL, &runHandle);

	if(status != ERROR_SUCCESS)
	{
		runHandle = NULL;
	}
	else 
	{
		// start a session?
		EventActivityIdControl(EVENT_ACTIVITY_CTRL_CREATE_SET_ID, &currentSession);
	}

	return status == ERROR_SUCCESS;
}
// simplify/remove duplication
void eventLogging_WriteEvent(PCEVENT_DESCRIPTOR evt, unsigned char eventDataCount, PEVENT_DATA_DESCRIPTOR evtData) 
{
	ULONG status;

	if(runHandle == NULL && eventLogging_RegisterHandle() == FALSE)
		return;

	status = EventWriteEx(runHandle, evt, 0, 0, NULL, &currentSession, eventDataCount, evtData);
}

void eventLogging_Exception(char* message)
{
	EventDataDescCreate(&desc, message, strlen(message));
	eventLogging_writeEvent(&ExampleProviderException, 1, &desc);
}

Register the manifest

This is where I ran into a few gotchas. The core of the problems, I already covered previously, but I’ll discuss how I addressed a couple of these problems for my situation.

1. Absolute Path to the compiled application
2. Rebuild your application – I listed it explicitly in the steps above, but just to reiterate. Windows will barf on you when you try to register if the event resources haven’t been compiled into the application
3. Security – you’ll need to grant Read & Execute permissions on the directory containing your application

The Absolute Path for the application proved to be a little tricky for us. While most of my team uses keep their code repositories in a standard location, some people had other preferences. So, I actually wrote a small app that I keep in the repository to handle the manifest registration. It knows the relative path to the application and manifest, so it opens the manifest file and fills in the absolute path based on the known relative path and then registers it that way. This allows me to keep the manifest in the repository generically and not force the few people who keep their code non-standard locations from having to edit the absolute path just to register the manifest. It also means that if someone accidentally commits the absolute path for their pc it doesn’t cause any difficulties for everyone else.

When it’s all written out like this, it doesn’t seem so hard. The truth is, it isn’t. But if, like me, you run into a problem at one of these steps finding the right information on how to fix it can be difficult. I’m all around pretty surprised and happy with the Windows ETW Apis. Everything was surprisingly easy to use. While I’m not a fan of xml as a solution to all problems, in this case it seems to fit fairly well and offer the flexibility you would want from such a situation. I wish Microsoft had some slightly better docs on the full process, but I also wish I had better documentation at work of my code too. Alas, I think we all know it is much harder than it sounds.

I love the growth and ever increasing adoption of continuous integration. At my new job I was thrilled to see how much they had already begun to move in that direction. It’s not perfect, that’s for sure, but we’ve got a solid start and we always keep our eyes out for ways to improve that process. That said, there’s still a lot of room for grabbing the low hanging fruit and Python/IronPython (two tools I’ve mentioned a number of times) come in to save the day for me.

One place I decided to tackle recently was our revision control. We use Hg which is a Python based tool itself, but they expressly discourage people from importing their libraries and using them, simply because they don’t guarantee they won’t change and therefore break everything. I had a short internal debate about if I thought it was worth it and eventually I opted against it, simply because it was just too easy to work around that issue (for my needs at least).

The problem came up that we have a pile of repositories that we all work in frequently. Keeping these repositories up to date was kind of a pain. That said the more familiar I get with the command line for Hg the easier, however I opted to write something with python to handle the task at hand. I came across this script which really accomplished what I needed, but I just wasn’t in love with the implementation. I wanted something a bit more flexible, I had in mind that I could use some of this for some tasks with our automated build too.

I started off by writing a simple “repo” class.

import os
PENDING_STATUS = ['A','M']
ERRORS = ['access denied', 'abort']

PULL = 'hg pull'
UPDATE = 'hg update'
STATUS = 'hg status'
TIP = 'hg tip'
NUDGE = 'hg push --rev .'

class repo:
    def __init__(self, path=''):
        self.path = path
        self.validdir = False

        if path:
            self.__validate_dir__()

    def __validate_dir__(self):
        if not os.path.exists(os.path.join(self.path,'.hg')):
            print self.path, 'is not an hg directory.'
            self.validdir = False
            return
        
	self.validdir = True

    def __haspending__(self, show=False):
        if not self.__prep__(): return False
        out = os.popen(STATUS)
        changes = [ln for ln in out if ln[0] in PENDING_STATUS]
        if show:
            for chg in changes: print chg

        return len(changes) > 0

    def __indir__(self):
        return os.getcwd() == self.path

    def __gotorepo__(self):
        if not self.validdir: return False
        if not self.__indir__():
            os.chdir(self.path)

        return True

    def __prep__(self):
        if not self.validdir: return False
        return self.__gotorepo__()

    def __checkerror__(self, output):
        for l in output:
            err = [e for e in ERRORS if e in l]
            if err:
                print err
                return True

        return False

    def setpath(self, path):
        self.path = path
        self.__validate_dir__()
    
    def issafe(self):
        if not self.__prep__(): return False
        if self.__haspending__(True):
            print 'Will not update. Pending changes found.'
            return False
        
        print 'No pending changes found... safe to update.'
        return True

    def getchg(self):
        if not self.validdir: return False
        changeset = [ln for ln in os.popen(TIP) if 'changeset:' in ln]
        return ln.split(':')[2]

    def anypending(self):
        return __safe_to_update__()

    def pull(self):
        if not self.__prep__(): return
	out = [ln for ln in os.popen(PULL)]
	if self.__checkerror__(out): return
	
	for ln in out:            
            if 'no changes found' in ln:
                print ln
                return False

        print 'pulled latest'
        return True
			
    def update(self, show=True):
	if not self.__prep__(): return
	out = [ln for ln in os.popen(UPDATE)]
	if self.__checkerror__(out): return
        for ln in out:
            if show: print ln
            
    def sync(self, update_if_safe = True):
        if not self.__prep__(): return
        
        print 'Getting latest for ' + os.getcwd()
        #only try to update if there were changes...
        if self.pull():
            if update_if_safe and self.issafe():
                print 'Updating...'
                self.update(False)
                
        print 'done'

    def nudge(self, show=True):
        if not self.__prep__(): return
        out = [ln for ln in os.popen(NUDGE)]
	if self.__checkerror__(out): return
        for ln in out:
            if show: print ln

It’s not the prettiest, in fact I’ll change it soon to have a ‘verbose’ property or something so I can ditch the “show=True” params from a number of those functions. The idea for the “nudge” function came from an hgtip.com post where they address what is honestly an annoying part of mercurial to me.

Once I had this repo class then I just needed a simple script to wrap around it. This is a bit closer to the script I mentioned earlier, but it’s got some options for command line params.

import os, hgutil, config, sys

def main(argv=None):
    if argv == None:
        argv = sys.argv

    arg_path = None
    # start at 1, 0 is the command
    for i in range(1, len(argv)):
        if argv[i] == '-p':
            arg_path = argv[i+1]
        if argv[i] == '-h':
            show_help()
            return True
        
    path = get_path(arg_path)

    if has_connection():
        for f in os.listdir(path):
            p = os.path.join(path,f)
            if os.path.isdir(p):
                repo = hgutil.repo(p)
                repo.sync()
    else:
        print 'No connection to hg found. Please ensure pageant is running.'
        return False

    return True

def has_connection():
    return len([ln for ln in os.popen('tasklist') if 'pageant' in ln]) > 0

def get_path(path=None):
    if path != None and os.path.exists(path):
        print 'starting from path:', path
        return path

    ##default
    print 'No valid path found... using default.'
    return config.get_home()

def show_help():
    print '--- sync.py ---'
    print 'Attempts to sync all hg repositories under a path.'
    print 'Returns errors if pageant is not running.'
    print 'Args:'
    print '-p : path to work from'
    print 'Example: sync.py -p c:\dev'
    print 'Done.'

if __name__ == "__main__":
    sys.exit(main())

Since we use pageant for our connections to mercurial it’s nice to have this be able to check and let me know in case I’ve forgotten to enable it.

Now that I had this full script, I create a windows scheduled task to run it behind the scenes for me every 10 mins. Now if I’m not actually working with a specific repository it just gets updated for me, and if I am, I pull the changes, but don’t update until I’m ready. This needs more work before it’s terribly useful on my automated build server, but the ground work is there and I’ve been really happy with it so far. It might not seem like it but this script saves me a solid 10-15 mins a day manually doing pulls and updates on my repos and a lot of help where I run into issues from running an older version of some code.

Now, it is possible that I’ll be working on something I don’t want automatically updated. Then I’ll just change it over to default to not update. The automated pull is still useful.

But what I’ve really enjoyed that’s been a fairly big change from the IDEs I’ve been using is vim. Not long ago I wrote not to long ago abouthow I like minimalism from my editor and while I’d used vim here and there in small chunks (mainly for commits with git) I never really spent a lot of time in it. Moving into a world of embedded code a friend of mine who also works in embedded software highly recommended it. He pointed me to the Vim Book which gives a really solid overview of not just the tool, but the concepts and workflow behind it.

While some people might not see it as minimal, to me is. It’s *just* an editor. It can be a little convoluted and the commands don’t always seem like the most obvious choices, but the more I use it the more the system makes sense. Yesterday I even spent sometime customizing my config file. For one, I like line numbers. They don’t seem to be a general favorite of vim users, but I like it. The focus on keyboard is great. I don’t need my hands to move from the keyboard for anything. I did turn on the mouse just because I don’t like the Ctrl-Ww for switching between windows. I know it’s possible to make a custom command for that, but I’m still new and haven’t made sense of it yet.

A few of my choice settings:

" Save a lot more of a history
history=1000
" Turn on line numbers
set number
" Show the file name in the title bar
set title
" Make tabs 4 spaces and convert tabs to spaces
set tabstop=4
set expandtab
" Enable mouse usage in all settings
set mouse=a

As I get better with vim and find more things that aren’t interacting the way I like, I’m sure I’ll add to this, but this seems like a good beginners config. Vim has a bit of a learning curve, but I think it’s worth it. I think it’s a bit like git that way. Weird at first, but the more you get used to it the more awesome it is. I still only know 1% of what’s possible, but so far, I love it.

For me, this is very exciting work. I’ll get the chance to make use of my existing skill-set to help out the team with implementing a number of tools to assist with testing. The team right now is primarily Electrical Engineers and they are missing the skills for OO programming. That happens to be my expertise right now. But what makes this great is that it puts me in a prime spot to start learning and expanding my skills to work in embedded software. Which is my goal for the not too distant future.

I just don’t really get excited about web development anymore. It’s given me a great chance to learn, but when it comes down to it, how many forms over data can I really build and have it be interesting? That’s not to say there isn’t innovation to be made or that there isn’t some really cool stuff happening on the web. Just that it turns out not to be exciting to me.

I’m grateful for the opportunity and glad to have a change of pace. I’ll be moving to the Denver, Colorado area which is beautiful and, if I have to move from Charlotte, it’s a pretty good option.

This is my really simple csv reader. I’m calling it, icsv.py. There’s a million things it doesn’t handle yet, but it covered what I happened to need today. Maybe I’ll expand it as the situation warrants.

class reader:
    def __init__(self,path,delimiter=','):
        self.File = open(path, "r")
        self.Delimiter = delimiter
        self.Header = self.File.readline().replace('\n','').split(delimiter)
        self.__curr_line = None
	
    def canread(self):
        return not self.File.closed
	
    def stop(self):
        self.File.close()
	
    def nextline(self):
        if self.File.closed: return None
			
        curr = self.File.readline()
		
        if not curr: 
            self.File.close()
            return None
		
        self.__curr_line = curr
		
        ## no quoted text return the easy way...
        if curr.find('"') == -1:
            return dict(zip(self.Header,self.__curr_line.replace('\n','').split(self.Delimiter)))
			
        line = dict()
        for h in self.Header:
            if not curr.startswith('"'):
                line[h], s, curr = curr.partition(',')
            else:
                clsq = curr.find('"',1)+1
                line[h] = curr[0:clsq]
                curr = curr[clsq+1:]
					
        return line

I’m no good with simply taking everyone’s word for it, so I wrote up some basic tests. They aren’t perfect for sure but it gets the job done (download the project), and I see some kind of large divergence in the times depending, I’m guessing, on what else my computer is doing at the time. However the overall trend of the results are consistent. Interfaces are faster until you get to an enormous number of iterations. I didn’t see Interfaces show as consistently slower until I was testing in the realm of 100mil or 1bil iterations, they started to become close enough as makes no difference around 10mil iterations.

Even with the divergence I saw, in the typical application I’m not sure this would play a real role. My approach to this will definitely be to not worry about it unless I am A) working on something where optimization is crucial or B) I start to notice some performance issues specifically related to my abstraction. Chances are that if you have performance issues, whether you are using an abstract class or an interface is not the problem. Of course that’s not to say you should ignore known performance issues, but there are enough non-performance related benefits around abstract classes and interfaces that you shouldn’t sacrifice good design or maintainability for an optimization you’re not likely to ever notice in the real world.

The thing I really wonder about now is if it’s a product of the CLR. If I get to it, I may write a similar test in java and post the results.

Cool eh?