Software Dependency Tracking

As we build more and more products, we share more and more code and components between them.  As we’re always told, code reuse is a good thing – though as we’re starting to learn, it’s not without its own set of challenges.

Our current setup is based on Subversion externals – shared libraries are pulled in and compiled into derived products.  Shared components, for example USB drivers, are pulled in by release tag and built (which feels like the right way to do it) or simply included as binaries, which is the lazy way.  Some products specify a particular revision of a shared library, while unfortunately most just pull the head revision (this is dangerous, see later).

Our automated build system tracks which projects use which components, and rebuilds all the non-versioned dependancies whenever a component changes (to shake out any build problems).

There are a number of problems we need to address if we want to share and reuse more:

Keeping builds repeatable

Pulling the head revision breaks repeatability of builds, so we must use revisions when specifying an external – though it’s a pain to go through and update them for all dependants when an external changes.  A helper script might be a way of doing this, though it could be risky.

Predicting the scope of a change

A simple way to identify all derived products of a given shared module allows you to make decisions about where, how or whether to do it (for example, does changing a badly named function  justify rebuilding six derived products?).

To achieve this, including compiled binaries in projects has got to stop, as the dependency tracking system has no way of identifying them.

A nice visual dependency graph would be good, but a flat list of externals would be OK.  There doesn’t seem to be a third-party tool which can do this, though there are several things which can make graphs from your repository – so  I decided to write one, which can be found here (internal only, sorry).

Sharing information about library code

Libraries are less useful if you don’t understand them or know about them.  We have a system which generates documentation from key repositories daily, using doxygen.  Now we just need to get into the habit of including doxygen markup in our source, to make using these libraries as easy as possible.

Additionally, changes to shared libraries should be automatically posted to the review server, and reviewed by the entire team (another way of identifying any consequences of changes).  At the moment, only changes to built projects are automatically posted for review.

Keeping dependants up to date with changes to shared code

This is the big one.  When a shared component changes, all dependants must be updated and tested.  One solution might be to always specify the revisions of externals, but have the build server do two builds – one against the head and one against the specified revisions.  Then, if a problem with building against the head of an external is found, it can be identified without losing the repeatability of specified-revision alpha builds.

All you would need is a notification in the build results or a warning flag on the build summary page (“warning – does not build/pass tests against external rev. #67“).

End-user issues

Where a shared component is actually a separate install, for example the USB drivers, this presents problems for end users as well as development – for example, if they install a new driver from our website, then install an older CD from a box for a different product, they will be confused by the “there’s a later version of this driver installed” message.  How can we resolve this?  I think that’s for another day!

Other people have thought about the same problems, and a theme emerges – it’s hard, and it’s often very domain-specific.  It’s not easy to Google for more, as most articles refer to compile-time dependency tracking or runtime dependencies.

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: