First steps in a new software project

This week I started a brand new project. This time it’s the start of product development, rather than just an internal project to get something running. And it really is new, so everyone is new and we’re setting up everything from scratch.

So this is my list of things to go from nothing to a fully fledged software engineering project.

Step one, getting the right tools installed for everyone off the bat makes things easier later on. Standardise upfront rather than try to merge half a dozen separate ideas about whats best later.
For this team we will be testing a GUI so Rational Functional Tester (RFT) with Rational Team Concert (RTC) is the basic tool set.

This gives us all the project management tools in RTC, plus source control, and easy collaboration. Then when we get as far as something to test RFT will already be there.

RTC is the best tool I’ve used for starting projects, it lets you start capturing requirements and high level ’stories’ and start hanging individual tasks off of them. Code sharing is very easy, and delivering code changes under tasks makes the project tracking pretty seamless.

Step two, a build server. It’s easy to link into RTC to have a build engine that allows you to schedule builds as frequently as you like. It also allows people to kick off builds of the mainline code plus their ‘local’ changes. So you can build and unit test changes before delivering them to the main codebase. The outline of this can be set up quickly, the detail of full build and kicking off unit tests takes longer but is a top priority.

Step three is a Rational Quality Manager (RQM) server. This can be linked to the RTC server so that it gets notified when builds complete and also allows you to create defects in RTC if tests fail. RTC can then view defects which block test cases. RQM lets you define what environments you will test in, how you will split things into test plans. And makes it easy to start defining test cases and where they will run. Ultimately it can be used to report on test status, with built in dashboards and report generation.

The last step are some test servers which run adapters which connect to the RQM server. For our needs some of these will use a command line adapter, others will use RFT. This allows RQM to kick off tests against the systems and gather the results.

As a set of tools they are not light weight. Requiring at least 4 servers (RTC, RQM, build, test) but then they are a very powerful combination. I don’t expect to have any spreadsheets or hand-crafted chart generation being done by the team. All of the information we require comes straight from the tools, when we report project status we will use the tools not hand-made slides or spreadsheets. I also expect to be able to get tests automated and managed from the start. So no trying to retro-fit automation onto manual tests.

After one week we have RTC setup with the first wave of requirements stories and tasks, and a code base setup with components for our new product. We have an RQM connected and being told about builds. We have the skeleton of a build system. It’s not doing much other than defining the build types we will have, but the framework is in place.

Next week I’ll setup my first test server. Hopefully in a very short space of time we will have everything we need to develop and test a product. Complete with project tracking and reporting, all while most of the team are focussed on what we will produce, not tied up with figuring out infastructure.

Retrospective on an agile project

Yesterday I finished my 5 week project. I still can’t believe it’s been 5 weeks, it’s gone so fast.
I wrote a bit about it a couple of weeks ago. And now that it’s finished I wanted to reflect a little on how it went. This, of course, is part of the agile process. They call it a retrospective, and in theory you do one every iteration to consider what worked, and what didn’t. Since my iterations were just one week long I didn’t write anything down for each one.

One thing I said in my previous post is that I wouldn’t work long hours to achieve results, as it is unfair on future projects to misrepresent what can be done in the time. Almost the day after posting that I found myself breaking that rule.

I worked several long evenings, but I have an excuse-ish. One of our big problems was getting a system set up so that we could start on some of our tasks. But it gave us enormous problems, and I was tired of burning days on what should have been a couple of hours of set up. So I started working into the evening in the hopes of putting us back on track. Regardless it still took almost 2 weeks to get a system we could use. This was incredibly frustrating and I broke my rule because I didn’t consider the task part of what the project had to achieve. It was just a pre-req. But an important reminder that sometimes the things that take the longest are things you don’t even officially plan or size. I should have taken half a day at most, but it burned more like eight and a half.

I also broke my rule for another reason, perhaps one I should have factored in. We often needed to work with a team in the US, specifically with us raising questions or problems, and getting responses. To that end I joined a weekly conference call from 17.30 till 18.30. I also found it was better to check e-mail into the evening, since it was the difference of responding to a question and getting a second response by the next day versus finding the question the next morning. It’s easy to burn through days bouncing question/answer/question back and forth at a rate of one request/response per day. The conference call helped to speed things up. However the reality of relying on a team in another timezone is that you make faster progress if you spend more overlapping work time.

The common theme to these two rule breakers is things that felt unrelated to the *work*. Just stuff we needed to be able to get on with the doing. That said I need to remember in future to factor this stuff into short projects.

In terms of project results, I have mixed feelings. We failed to achieve all I wanted. However, I did achieve the major strategic investment I wanted. I never expected to be ‘done’ in 5 weeks. Just to have kick started it enough that it becomes a long term commitment to the direction. Although my project is over, most of the team will continue. Everyone agreed that we were going in the right direction, and we did enough to convince people that it *can* work.

I’m very happy about achieving the important attitude shift, and establishing the relationships necessary to succeed in the long run. At the same time I’m very frustrated by some of the things we didn’t achieve already which I feel we could have.

For me personally a big part of the project was trying to lead. My natural instinct when people aren’t doing what I’d like, is to wish I had the AUTHORITY. You know? That feeling that it would be easier if I had been bestowed with power such that people must do as I say. Perhaps this is what attracts people to management. To have that explicit power defined. However, I do not want to be a manager. So the trick for me is trying to lead with no explicit authority. And boy can that be frustrating!

I feel I had a particularly challenging crew. On previous projects I’ve worked with keen, enthusiastic people. Willing to listen and discuss. Ultimately accepting the group decisions, and working furiously towards the agreed goals.
This project could not have been more different. One person in particular had very specific views on what they wanted to do, and how. Whilst this fit with what was necessary that was fine, great even as they made great progress. But they would sometimes decide randomly on another priority. They totally didn’t get the idea of agile and committing to deliverables for this iteration. On one particular occasion when I asked on a Monday about getting started on one of our committed items, I was told ‘I’m not going to look at that this week’, other times I asked for where they had gotten to, and what was blocking progress. Only to be deflected with questions about other things. The attitude being that their tasks were none of my business, they didn’t want to explain the problems or the specific progress. They just wanted to be left alone for as long as it took to do it, their way.
On one occasion I got the Instant Message equivalent of being hung up on. Rather than answer my question I was told ‘I need to concentrate’ and they logged off.

It would be easy to say that person was just difficult, and in future I’ll only work with easier people. But of course I realise I have to learn to get the most from people like this. Perhaps I should have been more direct and just asked ‘ why don’t you want to tell me what the problems are?’

In that regard I feel I did badly as a leader. However in other regards I was getting better. One of my problems as a techie is that I come up with the solution I want, and I try to get people to do it my way. For this project I made a conscious effort to define the result I needed, and accept solutions that weren’t ‘my’ way, so long as they would meet the requirements. Sometimes this means watching something take longer than I think it should take. But maybe that’s just because the solution will be better?

So I still have much to improve on. That said, I was explicitly thanked by the manager in charge of the project for my leadership. So it can’t have been all bad

Automation isn’t free, and change isn’t either.

When I started this blog I intended it to be about both woodworking and software engineering. It’s pretty clear that my focus has been woodworking, and more specifically woodturning.
However, that’s not to say I’m not busy thinking about software engineering, I just don’t often think to write about it.

This week I found myself starting another agile project. A little over a year since I wrote about being ‘so agile they’ll need another word‘ and so I thought I’d write up some more thoughts on the subject of software engineering and agile projects.

Last year the project was one I had spent some time building support for, to overhaul our automation infastructure to be more sophisticated. It required a change in the accepted way of delivering results. Which meant convincing project managers that the new information would be *better* than what they were used to. Then convincing management that thoe whole package would be worth the cost of about 4 person months.

I really enjoyed that project, and I learned a lot. Both about how to run an agile project, but also about how to get one off the ground in the first place. I came to the following conclusion

People don’t know what they’re doing.

By which I mean, all of those people who ‘control’, managers, project managers, strategists etc etc. They mostly don’t have grand plans and visions. They are now ochestrating clever and subtle schemes to reach their goals. They basically are just winging it, like the rest of us.

The implications of this realisation are simply that if you have an idea of what you want to change, and can formulate a half decent plan to achieve it. Then there is a good chance that managers and planners will seize upon it with joy. At last! A plan! I don’t mean to undermine what these roles do, they keep projects moving, they understand what is required to deliver a project. They remember what ahs gone before. But the important realisation is that they rarely have time to think about great changes from the ‘norm’ and they will gleefully accept that idea you’ve been toying with, if only you can pitch it in their language.

Last year the project was a great success. And one year on I believe people are still feeling the benefits, looking back over the project it is clear that things went better, ran smoother. The project delivered on it’s promises.

That makes for a good reference when talking again about a project to run. And so it is that I started a few months ago paying attention to general talk around adopting a new product to manage our test tracking and reporting. There was a general feeling that we ought to be looking at it, but by default everyone is already completely commited to plans. And so ‘looking at it’ means, occasionally having a meeting to talk about it, but not actually having any time to *do* anything. It seemed clear that this would be my next target.

Part of the message I try to get accross about why such projects are necessary is : Automation isn’t free. And neither is significant change for the better. In short, you can’t expect to acheive great things for nothing.

For too long auotmating tests was sold as expensive up front, but then you got a lot ‘free’ like platform converage, repeatability etc. I think that this message set the wrong picture. Because many automation infastructures are woefully under funded for on going work. The attitude is very much that ‘we paid the high up front cost, now this bit is free…right?’
Well no, automation isn’t free, it requires people to make sure things keep ticking along. It requires updates to support new things. It requires skill and knowledge in what is often a complex environment. However, it has such a high rate of return, such an amazing return on investment, that it is easy to think of many of the benefits as ‘free’

I struggle to push the message that you get what you pay for, and when you pay into your automation infastructure you get a LOT. But it DOES require on going investment.

In this case I’m not dealing with automation as such, but we are talking about a big change. It has the potential to make many areas of our project ‘governance’ much better. It has fantastic scope to open us up to options that simply can’t happen without the first big step.

And it is this that I told people when looking into adopting this new technology. Anything *less* than a couple of person months dedicated time to get things going is a joke. If you aren’t prepared to stump up the resource for at the minimum that much. Then stop talking about it, stop thinking about it, because you’d just be wasting your time. Sure there are plenty of other things that you need to acheive with that resource, and you need to understand the relative priority of this compared to those things. But don’t for a second kid yourself that the team can effect such large change without some real time to plan, test, try, deploy.

And so here I am, 1 week into a 5 week project, where I have been given time, and a student. To really get to grips with how we will adopt this new technology. My goal is to have it either going into production at the end of the project. Or have a very specific list of blockers, and associated defects and actions to resolve them.

I love working this way. I have a set of stakeholders, people who will be the real customers of the system. and I meet them weekly on Friday’s. In that meeting I show them what has been achieved during the week, and get their approval to mark certain tasks as ‘complete’ Then I work with them to define the tasks of the next week. Sometimes this means making them agree amongst themselves the priority order for new tasks. Then everyone knows where we are, and what we’re doing. Why we are doing this first instead of that. The list of requirements can grow as and when the stakeholders think of them and define what they want. But they also get to help chose which of those on the backlog they *really need* this week. The great advantage of this is that you could keep the project rolling until people run out of *must have* items. You can stop at any time that people feel they’ve had most of the value. In essence the stakeholders are the ones who will ultimately decide some fraction of the requirements they raised, really weren’t that important afterall.

It also means that last weeks work finished on Friday, and next weeks work starts on Monday. That sounds kinda obvious, but pychologically, I am not starting to think about the challenges of next week yet. And those of last week are done, for better or worse I met with our stakeholders and showed them what we had and that was the end of the iteration. Working this way keeps me focussed. Everything boils down to ‘what do I have to show the stakeholders on Friday’. It makes for an intense working week, and sometimes it’s hard to go home at the end of the day and NOT keep working. But I have to remember that if did that it would be a disservice to anyone in the future. It would be dishonest to show a project completing on schedule, if it actually took me a weeks worth of over time to do so.

Week one is always hard in terms of just ramping up, but week two is where things are going to get serious. The level of expecation goes up a few notches, and my little team needs to start delivering real change. I’m looking forward to it, I know that success in this project is more than just delivering what I said I could, it’s showing that this whole way of working delivers consistent results. It means that next time I pitch a project, my job is a little bit easier.

Many projects I’m sure, struggle to justify the cost in terms of the benefits they yeild. I find a different challenge, trying to remind people there is a *cost* and what that is. It easy to see the value, but automation isn’t free, and change isn’t either.

Agile is..doing only as much as you need

This weekend I discovered what I consider to be a massive design flaw in my Mazda3. The problem that lead to this discovery is actually something that has happened a few times, and I believed it to be an electrical fault.

I was happier in that belief.

Because so much worse is the realisation that this glaring fault, this stupid, damaging, security hole is a ‘feature’!

What am I talking about? Well, it turns out that if you press and hold the unlock button on the remote, not only does it unlock the car, but it also lowers all of the windows.

Yes, all of them.

Now you might be thinking this doesn’t sound so bad. But the problem with remotes, is that they work at a distance. Say…  the distance between me inside a house and my car parked outside.
And the problem with surface buttons on something in your pocket, is the ability to accidentally press those buttons.
And so I woke after a night of rain, to be told by a friend that my car had all the windows open. And was rather wet inside.

$£%*$^&£^$£*^$!!!

This got me thinking around to how such things happen. And I can easily imagine the development process. I can consider reasons why you might want to be able to shut all windows remotely. In much the same way, I realised, I can see why I want to be able to lock the car from a distance, i.e. “Whoops I forgot to lock the car and I’m now 50 yards away”.

I suspect the feature to be able to open all the windows was a ‘freebie’.  If you’ve done all the work to allow for up, why *wouldn’t* you do both?  – it’s free.

Well I’ll tell you why not, because shutting windows is secure. I don’t mind accidentally CLOSING the windows. Opening the windows is INSECURE and I do very much mind accidentally doing so. This also made me realise that what I really want is a remote that lets me lock my car at distance, but only responds to unlock when I’m within a meter or two.

This all brings me around to why I like the AGILE development process. It helps more than ever to focus on doing what your customers/stakeholders have actually asked for and NO MORE. Do just enough to meet the needs, and only add new things if someone actually wants it.

I often find myself  telling people to note an idea as possible, but not to progress it unless someone decides there is a good reason for it. Of course developers like to implement ideas. Sometimes ‘because I can’ is all the reason one needs. But what happens is you spent time and effort developing, and then testing a feature that no one wanted, and in bad cases people actively don’t want.

The other thing I’m a big fan of, is making things configurable. If I had found out about the horrible design flaw in my car, but discovered I could disable this ‘feature’ I would not mind. I guess there maybe some people in the world that really do see a need to lower all their windows at once remotely. But the fact that it cannot be disabled just makes it worse.

There is a lesson in all this somewhere. And I like to think that if I keep evangelizing the agile process to people I might get people to learn this lesson. Don’t implement something if the only reason for doing so is ‘because we can’ -  save yourself and your business time and money and focus on what people actually want, if you don’t know precisely what people want then let them configure.  And, of course, deliver it in the order they actually want it.

So Agile, they’ll need another word

This last week has been the first of a project planned to take four weeks. It is a great opportunity for me, since the project’s goal is to deliver on an idea I pitched, some months ago, to solve a problem with the way we test.

I have been very pleased with getting this far. When I first proposed what I called ‘a pretty radical change to the way we approach testing’ I wasn’t sure I’d be able to convince people it was the right thing to do. But as it turned out I had no real resistance. In fact so positive has been the response, that many have long forgotten that I got things moving in the first place.

Ironically I was busy constructing this plan when I was being told that I am ‘not perceived as having enough sense of urgency on the day job’ Apparently I should have been more head down and focussed on the job in hand. However I think that criticism was short lived, and short sighted. By planning the changes I wanted to put in place, whilst everyone else was too busy looking no further than the next day, I found myself with a plan and a proposal ready… just as the management team started to wonder what on earth people will be working on once the last version shipped.

My proposal got discussed, and sized. The team lead of the automation team, who will ultimately own the changes gave a sizing of four person months. And he was insistent that he get to pick the four. It’s no good just assigning four random people and hoping for the best. He wanted people he knew would be able to deliver. He also insisted that these people be allowed to focus on the job. This sizing was for four people working full time – not being asked to fit it in around other priorities. This was a pretty tall order, and the fact that we got granted those demands is both a factor of having sold the long term benefits. And the management team having little better planned.

As fortune would have it, I got selected as one of the four. I am certainly the person most interested in seeing the idea succeed. And I’m grateful for the unusual opportunity to actually spend some focused time on delivering my idea. I’ve often thought it would be cool if submitting patents would come with the possibility of being given funding to develop a prototype… but no.

So here we are, end of week one of this four week plan. We are attempting to follow the ‘Agile’ method.

For those unfamiliar, Agile is nothing new as such. it is a rebranding and repackaging of various ideas that have been evolving for some time. It is – in my mind – an attempt to have a software engineering process that fits with real business needs. Gone are the days when clever people could lock themselves away to produce something clever, and just deliver when they thought it was ready. Many of the older software development processes just don’t work in a world of ‘marketing splash dates’ and tactical deliveries. You have to be able to adapt to the market. Being half way through a 2 year development cycle when the market changes could be devastating. One of the main features of ‘Agile’ is that you construct just enough to deliver customer value every ‘iteration’ And those iterations are as short as possible. That said we’re doing 1 week iterations which is very very short.
But the idea is the same. At the beginning of the iteration you plan with your stakeholders/customers, what you will deliver in this iteration. Something they will be able to see, and use. At the end you demo that to them. This helps to keep you focussed on delivering just what you promised, and no more.

I can say after 1 week that it is *really* hard not to get drawn in to considering design and longer term goals. How should we do ‘this’ so that when we get as far as needing ‘that’ feature down the line it will work right, etc. Finding the path between bad design that will make life harder in the future, and over engineering a solution at the expense of delivering this iterations value. But the driving goal must be, that in the modern business world, you must be able to deliver value at any point. If the project got canned next week, we’d have something that would provide some function. It isn’t everything people want. but it’s also not just a pile of unusable code.

The aim for next week is to actually deliver enough to go live on our ‘customers’ website. It will not be ready for full fledged use. It will still have some serious limitations. But it will be able to do some key piece of the project goals. From there forward our delivery will be to update that live system with the latest set of capability. Every week our customer will get a little bit more of what they wanted.

This week when asked how it was going, one of the group replied, “We’re so Agile they’re going to need a new word”.
Which in part is a reflection that with this process you can apply it to greater and lesser degrees, depending on the project. With just five people in one room, and a single month, we can take it to the extremes of Agile development.

Another cool thing about the project, is that I am sitting in an office with four other guys (we ended up getting a fifth person for the first couple of weeks) And they are all super smart, capable developers. It’s unusual to work in this way, it’s not that I normally work with people that aren’t clever. It’s more that normally the way of working doesn’t let you collaborate, as much as just do your own piece without that much interaction.

This week I’ve been blown away by how fast we got something working, and doing clever things. And it’s really energising to work along side people that know how to do things I don’t, and (hopefully) fill my own roll of expertise. It’s an education looking at the code they write. I think nothing of throwing together the sorts of code and solutions I’m familiar with, and they all have different areas that they handle with the same ease. Together we seem to work pretty well. There are some differences of opinion on design, as is healthy. We all help to stop each other getting drawn into considering options and designs that are not part of our current deliverables. And so far we’ve not hit a problem that has stumped us all.

I haven’t been this happy at work for a long time. I’m actually finding it hard not to keep on working during evenings and weekends. However another key feature of Agile is ’sustainable pace’. It’s no good relying on people working every hour, they will burn out. As enthusiastic as we all are, I’ve been keen to remind us all that this project may be used as a template for others. And we owe it to future projects not to succeed only by working ourselves to death. We prove what we can do working just regular working hours.

Perhaps the strangest part of this agile project, is the acceptance that requirements are shifting. There is no hard and fast list of what we must deliver at the end of 4 weeks. There are key capabilities which form our first areas of focus. But at the start of each new iteration we review what, out of the possible options, is most important to deliver next. The expected output of the project will be the most important parts of a working system AND a list of the things we wanted to do, but were prioritised lower than what we delivered, possible enhancements that may form the basis of a separate project.

That’s probably more than enough of the subject for now. However I will post updates as we go through as to how we are managing this Agile project. And if we have hit the much touted problem with Agile, that of hitting a point where you realise you’ve designed yourself into a dead end by virtue of not thinking far enough ahead.

Also I may post about Rational Team Concert.. for now I will just say it’s awesome. (Yes I work for IBM, no I’m not paid to say that, nor is it praise I’d have for many other Rational tools).

Check it out at jazz.net