Early last month I started a new job as Head of Software at Biosite Systems. Biosite create and manufacture a variety of hardware and software for access control and resource management on constructions sites in the UK. This is a significant change in direction for me, until now I’ve always been a software developer first and a manager second. In this role I’m a manager first, and a software developer second - and so far the difference has been even more significant than I was expecting.

In this series I’m going to catalogue the challenges facing our team, and how we’re moving to overcome them. I’m a strong believer in using Agile methodologies for software development, and one of the core ideas is that of continuous improvement. More than that, there is the idea that Agile teaches principles, without mandating particular implementations - but this means that implementing Agile can be a challenge for those new to it or inexperienced with the systems that need to go around it. I hope this series can offer ideas and inspiration to others who are developing their own Agile processes, from a concrete real-world setting.

The Tenets

Anyone familiar with Agile will have probably seen these a hundred times, but I feel the tenets ¹ are worth repeating because they are the basis for how we make decisions in an Agile environment.

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

It is important to also bear in mind the footnote:

That is, while there is value in the items on the right, we value the items on the left more.

which reminds us that Agile does not say “throw away all your documentation”, it says we should prioritise working software over comprehensive documentation.

I like these tenets because, having worked in small businesses all my professional life, I can see that they summarise what ends up happening anyway (at least if the business is successful) because they emphasise what is important to the success of the business. That is, communication between the important parties, delivering working solutions to real problems, and collaborating with customers to make sure those solutions meet their real-world needs.

I find it very hard to argue against these tenets, and yet I still hear developers say they need fully detailed specifications before they can begin working on a project. This kind of big up-front design goes against all four of the tenets, and personally I’ve never seen a project that takes this approach go entirely to plan. I would much rather accept that some change is inevitable, and build a culture that is prepared to adapt to change than try to create a process that resists it.

The Principles

The Agile Manifesto also gives us some principles ² that offer more detailed guidance on how we can go about delivering on the promise the tenets make.

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• Business people and developers must work together daily throughout the project.

• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

• Working software is the primary measure of progress.

• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

• Continuous attention to technical excellence and good design enhances agility.

• Simplicity – the art of maximising the amount of work not done – is essential.

• The best architectures, requirements, and designs emerge from self-organising teams.

• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

What I find most interesting about the tenets and principles is what they don’t say. They don’t mention ideas like pair-programming or test-driven development or many other practices that have become almost synonymous with Agile teams. That’s not to say that these practices are bad or unnecessary of course, but it’s worth remembering that Agile is about these principles rather than about any specific processes or tools.

Scrum

My team currently uses the “Scrum” methodology. When I started they were using “Scrum, but…” which is a nice way of saying they weren’t Agile - “Scrum, but…” is what happens when a team adopts the Scrum framework, but leaves out certain aspects without fully understanding the consequences of doing so.

A classic example of this is the sprint goal. Many teams won’t set a sprint goal and instead will treat the sprint as a list of tasks to be completed, so they do “Scrum, but we don’t set a sprint goal”. The sprint goal serves two purposes. Firstly, it’s a way to communicate your why ³⁴. It gives the sprint a sense of purpose, a reason for being beyond moving some post-its from one side of the board to another or an electronic ticket through its workflow. Secondly, it helps your team to remain responsive to change and make independent decisions. Software development requires smart people, hiring smart people and then micro-managing or eliminating their ability to make decisions is not a recipe for success - by giving them the information they need to decide how to perform their work they become empowered to perform better.

It might seem hypocritical for me to talk about what Agile doesn’t specify in one paragraph, but then advocate following the Scrum Framework rigidly in the next - but there is a method to my madness. In order to make progress we need to start somewhere, and at Biosite I’m choosing to start with Scrum because others in the field have had success. However, Scrum is just the start - the first step on our path to agility.

Our Challenge

Perhaps the biggest challenge facing my team is the current lack of product leadership. The Scrum Guide ⁵ says almost nothing about long term planning. As a result, implementations of Scrum often confuse agility with a lack of long term product vision. Why plan ahead, one might argue, when those plans can be scrapped at any moment by unexpected change? Yet, when you read any in depth guide ⁶ it is obvious that long term planning is important. Indeed, after working with Scrum for long enough this becomes obvious in the standard of the product - a lack of focus becomes evident.

So how do we reconcile this need for long term planning with the agile approach, and who does it? Who is obvious, the product owner (or the product owner team) is clearly best placed to do this kind of work - they have all the information they need and the authority to turn it into a reality. How is given by one of Mike Cohn’s ⁶ guiding principles - “planning should be deferred to the last responsible moment”. The key here is last responsible alongside the realisation that we can do planning at difference scales ⁷. It would be irresponsible for us to do no long term planning, but equally we do not need to do long term planning to the same level of detail as we need to do sprint planning. Indeed, most depictions of the project backlog have the idea that stories and tasks get more vague the further away they are.

Until recently our CEO had being operating as product owner, and taking most of his input from the sales team. Unsurprisingly, he doesn’t have time to fulfil all of the responsibilities of a product owner, and listening solely to the sales team means that other stakeholders (such as support and operations) were left without much voice in the development of the software. It also meant the development was very short term focussed, sales tend to be very focussed on what they need for their next sale - whether or not it fits with some kind of long term plan. The product owner needs to think further ahead, and be thinking about how to meet the needs of many customers over a much longer term.

To solve this problem we’ve promoted an experienced developer from our team into the product owner role. We could have found an experienced product owner instead, but we felt we needed someone whose in-depth product knowledge would enable them to get started right away. Three weeks in and our product owner is still finding his feet, getting control of our current backlog is a massive task and only time will tell if this strategy works out for us. That said we’re optimistic that we’ll have stabilised that part of the process by the end of the year.

The other major problem facing the team surrounds our testing strategy. A prolonged period of pushing features into the development pipe without being aware of the need for a sustainable pace has left us seriously lacking in testing capabilities. Right now we’re handing over releases to a dedicated tester at the end of each sprint (the “Secret Waterfall”). We have unit tests, but we lack automated integration or end-to-end tests. Because it’s simply not possible for our test team to cover all the product features each sprint, it means we’re leaving a lot to chance each time we make a release. I’ve written extensively about what I believe is necessary to make testing work - and this isn’t it. It’s clear we need to automate a lot of our testing, but the challenge is adding a framework to our existing code-base that will let us do this in a sustainable way. We operate a curious mix of C++, Qt, and Javascript. Our architecture includes server components, client side applications (with embedded QWebViews), an online “cloud” portal, both local and networked hardware elements and some pretty complex synchronisation to tie it all together. Needless to say off the shelf solutions don’t quite cover what I believe we need, which is a comprehensive set of end-to-end tests.

Unlike my other series, I don’t have a plan for where this one will take us. Over the coming months I plan to keep the series updated with how we’re getting on, the problems we find and the solutions we choose.