Building the Idea: Planning for Success

This is the second post in our series Application Development from Inception to Production, which documents the complete lifecycle of a Grails project.  The first post is available here: Building the Idea: The Genesis of a Project.

I began working with startups a few years ago.  I had been an agile developer before I started working on fledgling ideas with small groups, but there was always something missing.  Something that kept Agile honest.  Something that really allowed me to talk about Agile with a new business owner in terms we all understood, and in terms that equated to a real shift in thinking and the bottom line.  I found this missing piece in Scrum.  Scrum is a framework for Agile development that provides the foundation of common terms and required actions that permit an Agile team to work succinctly towards a common goal.  It is elegant, yet rigid.  Scrum is based on very few principles, each that strives to empower the team itself while providing deep transparency to the stakeholders of any project.

I received my Scrum training from Jim York, owner of FoxHedge, Ltd back in 2008 or so.  I have logged thousands of hours in Scrum Master project time since then and can honestly ay that Jim thoroughly prepared me for real world implementation, and I highly recommend using Jim if you are looking to become Scrum certified, either as a product owner or a scrum master.  Jim said one thing that I’ll bring up here:

“Scrum is simple.  Doing Scrum is hard.”

This couldn’t be more true.  I won’t go too far into Scrum in this post as I want to just touch on the high level components for our current project discussion, but feel free to go over here to the CFHour podcast and listen to an interview I did on Scrum.   The core of this article will focus on introducing Scrum to your product owner, as well as building your project backlog and estimating your first Sprint.

First and foremost, you can think of the Scrum team as an exclusive club.  Those in the inner circle are those who have a direct impact on the project (see the fable of the chicken and the pig).  The Scrum Master sits as an emissary that can help to clear blockers, or things that prevent forward momentum and provide detail around specific tasks and goals.  The Scrum Master facilitates, but it is the team who leads.  The team comprises the developers, UX architects, etc that are responsible for the project output.  The team sets the pace, and the team agrees to the current work effort included in each Sprint (one iteration of tasks).  At the end of each Sprint, the client receives tested, deployable code.  This does not mean that the code is always deployed, but it in fact can be if the current level of functionality is useful to the end user and the business goals.  Generally the output from Sprints are rolled up into releases, which we are all familiar with from basic SDLC.

The client is certainly not absent from the team!  The client has one delegate, the Product Owner (hereafter referred to as the PO).  I’m sure you have been in meetings where each person on the client team has an opinion and, you, as the contractor have to agree to each one because it really isn’t your place to tell a shareholder their idea is jeopardizing the entire project, or tell the CEO that allocating a page to pictures of her cat is a ridiculous waste of the team’s effort, even if she thinks that it will bring Redditors to her site.  Well, in Scrum this task is relegated to the PO.  The PO is the single voice that drives the product development from the client team.  This works great, believe it or not.  It only works if two factors are present though:

1. The PO must be empowered to make all final decisions for the client.  He or she is the last voice on any subject.  He or she meets with the larger client team to get answers, but is empowered to move the team forward through the Scrum Master.
2. The PO must have complete transparency into the project.  This includes the backlog, the current Sprint velocity and tasks, and a immediate accounting of what changes will do to the velocity for the current Sprint.

That being said, it is sometimes difficult to identify the correct PO for a project.  The person has to have the clout to stand up to the C-level execs and board of directors, have a good head on their shoulders (not necessarily technical, but have sound business acumen and a complete understanding of the product), and most importantly MUST BE AVAILABLE.  If you can identify a person with passion for the project who meets this trifecta, you’ll have an easier ride.  If not, it might take some coaching and some pushing.  One other thing to make a note of from above, is that the PO does not drive the team, but rather works with the Scrum Master, who facilitates the teams actions.

Once you have identified the PO, the real fun begins.  After chatting it up as we saw in the first article of this series, it is time to create the product backlog.  The backlog is a compilation of user stories that comprise the functionality of the product.  These are your requirements, but written in a format anyone can understand.  User stories are written at a high level and do not include technical solutions.  This means that they don’t get into the weeds of the solution to the requirement like ‘We will use the Grails Authentication Plugin to handle logins.”  Instead, user stories follow the format:

  As a <role> I can <action> so that <goal>

An example of this is:

As an administrator I can disable a user account so that I can prevent further access for unpaid users without losing historical user data

Some Scrum proponents argue that the ‘so that’ clause is optional.  I am not one of those people.  In fact, without the so that clause I won’t prioritize a story.  If the PO can’t tell me why something needs to be done and what the goal of that action is, it must not be important.  Remember that Agile strives for ‘just enough’.  You don’t need to use valuable cycles working on tasks that have no clearly defined goal.  Take the story above for instance.  Not only does the goal remind us of why we need the story itself, to prevent user access on an individual account level, but it also tells us how to prioritize it.  We don’t need to perform this story until after we have paying users created that have the need to be disabled, but also that we don’t just want to delete them!  We want to keep all data around in case they want to pay again.

User stories also define all the roles in a system.  In the story above we see we need an administrator role.  We might also have a registered user role, a visitor role, etc.  It is important to denote the roles clearly in our user stories.  User stories can be written on a 3×5 note card (as I really like to do) or you can use one of the many great Agile planning sites out there.  I do caution you to find an Agile site that permits the use of Scrum terminology.  I use Jira right now with the Greenhopper plugin, which allows for Scrum cards, backlogs, sprints, burndown charts, etc.  I have also used VersionOne and RallyDev in the past with decent results.  In my opinion, nothing beats 3×5 sticky backed note cards.

Once the backlog has been “completed” (remember this is a fluid process, a backlog is in constant flux), work with the PO to prioritize the stories from a business perspective.  I also like to group cards into feature sets so we can track burndown (completed tasks versus remaining tasks) against individual product goals.  VersionOne allows you to do this online, as does Greenhopper to a lesser extent.  On cards you can use a colored marker to group features into sets like “security”, “social media integration”, etc.

Once you have worked with the PO to both define the backlog and prioritize the backlog, it is time to involve the rest of the team.  A Sprint Planning Session occurs before each Sprint.  This can be clumsy the first few Sprints while everyone is getting used to each other and learning the ins and outs of Scrum and the product itself, as well as the abilities of each team member, but as the team grows together, the process becomes more succinct and the teams estimates become accurate.

During a Sprint Planning Session, every team member has a voice.  I play planning poker using either the Fibonacci sequence, or more frequently 0,1,2,4,8,20,? for estimating stories.  The gist of planning poker is that each team member holds up a card at the same time of a level of difficulty (known as “story points”).  The Scrum Master then facilitates the discussion and the team comes to a unanimous decision on the value of the current story.  The group nature of this activity allows each member to discuss why a story could be more complex than others are thinking.  Since you generally have a diverse set of skills on the team, such as UX, UI, DBA, security, code monkey, etc, it is much harder to overlook buried problems.  If everyone throws up a 1 or 2 and the DBA throws up a 8, it could mean that the DBA is over thinking the problem, or more frequently it could be that non-DBAs overlooked the fact that they have no database server installed to start the task and he is accounting for the additional complexity.  The team can then rework their estimates and come to a consensus.  During this estimation process it is also important to define the acceptance criteria if it was not already defined for the story during backlog planning.  This acceptance criteria spells out what it takes for a task to be “done”.  This can equate to a functional test, and is usually a description of setup and expected results in plain English (or plain French if you are programming in Quebec).

I get asked all the time, “what is one story point?”.  Great question, and one I can’t answer!  Sometimes story points equate to days, sometimes story points equate to problem complexity, but the magic is as a team goes through a planning session, they silently come to an agreement on what a story point means to them.  In all honesty, it doesn’t matter what a story point actually means.  After the team has estimated” just enough” stories, they will agree to the stories that they feel they can complete during the next Sprint.  A line is drawn in the backlog, all of the stories above the line are pushed into the next Sprint, and the team can start defining tasks and estimating them (generally in hours) to begin work.  When the Sprint is over, the Scrum Master will look at the stories completed (known as the velocity) and can then use that number to determine the next Sprints task during the next planning meeting after estimation.  That velocity is then tracked for each Sprint as the project continues.

The important gotcha of the above process is that at no point did the PO or Scrum Master tell the developers what they are going to complete during the next Sprint.  The developers unanimously determined what they can get done and have agreed to start working on those stories.

The beauty of Scrum lies in transparency across the entire team.  This is especially true for the client.  Since the PO can see what tasks are being worked on, what tasks are “done” and what tasks are not going to be worked on at any given time, they have the knowledge of what they are getting (see burndown charts), when they are getting it, and what changing the current workload will do to the current Sprint.  Lets say the CEO has a brilliant idea, or even better yet, a bug was found that needs to be addressed.  The new story or bug is estimated (story points/complexity), and the PO can see that if that story is addressed n number of story points needs to be removed from the current Sprint (as well as any ancillary effects caused by disrupting the Sprint’s rhythm).  They are welcome to make the choice to reprioritize that story into the current Sprint, and they can easily see how that affects the next deliverable.

One more thing I’d like to mention is that Sprints can be of any length.  I find that 1-2 week Sprints are best for most projects.  Longer Sprints start inheriting the problems of waterfall development, such as scope creep and black box development.  For a new project I try to start with a 2 week Sprint and adjust accordingly based on burndown and the results from the Sprint Retrospective (an open and honest meeting to discuss what worked and what didn’t in the prior Sprint).

Here are a list of stories for our current project:

• As a user I can log in so that I can use system features
• As a user I can upload an entry from my mobile device so that I can rate my beer and show a picture
• As a visitor I can view pictures via a web browser so that I can get an idea of system functionality and review ratings
• As a user I can create an account so that I can tie my activity to myself
• As a user I can flag a post as inappropriate so that the user base can police itself
• As a user I can rate other user’s posts so that other users can see how useful/accurate a rating is
• As a user I can search for a beer so that I can see if I want to try it
• EPIC: As a user I can earn badges for performing actions and receive notifications on the system so that I am drawn back into the system

These stories are prioritized by business need.  Notice that creating an account falls pretty far down the list.  I did this to point out MVP (minimum viable product).  If this were a real product and I was going for investment dollars, I wouldn’t need the ability to create accounts to talk to investor groups.  All I need are the basics: log in, take a picture, rate it, and view the results on a nice web page.  I can add accounts directly into the database for demos.  If I was going to build a user base and launch the product to a beta group first to generate interest, I might reprioritize the create user step higher.  It all depends on the current needs of the business as defined by the PO.

I also introduced the concept of the Epic above.  An Epic is a story that needs broken down and is too complex in its present form to begin working on it.  It may be marked as an Epic when it is entered into the backlog, or it might be marked as an Epic during the Sprint planning session.  This generally happens if the story point number is high (many 8s and 20s or ?s – meaning I have no idea how to estimate this story), and is a flag to the PO and Scrum Master to refactor the story into workable pieces.

As you can see, there are immediate benefits that result from application of the Scrum framework.  The transparency that Scrum brings to a project is both good and bad, but always works to the advantage of the project.  The efficiency (or deficiency) of the team becomes immediately evident as does project impact when new requirements are introduced.  User stories document requirements in an easy to trace and easy to understand format.  They have worked much better for me than any requirements traceability matrix.  Scrum also provides the benefit of coalescing the team, where everyone has an equal voice and a domineering project manager or client cannot impose artificial deadlines.  This article stops just short of talking about how a Sprint should be run, how to facilitate daily stand-up meetings and other key components of Scrum, but hopefully this will spawn some questions for further blogging.  I am always happy to go into more detail on any part of Scrum so please feel free to ask in the comments below.

In my next article we will dive into the design side of this project, focusing on user-centric design principles.

NOTE: During peer review, Marc had asked about remote Scrum in regards to planning poker.  Scrum purists believe that Scrum should only be performed in a onsite location (I am not a purist BTW).  There are many reasons for this, but I have never had the fortune of having a centralized team.  When possible I prefer to have planning sessions in person, and for older contracts we would actually fly the team in.  This generally meant longer sprints, but the results were actually better than remote planning sessions.  Once the team really starting jiving, we moved to occasional offsite planning sessions.  The largest problem I have found with performing Scrum with remote teams is that you start to drop some core activities.  Scrum is defined as an all or nothing framework, and I have been told that it might be better to go with Lean instead of Scrum for remote developers.  That being said, there are plenty of good tools available for remote Scrum, such as the Agile planning software I recommended above and various other tools to enable sprint planning and collaboration.  I have seen Scrum work remotely many times, as do most other Scrum practitioners.  If you are interested in hearing more about Scrum in a remote environment, shoot me an email or comment and I will draft up a new post.