Agile and Lean Convergence in software and creative development practices

Looking under the covers of the popular principles of Lean and Agile it’s very obvious why these principles are so very important to progressive organizations, but these two principles also seem to be on a path of convergence into a new and more widely accepted set of principles and methodologies.

In larger scale projects the more significant advances in performance outcomes in recent years seem to be coming from the principles of Lean manufacturing which aims to eliminate waste and by using more reactive demand based communications between the start and finish of an operation “pulls” through the supply of inputs (i.e. information, software components, functionality, content) as required by the next stage of a given operational function.

On the other hand no one can dispute that software development has benefited greatly by it’s departure from Waterfall principles to the widespread the use of Agile. The problem with waterfall is that whilst there is considerable care and effort placed in gathering and documenting requirements these requirements are ultimately; incomplete, insufficient or ill-conceived by the time they reach the development stages. Agile has the ability to make corrections on the fly, encourages mid stream corrections, customer intervention and tests outcomes at the end of short intervals. With agile you build the smallest possible components, and progressively assemble an MVP from the inside out.

Agile is of particular benefit in creative operations such as software, digital media and design, but it still has its drawbacks especially its inability to facilitate working with exact budgets or time frames. As a result experienced agile practitioners are adopting principles that seem more Lean than Agile in origin. Some Agile projects must also allow deviation from core principles where smaller items of development are impossible to test until combined with other components to form a sub assembly or MVP which could be interpreted as the Lean principle of converging streams.

In Agile there is the principle of rapid feedback between the users providing the “information” which in Lean manufacturing is the raw materials, to the transformation process (expert that transforms this into a product in some form), and instead of there being requirements (inventory in manufacturing terminology) there are story boards (orders for a finished product or desired outcome).

Agile is essential in creative environments as it gives the skilled expert the opportunity to use their knowledge and intuition to balance “what is technically possible” with “what the customer really wants”.

Hall et al “Harvard Business Review” (2009, Vol 87) reminds us ‘creative activities are often described as “judgment-based” work,” “craft work,” or “professional work.” The common thread in such work is variability in the process, its inputs, and its outputs.’ and in this “Highly variable environment: where attempting to use structured process management to blindly reduce variability. Not only does that reduce accountability, but it often causes workers to switch to autopilot instead of trying to understand the specifics of each job.” we must remember that the “variations of output is what the customer sees as value”
“creative processes can and should reliably produce innovative products and services that many structured [Waterfall style] processes cannot mimic.”

In large teams Agile has to give way to Lean principles ability to coordinate the work of different teams (streams) working either sequentially or in parallel. The Kanban (a lean principle) is widely used for managing Agile iterations.
Since some iterations are completed faster than others, in larger projects there still needs to be multiple points of consolidation or synchronization.

Many larger technology based cloud service providers are adopting Devops practices (a convergence of agile system administration with agile operations)  to reduce time to production increments for new capabilities, where agile has been adapted to deliver finished goods in smaller increments directly to the production systems accessed by customers. This in itself is giving rise to new operational capabilities best described as Lean principles. The Lean continuous improvement practice of Kaizen in conjunction with Agile and Devops teaches the teams to seek out improvements to both the product and the process used to produce the product.

Project Methodologies and Principles

Recently I have come across a number of people grappling with the choice of methodologies used within their organization, and keen to debate the pros/cons of their existing methodologies.

Selecting methodologies is a tricky task, and if the organization has established procedures thinking through the adoption of new techniques and their impact on policy and process is not trivial.

When it comes to selecting an appropriate one, there are a few dozens of factors you should consider,  management methodology carries its own strengths and weaknesses.

Project Principles and Methodologies

The following are the most frequently used management methodologies in the project management practice:

1 – Adaptive Project Framework

Agile software development methodology is for a project that needs extreme agility in requirements. The key features of agile are its short-termed delivery cycles (sprints), agile requirements, dynamic team culture, less restrictive project control and emphasis on real-time communication.

3 – Crystal Methods

In crystal method, the project processes are given a low priority. Instead of the processes, this method focuses more on team communication, team member skills, people and interaction. Crystal methods come under agile category.

4 – Dynamic Systems Development Model (DSDM)

This is the successor of Rapid Application Development (RAD) methodology. This is also a subset of agile software development methodology and boasts about the training and documents support this methodology has. This method emphasizes more on the active user involvement during the project life cycle.

5 – Extreme Programming (XP)

Lowering the cost of requirement changes is the main objective of extreme programming. XP emphasizes on fine scale feedback, continuous process, shared understanding and programmer welfare. In XP, there is no detailed requirements specification or software architecture built.

6 – Feature Driven Development (FDD)

This methodology is more focused on simple and well-defined processes, short iterative and feature driven delivery cycles. All the planning and execution in this project type take place based on the features.

7 – Information Technology Infrastructure Library (ITIL)

This methodology is a collection of best practices in project management. ITIL covers a broad aspect of project management which starts from the organizational management level.

8 – Joint Application Development (JAD)

Involving the client from the early stages with the project tasks is emphasized by this methodology. The project team and the client hold JAD sessions collaboratively in order to get the contribution from the client. These JAD sessions take place during the entire project life cycle.

9 – Lean Development (LD)

Lean development focuses on developing change-tolerance software. In this method, satisfying the customer comes as the highest priority. The team is motivated to provide the highest value for the money paid by the customer.

10 – PRINCE2

PRINCE2 takes a process-based approach to project management. This methodology is based on eight high-level processes.

11 – Rapid Application Development (RAD)

This methodology focuses on developing products faster with higher quality. When it comes to gathering requirements, it uses the workshop method. Prototyping is used for getting clear requirements and re-use the software components to accelerate the development timelines.

In this method, all types of internal communications are considered informal.

12 – Rational Unified Process (RUP)

RUP tries to capture all the positive aspects of modern software development methodologies and offer them in one package. This is one of the first project management methodologies that suggested an iterative approach to software development.

13 – Scrum

This is an agile methodology. The main goal of this methodology is to improve team productivity dramatically by removing every possible burden. Scrum projects are managed by a Scrum master.

14 – Spiral

Spiral methodology is the extended waterfall model with prototyping. This method is used instead of using the waterfall model for large projects.

15 – Systems Development Life Cycle (SDLC)

This is a conceptual model used in software development projects. In this method, there is a possibility of combining two or more project management methodologies for the best outcome. SDLC also heavily emphasizes on the use of documentation and has strict guidelines on it.

16 – Waterfall (Traditional)

This is the legacy model for software development projects. This methodology has been in practice for decades before the new methodologies were introduced. In this model, development lifecycle has fixed phases and linear timelines. This model is not capable of addressing the challenges in the modern software development domain.

Choosing the best methodologies

The first point to remember is that you don’t need to choose just one. LEAD for example is more a principle used in conjunction with other methods. TQM (total quality management) whilst not methodology itself is an approach to quality that you would weave into the framework of the organization.

To help you choose the best fit methodologies and principles here are some considerations to start with:

  • If your project a major project involving multiple stakeholders across departments or organizations then you will need a project manager with PMBOK/PRINCE2 experience to run the project.
  • Is the project a major systems implementation for a customer, and whilst PMBOK mostly applies, some of the principles of your integration plan are a duplication of contract terms, and in these cases a modified PMBOK is needed.
  • Is your organization delivering standard applications using Microsoft SureStep and need the structure and process management of PMBOK, if so then use both.
  • Does your organization implement a fixed set of products in its projects, and the strict use of PMBOK, preparing details plans at each phase, that seem to be a duplication of the plans from past projects, then you should convert PMBOK steps to procedures and check lists (Project Controls). You can create simple meeting check lists or use a Workflow or BPM product to encapsulate these into organizational procedures or email prompts to team members. This will give your projects a greater predictability of outcome and reduced risk.

    Sample outcomes from project phases
    Sample outcomes from project phases
  • If the project is for the internal development of a product feature from your product road map that will be added to the core product offering, but because the teams are working multiple projects, work can stop/start and you will need to have designs produced in specifications so the team can pick up where they left off when returning to work on the project, so a modified Waterfall / Agile works in these cases.
  • If the design of the product is evolving and there is need for short bursts design/build sprints to develop the end product, then a small team (2 or 3) using Agile methodologies will work best.
  • If running a major business application development involving multiple teams and sub-projects (some of which may use Agile within that sub project) then Spiral would help to run the Umbrella project, with different techniques used in sub projects and tasks within the WBS.
  • When developing software for resale, the developers and team members can tend to develop functions that they consider will be important, and loose touch with what the customer considers to be of value, in these case using LEAD principles in the projects is important to ensure the product is quick to market, provides sponsors with a ROI and is continuously improving.

The point being there are “horses for courses” and no reason why you cant use multiple methods within an organisation, what is important is making the choice, setting up the procedures and communicating it well.

More helpful explanations:

Diploma of Project Management (BSB51413)

Recently I started working with Lou Marks of the Institute of management who provided a consolidated list of the subjects within this Diploma and by setting up a spreadsheet using the training elements provided by the Australian governments training website, I was able to map steps with our current Project management methodology to the elements of competency within this Diploma course.

Why is that important .?

As a supplier its important to our customers that we provide PM services using the best available PMBOK methodologies available to us at a given point in time, and to periodically compare your companies procedures to what is at that time “best practice” is a healthy discipline and self check.

Secondly PM courses are provided as a basis of knowledge for people who will then apply that knowledge in many different fields, and having a direct comparison of the course outline to the procedures used by a software company performing software project implementations is of benefit when recruiting and inducting new project managers brought into the organisation periodically.


Part 1

Part 2