Product Development Best Practices

Guy Kawasaki on product development ... keep it dicee. ...

PMO Success Metrics: Proceed With Caution

Based on Benjamin Disraeli's well-known statement about the three types of lies, "lies, damned lies, and statistics," Jeannette Cabanis-Brewin wrote an interesting article in Developer.com about the many faulty assumptions people make based on so-called statistics about PMOs.

Cabanis-Brewin is editor-in-chief for PM Solutions' Center for Business Practices, so she's seen her share of statistics. Some key points:

- There are many interpretations on what constitutes a PMO, so many statistics on PMOs are skewed from the start.

- Some reports indicated high project failure rates in organizations with PMOs. That's likely due to the fact that organizations without PMOs don't tend to measure project success. Without measures, there are no failures.

- Many surveys are poorly designed and miss crucial clarifying questions.

Cabanis-Brewin recommends going to the source and digging into the supporting details to draw your own conclusions. She also reminds us that surveys usually do not contain the definitive answer, but rather serve as a starting point for more research. Finally, she cautions us to beware of the Hawthorne Effect, which states that the act of observing often changes the observed.

I would add that it's also important to be careful what you ask for. Many organizations want to begin using metrics, but are surprised to see success rates so low. They pressure project managers too soon and expect success rates to instantly soar to above 90%. It's vital to give the organization time to address problem areas and develop maturity.

Yes, it's important to capture metrics, but it's equally important to create a blameless reporting environment, by which people will report accurate data without fear of retribution. It's also critical to think about how you measure success. True success doesn't always correspond to on-time and on-budget. But that's another story.

Here's the article...

Lies, Statistics, and the PMO

Project Forecasting and Uncertainty

Yesterday, I posted about the need to focus on time remaining, using a driving analogy. The premise was that, if we were traveling from Philadelphia to New York and we were at the New Jersey Turnpike entrance (and assuming arriving on time is a critical success factor), the best predictor of success would be to determine how much time remains on our trip.

But what if arriving on time were not the most pressing need? And what if we increased the uncertainty factor? Is "time remaining" still something we can and should monitor?

Let's take IT research projects, for example, which tend to carry great uncertainty. Using an agile approach, the team builds prototypes, and the deliverables get ever closer to "the truth" as the project progresses. If we used a fixed set of iterations, and/or fixed-time iterations, we still benefit from focusing on time remaining, possibly even more so. The only difference is, we're operating in increments.

And even if time is not the ultimate concern, any customer will still want to know "How long will it take before I see something of value?", and usually, "How much will it cost me?" These are the basic fundamentals of good customer service. We still need objectives and scope statements, even on agile projects. It's just that we recognize uncertainty more.

And speaking of uncertainty, proper preliminary research and planning can often help reduce it, and contingencies or buffers can help prepare for the unexpected. Piecemeal deliverables (iterations) can also help, as learnings can be applied to the next phase.

Project Forecasting: More Lessons from Driving

A while ago, I entered a post about the importance of staying tuned in, drawing an analogy to driving. Well, another driving analogy had occured to me, this time about the need to focus on remaining time.

Let's put it this way. If you're driving from Philadelphia to New York City and you're at the entrance to the New Jersey Turnpike, what percent complete are you on your trip?

Some of you may guess certain percentages based on distance, but that's as foolish as basing project percent complete on the percent of budget or time that's been spent, without regard for work accomplished.

The quick answer is: Who cares what percent complete we are? What we really should be concerned with is how much time is left, assuming we care about what time we arrive to begin with.

But let's say that we DO care (i.e. schedule is a priority for us, as opposed to some other success factor). How can we measure whether we'll be there on time?

Simply using a percent complete tells us nothing. It's too subjective. What we need to know how much time is remaining. And that will depend on how fast you're going, how many miles are left, what barriers may arise (i.e. road closings, flat tires, etc.), how many stops you make, and a number of other variables. It's no different for projects.

For project schedule control, capturing percent complete is too theoretical, so that's not of much use to us. And capturing time spent tells us very little, except perhaps how long it took us to do prior work, which may not be an accurate indicator of future work. Besides, we can probably determine future work estimates more accurately through expert opinion and/or statistical sampling (combined with good planning).

Of course, there's no harm in entering time spent as long as people are disciplined to always include time remaining. Then a percent-complete can be calculated based on that. But the percent-complete itself is not a leading indicator, so is still of questionable value.

If we focus instead on time remaining at the task level, and combine that with barrier removal, risk planning, and regular reforecasts, we'd have much better control over whether we "arrive on time."

We can improve our ability to estimate in the future by capturing lessons learned, doing spot checks, and using the information to create project schedule templates and checklists, so future projects can avoid running over the same potholes.

Some may say, "Oh, we still need the percent-complete for Earned Value calculations."

Do we really? By putting a dollar amount to the time remaining, we can solve the same problem in a simpler fashion, answering the question: How much is it going to cost us to complete this project and what's our estimated time to arrival?

Just some food for thought. See my followup post on Project Forecasting and Uncertainty as well.

Project Success Measures: What Really Counts?

Finally, people are starting to say what I've been saying for some time. Often , the most telling measures of project success are qualitative, not quantitative. Consider this quote from an excellent article on PMI's latest issue of PMP Passport:

“Measures like on time, on budget and on spec are part of the success criteria, but one could achieve these three and still have a dissatisfied customer,” says Ernie Baker, PMP, president of Start to Finish PM Inc., Verona, N.J., USA."

Halleluyah! Indeed, qualitative measures such as customer satisfaction, customer engagement, and team engagement can often tell us more about the "success" of the project, and can even shed light on why the project wasn't on budget, schedule, or spec.

Here's the article...

PMP Passport - Features

The Enemy of Simplicity: The Thud Factor

We've all heard about the benefits of simplicity, whether in our processes, our communication, or in our objectives. In all its forms, simplicity is a way to reduce confusion, boost morale, and encourage speed and flexibility. In fact, simplicity, speed, and flexibility are three of the "Six Winning Principles" I wrote about in Napoleon on Project Management (the other three being exactitude, character, and moral force).

But there's a lurking enemy of simplicity, and it often goes unnoticed. It can be found in the motives of individuals creating the processes, communications, or objectives. I'm talking about job protection. I don't mean the blatant kind that results from grandiose thinking, egotism or turf wars. It's much more subtle than that.

It can happen if an individual or department is placed in charge of creating a process or devising a plan. Or it can happen if a consulting company is brought in to do a study or offer advice. Common sense says that these people, while not necessarily devious, will hesitate to come up with anything too simple, lest they feel they're not doing their job. The result is often something that is way more detailed, complex, and expensive than it needs to be.

What can we do about it? We need to be very aware of motives and rewards, and make sure we don't consiously or unconciously reward people for complexity. We need to send a message that the shortest, simplest way to meet the goal wins (even offering incentives if possible). This can avoid what many consultants jokingly refer to as "the thud factor"----the customer's perception of the value of the service as judged by how much of a noise the report makes when it's dropped on their desk.

Whether it's a consulting company, a PMO, an internal process center, or a project team, we need to find a way to head off the thud factor and insure simplicity. We can do this by understanding motives; sending the right message; insisting on brief, simple reports; and creating the right reward system.

Just the Facts: Evidence-Based Management

I recently read an enlightening book by Jeffrey Pfeffer and Robert I. Sutton, titled, Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management.

The premise of the book is that many organizations follow the guru du jour, or manage according to the book of "someone said so." As the book points out, if we only looked at the evidence, we'd see that may of these so-called truths are anything but.

Here are some examples of the lessons the book has to offer, always supported by evidence:

1) Forced ranking of employees doesn’t work, especially where people’s performance depends on interdependence with others. Furthermore, a survey of over 200 HR professionals by the Novations Group found that forced ranking (employed by more than half of the companies) resulted in lower productivity, injustice, skepticism, less employee engagement, reduced collaboration, lower morale, and mistrust in leadership.

The authors add that, if an organization trains people right and places them in an effective system, there’s no reason why 10 or 20 percent would automatically become incompetent every year.

2) Beware of your biases as a manager. Studies of NBA drafts showed that players picked earlier and paid more were less likely to be traded and had longer careers, regardless of their actual performance.

3) In the war for talent, don't forget that bad systems cause far more damage than bad people. Try redesigning systems and jobs before judging individuals. And don’t give people objectives unless the system and staffing can support it.

4) Watch out for dangerous incentives. One organization's salespeople shipped too far ahead of schedule just to win a prize. Some salespeople would hold customer returns in the trunk of their car so they still get their commission for that period. Others opened bad credit accounts because any order counted as a good order. In another company, incentives to complete truck routes early led to increased accidents and overloading of trucks to avoid multiple trips.

5) Strategy isn’t all it’s cracked up to be. Operational execution often has a greater impact on performance. The CEO of Wells Fargo once said, “I could leave our strategic plan on a plane and it wouldn’t make any difference. No one could execute it.” In U.S. football, virtually every play is designed to go for a touchdown. Unfortunately, reality gets in the way, as do mistakes in execution.

The authors point out that time spent pursuing strategic options could be better spent solving operational problems or focusing on customer needs. Organizations such as eBay and Intel use a “learn as you go” approach, putting something in the market and tweaking accordingly. Doing the right things is important, but not at the expense of doing them effectively.

6) Many changes, including mergers and acquisitions, ERP implementations, Six Sigma programs, Business Process Reengineering, cost cutting initiatives, and others, carry risks that outweigh the benefits and can be easily misapplied. People tend to underestimate the costs and overestimate the gains.

However, if it is determined that the change is still needed, the authors suggest we:

a) Ensure dissatisfaction with the status quo (i.e. the burning platform)
b) Communicate the same message repeatedly about the need for the change
c) Express extreme confidence in the change, but listen to concerns and adjust accordingly
d) Expect setbacks, errors, and miscommunication; Learn from it and revise processes. Never point fingers.

7) Based on proven evidence, in order to gain respect and trust, leaders should:

a) Act "as-if" - Be sure to act and talk like a leader
b) Have some sense of modesty. Understand the difference between knowledge (knowing things) and wisdom (knowing what you know and knowing what you don’t know).
c) Know when to get out of the way.
d) Above all, be an architect of systems, teams, and cultures.

These are but a few of the valuable nuggets in the book. The book offers additional tips as well, plus loads of supporting stories, examples, and research. Perhaps most valuable is the chart on the various types of changes and risks associated with them. I highly recommend this book to all leaders.

Project Management: Definition of Success

Is cost performance or estimate accuracy the measure of success for your project? What about usage, adoption, or percent of target value achieved? ...

... "when we talk about a project failing when it cost more than somebody said it would cost, or takes twice as long, how can we be sure those estimates were anywhere near the right ball park in the first place? " ...

Via CIO Austalia: Define Success