I’m not sure, but I’m told some famous guy back in Biblical liturgy was once asked to explain the point of the Pentateuch (aka, the Torah, aka, The Five Books of Moses) while "standing on one leg".  

I now undertake a task, possibly no less daunting, regarding CMMI.  And, if there ever were anyone more appropriate to try it, I doubt I’ve met them.

Seriously though, much has been written here and many other places (not to mention eons of conference and user group content) about a number of "universal truths" about CMMI.  Let’s get these out there first, but without dwelling on them:

• There are no "processes" in CMMI, only practices, and there’s a difference.
• The practices in CMMI are "what" but not "how".
• These practices are use to improve your processes, not to define them.
• The CMMI does not require the SCAMPI appraisal to be effective.  You can use CMMI to improve your operation without ever using the SCAMPI to appraise your use of CMMI.
• 42.  OK.  Not really.

However, not a single one of these "truths" explain the point of CMMI, or,  how to actually use CMMI.  So, here it goes:

Each one of the practices in CMMI improves some aspect of your organization’s performance resulting from how you do your work.  It doesn’t matter whether it’s providing a service or developing a product.  And, it doesn’t matter whether you do so using so-called traditional development methods or Agile approaches.  If you have performance issues in an area of your operation (called, "Process Areas" in CMMI), Check each of the practices in that area for activities in your operation that might be causing those performance issues. 

It’s assumed, then, if you don’t have any issues covered by a practice then you don’t need to do anything about a practice, because you’re already doing it.  This says nothing of how well you do it, why you do it, how you do it, whether you recognize that you do it, or whether the fact that you do it is a complete coincidental freak of nature, but, if you read a practice, understand the risk it avoids, and you don’t encounter that risk, you’re somehow performing that practice.  Pretty simple.

I’ll repeat and summarize that two-step thought experiment:

1. Look in the process areas for practices that address performance issues you’re experiencing with the operation of your work.  When you encounter a practice (or more than one), the absence of which can explain why you’re seeing those issues, make appropriate changes to your operation so that you incorporate that/those practice(s) into your operation.  Rinse and repeat.
2. Practices that don’t represent risks or issues you’re not seeing are (pretty much, by definition) practices you’re somehow managing to accomplish.  Don’t bother with them — unless you notice that you don’t like something about how you do it, but that’s a different priority/matter.

Keep in mind, this says nothing of

• whether what you do/don’t do will suffice as "evidence" for an appraisal
• how well you perform the practices (regardless of whether or not you perform them or believe you can use them to improve),
• what it takes to incorporate practices or make change, in general, happen in your operation,
• whether an appraisal team will concur with whether you do/don’t perform practices, or
• you interpret practices in constructive ways.

Nonetheless, if you internalize the significance of the above 2 steps, you can (I dare say, "will") save yourselves a lot of time and grief when using CMMI.  This approach can certainly help you prioritize the practices for which to focus on, appraisal or not.  And, if you do take this approach towards preparation for an appraisal, keep in mind the bulleted caveats and don’t try this alone.

Say you’re in a truly disciplined, lean and agile operation and your processes are so deeply ingrained in what you do that putting your finger on tangible evidence is a challenge, and not for lack of process.  Just lack of being able to step back far enough from the canvas to see the whole picture.  What do you when it comes to demonstrating your practices for a CMMI appraisal, for example?

Well… the best advice I can give companies in such situations is to work early and closely with a consultant and/or lead appraiser to elicit the best evidence for the appraisal long before the appraisal event itself is planned or carried out.  It’s important to be clear about what the evidence is, and, you want the appraiser and the appraisal team on board with how the evidence will “show up”.  This is not something you want to surprise anyone with come appraisal time.

Working early and closely with a lead appraiser will not only help everyone understand the context, and not only will it provide an opportunity to strengthen practices and identify operational risks, but it will give you a good idea about whether or not the lead appraiser has the wherewithal to think broadly about practices and to assemble the contextual picture for how practices would “show up” in the context of your operation.

Sadly, not all appraisers have this skill set.  In fact, in my experience, the great majority do not have the skills to make contextually relevant model interpretation such that actual, naturally-occurring evidence from an operation can take its most natural form and still be recognized as implementation of CMMI practices.  In my experience, most lead appraisers expect evidence to come in very specific shapes, sizes, and colors and they don’t recognize the evidence when it doesn’t meet their pre-conceived notions of what particular evidence should look like. 

That being said, this does not give carte blanche for not having evidence.  I’m not saying that the evidence isn’t there, I’m just saying that the evidence may not be what’s traditionally thought-of as evidence from larger or more traditional development operations.

Process evidence from operations whose processes are deeply ingrained can often show up as very clear, obvious artifacts.  Especially from traditional development operations.  However, in small, lean, and agile operations, the evidence can be much less obvious.  It is a special skill set to be able to recognize the outputs of such operations as evidence of CMMI practices and organizations are served well to work with the lead appraiser early to determine whether or not their operation produces evidence as well as whether or not the appraiser can see more broadly than the evidence they’re used to seeing from traditional operations.

Since few organizations know how to pick a lead appraiser, perhaps this “litmus test” for a lead appraiser can serve to help them through the process.  The alternative could be a disastrous paper-chase to create evidence on top of the evidence that’s already there.

A friend who consults in program, project and risk management (typically to parachute-in and save wayward technology projects) is working with a client whose project is dreadfully behind schedule and over budget, and, not surprisingly, has yet to deliver anything the end-client or their users can put their hands on.  It doesn’t help that his client isn’t actually known for being technology heroes.  In fact, this is not the first time his client has tried to get this project off the ground.

Looking everywhere but in the mirror, my buddy’s client decided to have the developer put under a microscope.  After all, reasoned the client, they hired the developer on, among other attributes, touts that they were rated at CMMI Maturity Level 3!  So, they had the developer and the product undergo a series of evaluations (read: witch hunts) including a SCAMPI (CMMI) appraisal.  Sadly, this tactic isn’t unusual.

Afterwards, trying to help his client make sense of the results, my pal asked me to review the report of the appraisal which was fairly and (more or less) accurately performed by someone else (not us).  The appraisal was quite detailed and revealed something very interesting.

Lo-and-behold, the company had processes!

However, the development organization nonetheless failed to demonstrate the necessary performance of the Maturity Level 3 (ML3) practices they were claiming they operated with!  In other words, they had processes, but they were still not ML3!  In fact, they weren’t even Maturity Level 2 (ML2)!

How could this be?

While the details bore some very acute issues, what was more interesting were the general observations easily discernable from far away and with little additional digging.  The appraisal company created a colorful chart depicting the performance of each of the practices in all of ML3.  And, as I noted, there were important practices in particular areas with issues that would have precluded the achievement of ML2 or ML3; but, what was more interesting were the practices that were consistently poor, in all areas as well as the practices that were consistently strong in all areas.

One thing was very obvious: the organization, did, in fact, have many processes.  Most of the processes one would expect to see from a CMMI ML3 operation.  And, according to the report, they even had tangible examples of planning and using their practices.

What could possibly be going on here?

Seems awfully much like the development group had and used processes.  How could they not rate better than Maturity Level 1 (ML1)?!  Setting aside the specific gaps in some practices that would have sunk their ability to demonstrate anything higher than ML1 – because this isn’t where the interesting stuff shows up, and, because even were these practices performed, they still would have rated under ML2 – what the report’s colorful depiction communicated was something far harder to address than specific gaps.  The developers’ organization was using CMMI incorrectly.  A topic I cover at least in the following posts: here and here.

In particular, they were using CMMI to “comply” with their processes but not to improve their processes.  And, *that* is what caused them to fall far short of their acclaimed CMMI ML3.

How could I tell?

Because of where the practices were consistently good and where they were consistently gap-worthy.

I was reviewing the report with my friend on the phone.  As I was doing so he commented, “Wow!  You’re reading that table like a radiologist reads an X-ray!  That’s very cool!”  The story the chart told me was that despite having processes, and policies, and managing requirements and so on, the company habitually failed to:

track and measure the execution of their processes to ensure that the processes actually were followed-through as expected from a time and resource perspective,

objectively evaluate that the processes were being followed, were working, and were producing the expected results, and

perform retrospectives on what they could learn from the measurements (they weren’t taking) and evaluations (they weren’t doing) of the processes they used.

It was quite clear.

So, here’s the point of today’s post… it’s a crystal clear example of why CMMI is not about process compliance and how it shows up.  There are practices in CMMI that definitely help an organization perform better.  But, if the practices that are there to ensure that the processes are working and the lessons are being learned aren’t performed, then the entire point to following a process has been lost.  In Shewart’s cycle, this would be akin to doing P & D without C & A.

The only chance of anything that way is compliance.  There’s no chance for improvement that way except by accident. 

CMMI is not about “improvement by accident”.  (Neither is Agile for that matter.)

Interestingly enough, while there were clearly issues with the developer’s commitment to improvement, there may not necessarily have been any clear issues with either the product or the results of their processes.  While the experience may not have been pleasant for the developer, I don’t know that by buddy’s client can say to have found a smoking gun in their supplier’s hands.  Maybe what the client needs is a dose of improving how they buy technology services – which they might find in CMMI for Acquisition.

Sorry, folks, no fun (or not-so-fun as you may prefer) video today.  Not even any pictures I took at SEPG.  In fact, as far as today went, I don’t have much to report from the sessions.

Again, I missed the plenary session.  This time on account of a phone meeting with a client in another time zone.  So, my first session to attend was the other of my two collaborative efforts with Judah Mogilensky on SCAMPI Evidence from Agile Projects. As anything with Judah in it, it went rather nicely.  Many generous bits of feedback.  I felt really good about my role, and Judah was his usual incomparable self.

My friend and colleague, Eileen Forrester of the SEI was kind enough to give me some supremely powerful feedback.  I am, and will be, grateful for it.  I was then roped into shop talk about CMMI for Services in advance of the 2nd half of the orientation workshop I’m helping her with.  Thus, my missing out on my buddy, Jeff Dalton’s, excellent (so I’m told from many reports) job with Encapsulated Process Objects.

One point made to me later by another of the few “agile-friendly” lead appraisers, Neil Potter, about a bit of content in the presentation does require some follow-up.  In the presentation we short-cutted the details on a discussion regarding the potential design aspects of test-driven development with an engineering design.  I should say that TDD is NOT the same as a design, but that depending on how TDD is planned and performed, it can include design-like attributes which could accomplish design expectations in the engineering process areas of CMMI-DEV.  So don’t anyone out there go around blabbing some “Hillel said TDD is Design!” crap.  Mm’K?

After lunch, my job was to keep people from falling asleep with a session on Love and Marriage: CMMI and Agile Need Each Other.  From the response, I think it went went rather well.  I, personally, was quite pleased with how it came off from a “talk per slide” metric.  A good friend, Tami Zemel, later admitted that she “takes back” her earlier criticism of Monday’s presentation.  She said it had too many words and didn’t believe me when I told her why.  She complemented not only the picture-centricity of today’s pitch but also the delivery, style, and content.  That was very generous, thank you.

From then to the end of the day, I spent scheming, strategizing, shmoozing, and networking with too many people to mention.  (No offense.)  A client who came to the conference (who never holds back and only inflates the truth when it’s funny to do so) got very serious when a prospect I’d recently met off-the-cuff asked whether he’d recommend me.  I won’t repeat his answer because it really was just crazy nice.  Today’s interesting photo is in his honor.  (And also because my boys love transportation.)

The last “session” was a Peer 2 Peer double-header on the topic I mentioned on Monday which I co-created with Michele Moss.  She and I are also on the SEPG Conference Program Committee.  We used the feedback and other data from the Peer 2 Peer as input to a retrospective on this year’s conference, which will be used for strategies for next year’s conference in Portland, OR.

You can also read an entry I gave to the SEI for their official blog about my impressions of this year’s conference-goers.

Dinner conversation back at the hotel with Michele was back on the subject of our Peer 2 Peer session.  Net result: We single-handedly wrote the 1-3-5 year plan for all SEPG’s.  Or at least we think so. 

Imagine what would happen if you went to a doctor (or any specialist) who had no experience in your specificcondition or situation.  Has this every happened to you?  It has to my family when I was young.  Let me tell you, it wasn’t pleasant.  What was frightening was that the “professional” didn’t know that they didn’t have the right experience.  What was just as bad was that my family didn’t have the knowledge or experience to know that the person we went to was not qualified.

This is a situation encountered by many organizations when seeking advice and/or appraisal services from a CMMI consultant / appraiser.  However, in business, you should at least know enough about your organization and ways of operating to do your homework before picking someone to help you with CMMI.

What you may not have known is that CMMI and the appraisal method are not as clear and obvious as other means of performance evaluation and that you must choose your consulting firm and appraiser very carefully, and among other factors, consider their contextually relevant experience. . . .

In this quote, CAPT Kirk wants Dr. Bones McCoy to do something he feels he’s not-qualified to do because he doesn’t know how to treat the species.

I’m using it to explain that organizations looking for a lead appraiser to work with them towards an appraisal and/or to perform an appraisal ought to think of what we do as they would think of a doctor, not a laborer or vendor.

Do you really want the lowest price doctor?

For that matter, is the highest price doctor necessarily the best in town?

When reaching out and interviewing for a lead appraiser or CMMI consultant, you:

• Want the person who is the right person for the job.
• Want someone who is qualified (definitely not under-, but preferably not over- either).
• Not the lowest bid.

Seriously, whoever you hire for this effort has in their power the ability to make or break your future.  They literally have the health and well-being of your organization in their hands.  They can put you in the dump just as easily as they can take you to the next level.

They should see themselves that way as well.

Unfortunately I’ve got too many sad stories of appraisers/consultants who definitely see that they can make or break you, but they don’t feel like they personally own the responsibility for what happens to you when they’re done.

If it costs too much?  So what?
If you get no value?  Not their problem.
Didn’t see any benefit?  Didn’t learn anything?  Things take longer and cost more and you’re not seeing internal efficiencies improve?
YOU must be doing something wrong, not them.

In an AgileCMMI approach, your CMMI consultant and/or lead appraiser would see themselves as and act like a coach, and would put lean processes and business value ahead of anything else.  And, an AgileCMMI approach would know that when the processes work, they add value; when they add value people like them and use them; when people like and use them, the next “level” is a big no-brainer-nothing.  You get it in your sleep.

Let me know if you want help finding the right lead appraiser or consultant.

Your people with prior CMM/CMMI experience are probably worse than worthless, they’ll probably cause you to fail.

Why?

Because what they (or you) think they (or you) know is probably wrong and the advice you’re getting, the expectations being generated are entirely off base.

It all goes back to the many ways in which CMMI can be done poorly and the few, simple, but hard work ways in which it can be done correctly.

Every time I meet with a new prospect I’m confronted with reams of inaccurate assumptions and assertions about what it will take to implement CMMI and how am I expected to “do all that” and still claim to be “agile”.

My simple answer: I’m not going to do all that.  And, you shouldn’t be doing it either.

Seriously, you’ve got to wonder about executives who will force their company into doing stupid things for the sake of a rating instead of doing their homework to learn about CMMI before they head out on an implementation journey.

A recent client didn’t know any better.  They hired a consultant and an appraiser to evaluate their work against CMMI and to help them prepare for a SCAMPI appraisal.  Unfortunately, they got as far as the appraisal only to realize they weren’t going to get the target Maturity Level.  (I won’t get into some of the inappropriate behavior of the firm they hired.)

However, when this client was confronted with:

1. Do something stupid, or
2. Find a better way to do something smart.

They took option B and found a consultant and an appraiser who understood their context and found how to both be on a disciplined improvement path while also remaining true to their own business.

Fortunately for them, this client had a strong engineering backbone and knew what they did worked and were confident in their processes.  Many companies have a while before they can claim that much.

Next week:

Picking a Lead Appraiser:  “Dammit, Jim!  I’m a doctor not a bricklayer.”

What most people (80/20) seem to “know” about CMMI and the SCAMPI appraisal method comes from what people learned and how they used CMM and CMMI in the early adoption phase.

However, instead of innovating and using engineering to create appropriate processes, they just reused old and often poorly-fitting processes and approaches to situations they never dreamed of in the 1980s.

Even people with positive experiences with CMM/CMMI tell us that we challenge what they once believed to be “true” of CMMI … but that they’re relieved because many always felt that what they thought was “true” made little sense.

Recommend:

• Entinex CMMI Crash Course™ (Slides, Description)
• CMMI FAQ

Next week:

Your people with prior CMM/CMMI experience are probably worse than worthless, they’ll probably cause you to fail.

10. Four months ago you couldn’t spell “CMMI”.

9. No one in your organization has ever had any training of any kind whatsoever in CMMI, appraisal planning, or process improvement.

8. You haven’t worked with anyone (in-house or hired) who knows what they’re doing with CMMI.

7. Price-shopping for lead appraisers seems like a good idea. Kind-a like price shopping for a heart-surgeon.

6. After months of work, you switch from one constellation to another and think your appraisal is still on schedule (see #9 and #8). Kind-a like switching your team from field hockey to ice hockey mid-season.

5. Your CEO is petulant about the delay of your appraisal yet has no idea that your actual process performance is in the toilet.

4. You’ve waited until it’s time to start planning for your SCAMPI to start looking for a consultant to help you implement CMMI in an “agile” way.

3. The only people you’re sending to Introduction to CMMI are the ones you plan to have on the appraisal. You have no back-up plan if they can’t make it to training and/or to the appraisal. Class isn’t for another month, and, they’re the same ones who’ve been working on your processes for the past several months, but until now, see #9 and #8.

2. You haven’t qualified the people in #3 with your lead appraiser (which you haven’t hired yet + see #9 and #8), you haven’t qualified the projects to be appraised with your lead appraiser (which you haven’t hired yet + see #9 and #8), and nonetheless, you have established a level of effort for the appraisal despite all of the above.

And,

The Number One Clue You’re NOT Ready for a SCAMPI:

1. After months of work, you still don’t see there’s a fundamental flaw in committing to or expecting others to commit to (including your appraiser no less!) a firm-fixed price contract without knowing the requirements.

I wish the above list of clues were tongue-in-cheek.

If you look at some of the other behaviors of organizations not really ready for a SCAMPI, you’ll find that they continue to:

• accept more work than they can handle,
• be unpredictable in what they will deliver and when,
• measure little other than billable hours, and
• have no insight into where their defects come from.

Despite the condition in which many organizations may have artifacts for an appraisal, they have seen no intrinsic benefits to their new set of processes. They’ve been just “chasing a level”, which results in lots of work for no real benefit. You can guarantee these organizations will drop their processes shortly after the appraisal.

Really, what these clues point to is the dreadful lack of realization that first and foremost, process improvement requires the right culture for process excellence. The above list points to a fundamental absence of the culture for process excellence. What people don’t realize (especially in the USA), is that process improvement is a total JOKE for companies with the right culture of process excellence. If people want the truly idiot-proof, guaranteed, easy path to just about any CMMI Maturity Level, they would need to go no further than to foster a culture of process excellence, then whatever they did from that point forward would likely cause just about every practice in CMMI to form on its own. In other words, these clues aren’t about process, they’re about culture and leadership. You don’t have to know anything about CMMI to pick up clues about culture and leadership.

One of the failures of CMMI is that it fails to press the basic importance of culture and leadership for process improvement. It fails to communicate in no uncertain terms that an absence of the right culture (and what that looks like) and the absence of leadership (and how that shows up) will lead to a failure in process improvement — CMMI or otherwise. I’m not blaming CMMI for not including this information in the text, because much of it is there. What I’m saying is that despite what is there about the importance of culture and leadership, most CMMI users fail to grasp these important points. The text, therefore, fails to communicate this in a way that people will pay attention.

CMMI should come with a warning similar to, “Don’t Try This At Home!” or “Use Only as Directed!”, or “Check with a Qualified Professional Before Beginning Any Process Improvement Program”, or “You Must Be *This* Tall to Use this Book”. Or simply, “Danger Ahead!”. Something to get people’s attention and direct them to some of the fundamentals of any improvement program.

This accusation is not just leveled at garden-variety CMMI adopters. Often, they’re the hapless “stuckees” forced to “make CMMI happen” against all odds. At least there’s ample reason to be sympathetic to their plight. What’s inexcusable are the too-many consultants, instructors and appraisers who are willing to ignore these fundamental requirements-for-success and who are unwilling or unable to posture with prospects and clients in such a way as to impart the importance of culture and leadership to success with CMMI. So, that translates to a pathetic statement about the consulting abilities (and possibly the ethics) of too many people who take money to work with others on CMMI.

Sorry for the rant. But people need to be warned. Attempting a CMMI effort without the requisite culture and leadership attitude may yield a short-term appraisal rating, but will ultimately lead to medium and long-term failure. *THAT* I can guarantee.

This is why I’m such an advocate for agile methods. Agile methods impart some of the basic needs for long-term process success. And, it imparts them at a level of abstraction usable by people who aren’t process experts, yet establishes many of the appropriate culture and leadership traits that so many CMMI-only efforts fail to recognize. Agile values, methods, and practices empower their teams, cause leadership to eliminate obstacles, value the input of customers and practitioners alike, values learning, develops multi-disciplined teams, and most importantly (as far as processes are concerned) promotes lean thinking. While Agile ideas may not change leadership and culture over night, they contain many of the right activities that can eventually win over those whose hearts and minds need winning over. And, with the appropriate use of CMMI, agile ideas can kick-start the motion and direction needed for long-term and ongoing improvement.

I’ll be writing more about why CMMI and Agile need each other for a while.
Stay tuned.