04 May 2009

Reintroducing "engineering" to software.

I've been noticing an interesting "convergence" going on in this corner of the universe:  "engineering" is being re-introduced to the idea of software.  It's fascinating how no sooner do I have the idea to write this blog (while entirely not connected to the Internet), then I re-connected to get an article that's entirely speaking of the same root issue.image

Of course, I'm not saying that engineering had completely left the software universe, though, a strong argument can be made that for the last decade and then some, software has allowed engineering to escape from the premises.  In particular, architecture, analysis, systems thinking, design, hardware and other integration issues, and planned, deliberate, methodical testing have largely been allowed to merely "emerge" from the work that was completed.

Again, I'm not advocating for Big ____ Up-Front as a solution, what I'm pointing out is that many people who embrace agile methods incorporate engineering practices into how they organize and perform their work, but enough don't that it raises issues with agile scalability.

And here's where I *am* expecting to annoy some people...

Programming and Development are NOT the same.

Development is an engineering function.  Developers ought to be using engineering practices in what they do.  Just look at the word "development".  The connotation is that something is "grown" or "evolved".  The denotation of 'development' in the technical sense is that it is done deliberately, not by happenstance.

This idea is where I believe software, in general, not limited to agile practices have short-changed themselves.  Too often, activities that amount to nothing more than programming are called development when no actual engineering is happening.  In other words, programming is allowed to take place without any (or at best without enough) engineering, and therefore what's really happening is the building of something without any/enough forethought about the thing itself that is to be built.  Instead, what happens too often is all the focus is on "staying busy" (albeit on ostensibly priority work), but what is worked on is absent sufficient technical rhyme or reason.

Is this true of all agile development?  NO.  But, it is what happens in many organizations when they don't have sufficient technical leadership.  For what it's worth, many development projects don't need much engineering, and product development is sufficiently described in tasks defined by few people.  So the jump from development to programming is small and fast. 

image However, there are projects (or tasks) of sufficient technical complexity that skipping the engineering and handling such projects/tasks as programming alone is where I believe a space is created for the unfair reputation for agile and its scalability, as well as some of the anti-process bias among agile proponents.  When I read (and sometimes contribute) to agile and non-agile software groups, I'm often struck by the same thought: where's this person coming from?  This is basic engineering!

But that's the matter, isn't it?  Programming isn't development without engineering and too many programmers aren't engineers (not should they be) but are being told to "develop" without given the time or resources or something to do the engineering.  And so what they're really being told to do is "program", not "develop".  Someone, somewhere doesn't see and/or understand that what many projects need are to be engineered.

I think what this points to is a persistent phenomenon plaguing software: it's not being taken seriously as an engineering discipline.  Sometimes by leadership in organizations where software is being worked on, sometimes by programmers and sometimes by customers.  I'm sure there's plenty of blame to spread around, and spreading the blame is both a waste of time and not the point at all.

Programming is to software as assembly is to construction.  Not image everyone swinging the hammer needs to be the civil engineer nor the architect, and not everyone with a nail gun can be the foreman (and no, I'm not likening the skill set of programmers to those of construction workers, and no, I'm not saying construction workers aren't smart....geez).  There has to have been engineering taking place before software can be actually developed, and as evinced by the kinds of challenges I encounter regularly, enough software shops are going about their work absent acknowledgement or awareness or consideration for the engineering that has taken place or has yet to take place (or should have taken place but didn't[!]).

Process stuff generally finds its roots in engineering.  Especially process stuff as found in CMMI for Development.  Excepting processes that are over-engineered, are themselves lacking in engineering, or are odious even alone in a room, I'm beginning to piece together that resistance to processes in general, and CMMI in particular, is actually from a lack of engineering discipline in the software practice and not from anything intrinsic to process as a topic.

It's no wonder CMMI is so hard to use by so many, it assumes peopleimage are not only experts in process improvement, it also assumes everyone using it is an engineer.  Some people are nail-gun swingers, worried about getting enough done that day to avoid having to work on the weekend.  Meanwhile, someone else already worried about in what order to build piece the trusses together and someone before that worried about the right number of trusses and their thickness and someone before that worried about its shape.

It's becoming fairly clear that anyone fooling themselves into believing that agile advocates not doing an architecture at all, or a design at all, or other engineering activities at all are doing themselves a disservice.  In fact, I'd go so far as to say that once an architecture has been settled upon and once a design becomes clear, that agile practices can happen more freely and effectively.  More so, I'd assert that the future of agile "scalability" depends on these.

What I and a colleague are setting out to do over the next few months is help agile scale by re-introducing engineering to software, and while we can't fix the software universe, we hope to help agile out by giving it some engineering practices that software (as a whole) lacks -- not everywhere, just in too many corners -- but that we believe agile can really take and run with.

Let me know if you want to play with us.

Labels: , , , ,

3 Comments:

At 04 May, 2009 22:32 , Blogger John said...

Hillel,

I think you've hit right at the spot where large organizations like the big DoD contractors run into problems using agile methods. They have trouble finding the sweet spot between over-heavy process and tossing engineering out the window. The successful teams on agile, I think, are the ones who can think critically about this and make the appropriate decisions about when analysis is needed and when overanalysis is value-destroying.

What I think you're missing in this essay, though, is this. You make a point that programming is like assembly. I don't find the analogy as apt as you do, because construction assembly (to my knowledge) doesn't involve nearly the sort of uncertainty, discovery, and adaptability that we find at all levels of software development, from the method-and-line level on up. I can't think of a construction assembly job in the modern day that reminds me at all of the process of test-driven development. Perhaps a better analogy would be craftsmanship?

I found myself missing the other half of the analogy - if programming is like assembly, what's engineering?

Let me know if I can help you out.

- John Stoneham

 
At 04 May, 2009 22:56 , Blogger Hillel said...

Hi John,

The distinction I'm making is between programming and development. I'm saying that software development without engineering is just programming.

So, if programming is like assembly, development is like engineering.

What you describe when you note, "...the sort of uncertainty, discovery, and adaptability..." is found when you are actually performing development. That is, programming + engineering. (In the other order, most likely.)

So, we're in agreement, that if we eliminate engineering wholesale, we end up short-changing development -- and often expect non-engineering-types to somehow think and analyze like engineers. In the software world, not every programmer can also be a good engineer. (Duh.)

I'm reminded of a line from the Agile principles, found behind the manifesto:
Continuous attention to technical excellence and good design enhances agility.Which, to me, acknowledges the importance of engineering -- which so many organizations, as you note, try to toss out the window when applying agile methods.

You can certainly help. One of many ways, as a practitioner, is by keeping an eye out for examples of agile practices being enhanced, enabled, and/or facilitated by the application and/or existence of architecture and design as a precursor to much (but not necessarily all) of the agile-oriented "programming" steps.

 
At 05 May, 2009 06:37 , Anonymous Christian said...

Hi John,

I was reading the post and just thought:

agree,
agree,
agree ...

Struguling myself explaining customers the need for sound engineering practices on one side and being agile on the other side.

I think the reason why companies build a bureaucracy around development and the reason why some agile shops toss out some engineering stuff is the same - missing knowledge of and how to apply engineering practice in software development.

Is it missing education or just slopy practice?

Just a thought: There is an education difference between engineers and workers in construction and car manufacturing.

Applying tight engineering practices is sometimes seen as a constraint to the creativity of the software developer.

That's been just some thoughts not a solution. Hope for feedback. I am still thinking about the topic.

 

Post a Comment

Links to this post:

Create a Link

<< Home