Is it worth separating between the essential and accidental difficulties of software development?
Upon reading the chapter about productivity tools in the classic Rapid Development of Steve McConnell, I notice an interesting point Steve made about the role of high-level programming languages in the context of Fred Brooks’ No Silver Bullet. In particular, Steve wrote
As a rule of thumb, the more a technology strikes at the essence of what makes software development difficult, the more effort it is likely to save.
A high-level language compiler saves a lot of effort because it moves the whole set of concepts that you have to be concerned with to a higher level. That’s what the move to higher-level programming languages has accomplished over the years. The switch from low-level languages like assembler to higher-level languages like C and Pascal freed you from having to think about what the underlying machine was doing when it ran your program. The move to visual-programming languages such as Delphi, PowerBuilder, and Visual Basic provides a similar kind of simplification, allowing you to forget about many of the tasks a windowing environment performs when it runs a graphically oriented program.
Tools that do not strike at the conceptual essence, the hard part of programming don’t help as much.
Basically, Steve thought that the essence of software development was being tackled by high-level languages. However, that would not be agreed by Brooks, who wrote:
What does a high-level language accomplish? It frees a program from much of its accidental complexity. An abstract program consists of conceptual constructs: operations, datatypes, sequences, and communication. The concrete machine program is concerned with bits, registers, conditions, branches, channels, disks, and such. To the extent that the high-level language embodies the constructs wanted in the abstract program and avoids all lower ones, it eliminates a whole level of complexity that was never inherent in the program at all.
I am not going to discuss about whether Brooks or Steve makes more sense here, since I think the very concept between accidental and essential difficulties is not necessary and it is much more important to understand about the fact that there is no silver bullet, and that tools and methodologies can only incrementally, not revolutionarily, increase development productivity.
In fact, why should we care about whether high-level languages are addressing the essential or accidental difficulties as long as we all agree that they really help software development easier and faster and we are trying to improve them more and more? Can we tell whether Steve or Brooks makes more sense, or the line between what is part of the essential and accidental difficulties is just so blurred that there is no black and white answer to that? Before developing a new productivity tool or method, do we ask ourselves “is this going to tackle the accidence or essence of software development”? Or we just simply see a pain and try to have a tool to relieve it, regardless of whether it is accidental or essential? Is it true that the “essential VS accidental” concept helps explain why there is no silver bullet? Or are the inherent properties of software systems (described by Brooks as consisting of complexity, conformity, changeability, and invisibility) and what the recent history of software development has taught us those which really explain this whole silver bullet thing?
After thinking about all the above questions, if you still believe the separation between the essential and accidental difficulties is necessary, please offer your reasons.
From what I can tell from reading this, I’d have to say either McConnell is wrong or just misunderstood as a result of being taken out of context. The whole point of the accidental/essential difficulty idea is that you can’t really make tools/programming languages/etc. that let the programmer avoid essential difficulty. The essential difficulty is the core of the problem, and therefore it is the part that the programmer needs to work on.
Honestly, I think both McConnell and Brooks are trying to say the same thing in two different ways. The problem is just that McConnell talks about “striking at the conceptual essence” when, from what I can tell, he really means “eliminating any accidental difficulties of the problem”. If you look at the examples he gives, they all have to do with removing as much accidental difficulty as possible (window environment constraints, low-level machine operations, etc.). So I think he’s really contrasting more tools which only remove some of the accidental difficulty (say, assembly language instead of machine code), and those which remove a large amount of accidental complexity (high-level languages instead of machine code).
Finally, yes, I do think that the difference between accidental and essential difficulties is important, because programmers should focus as much of their attention as possible on solutions for the essential difficulties, and apply other tools and languages which help them ignore the accidental difficulty.
My take on this has always been that the distinction between accidental and essential difficulties is a function of time. I am sure there would have been times when people would have argued that garbage collection and a bunch of other things we take for granted were essentially difficult. I am not convinced that essential difficulties exist. For example, for a class of programming problems, an expert system may well be able to be developed to automate the entire programming process.
@Jon: thanks a lot for your comments
I’ll respond to this in a below paragraph…
I don’t think so, nor do I think Brooks thought so, since he wrote the following paragraph, in response to a 1990 paper of Brad Cox, in “No Silver Bullet – Refired”
In the same paper, Brooks also mentioned that reusability is an attacker of the essential difficulties. I can infer from it that if a language makes it easier to create reusability components, it is attacking the essence.
I can understand your point. I should have added another paragraph, right after the last sentence in Steve’s quote since I think that would make his point more obvious to us.
I think it should be clear by now that Steve was really making a compare and contrast between things attacking the essential (high-level languages) and things attacking the accidental (editors, automated testing tools etc.)
This does not seem to me as a good argument. I use a tool, a language, or a methodology because it addresses a pain that I have, and I do not see me or anybody I know consider whether those things tackling the essential or accidental difficulties, or why we need to care about that.
@Peter: thanks for your comments. I totally agree with your opinion that the distinction between accidental and essential difficulties is a function of time.
Is Steve attacking the essentials? It appears to me that (as the other poster had mentioned) he is accidential difficulties, albeit at different levels.
High level languages and the tools appear to me as accidental difficulties addressed, instead of essential and accidental respectively.
Thanks for your comment, Fadzlan
I believe he was. I also responded about that to Jon. Let me pull all Steve’s paragraphs together
I think the above tells what Steve thought clearly.
I think it is debatable as to whether a high-level language can address the essence (defined by Brooks as “a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions”) – Steve’ view VS Brooks’ view is one example. Are the algorithm s, data sets, function invocations etc. floating in the mind of a Assembly designer the same with those in the mind of a Ruby designer? How about a future language or tool which completely hides most of the algorithms and function invocations?