What is the Scope of Software Engineering?

2010/01/26

Looking at the discussions that have been appearing in response to the initial SEMAT-related blog posts, there appears to be little shared understanding of what we in the software world mean by the term “software engineering” or why it was selected ahead of other candidates such as “software development” or “computer science”.

The importance of the wording is also illustrated by the other discussions that touch on topics such as ”what is the relationship between software engineering and the management of software engineering projects?”, and “what is the relationship between software engineering and systems engineering?” There have even been other people publishing blogs on the suitability of the term software engineering as the banner for an initiative of this sort, one of our favourites being http://parijatmishra.wordpress.com/2010/01/08/188/.

Within the SEMAT group, we already have differences of opinion as to what the term “software engineering” should mean.

Some people want to start with the Wikipedia (http://en.wikipedia.org/wiki/Software_engineering) definition, which comes from the Software Engineering Body of Knowledge:

Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.

Alistair Cockburn is on record disagreeing with that definition, noting the difference between that definition and the the American Engineers’ Council for Professional Development (ECPD, the predecessor of ABET[1]) definition of engineering as:

[T]he creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.

Alistair develops his criticism of the term “software engineering” in his article http://alistair.cockburn.us/The+end+of+software+engineering+and+the+start+of+economic-cooperative+gaming). He proposes a model for software engineering (and engineering itself) that is comprised of

  • craft,
  • cooperative gaming,
  • lean processes, and
  • design as knowledge acquisition.

You can see his talk on the subject at http://alistair.cockburn.us/Software+engineering+in+the+21st+century.ppt.

The two of us take working through this disagreement as a part of the SEMAT assignment. We hope that by bringing all of the signatories and supporters together, we can refound software engineering to reflect these and other important perspectives.

This gets us to the question: Should SEMAT address “software engineering” or all of software development?

The term “software development” includes every type of software development, from simple spreadsheet macros to the programming of personal websites, up the scale to drawing packages, game development, massively  parallel scientific calculators, civil-engineering structural simulations, and the running of nuclear submarines. There are so many differences in the optimal working habits across these activities that we think it is too broad of a scope for the SEMAT charter.

“Software engineering” points to a subset of software development where the term “engineering” can be felt as appropriate. That is likely to exclude small efforts where the danger of failure is only loss of comfort. It should include efforts where lives are at stake, from structural simulators for civil engineering projects to medical devices and power control systems. In government and defence projects, there will be many projects – even non-life-critical projects – that need the “engineering” aspects. There will be others that don’t. The two of us recognize those differences.

We choose to stay with the term “software engineering” for SEMAT because we believe that the growth and success in of the targeted category of projects depends on the elaboration and evolution of an engineering perspective with its emphasis on evidence-based practices grounded in sound theory.

It should be obvious that if we SEMAT supporters succeed with our goals, many of the other software development projects may find it useful to adopt parts of our result.

One of the goals of the SEMAT initiative is to establish a baseline theory that can be tested and evolved over time. The SEMAT initiative will clarify what we mean by “software engineering” and define a kernel that embodies that definition.

Ivar Jacobson proposes that that theory, amongst other things, clearly:

  • Identifies the bounds of software engineering.
  • Allows software engineering practices and principles to be defined, studied, and improved.
  • Shows that software engineering is a form of collaborative activity.
  • Enables the measurement and comparison of the effectiveness of different practices and teams.

Alistair’s proposes that when we are done, the term “software engineering” should

  • explain why successful projects succeed and failing projects fail,
  • intrinsically name topics known to be important to project success,
  • lead a person on a live project to derive sensible advice as to how to proceed,
  • provide a sound pedagogical base for teaching newcomers to the field.

Now these are not conflicting goals. The two of us are both working together on SEMAT because we both believe it is possible to refound software engineering in a way that addresses all our concerns. Starting to address the question “What is Software Engineering?” will be a key part of the initial SEMAT meeting in Zurich. But there is no reason to wait to start our discussions, please feel free to support our working definitions or propose others that you prefer and that we all can learn from.  Remember the goal here is but to create an environment where we can learn from each other and build a new, less partisan consensus.

– Ivar Jacobson and Alistair Cockburn

with thanks to Ian Spence and Larry Constantine for their help in pulling this blog entry together.

About these ads

10 Responses to “What is the Scope of Software Engineering?”

  1. Phil Armour Says:

    Excellent synopsis of the story so far. I very much like the laying out of options and opinions. I will be interested to see how the deciding process works between these options.

    For my part, I think the focus will need to be on the nature of the knowledge (as product) and the mechanisms of individual and collective cognition (as process) aspects of building systems.

    The perennial attention paid to the “work products” in software engineering has not paid the dividends we had hoped–attention always seems to be drawn to the format: the “menu” rather than the “meal” (as J.W. Smith described it). Similarly, the focus on progressively defining “process” as a more and more detailed mechanistic stepwise assembly process has been tried over and over with quite poor results.

    The work in systems development does not occur in processes or on paper or even (paradoxically) much in software–it occurs in and between peoples’ brains. At best the processes and formats are rough templates representing (sometimes) the contextual framework for the knowledge being acquired and “assembled” into a system-level form. The extent to which the knowledge being acquired then differs in structure from these templated contexts determines the usability and usefulness of the process and work product format. We can be highly definitive in narrowly-defined knowledge domains, but applying definitions, rules, processes, and templates across widely differing knowledge areas can be ineffective and even dangerous. That is unless the contexts are very high level, in which case they are not very useful (read: don’t contain much knowledge themselves).

    There are key external domains (eg., organizational structures, system state behavior) that we might usefully define at some level, particularly if we can prescribe the domain in which they should be used, but I think we will need to look inward at human information processing (particularly cooperative understanding) for much of this initiative. There have been many references to this challenge over the years since the Garmisch conference, from Jerry Weinberg’s “Psychology of Computer Programming” through Alistair’s “cooperative game” ideas. Maybe we need to consult some psychologists?


  2. I want to keep this brief enough to be read and yet summarize my views on a Theory of Software.

    I agree that a Theory of Software would be valuable. I see the development as having 2 parts:

    1) observe what works and what doesn’t in practice and extract workable principles based on it — the Agile approach strives to do this

    2) begin with first principles and form hypotheses, develop practices based on the hypotheses, and return to step 1.

    I will attempt a summarization of a majhor set of reasonig from first principles.

    Assert: Software development (of the sort that needs a Theory of Software) is done by teams of people using practices and processes in an organizational / management / cultural context.

    Assert: These people are knowledge workers, and much can be learned from a study of the work done with other knowledge workers. The book “Changing Softeare Development” Learning to Become Agile” by Allan Kelly makes this case very well.

    Assert: Software encapsulates knowledge: software allows us to do what the developer does without knowing what he knows.

    Corollary: Since software is developed by people, the process sthould take into account their strengths and weaknesses. First and foremost, software development is about strategies for managing complexity and supporting attention to detail, both of which are weak in people. At any level of abstraction and at any point in the space, the size of the “working set” has to be small enough to be encompassed by a single mind. We use various mechanisms of decomposition, chunking, and pattern recognition to reduce the complexity to a level that we can manage.

    Corollary: Software development can never proceed in a totally top-down manner. People think associatively, possible even holographically. Our systems whould allow us to capture knowledge whenever it occurs and make us of it later. David Parnas in “A Rational Design Process: How and Why to Fake it” contends that no system of any size was ever built strictly top-down and none likely ever will be, but that we can endeavor to construct the trail of documentation that would have resulted had we been able to do so.

    Corollary: Since software is built by teams, all of the team-building and communication processes are of great importance. As the productivity of individual varies dramatically, so does the productivity of teams. When a team jells, individuals do what is needed when it is needed with little or no direction, at any point the focus is on the person who is “on” and has something to contribute at that time. Individual strengths are capitalized upon, and individual weaknesses are strengthened.

    Corollary: Since development is done in a management context, a workable Theory of Software must address that context as well as the skills of the developers. A team whose management requires that 100% of the requirements be done and frozen before development can begin starts out with a tremendous handicap. The Future Combat Systems Program run by Boeing for the Army was planned with a single requirements phase to cover nearly 3 decades worth of development, and it doomed the project from the start.

    Corollary: Since this is knowledge work, our tools and practices should facilitate that work. There is a reason that wikis work well in discussing a proposed system. There are reasons why Use Cases work well. Humans have been using stories and narrative to communicate for as long as there has been spoken language, and we need to facilitate and utilize that communication. Note that all such collaborative efforts have found the need for regular refactoring when an are gets too jumbled, or enough work has been done to be able to discern usable patterns in the material. Conventional documents have more in common with PhD theses than with communication tools. Requirements as merely lists of shall statements lose all context, grouping, rationale, and the coherence that makes them understandable.

    And now we can begin to look at technical issues.

    Assert: At every level of abstraction in the development of a system, the general activity is the same: Given a source of system objectives (needs, desired outcomes, functions, and features) evolve concurrently a system concept (concept of operations), an architecture (what is doing the work), and an expanded description at the new, lower level of abstraction. The process is essentially the same at the high end of system concept development from needs expressed by potential users and domain experts to software coding based on design documents. We don’t need different practices for each level of abstraction.

    Corollary: Where we use natural language, we must endeavor to use it with the same precision as is demanded during coding. Names need t be unambiguous, statements need to be clear and complete, we need data dictionaries to support controlled vocabularies and possibly even full ontologies in the problem domain.

    Assert: The approaches we apply to code to improve clarity, modularize and group descriptions, and reduce duplication work at all levels of abstraction. The application of aspects to Use Cases is one good example, but there are others. Use of interfaces and patterns in requirements can reduce volume, improve clarity, and improve maintainability. For example, for disk-based data, we used to use CRUD (Create, Read, Update, Delete). I routinely see these called out as separate actions on individual data items today. These things cannot be tested independently. They need to be grouped into the equivalent of an interface and specified with a single requirement.

    There is much more to be said, clearly, but I believe that detailed effort on the above with application of experience in software development can provide the basis for a Theory of Software Development that is robust and workable because it si based in reality.

    I first got into computers to get away from people, and discovered during a 45-year career that is is all about people. It is about more than psychology, which Gerald M. Weinberg has studied and written on so effectively I believe passionately that appropriate incorporation of knowledge management can bring a next jump in productivity.


    Thanks,

    Garold L. Johnson

  3. Magnus Timmerås Says:

    Regarding the discussion about what software development needs “engineering”:

    You point out life-critical applications, and add that “In government and defence projects, there will be many projects – even non-life-critical projects – that need the “engineering” aspects”.

    I would like to add “money” to this. Apart from life-critical systems, the main driving force for development is money (in some form). The need for “engineering” would be based on a wish to not waste money, by developing software that needs to be re-developed, or thrown away because it doesn’t serve its purpose.

    If you risk losing only a small amount of money, “engineering” is not needed (maybe even for a life-critical system, because you can re-develop it until you get i right). If you risk losing massive amounts of money, you will need “engineering” to ensure the complex system will work at all, and reduce the financial risks.

    This makes me think of other aspects of financing software development, for example as described above when it comes to military systems. Many decisions affecting the development process are not made because it will serve the development work best, but because it is a part of the fundamentals of that market. If the people with the money requires you to present a requirements document up front, you will try to do that, if that is the only way to get the resources to do the actual work. So the “software engineering” theory also needs to cover the aspects of funding and final decisions whether or not to do the development work at all.


  4. If you look at accreditation (at least in Canada) for engineers (Professional Engineers – P. Eng), the application of science in a creative and innovative way (which usually implies unprecidented endevours containing uncertainty/risk) carries with it a fiduciary duty. In other words Engineering implies a standard of care from a Tort Law perspective. However, the term is often misused.

    The problem is that we do not have a “common law” that establishes the precidents through a court registry to establish this standard of care for software. And yet the creative medium of software as the output of “pure thought stuff” is integral to the systems that run the world we know. Failure to meet a fiduciary duty can lead to material economic loss and/or loss of life, each of which are remedied by the courts after the fact. With engineering being involved for a subset of software developments, we do not have as robust a precident library to consult, one that is associated with success or failure (patterns). Perhaps this needs to be established if a Software Engineering Profession is to stand alongside the other engineering subfields, or medicine or even law.

    I think Philippe Kruchten has been exploring this related issue to “Software Engineering” at the University of British Columbia in his position in the Faculty of Applied Science, Electrical Engineering department. Perhaps he will comment further in the blog.

    Also, Grady Booch recently put out a short and interesting video titled “Why Engineering”. It might stimulate the discussion further. http://www.youtube.com/watch?v=d0HVjoTD_ro

    Cheers.

    Mark Kennaley
    President & Principal Consultant
    Fourth Medium Consulting Inc.

  5. Goto Shoichi Says:

    I expect that Software Engineering is basic common recognition shared by tech-guys, managers and all stakeholders in software development.

    It tells logical, well-grounded sequences or relations between causes and results of most software development activities. It’s able to have software engineers explain “what they do” and “why they do” for success of their project.

    It may include software development processes, concrete practices, mindsets, disciplines and so on. I think it’s should be emphasized that Software Engineering is basic “knowledge” for software engineers to co-work well, to improve their own work and to get happiness in their office.

    thanks,


  6. One more aspect that we should take care:
    Different project teams may follow different models like waterfall, iterative or agile. There are certain essential attributes which make a project a success or failure irrespective of which model is being followed. There may be different ways to achieve the essential depending on the situation or model being followed. For example, some of the practices of agile development may be useful for iterative or waterfall in certain situations.

    So the final outcome of this exercise should be to outline the essentials for success along with different options and suggested situations where each can be used.

  7. Lee Riemenschneider Says:

    The term “software development” includes every type of software development, from simple spreadsheet macros to the programming of personal websites, up the scale to drawing packages, game development, massively parallel scientific calculators, civil-engineering structural simulations, and the running of nuclear submarines. There are so many differences in the optimal working habits across these activities that we think it is too broad of a scope for the SEMAT charter.

    “Software engineering” points to a subset of software development where the term “engineering” can be felt as appropriate.

    This statement is just wrong! If we look at both definitions presented for software engineering, we can see that engineering boils down to the application of knowledge. A web developer might not use the title, software engineer anymore than a farmer, who bridges a ditch for his tractor, would call himself a civil engineer, but they are both engineering a solution to a problem. i.e., applying knowledge.
    The difference between obfuscated C code and a “good” C module is the software engineering principles applied. IOW, there is no difference between “software development” and software engineering”. There are just different classifications of software engineering principles. The classifications should target specific goals. e.g., obfuscation could be considered a type of security engineering.


  8. A colleague of mine recently came across an article on the IEEE Computer Society’s website (http://shar.es/maw9I) talking about the importance of software engineering degrees. It loosely defined and contrasted software engineering as:

    “Software engineering is loosely defined as the application of engineering concepts, techniques, and methods to the development of software systems. It is distinct from the field of computer sciences, which draws on a strong theoretical foundation in mathematics, statistics, and computing to develop new, cutting-edge computing solutions.”

    While I am not advocating this definition, I found it interesting as a possible indication of how some people view software engineering.


  9. “systematic, disciplined, quantifiable” seems as Engineering is a job for the brain dead.

    “creative application of scientific principles …” is much closer to what one can exprience in every engineering department. (of course not without an appropriate level of discipline etc.)


  10. Thanks for some other magnificent article. Where else could anyone get
    that type of information in such a perfect way of writing?
    I’ve a presentation next week, and I’m at the search for such information.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 49 other followers

%d bloggers like this: