As you may know from our Three Year Vision, Semat’s first step was to create a common ground for software engineering. This common ground would be manifested as a kernel of essential elements that are universal to all software development efforts, and a simple language describing methods, practices and the kernel elements. Both the kernel and the language should be widely accepted.  Right now, Semat is in the process of completing the first step, which has resulted in the following:

  • A submission, called “Essence — Kernel and Language for Software Engineering” to the OMG’s RFP (Request for Proposal) “A Foundation for the Agile Creation and Enactment of Software Engineering Methods“. Two other works have been submitted to OMG. The Semat results were very well received by OMG. The Semat submission will incorporate some ideas from the other two submissions into the new “Essence” submission. Semat will also incorporate ideas from the existing standard Spem.
  • Semat will be presented at the International Conference of Software Engineering (ICSE 2012) in Zurich on June 6, 2012. This will happen in the “Refounding Software Engineering” session. It is Semat’s first international publication and more publications are on the way.
  • A new book `The Essence of Software Engineering – Applying the Semat kernel’ is coming – now in peer review. The book’s objective is to put the kernel into a context.  The book describes how the kernel can be used by the practitioners and how they can measure progress in an endeavor independent of which method they use. It also provides suggestions for how they can scale up, scale out or scale in their methods.

The Semat kernel is already being used on two academic project courses at KTH Royal Institute of Technology. On one project course, Mira Kajko-Mattsson is teaching the Semat kernel and evaluates its usefulness within software education. Using the kernel elements, her students continuously evaluate the progress and health of their projects.  On the other KTH project course, Anders Sjögren uses the Essence book as a base for education in software engineering with extensive lab work. Moreover, two other universities use or plan to use the Semat approach when training their students. These are Peking University and Oslo University. Experiences on the use of the Semat kernel in education will be published on Semat’s website in the next couple of months.

Getting a widely agreed upon kernel is just the beginning.  The kernel is a fundament that constitutes a critical milestone for moving forward in refounding software engineering.  To be able to systematically work with practices and methods both in the industry and academia, more work needs to be done in form of creating products, creating practice market places, constructing supporting tools, revising software engineering curricula, doing research and writing books and papers.

Thanks to the kernel, our less experienced developers (and most of us belong to this group) will become professional much easier and much faster.  Standing on the common ground, they will be able to learn from one another much better and move from one team to another much easier.  Since we focus on you, the developers, and not the process engineers, the adoption of the Semet kernel is more of a pull than a push.

The Semat kernel stands on two important principles, which are ‘agile in everything we do’ and ‘separation of concerns’.  For example, we value:

1)      What helps the least experienced developers over what helps the experts?

2)      What helps the practitioners over what helps the process engineers?

3)      Method usage over method definition?

Of course, we must support the experts and process engineers as well, but not by having the practitioners to pay a price. And, of course, we can define methods, but most importantly, we have to make methods useful while we actually work in real endeavors.

Compared to earlier attempts, the Semat approach has several differentiators that make developers more motivated to work with methods.  Maybe the most important one is that the whole approach is actionable. As already described in the Three Year Vision, the developers do not just use it for studying. They use it in their actual daily work when enacting their ways of working and when measuring their progress.   part of the whole game.  All this will help the industry.  The education at a university level will become more fundamental, more systematic and effective. The research will become more disciplined and more useful to the software community.

In parallel with working on the proposal, a small team of people has been working on a new book ‘The Essence of Software Engineering – Applying the Semat kernel’.  The authors are Ivar Jacobson, Pan Wei Ng, Paul Mc Mahon, Ian Spence, Svante Lidman. The book is now in its 3rd draft and about 30 people have reviewed it and given constructive feedback.  We expect that the 4th draft will be widely published.  If you are interested in reviewing and providing feedback, please send an email to ivar@ivarjacobson.com. A copy of the book will be sent to you asap.

— Ivar Jacobson and Mira Kajko-Mattsson

Advertisements

Given the ambition of the Semat project it would be surprising if it did not cause some controversy. Alistair Cockburn wrote a long critique at http://alistair.cockburn.us/A+Detailed+Critique+of+the+SEMAT+Initiative. Martin Fowler, at http://martinfowler.com/bliki/Semat.html, echoes Cockburn and states that the effort is “fundamentally doomed”.

The issues are complex and we welcome controversy. We regret, however, the strident tone of these comments and their authors’ decision to criticize from the outside rather than helping with Semat’s mission of improving software engineering. This decision is for a large part based on rather unconvincing generalities. For example Fowler’s argument is that “I spent a fair bit of time in the 80’s and 90’s mucking around with this idea. In the end I decided that it was too hard and of limited value.” It is good to know that Fowler gave some thought to the problems of software engineering twenty years ago and gave up, but others might be forgiven for not finding this a good enough reason for the rest of the world to stop looking for solutions.

Fowler writes: “I got the distinct impression that the central thrust of the initiative is to create a software meta-method-kernel – essentially a set of common process elements for software developments that you can rigorously compose into a method for your own project.”  This is a misreading. The Semat kernel is not at the “meta” level; it is a concrete kernel of practices and patterns. The kernel is small, and includes universal elements relevant to all development projects.  It will be used to describe methods.

Cockburn’s critique exhibits a problem that we had foreseen and explicitly tried to forestall: the prima donna phenomenon. In setting up Semat we have contacted thought leaders in software methodology (Cockburn and Fowler included) and asked them to accept that each has a piece of the truth but none has the entire truth. We understand that this is difficult to accept (none of us is particularly modest either), but that accepting to work together regardless of guru status is essential to progress. Cockburn’s main argument is the eight years of carefully constructed literature including his own book, and that his 2002 doctoral dissertation pointed out that ”methodologies will inevitably increase in number, not decrease”. Much as we enjoy Cockburn’s contributions, they are neither the only word nor the last word.

Nowhere do the Semat documents suggest that Semat would result in fewer methods.  The Call for Action’s mention of the “huge number of methods and method variants“; Alistair cites this, but ignores the following comment:  “with differences little understood and artificially magnified.”  The problem is not the large number of methods, but their monolithic nature and the difficulty of comparing and combining them The Semat initiative should in fact make it easier to create new methods out of a smaller number of clearly defined practices.

Cockburn’s complaint also embodies a strangely thin-skinned reaction to suggestions that previous ideas, in particular Agile ideas, may need further refinement. Cockburn objects to our comment on the Agile manifesto, which stated that:

“… the elegantly written foundational document of the [Agile] approach is a “manifesto”, long on emotional appeals in the first person plural and short on facts. This style may be effective to capture attention, but as ideas mature it should yield to more traditional (even if also more boring) forms of argumentation.” (From an article by Ivar Jacobson and Bertrand Meyer)

He states that “The authors once again use precisely the polemical language they decry”. This is a bizarre criticism: first we don’t object to “polemical” language (discussion is a normal part of progress in science and technology), only to assertions that are not backed by facts and theory. Second, the article from which this is extracted was published long before the Semat documents and the first Semat meeting, so it is not like he discovered it after attending the meeting. But third and most importantly, our cited extract is quite laudatory of the Agile Manifesto; it simply points out that manifestos are not enough (a comment that also applies to the Semat Call for Action). It is one of the fundamental premises of the Semat effort that there are no sacred cows: we want to get the best of all existing ideas, agile included, and understand their limitations as well. We fear that Cockburn’s reaction shows a lack of willingness to have any aspect whatsoever of agile methods questioned in any way. That is not justifiable: everyone’s ideas, including ours, Cockburn’s and Fowler’s, should be assessed at face value.

Cockburn is also critical of the Zurich meeting. He complains that no agenda was presented.  Of course the meeting had an agenda; If he had read that agenda, the Call for Action and the vision statement, he would have known the goals and principles of the Semat initiative before he decided to come.  Moreover, He omits one of the main difficulties that this meeting (altogether very successful) encountered. The cause of that difficulty was Cockburn’s own decision, unannounced, to comment on the discussions in real time on Twitter; many of the comments were sharply critical of other participants. One of these participants suddenly found out and was upset at what he viewed as a betrayal of trust. A discussion ensued, moderated by us (we maintained a neutral view); it led first to a decision to allow tweeting and soon thereafter to the reverse decision, as a majority felt the tweeting compromised the quality and integrity of the discussions. This was a side issue, and was taken care of, but it did divert the meeting from its focus and cause tempers to rise for a while.

Cockburn states that the “state of [the research literature] more indicates that we don’t yet understand what is happening on software projects”. This is greatly exaggerated; of course we as a community do not understand everything — if we did, there would be no need for Semat — but a vast amount of literature, including both process-oriented (CMMI etc.), object-oriented and agile sheds considerable light on the reality of software development. The attitude that we don’t understand anything anyway is good for gurus (just as ignorance of medicine is good for witch doctors), but Semat takes a different view: that we know quite a bit already, albeit in a messy and often inconsistent way; that we must organize and classify this knowledge; that we must assess every idea, existing or new, through objective criteria, without fear or favor; and that we must apply our best efforts to solve the remaining open problems. Proponents of agile methods have a major role to play in this process.

— Ivar, Bertrand, Richard

The Vision

2010/02/16

Today as promised, 15 February 2010, we are delighted to publish our Vision Statement for the Semat project.  We appreciate all of the comments that we have received privately and on the blog, which influenced our thinking and sharpened our focus.  We hope that you will find the result coherent and focused, and a good baseline for the project.  We have identified and defined five tracks (or working groups) to manage parts of the effort.
Please see http://www.semat.org/pub/Main/WebHome/SEMAT-vision.pdf for the Vision Statement.  We look forward to continuing the constructive dialog.
— Ivar, Bertrand and Richard

One of the key foci of the SEMAT Call for Action is to agree “a kernel of widely-agreed elements, extensible for specific uses”.  In this discussion we will explore the idea of a kernel. The goal is to gather requirements related to the kernel.  Please join the debate by commenting on this entry.

What is the Kernel

An important step in fulfilling the promise of the SEMAT initiative to “refound software engineering” is to establish a framework for thinking about software development.

This framework must be concrete representations of the acts & artifacts of software development, representations that are applicable no matter the size or scale of the software under development, nor the size, scale or style of the team involved in the development.

We call this framework the “kernel” as it captures the set of elements that are inherent to all software development efforts. In essence , it provides a practice independent framework for thinking and reasoning about the practices we have and the practices we need.

The goal of the kernel is to establish a shared understanding of what is at the heart of software development.   We must discover the set of elements that are essential to all software development efforts, a shared body of knowledge for academics, researchers & practitioners.

The kernel will allow us to think about software development in a number of different styles.  It might comprise:

  • A list of the key elements that always need to be addressed, in all projects;
  • A map of the practice territory so that we understand how to build from the kernel
  • An understanding of the skills needed to build teams and software
  • A framework for using the kernel, both to develop extensions to the practice and to develop software
  • A set of definitions to be shared by all practices

The kernel is the core of the underlying theory for software development. What rigor is important, it is also usable; that is, it can be taught, measured and put into practice in real projects. To academics, the kernel might be seen as a language to describe practices.  To the developer the kernel underlies what he does every day.

It’s important to remember that SEMAT is focused on building theory that allows different practices to work together – not defining new or “best” practices.  As Leonardo da Vinci once said, “He who loves practice without theory is like the sailor who boards a ship without a rudder and compass and never knows where he may cast.”

So what are the key components of such a kernel?  What is extraneous, or unnecessary in all descriptions of practice?  What are the right starting points in existing research or practice?

– Ivar, Bertrand and Richard

Welcome!

2009/12/01

Barely a week after setting up www.semat.org, we are close to two hundred people supporting our initiative. More are joining by the hour. The initiative appears ready to mushroom!

We don’t expect SEMAT to solve all the problems of software engineering. It addresses the development of a common knowledge base; other initiatives are needed for example to increase professionalism in the industry and to close the gap between computer science research and software engineering practice. SEMAT is a great start towards recognition and resolution of fundamental problems. We have committed to working together with developers, researchers, methodologists, and leaders at all levels to make the software world better.

The Call for Action points in the direction of a solution by specifying “a kernel of widely-agreed elements, extensible for specific uses.” It avoids prescribing a specific solution, not because none exist, but because none have been widely accepted.

Now the hard work will start. The first step is a vision statement, specifying what we as a group want to achieve in the next twelve months. The three of us will prepare a skeleton of the vision statement, and will submit it to the community so that you can provide your insights.

Please prepare yourself by reading the material on the semat.org site: the two papers and the Questions and Answers. Additional material is also available on the site.

The biggest difficulty will be to achieve consensus without compromising ourselves to death. We can do it!

Now, we open up the floor for you to express your ideas.

— Ivar Jacobson, Bertrand Meyer, Richard Soley