In Medellin a few weeks ago in a workshop held following my keynote address at the Latin American Workshop on Software Engineering and Knowledge Engineering (JIISIC) one of the first questions I received was:
How does SEMAT relate to SWEBOK?
I answered by focusing on the fact that Essence is small. It contains only the essentials required on all successful software engineering endeavors and it includes no practices. SWEBOK, on the other hand, is large and covers many best software engineering practices.
This past week I decided to go back and refresh my understanding of SWEBOK. Part of my motivation was based on the question I received in Medellin, but I was also motivated because recently there has been some criticism of Essence (the first product of the SEMAT community) because of its lack of emphasis on software engineering architecture and testing practices.
What immediately struck me in the preface to the SWEBOK guide (2004 Edition) was the phrase:
“the emphasis of the guide is placed on the construction of useful software artifacts.”
SWEBOK focuses on best practices in software engineering knowledge areas including requirements, design, construction, testing, maintenance, configuration management, software management, process, tools, methods, and quality. Clearly these are all essential areas software engineering practitioners should know.
SWEBOK, like SEMAT, intentionally does not address rapidly changing technologies such as programming languages. SEMAT, however, unlike SWEBOK, has intentionally not focused on the construction side of software artifacts.
So why did the SEMAT volunteers make this choice?
While constructing useful software artifacts is clearly an essential competency all software practitioners need to know, experience has shown that a project’s true progress and health cannot be measured by work products (or artifacts) alone. It is the essentials of project success that transcend the work products themselves, such as ultimate stakeholder satisfaction, where SEMAT places its focus.
As an example, a key goal of the Essence kernel is to guide practitioners in asking the right questions that can help their team focus on what is most important right now leading to ultimate endeavor success. While all software practitioners need appropriate skills in producing useful software artifacts, they also need appropriate skills in making critical on the job decisions related to the progress and health of their overall endeavor. When you look at it this way, SEMAT and SWEBOK just might make a perfect marriage.
by Paul E. McMahon
There has been quite a bit of discussion recently about why we need SEMAT, or more specifically the Essence kernel which is the first product of the SEMAT community.
This past week I gave a keynote address at the Latin American Workshop on Software Engineering and Knowledge Engineering (JIISIC) held at the University de Medellin, in Medellin, Columbia– a great city to visit and a great University full of friendly bright students eager to learn.
Because I view the question above as a key challenge the SEMAT community must address, I decided to focus my Medellin presentation specifically on this topic.
But before I tell you more about what happened in Medellin, let me tell you a little more about the challenge SEMAT faces. The goal of SEMAT has always been to produce a common ground kernel that we all can agree is essential to all software engineering endeavors. But many people– including some of SEMAT’s own volunteers– have recently questioned how the Essence kernel is any better than Scrum, or the CMMI, or any other method an organization chooses to use.
A few months ago Capers Jones was asking questions related to comparing the SEMAT approach with other well known methods, and in the workshop held after my presentation in Medellin last week one of the first questions I received was about how SEMAT compared to SWEBOK, The Software Engineering Body of Knowledge.
In Medellin I decided to take a different approach to explain what Essence is, and why you should care. For the first twenty minutes I didn’t talk about SEMAT or Essence at all. Rather I talked about the problem the software community faces today as I have observed many organizations spending literally millions of dollars each year on process improvement efforts that too often fall short of their goals.
In my presentation in Medellin I provided a number of common patterns I have observed that can help us understand why many organizations, regardless of their chosen approach to improve– Scrum, CMMI, Lean, Lean Six Sigma, RUP…– fail to gain and sustain the improvements they seek.
I challenged what many of us have been taught and what many organizations continue to do today related to collecting large data samples before acting on improvements. And I gave examples of a much simpler and faster way to determine where the real pain points are in an organization.
Once you know where the pain is the next step is to figure out how to resolve it in a lasting way.
And this is where I explained what Essence offers that can help any organization have a much better chance of achieving their improvement goals in a way that is lasting, and they can continue to build on in the future.
It is natural for people to want to compare anything that is new to what came before. It is also natural to resist change as change is frequently viewed as risky. But as I explained in Medellin, the Essence approach is not about adding risk to whatever you do today, or changing whatever approach you use today.
In Medellin I gave specific examples demonstrating why the way the CMMI is used today in many organizations fails to help real practitioners with the real common difficult situations they face each day on the job.
And I gave specific examples showing how key checklists within the Essence kernel lead teams to ask the deeper questions that are needed to guide them to the right decisions that need to be made now to keep their software endeavors on a path to success with satisfied stakeholders.
Essence is not in competition with any method or improvement approach. It can help your team implement whatever approach you choose by helping them ask the right questions leading to better choices based on where their team is now and where they need to go next.
Essence doesn’t add risk. Rather it can help your team reduce the risk of failing to achieve a successful outcome and it can do it independent of whatever way of working you have chosen.
The SEMAT community will continue to face questions related to comparing Essence with other approaches and we will need to gather measures to demonstrate the value of the Essence approach. But the comparisons– I suggest– should not be Scrum versus Essence, or CMMI versus Essence, or RUP versus Essence. But rather they should be Scrum versus (Scrum + Essence), or CMMI versus (CMMI + Essence), or RUP versus (RUP + Essence).
Essence is not in competition with any existing practice or method. It is agnostic to your chosen approach and can be implemented by your team today without changing what you are currently doing. It is more about helping you do what you are already doing so you can keep doing it even better in the future.
As you may know from our Three Year Vision, Semat’s first step was to create a common ground for software engineering. This common ground would be manifested as a kernel of essential elements that are universal to all software development efforts, and a simple language describing methods, practices and the kernel elements. Both the kernel and the language should be widely accepted. Right now, Semat is in the process of completing the first step, which has resulted in the following:
- A submission, called “Essence — Kernel and Language for Software Engineering” to the OMG’s RFP (Request for Proposal) “A Foundation for the Agile Creation and Enactment of Software Engineering Methods“. Two other works have been submitted to OMG. The Semat results were very well received by OMG. The Semat submission will incorporate some ideas from the other two submissions into the new “Essence” submission. Semat will also incorporate ideas from the existing standard Spem.
- Semat will be presented at the International Conference of Software Engineering (ICSE 2012) in Zurich on June 6, 2012. This will happen in the “Refounding Software Engineering” session. It is Semat’s first international publication and more publications are on the way.
- A new book `The Essence of Software Engineering – Applying the Semat kernel’ is coming – now in peer review. The book’s objective is to put the kernel into a context. The book describes how the kernel can be used by the practitioners and how they can measure progress in an endeavor independent of which method they use. It also provides suggestions for how they can scale up, scale out or scale in their methods.
The Semat kernel is already being used on two academic project courses at KTH Royal Institute of Technology. On one project course, Mira Kajko-Mattsson is teaching the Semat kernel and evaluates its usefulness within software education. Using the kernel elements, her students continuously evaluate the progress and health of their projects. On the other KTH project course, Anders Sjögren uses the Essence book as a base for education in software engineering with extensive lab work. Moreover, two other universities use or plan to use the Semat approach when training their students. These are Peking University and Oslo University. Experiences on the use of the Semat kernel in education will be published on Semat’s website in the next couple of months.
Getting a widely agreed upon kernel is just the beginning. The kernel is a fundament that constitutes a critical milestone for moving forward in refounding software engineering. To be able to systematically work with practices and methods both in the industry and academia, more work needs to be done in form of creating products, creating practice market places, constructing supporting tools, revising software engineering curricula, doing research and writing books and papers.
Thanks to the kernel, our less experienced developers (and most of us belong to this group) will become professional much easier and much faster. Standing on the common ground, they will be able to learn from one another much better and move from one team to another much easier. Since we focus on you, the developers, and not the process engineers, the adoption of the Semet kernel is more of a pull than a push.
The Semat kernel stands on two important principles, which are ‘agile in everything we do’ and ‘separation of concerns’. For example, we value:
1) What helps the least experienced developers over what helps the experts?
2) What helps the practitioners over what helps the process engineers?
3) Method usage over method definition?
Of course, we must support the experts and process engineers as well, but not by having the practitioners to pay a price. And, of course, we can define methods, but most importantly, we have to make methods useful while we actually work in real endeavors.
Compared to earlier attempts, the Semat approach has several differentiators that make developers more motivated to work with methods. Maybe the most important one is that the whole approach is actionable. As already described in the Three Year Vision, the developers do not just use it for studying. They use it in their actual daily work when enacting their ways of working and when measuring their progress. part of the whole game. All this will help the industry. The education at a university level will become more fundamental, more systematic and effective. The research will become more disciplined and more useful to the software community.
In parallel with working on the proposal, a small team of people has been working on a new book ‘The Essence of Software Engineering – Applying the Semat kernel’. The authors are Ivar Jacobson, Pan Wei Ng, Paul Mc Mahon, Ian Spence, Svante Lidman. The book is now in its 3rd draft and about 30 people have reviewed it and given constructive feedback. We expect that the 4th draft will be widely published. If you are interested in reviewing and providing feedback, please send an email to email@example.com. A copy of the book will be sent to you asap.
– Ivar Jacobson and Mira Kajko-Mattsson
Ivar Jacobson, Shihong Huang, Mira Kajko-Mattsson, Paul McMahon, Ed Seymour
The work on Semat has been going on for almost two years. We have not published much that can be of any practical value to our target groups, which are the industry, represented by developers and their teams, and the academics, represented by instructors and researchers. This is not because we want to be secretive. On the contrary, we want to be as open as possible. We have worked on implementing the vision we laid out almost two years ago, and while implementing it, there has been little to discuss widely. However, we are now reaching a point where meaningful results can be shared with our community.
On February 22, 2012 members of Semat will (as a response to an OMG RFP) publish a proposal for a new standard in working with methods, a proposal that is fundamentally different from anything we as a community have seen in the past. Other parties will most likely also submit proposals and at the end of this year we should have reached an agreement on what OMG in its rather formal style refers to as “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”.
At the end of 2012 we should expect to will have a kernel in software engineering. The kernel has three faces, which are important as a starting point for the whole initiative:
1. The kernel is executable.
The kernel is not a static dictionary that you read. It is dynamic and executable. By that we mean you can make the kernel come alive. You can use it in real life to run an endeavor (e.g. a project or sprint). It includes the essential elements always prevalent in every endeavor such as Requirements, Software System, Team, and Work. These elements have states representing progress and health, so as the endeavor moves forward the states of these elements progress from state to state.
2. A blueprint for growth.
The kernel includes just the essentials, but it is extensible. On top of the kernel practices can be added and composed to support any existing method (to the best of our knowledge). Some larger endeavors may extend the kernel with more essential elements. The blueprint helps us to grow the kernel to whatever size is needed.
3. Principles and values.
The principles and values help us to make decisions that are not guided by the blueprint. In everything we do we apply the principle of ‘separation of concerns’. For instance, we separate what the practitioner wants to work with from what a process engineer needs, in such a way that the practitioner doesn’t need to ‘see’ more than what is of interest to her/him. We also separate practitioners from one another; all developers do not have the same level of interest.
An example of a fundamental principle applied in everything we do, is ‘Agile and lean in working with methods’. We focus on the essentials. When changing your method take evolutionary steps starting from your existing method, for example.
“Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning”. It is a very important base on top of which we now can move forward. The attached ‘Semat – Three Year Vision’ attempts to paint the picture of where we believe we will be in three year’s time. We don’t just want to tell you where we want to be but we also want to be able to measure where we are.
It seems clear that we will get a new standard. The real question is whether this new standard will make a difference to the world. Will we be able to reach the millions of developers, which so far have had no or very little interest in working with methods? We believe so of course, but there are many things we need to do different. These differentiators will be discussed in another paper. One thing is for sure, we won’t get there without the help and contributions by many of Semat’s supporters.
To read more please visit
 Winston Churchill.
Semat has reached new regions in the world. In April 2010 a new chapter of Semat was founded in China—the Semat China—and this chapter is now ready to function.
This week another chapter of Semat was founded in Colombia—The Semat Latin America. It covers Latin America, which includes South America and Central America.
During the Latin American Software Engineering Symposium (LASES 2011), which took place in Medellin (Colombia), Semat and its mission were introduced by Ivar Jacobson. The symposium was attended by senior academics from many universities in different countries in Latin America. Apart from Colombia, there were representatives from Argentina, Brazil, Chile, Mexico, Peru, and Venezuela.
With a resounding support from the vast majority of the symposium participants Semat Latin America was founded. More than 80 people signed a foundation statement. An initial executive committee with nine members representing the academic world was selected:
- Carlos Mario Zapata J. (Colombia)
- Raquel Anaya (Colombia)
- José Antonio Pow-Sang (Perú)
- Jonás Arturo Montilva C. (Venezuela)
- Ramón García-Martínez (Argentina)
- Claudio Meneses Villegas (Chile)
- Hanna Oktaba (México)
- Edison Spina (Brasil)
- Marcel J. Simonette (Brasil)
Professor Carlos Mario Zapata Jaramillo from the National University of Colombia was elected as the chairman. The committee’s immediate task is to get informed about what Semat has achieved so far and to agree on how to move forward. The committee’s members agree on searching for a more balanced constitution of the committee itself. Representatives from industry and practitioners must be included in the committee, in order to reach the desired balance of the Semat Latin America.
Semat Blog Release by Shihong Huang
We are now happy to declare that Semat has reached a milestone in its history. The OMG RFP (Request for Proposal), entitled “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”, was approved by OMG on June 24, 2011. The RFP is heavily inspired by the work of Semat. Becoming adopted as part of a standard body is a great achievement. The full version of OMG RFP can be found in .
1. Challenges we face
The number of people developing software is exceeding 10 millions. Some sources (for instance Evans Data Corporation) estimated in 2010 the number was 20 millions. Most of these people have no clearly defined method to follow; they just do it. Most of them think they know why, what and how they work. So even if not explicitly defined, they have a method. However, improving these methods to help developers build software better, faster and with happier clients and developers, we need to better understand what software engineering is. This is why Semat was created with the vision to refound software engineering as a rigorous discipline based on a solid theory, proven principles and best practices.
Some Semat theses include:
(1) Every method is just a composition of practices, either human- or technology-related.
(2) Practices are reusable over many application areas and technologies.
(3) Underneath all practices is a small kernel of universals, things that we always have, do or produce when building software.
(4) Methods and practices are described by a light Language.
Could we address the need of millions of practitioners, could we bridge the vast chasm between academic research and industrial practices? Today, by reaching this milestone of Semat, we are more confident than ever – yes, we can. However, it won’t be easy. We need to create something fundamentally new.
2. Two principles we follow
First, we will rely on two leading principles – ‘separation of concerns’ and ‘agile in everything’, inspired by which some distinctive features have been found. We believe these features are essential to what we would like to achieve.
2.1. Separation of Concerns
A key principle of the Semat initiative is the principle of Separation of Concerns (SoC) (http://en.wikipedia.org/wiki/Separation_of_concerns). It penetrates much of what we do. It allows us to specify a core, and then through extensions without changing or complicating the core, add what is needed for the more specific cases. In this way we can be inclusive of all relevant work in software engineering and not excluding anything that is or will be beneficial to someone. Below are some examples of the application of this principle, more to come later.
(1) It separates at least two different views of methods: the practitioners’ view and the method engineers’ view. The primary users of methods are project practitioners (developers, testers, project leads, etc.). We should target the practitioners uncompromisingly. Through extensions the result will also support method engineers efficiently without complicating its usage for the practitioners.
(2) It separates the essentials from the non-essentials, such as key guidance from detailed guidance, or explicit knowledge from tacit knowledge.
(3) It separates the generalized definitions of terms from specialized definition details, allowing for the inclusion, rather than the exclusion of earlier work on methods.
2.2. Agile in everything
Being agile and light-weight is the key to be successful. We should be agile in whatever we do, nothing else would be acceptable. In a separate blog we have made an attempt to capture what we mean by ‘agile in everything’. Please, read and give your comments and suggestions.
3. Three distinctive features we identify
In our previous blog “The key points of the OMG RFP draft: call for a widely accepted kernel” (http://sematblog.wordpress.com/), we highlighted three striking features of this initiative:
(1) Finding a kernel
It emphasizes that the RFP is not creating a new method; instead, it is to build a foundation that “consists of a kernel of software engineering domain concepts and relationships that is extensible (scalable), flexible and easy to use”.
(2) Practitioners focused
The kernel has to be agile and lightweight to be successful. It focuses on people who do the work: the practitioners (e.g., analysts, developers, testers, etc). This foundation is created by practitioners, and serves the practitioners. Without getting the practitioners to adopt the result of this initiative, it will frankly just be an intellectual exercise.
(3) Focus on the usage of methods, not the definition of them
The priorities of practitioners are to actually develop software and to get the job done. They can do this without a lot of descriptions. This is why our language must primarily focus on method usage, whereas method definition is a secondary objective. The use of a method can be defined as the carrying out of that method in the context of a specific project effort.
4. Call for participation
Getting the RFP approved by OMG was one of the major milestones of Semat. Quoting Churchill: “Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Now we need to create something that will go beyond anything previously done by any standards body working with methods: getting the standard adopted by the broad developer community.
This is a challenge that cannot be overestimated. This requires new fresh ideas that are not typical for standard bodies and methods work. Fortunately, the Semat teams have several such new ideas. ‘Separation of concerns’ and ‘agile in everything’ will guide us, but more is needed. Thus we would like to have more very talented people involved in our work.
Please let’s us hear from you – your feedback and comments on our blogs. Your involvements are critical in making a difference to the community.
 OMG RFP “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”, online at
(Reading tips: You can skip sections 1—5, since they just introduce general rules for OMG proposals and standards; you can start reading section 6, since the actual technical content starts here.)
In our experience, to be successful with methods you need to be agile, light and lean in everything you do while working with methods. This means, for example, that how you build your method, adapt it as you learn more and use it while developing real software all need to be agile, lean and light. Even if the method you work with is not agile, for instance it is based on a waterfall approach, you need to be agile in how you improve it over time.
To some extent inspired by the principles behind the agile manifesto, (see http://agilemanifesto.org/principles.html), we have come up with 9 principles we believe are useful in working with methods:
(1) Active practitioner engagement is a must. Everyone in the team is important and should make valuable contributions to its way of working.
(2) The best method to start from is the one the team already has. Don’t be revolutionary, be evolutionary – focus on delivery and your customer while improving the way you work.
(3) Empower the team to change the method to fit their experience and the project context. The most appropriate method emerges from the team itself.
(4) Replace one practice at a time and make sure it works before moving on to the next. When you change, you experiment. Make sure your experiments have value, and that you understand the cause and effect.
(5) Continually inspect and adapt your practices to address the challenges facing the team.
(6) Make methods as simple as possible, but not simpler. Focusing on the essentials, the things that if not done correctly threaten the success of the project.
(7) Build trust with your customer, deliver on your promises, and be consistent.
(8) Better, faster, cheaper, and happier (teams as well as customers) are the primary measure of success.
(9) Promote continuous and sustainable improvement.
We are sure these principles can be improved, maybe even replaced with better ones. Please, contribute by suggesting changes. At some point in time we want to be able to publish something stable for many years to come.
– Ivar Jacobson, Paul McMahon, Ed Seymour, Ian Spence