Another step forward for Semat

As you may know from our Three Year Vision, Semat’s first step was to create a common ground for software engineering. This common ground would be manifested as a kernel of essential elements that are universal to all software development efforts, and a simple language describing methods, practices and the kernel elements. Both the kernel and the language should be widely accepted.  Right now, Semat is in the process of completing the first step, which has resulted in the following:

• A submission, called “Essence — Kernel and Language for Software Engineering” to the OMG’s RFP (Request for Proposal) “A Foundation for the Agile Creation and Enactment of Software Engineering Methods“. Two other works have been submitted to OMG. The Semat results were very well received by OMG. The Semat submission will incorporate some ideas from the other two submissions into the new “Essence” submission. Semat will also incorporate ideas from the existing standard Spem.
• Semat will be presented at the International Conference of Software Engineering (ICSE 2012) in Zurich on June 6, 2012. This will happen in the “Refounding Software Engineering” session. It is Semat’s first international publication and more publications are on the way.
• A new book `The Essence of Software Engineering – Applying the Semat kernel’ is coming – now in peer review. The book’s objective is to put the kernel into a context.  The book describes how the kernel can be used by the practitioners and how they can measure progress in an endeavor independent of which method they use. It also provides suggestions for how they can scale up, scale out or scale in their methods.

The Semat kernel is already being used on two academic project courses at KTH Royal Institute of Technology. On one project course, Mira Kajko-Mattsson is teaching the Semat kernel and evaluates its usefulness within software education. Using the kernel elements, her students continuously evaluate the progress and health of their projects.  On the other KTH project course, Anders Sjögren uses the Essence book as a base for education in software engineering with extensive lab work. Moreover, two other universities use or plan to use the Semat approach when training their students. These are Peking University and Oslo University. Experiences on the use of the Semat kernel in education will be published on Semat’s website in the next couple of months.

Getting a widely agreed upon kernel is just the beginning.  The kernel is a fundament that constitutes a critical milestone for moving forward in refounding software engineering.  To be able to systematically work with practices and methods both in the industry and academia, more work needs to be done in form of creating products, creating practice market places, constructing supporting tools, revising software engineering curricula, doing research and writing books and papers.

Thanks to the kernel, our less experienced developers (and most of us belong to this group) will become professional much easier and much faster.  Standing on the common ground, they will be able to learn from one another much better and move from one team to another much easier.  Since we focus on you, the developers, and not the process engineers, the adoption of the Semet kernel is more of a pull than a push.

The Semat kernel stands on two important principles, which are ‘agile in everything we do’ and ‘separation of concerns’.  For example, we value:

1)      What helps the least experienced developers over what helps the experts?

2)      What helps the practitioners over what helps the process engineers?

3)      Method usage over method definition?

Of course, we must support the experts and process engineers as well, but not by having the practitioners to pay a price. And, of course, we can define methods, but most importantly, we have to make methods useful while we actually work in real endeavors.

Compared to earlier attempts, the Semat approach has several differentiators that make developers more motivated to work with methods.  Maybe the most important one is that the whole approach is actionable. As already described in the Three Year Vision, the developers do not just use it for studying. They use it in their actual daily work when enacting their ways of working and when measuring their progress.   part of the whole game.  All this will help the industry.  The education at a university level will become more fundamental, more systematic and effective. The research will become more disciplined and more useful to the software community.

In parallel with working on the proposal, a small team of people has been working on a new book ‘The Essence of Software Engineering – Applying the Semat kernel’.  The authors are Ivar Jacobson, Pan Wei Ng, Paul Mc Mahon, Ian Spence, Svante Lidman. The book is now in its 3^rd draft and about 30 people have reviewed it and given constructive feedback.  We expect that the 4^th draft will be widely published.  If you are interested in reviewing and providing feedback, please send an email to ivar@ivarjacobson.com. A copy of the book will be sent to you asap.

– Ivar Jacobson and Mira Kajko-Mattsson

Semat Three-Year Vision

Ivar Jacobson, Shihong Huang, Mira Kajko-Mattsson, Paul McMahon, Ed Seymour

The work on Semat has been going on for almost two years.  We have not published much that can be of any practical value to our target groups, which are the industry, represented by developers and their teams, and the academics, represented by instructors and researchers.  This is not because we want to be secretive.  On the contrary, we want to be as open as possible. We have worked on implementing the vision we laid out almost two years ago, and while implementing it, there has been little to discuss widely. However, we are now reaching a point where meaningful results can be shared with our community.

On February 22, 2012 members of Semat will (as a response to an OMG RFP) publish a proposal for a new standard in working with methods, a proposal that is fundamentally different from anything we as a community have seen in the past. Other parties will most likely also submit proposals and at the end of this year we should have reached an agreement on what OMG in its rather formal style refers to as “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”.

At the end of 2012 we should expect to will have a kernel in software engineering.  The kernel has three faces, which are important as a starting point for the whole initiative:

1.    The kernel is executable. 

The kernel is not a static dictionary that you read. It is dynamic and executable. By that we mean you can make the kernel come alive. You can use it in real life to run an endeavor (e.g. a project or sprint). It includes the essential elements always prevalent in every endeavor such as Requirements, Software System, Team, and Work.  These elements have states representing progress and health, so as the endeavor moves forward the states of these elements progress from state to state.

2.    A blueprint for growth. 

The kernel includes just the essentials, but it is extensible. On top of the kernel practices can be added and composed to support any existing method (to the best of our knowledge). Some larger endeavors may extend the kernel with more essential elements.  The blueprint helps us to grow the kernel to whatever size is needed.

3.    Principles and values. 

The principles and values help us to make decisions that are not guided by the blueprint.  In everything we do we apply the principle of ‘separation of concerns’.  For instance, we separate what the practitioner wants to work with from what a process engineer needs, in such a way that the practitioner doesn’t need to ‘see’ more than what is of interest to her/him.  We also separate practitioners from one another; all developers do not have the same level of interest.

An example of a fundamental principle applied in everything we do, is ‘Agile and lean in working with methods’.  We focus on the essentials.  When changing your method take evolutionary steps starting from your existing method, for example.

“Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning”[1]. It is a very important base on top of which we now can move forward.  The attached ‘Semat – Three Year Vision’ attempts to paint the picture of where we believe we will be in three year’s time.  We don’t just want to tell you where we want to be but we also want to be able to measure where we are.

It seems clear that we will get a new standard. The real question is whether this new standard will make a difference to the world. Will we be able to reach the millions of developers, which so far have had no or very little interest in working with methods? We believe so of course, but there are many things we need to do different.  These differentiators will be discussed in another paper.  One thing is for sure, we won’t get there without the help and contributions by many of Semat’s supporters.

To read more please visit

http://www.semat.org/pub/Main/SematDocuments/Semat_-_Three_Year_Vision13Jan12.pdf

Thank you!

---

[1] Winston Churchill.

Semat has reached Latin America

Semat has reached new regions in the world. In April 2010 a new chapter of Semat was founded in China—the Semat China—and this chapter is now ready to function.

This week another chapter of Semat was founded in Colombia—The Semat Latin America. It covers Latin America, which includes South America and Central America.

During the Latin American Software Engineering Symposium (LASES 2011), which took place in Medellin (Colombia), Semat and its mission were introduced by Ivar Jacobson. The symposium was attended by senior academics from many universities in different countries in Latin America. Apart from Colombia, there were representatives from Argentina, Brazil, Chile, Mexico, Peru, and Venezuela.

With a resounding support from the vast majority of the symposium participants Semat Latin America was founded. More than 80 people signed a foundation statement. An initial executive committee with nine members representing the academic world was selected:

• Carlos Mario Zapata J. (Colombia)
• Raquel Anaya (Colombia)
• José Antonio Pow-Sang (Perú)
• Jonás Arturo Montilva C. (Venezuela)
• Ramón García-Martínez (Argentina)
• Claudio Meneses Villegas (Chile)
• Hanna Oktaba (México)
• Edison Spina (Brasil)
• Marcel J. Simonette (Brasil)

Professor Carlos Mario Zapata Jaramillo from the National University of Colombia was elected as the chairman. The committee’s immediate task is to get informed about what Semat has achieved so far and to agree on how to move forward. The committee’s members agree on searching for a more balanced constitution of the committee itself. Representatives from industry and practitioners must be included in the committee, in order to reach the desired balance of the Semat Latin America.

A Major Milestone: On the way to a new standard

Semat Blog Release by Shihong Huang

We are now happy to declare that Semat has reached a milestone in its history. The OMG RFP (Request for Proposal), entitled “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”, was approved by OMG on June 24, 2011. The RFP is heavily inspired by the work of Semat.  Becoming adopted as part of a standard body is a great achievement. The full version of OMG RFP can be found in [1].

1. Challenges we face

The number of people developing software is exceeding 10 millions.  Some sources (for instance Evans Data Corporation) estimated in 2010 the number was 20 millions.  Most of these people have no clearly defined method to follow; they just do it.  Most of them think they know why, what and how they work. So even if not explicitly defined, they have a method.  However, improving these methods to help developers build software better, faster and with happier clients and developers, we need to better understand what software engineering is.  This is why Semat was created with the vision to refound software engineering as a rigorous discipline based on a solid theory, proven principles and best practices.

Some Semat theses include:

(1)        Every method is just a composition of practices, either human- or technology-related.

(2)        Practices are reusable over many application areas and technologies.

(3)        Underneath all practices is a small kernel of universals, things that we always have, do or produce when building software.

(4)        Methods and practices are described by a light Language.

Could we address the need of millions of practitioners, could we bridge the vast chasm between academic research and industrial practices? Today, by reaching this milestone of Semat, we are more confident than ever – yes, we can.  However, it won’t be easy.  We need to create something fundamentally new.

2. Two principles we follow

First, we will rely on two leading principles – ‘separation of concerns’ and ‘agile in everything’, inspired by which some distinctive features have been found.  We believe these features are essential to what we would like to achieve.

2.1. Separation of Concerns

A key principle of the Semat initiative is the principle of Separation of Concerns (SoC) (http://en.wikipedia.org/wiki/Separation_of_concerns).  It penetrates much of what we do. It allows us to specify a core, and then through extensions without changing or complicating the core, add what is needed for the more specific cases.  In this way we can be inclusive of all relevant work in software engineering and not excluding anything that is or will be beneficial to someone.  Below are some examples of the application of this principle, more to come later.

(1)   It separates at least two different views of methods: the practitioners’ view and the method engineers’ view. The primary users of methods are project practitioners (developers, testers, project leads, etc.).  We should target the practitioners uncompromisingly. Through extensions the result will also support method engineers efficiently without complicating its usage for the practitioners.

(2)   It separates the essentials from the non-essentials, such as key guidance from detailed guidance, or explicit knowledge from tacit knowledge.

(3)   It separates the generalized definitions of terms from specialized definition details, allowing for the inclusion, rather than the exclusion of earlier work on methods.

2.2. Agile in everything

Being agile and light-weight is the key to be successful. We should be agile in whatever we do, nothing else would be acceptable. In a separate blog we have made an attempt to capture what we mean by ‘agile in everything’.  Please, read and give your comments and suggestions.

3. Three distinctive features we identify

In our previous blog “The key points of the OMG RFP draft: call for a widely accepted kernel” (http://sematblog.wordpress.com/), we highlighted three striking features of this initiative:

(1)   Finding a kernel

It emphasizes that the RFP is not creating a new method; instead, it is to build a foundation that “consists of a kernel of software engineering domain concepts and relationships that is extensible (scalable), flexible and easy to use”.

(2)    Practitioners focused

The kernel has to be agile and lightweight to be successful. It focuses on people who do the work: the practitioners (e.g., analysts, developers, testers, etc). This foundation is created by practitioners, and serves the practitioners. Without getting the practitioners to adopt the result of this initiative, it will frankly just be an intellectual exercise.

(3)   Focus on the usage of methods, not the definition of them

The priorities of practitioners are to actually develop software and to get the job done. They can do this without a lot of descriptions.  This is why our language must primarily focus on method usage, whereas method definition is a secondary objective. The use of a method can be defined as the carrying out of that method in the context of a specific project effort.

4. Call for participation

Getting the RFP approved by OMG was one of the major milestones of Semat. Quoting Churchill: “Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Now we need to create something that will go beyond anything previously done by any standards body working with methods: getting the standard adopted by the broad developer community.

This is a challenge that cannot be overestimated.  This requires new fresh ideas that are not typical for standard bodies and methods work.  Fortunately, the Semat teams have several such new ideas. ‘Separation of concerns’ and ‘agile in everything’ will guide us, but more is needed.  Thus we would like to have more very talented people involved in our work.

Please let’s us hear from you – your feedback and comments on our blogs. Your involvements are critical in making a difference to the community.

Reference:

[1]   OMG RFP “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”, online at

http://www.semat.org/pub/Main/WebHome/Foundation_for_SE_Methods_RFP_ad-11-06-26.pdf.

(Reading tips: You can skip sections 1—5, since they just introduce general rules for OMG proposals and standards; you can start reading section 6, since the actual technical content starts here.)

Agile in Everything

In our experience, to be successful with methods you need to be agile, light and lean in everything you do while working with methods. This means, for example, that how you build your method, adapt it as you learn more and use it while developing real software all need to be agile, lean and light. Even if the method you work with is not agile, for instance it is based on a waterfall approach, you need to be agile in how you improve it over time.

To some extent inspired by the principles behind the agile manifesto, (see http://agilemanifesto.org/principles.html), we have come up with 9 principles we believe are useful in working with methods:

(1)      Active practitioner engagement is a must. Everyone in the team is important and should make valuable contributions to its way of working.

(2)      The best method to start from is the one the team already has. Don’t be revolutionary, be evolutionary – focus on delivery and your customer while improving the way you work.

(3)      Empower the team to change the method to fit their experience and the project context.  The most appropriate method emerges from the team itself.

(4)      Replace one practice at a time and make sure it works before moving on to the next. When you change, you experiment. Make sure your experiments have value, and that you understand the cause and effect.

(5)      Continually inspect and adapt your practices to address the challenges facing the team.

(6)      Make methods as simple as possible, but not simpler. Focusing on the essentials, the things that if not done correctly threaten the success of the project.

(7)      Build trust with your customer, deliver on your promises, and be consistent.

(8)      Better, faster, cheaper, and happier (teams as well as customers) are the primary measure of success.

(9)      Promote continuous and sustainable improvement.

We are sure these principles can be improved, maybe even replaced with better ones.  Please, contribute by suggesting changes.  At some point in time we want to be able to publish something stable for many years to come.

– Ivar Jacobson, Paul McMahon, Ed Seymour, Ian Spence

You are a developer – what is in Semat for you?

Ivar Jacobson and Paul McMahon

As a developer, you care about building great software quickly and making your customers happy. A developer is anyone involved in developing software. This includes architects, analysts, programmers, testers, etc. As time goes by developers want to become more and more professional working with software. When they move from team to team or even organization to organization they want to hit the ground running and be productive as quickly as possible.

If you agree with this, then Semat is for you. We want to explain why we think so, but first we need to brief you on some key ideas of Semat. These ideas will, as we will explain further down, help you work better, faster, and be happier, which we believe is your ultimate goal. But before getting there let us tell you a story, which you as a developer might appreciate.

The Zigzag Story of Software.
In the software world we have been moving in a zigzag path for close to 40 years. We zig to one method and then zag to the next. Often it feels like we are moving from one extreme to another. While there has been evolution we have also thrown out valuable ideas along the way only to later rediscover them. Some developers are tired of the zigzag story of software. They have had enough of it.

Once, about ten years ago, when I (Ivar) spoke to a small group of 50 people, a senior guy stepped up, very frustrated, and told me that UML and RUP would be dead in five years and I would be gone as well. I like provocations so I asked him calmly what he based that upon. He told us that he had worked with software his whole life. ‘In the 1960s we all worked with assemblers, then we got Cobol and Fortran, then we got database design’ and he continued describing a zigzag path with new methodologies in every step: ‘structured programming, structured analysis and design, object-orientation, components and so on.’ He felt it was awkward hearing about yet another silver bullet. If he had continued to be in business he would have been zigging and zagging with even more bullets: UML, RUP, CMMI, XP, and now Scrum, Kanban, Lean. And, if we don’t do anything about it we will find new bullets to zigzag with forever and ever.

Semat is addressing this problem by identifying a common ground of software engineering including the essential elements we always have to work with. This common ground is made concrete in the form of a kernel. The kernel is small. It is expected to just include 10-20 elements. Our initial kernel, which we are providing to a small group for review and feedback this month, contains four elements (Team, Work, Requirements, Software System), each with a small set of progress or health states and checklists. Thus, you can easily learn the basics about the kernel in a couple of hours, as it should be.

To keep this blog short we have had to make some claims that we unfortunately can’t provide detailed evidence for here. If you follow the work of Semat (www.semat.org), you will find our claims being supported by facts.

Based on this kernel your team can describe their method in a simpler way than ever before. Even more important than describing their method is that your team will get support in using their method in a real endeavor and they can adapt it as they learn more.

Semat is focused on you – the developer.
In contrast to most previous initiatives, which have attempted to get a grip on software engineering, Semat has recognized that it is the developers who best know how to build (architect, analyze, design, code, test, manage and deploy) software. They can learn from others but eventually it is up to them to select and be responsible for succeeding with their own method. To do this, they need just enough help to know the approach they take is sound.

Now back to our statement that Semat is for you:

Semat helps you become professional
Today almost all developers learn software engineering by example. At school (universities) you learn one or two specific methods. For instance, many universities teach XP, Scrum, and UP today. Other universities are more ambitious trying to teach formal techniques in software engineering. At work you will learn the method used for the project you will participate in.

All of these different approaches are great as examples, but these examples will not give you what you need to know to work better, faster and be happier. Many who have been involved in Semat believe with good reasons that it will be significantly easier to achieve this goal once the essentials of software engineering are widely accepted. Then developers will learn software engineering at its roots. You will learn the kernel. You will also learn how to select practices that are defined on top of the kernel, to create your own method. This is what Semat will help you do. Then you have taken the first steps to achieve your goal.

Semat recognizes that each team involved in developing software has its own method. Semat can help developers make their own method better in multiple ways. Once you as a developer understand the kernel and how to apply it:
- You will be able to easily compare methods. For example, you will learn how to objectively compare Scrum with Kanban, or user stories with use cases, and select what works best in your context to help you get your job done. The kernel helps you to see the differences.
- You will know how your team can improve their existing method by considering new ideas, such as backlog-driven development. Your team’s existing method will be represented on top of the kernel and you will know how to improve by replacing an existing practice with a better one.
- Understanding the kernel will allow you to learn and evaluate new ideas quickly.
- You will easily be able to detect when a new approach is really just a minor variation on an old theme. Because you have a solid understanding of the kernel you will be able to quickly determine whether for instance iterations and sprints are really two different things, or just variations on the same thing.

Even more important is that you will actually get support while you run a software endeavor. You will get more value out of your method than ever before. This is made possible since the kernel includes essential elements prevalent in every software endeavor, such as Requirements and Software System. These elements have states with checklists to ensure you have reached a particular state. The kernel elements progress from state to state as your project progresses over its lifetime. Please do not read “waterfall” into the idea of states – modern software development relies on building a little at a time. Measuring progress is essential to all software endeavors regardless of method. Thus you will get these benefits as well:
- You will be able to plan your work based on your chosen method described on top of the kernel. This comes from the fact that it is your method, which represents what your team actually does. It is not just a description of what someone thinks you should do.
- You will be able to assess the progress of your work more objectively based on the kernel element states and checklists.
The states of the kernel provide a Sat-Nav for software endeavors. By Sat-Nav we mean a system that helps everyone understand where she is and where she is going.
- You will be able to communicate more effectively with your teammates and stakeholders because you will all use a commonly understood and well-defined terminology coming from the kernel.
- You will get standard measurement and reporting because everyone will understand and use the kernel and its well-defined terminology.
- You will, thanks to the kernel, be able to clearly identify when you are really done with what you are doing.

Referring back to the zigzag story of software, the problem isn’t that we haven’t made progress. It isn’t that UML, CMMI and XP are wrong. What is wrong is that over the years as progress has been made we have failed to build our new ideas on a kernel that allows the developer to see precisely what is new, and what isn’t. You should be able to keep what works, and replace what doesn’t work with new and better practices.

Ultimately Semat’s kernel means that you, as a developer, will be able to grow your competency year by year, rather than feeling you are always relearning the same thing over and over when each new idea comes along. This will allow you to confidently move from one team to another or one company to another without worrying that your level of professionalism has been jeopardized just because you are changing jobs. You will no longer feel that you are starting all over just because a new methodology has come along. You will achieve your goal of working better, faster, and being happier, and nothing will ever again be able to take that away from you.

The key points of the OMG RFP Draft: Call for a widely accepted kernel

The OMG RFP is entitled “A Foundation for the Agile Creation and Enactment of Software Engineering Methods.” This title is selected to loudly and clearly strike for three distinctive key points driven by Semat:

(1)  It is about finding a kernel.

It emphasizes that the RFP is not creating a new method; instead, it is to build a foundation that “consists of a kernel of software engineering domain concepts and relationships that is extensible (scalable), flexible and easy to use”.

(2)  Its target group is the practitioners, not the process engineers.

The kernel has to be agile and lightweight to be successful. It focuses on people who do the work: the practitioners (e.g., analysts, developers, testers). This foundation is created by practitioners, and serves the practitioners.

(3)  Its focus is on the usage of methods, not on the definition of them.

Methods are enactable. The enactment of a method can be defined as the carrying out of that method in the context of a specific project effort.

These are the critical features that separate this initiative from previous and existing efforts in this space.  These features cannot be achieved by simply extending previous and existing work. These are the aspects that will fundamentally change our understanding of how to work with methods and processes.

Please refer to an extract of the OMG RFP draft for more detailed information:

http://www.semat.org/pub/Main/WebHome/ADTF_SEMAT_RFP_Brief_version.pdf

You can also read the Highlights of the OMG RFP Draft at:

http://www.semat.org/pub/Main/SematDocuments/OMG_RFP_Highlights.pdf

Do you agree with the three key differentiators mentioned in this blog? We would love to hear your feedback and comments. Your involvements ensure we are doing the right things and keep us in the right direction.

Thank you!

Best regards,
Shihong Huang