The work on Semat has been going on for almost two years.  We have not published much that can be of any practical value to our target groups, which are the industry, represented by developers and their teams, and the academics, represented by instructors and researchers.  This is not because we want to be secretive.  On the contrary, we want to be as open as possible. We have worked on implementing the vision we laid out almost two years ago, and while implementing it, there has been little to discuss widely. However, we are now reaching a point where meaningful results can be shared with our community.

On February 22, 2012 members of Semat will (as a response to an OMG RFP) publish a proposal for a new standard in working with methods, a proposal that is fundamentally different from anything we as a community have seen in the past. Other parties will most likely also submit proposals and at the end of this year we should have reached an agreement on what OMG in its rather formal style refers to as “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”.

At the end of 2012 we should expect to will have a kernel in software engineering.  The kernel has three faces, which are important as a starting point for the whole initiative:

1.    The kernel is executable. 

The kernel is not a static dictionary that you read. It is dynamic and executable. By that we mean you can make the kernel come alive. You can use it in real life to run an endeavor (e.g. a project or sprint). It includes the essential elements always prevalent in every endeavor such as Requirements, Software System, Team, and Work.  These elements have states representing progress and health, so as the endeavor moves forward the states of these elements progress from state to state.

2.    A blueprint for growth. 

The kernel includes just the essentials, but it is extensible. On top of the kernel practices can be added and composed to support any existing method (to the best of our knowledge). Some larger endeavors may extend the kernel with more essential elements.  The blueprint helps us to grow the kernel to whatever size is needed.

3.    Principles and values. 

The principles and values help us to make decisions that are not guided by the blueprint.  In everything we do we apply the principle of ‘separation of concerns’.  For instance, we separate what the practitioner wants to work with from what a process engineer needs, in such a way that the practitioner doesn’t need to ‘see’ more than what is of interest to her/him.  We also separate practitioners from one another; all developers do not have the same level of interest.

An example of a fundamental principle applied in everything we do, is ‘Agile and lean in working with methods’.  We focus on the essentials.  When changing your method take evolutionary steps starting from your existing method, for example.

“Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning”[1]. It is a very important base on top of which we now can move forward.  The attached ‘Semat – Three Year Vision’ attempts to paint the picture of where we believe we will be in three year’s time.  We don’t just want to tell you where we want to be but we also want to be able to measure where we are.

It seems clear that we will get a new standard. The real question is whether this new standard will make a difference to the world. Will we be able to reach the millions of developers, which so far have had no or very little interest in working with methods? We believe so of course, but there are many things we need to do different.  These differentiators will be discussed in another paper.  One thing is for sure, we won’t get there without the help and contributions by many of Semat’s supporters.

To read more please visit

http://www.semat.org/pub/Main/SematDocuments/Semat_-_Three_Year_Vision13Jan12.pdf

Thank you!

This week another chapter of Semat was founded in Colombia—The Semat Latin America. It covers Latin America, which includes South America and Central America.

During the Latin American Software Engineering Symposium (LASES 2011), which took place in Medellin (Colombia), Semat and its mission were introduced by Ivar Jacobson. The symposium was attended by senior academics from many universities in different countries in Latin America. Apart from Colombia, there were representatives from Argentina, Brazil, Chile, Mexico, Peru, and Venezuela.

With a resounding support from the vast majority of the symposium participants Semat Latin America was founded. More than 80 people signed a foundation statement. An initial executive committee with nine members representing the academic world was selected:

• Carlos Mario Zapata J. (Colombia)
• Raquel Anaya (Colombia)
• José Antonio Pow-Sang (Perú)
• Jonás Arturo Montilva C. (Venezuela)
• Ramón García-Martínez (Argentina)
• Claudio Meneses Villegas (Chile)
• Hanna Oktaba (México)
• Edison Spina (Brasil)
• Marcel J. Simonette (Brasil)

Professor Carlos Mario Zapata Jaramillo from the National University of Colombia was elected as the chairman. The committee’s immediate task is to get informed about what Semat has achieved so far and to agree on how to move forward. The committee’s members agree on searching for a more balanced constitution of the committee itself. Representatives from industry and practitioners must be included in the committee, in order to reach the desired balance of the Semat Latin America.

We are now happy to declare that Semat has reached a milestone in its history. The OMG RFP (Request for Proposal), entitled “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”, was approved by OMG on June 24, 2011. The RFP is heavily inspired by the work of Semat.  Becoming adopted as part of a standard body is a great achievement. The full version of OMG RFP can be found in [1].

1. Challenges we face

The number of people developing software is exceeding 10 millions.  Some sources (for instance Evans Data Corporation) estimated in 2010 the number was 20 millions.  Most of these people have no clearly defined method to follow; they just do it.  Most of them think they know why, what and how they work. So even if not explicitly defined, they have a method.  However, improving these methods to help developers build software better, faster and with happier clients and developers, we need to better understand what software engineering is.  This is why Semat was created with the vision to refound software engineering as a rigorous discipline based on a solid theory, proven principles and best practices.

Some Semat theses include:

(1)        Every method is just a composition of practices, either human- or technology-related.

(2)        Practices are reusable over many application areas and technologies.

(3)        Underneath all practices is a small kernel of universals, things that we always have, do or produce when building software.

(4)        Methods and practices are described by a light Language.

Could we address the need of millions of practitioners, could we bridge the vast chasm between academic research and industrial practices? Today, by reaching this milestone of Semat, we are more confident than ever – yes, we can.  However, it won’t be easy.  We need to create something fundamentally new.

2. Two principles we follow

First, we will rely on two leading principles – ‘separation of concerns’ and ‘agile in everything’, inspired by which some distinctive features have been found.  We believe these features are essential to what we would like to achieve.

2.1. Separation of Concerns

A key principle of the Semat initiative is the principle of Separation of Concerns (SoC) (http://en.wikipedia.org/wiki/Separation_of_concerns).  It penetrates much of what we do. It allows us to specify a core, and then through extensions without changing or complicating the core, add what is needed for the more specific cases.  In this way we can be inclusive of all relevant work in software engineering and not excluding anything that is or will be beneficial to someone.  Below are some examples of the application of this principle, more to come later.

(1)   It separates at least two different views of methods: the practitioners’ view and the method engineers’ view. The primary users of methods are project practitioners (developers, testers, project leads, etc.).  We should target the practitioners uncompromisingly. Through extensions the result will also support method engineers efficiently without complicating its usage for the practitioners.

(2)   It separates the essentials from the non-essentials, such as key guidance from detailed guidance, or explicit knowledge from tacit knowledge.

(3)   It separates the generalized definitions of terms from specialized definition details, allowing for the inclusion, rather than the exclusion of earlier work on methods.

2.2. Agile in everything

Being agile and light-weight is the key to be successful. We should be agile in whatever we do, nothing else would be acceptable. In a separate blog we have made an attempt to capture what we mean by ‘agile in everything’.  Please, read and give your comments and suggestions.

3. Three distinctive features we identify

In our previous blog “The key points of the OMG RFP draft: call for a widely accepted kernel” (http://sematblog.wordpress.com/), we highlighted three striking features of this initiative:

(1)   Finding a kernel

It emphasizes that the RFP is not creating a new method; instead, it is to build a foundation that “consists of a kernel of software engineering domain concepts and relationships that is extensible (scalable), flexible and easy to use”.

(2)    Practitioners focused

The kernel has to be agile and lightweight to be successful. It focuses on people who do the work: the practitioners (e.g., analysts, developers, testers, etc). This foundation is created by practitioners, and serves the practitioners. Without getting the practitioners to adopt the result of this initiative, it will frankly just be an intellectual exercise.

(3)   Focus on the usage of methods, not the definition of them

The priorities of practitioners are to actually develop software and to get the job done. They can do this without a lot of descriptions.  This is why our language must primarily focus on method usage, whereas method definition is a secondary objective. The use of a method can be defined as the carrying out of that method in the context of a specific project effort.

4. Call for participation

Getting the RFP approved by OMG was one of the major milestones of Semat. Quoting Churchill: “Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Now we need to create something that will go beyond anything previously done by any standards body working with methods: getting the standard adopted by the broad developer community.

This is a challenge that cannot be overestimated.  This requires new fresh ideas that are not typical for standard bodies and methods work.  Fortunately, the Semat teams have several such new ideas. ‘Separation of concerns’ and ‘agile in everything’ will guide us, but more is needed.  Thus we would like to have more very talented people involved in our work.

Please let’s us hear from you – your feedback and comments on our blogs. Your involvements are critical in making a difference to the community.

Reference:

[1]   OMG RFP “A Foundation for the Agile Creation and Enactment of Software Engineering Methods”, online at

http://www.semat.org/pub/Main/WebHome/Foundation_for_SE_Methods_RFP_ad-11-06-26.pdf.

(Reading tips: You can skip sections 1—5, since they just introduce general rules for OMG proposals and standards; you can start reading section 6, since the actual technical content starts here.)

To some extent inspired by the principles behind the agile manifesto, (see http://agilemanifesto.org/principles.html), we have come up with 9 principles we believe are useful in working with methods:

(1)      Active practitioner engagement is a must. Everyone in the team is important and should make valuable contributions to its way of working.

(2)      The best method to start from is the one the team already has. Don’t be revolutionary, be evolutionary – focus on delivery and your customer while improving the way you work.

(3)      Empower the team to change the method to fit their experience and the project context.  The most appropriate method emerges from the team itself.

(4)      Replace one practice at a time and make sure it works before moving on to the next. When you change, you experiment. Make sure your experiments have value, and that you understand the cause and effect.

(5)      Continually inspect and adapt your practices to address the challenges facing the team.

(6)      Make methods as simple as possible, but not simpler. Focusing on the essentials, the things that if not done correctly threaten the success of the project.

(7)      Build trust with your customer, deliver on your promises, and be consistent.

(8)      Better, faster, cheaper, and happier (teams as well as customers) are the primary measure of success.

(9)      Promote continuous and sustainable improvement.

We are sure these principles can be improved, maybe even replaced with better ones.  Please, contribute by suggesting changes.  At some point in time we want to be able to publish something stable for many years to come.

– Ivar Jacobson, Paul McMahon, Ed Seymour, Ian Spence

As a developer, you care about building great software quickly and making your customers happy. A developer is anyone involved in developing software. This includes architects, analysts, programmers, testers, etc. As time goes by developers want to become more and more professional working with software. When they move from team to team or even organization to organization they want to hit the ground running and be productive as quickly as possible.

If you agree with this, then Semat is for you. We want to explain why we think so, but first we need to brief you on some key ideas of Semat. These ideas will, as we will explain further down, help you work better, faster, and be happier, which we believe is your ultimate goal. But before getting there let us tell you a story, which you as a developer might appreciate.

The Zigzag Story of Software.
In the software world we have been moving in a zigzag path for close to 40 years. We zig to one method and then zag to the next. Often it feels like we are moving from one extreme to another. While there has been evolution we have also thrown out valuable ideas along the way only to later rediscover them. Some developers are tired of the zigzag story of software. They have had enough of it.

Once, about ten years ago, when I (Ivar) spoke to a small group of 50 people, a senior guy stepped up, very frustrated, and told me that UML and RUP would be dead in five years and I would be gone as well. I like provocations so I asked him calmly what he based that upon. He told us that he had worked with software his whole life. ‘In the 1960s we all worked with assemblers, then we got Cobol and Fortran, then we got database design’ and he continued describing a zigzag path with new methodologies in every step: ‘structured programming, structured analysis and design, object-orientation, components and so on.’ He felt it was awkward hearing about yet another silver bullet. If he had continued to be in business he would have been zigging and zagging with even more bullets: UML, RUP, CMMI, XP, and now Scrum, Kanban, Lean. And, if we don’t do anything about it we will find new bullets to zigzag with forever and ever.

Semat is addressing this problem by identifying a common ground of software engineering including the essential elements we always have to work with. This common ground is made concrete in the form of a kernel. The kernel is small. It is expected to just include 10-20 elements. Our initial kernel, which we are providing to a small group for review and feedback this month, contains four elements (Team, Work, Requirements, Software System), each with a small set of progress or health states and checklists. Thus, you can easily learn the basics about the kernel in a couple of hours, as it should be.

To keep this blog short we have had to make some claims that we unfortunately can’t provide detailed evidence for here. If you follow the work of Semat (www.semat.org), you will find our claims being supported by facts.

Based on this kernel your team can describe their method in a simpler way than ever before. Even more important than describing their method is that your team will get support in using their method in a real endeavor and they can adapt it as they learn more.

Semat is focused on you – the developer.
In contrast to most previous initiatives, which have attempted to get a grip on software engineering, Semat has recognized that it is the developers who best know how to build (architect, analyze, design, code, test, manage and deploy) software. They can learn from others but eventually it is up to them to select and be responsible for succeeding with their own method. To do this, they need just enough help to know the approach they take is sound.

Now back to our statement that Semat is for you:

Semat helps you become professional
Today almost all developers learn software engineering by example. At school (universities) you learn one or two specific methods. For instance, many universities teach XP, Scrum, and UP today. Other universities are more ambitious trying to teach formal techniques in software engineering. At work you will learn the method used for the project you will participate in.

All of these different approaches are great as examples, but these examples will not give you what you need to know to work better, faster and be happier. Many who have been involved in Semat believe with good reasons that it will be significantly easier to achieve this goal once the essentials of software engineering are widely accepted. Then developers will learn software engineering at its roots. You will learn the kernel. You will also learn how to select practices that are defined on top of the kernel, to create your own method. This is what Semat will help you do. Then you have taken the first steps to achieve your goal.

Semat recognizes that each team involved in developing software has its own method. Semat can help developers make their own method better in multiple ways. Once you as a developer understand the kernel and how to apply it:
- You will be able to easily compare methods. For example, you will learn how to objectively compare Scrum with Kanban, or user stories with use cases, and select what works best in your context to help you get your job done. The kernel helps you to see the differences.
- You will know how your team can improve their existing method by considering new ideas, such as backlog-driven development. Your team’s existing method will be represented on top of the kernel and you will know how to improve by replacing an existing practice with a better one.
- Understanding the kernel will allow you to learn and evaluate new ideas quickly.
- You will easily be able to detect when a new approach is really just a minor variation on an old theme. Because you have a solid understanding of the kernel you will be able to quickly determine whether for instance iterations and sprints are really two different things, or just variations on the same thing.

Even more important is that you will actually get support while you run a software endeavor. You will get more value out of your method than ever before. This is made possible since the kernel includes essential elements prevalent in every software endeavor, such as Requirements and Software System. These elements have states with checklists to ensure you have reached a particular state. The kernel elements progress from state to state as your project progresses over its lifetime. Please do not read “waterfall” into the idea of states – modern software development relies on building a little at a time. Measuring progress is essential to all software endeavors regardless of method. Thus you will get these benefits as well:
- You will be able to plan your work based on your chosen method described on top of the kernel. This comes from the fact that it is your method, which represents what your team actually does. It is not just a description of what someone thinks you should do.
- You will be able to assess the progress of your work more objectively based on the kernel element states and checklists.
The states of the kernel provide a Sat-Nav for software endeavors. By Sat-Nav we mean a system that helps everyone understand where she is and where she is going.
- You will be able to communicate more effectively with your teammates and stakeholders because you will all use a commonly understood and well-defined terminology coming from the kernel.
- You will get standard measurement and reporting because everyone will understand and use the kernel and its well-defined terminology.
- You will, thanks to the kernel, be able to clearly identify when you are really done with what you are doing.

Referring back to the zigzag story of software, the problem isn’t that we haven’t made progress. It isn’t that UML, CMMI and XP are wrong. What is wrong is that over the years as progress has been made we have failed to build our new ideas on a kernel that allows the developer to see precisely what is new, and what isn’t. You should be able to keep what works, and replace what doesn’t work with new and better practices.

Ultimately Semat’s kernel means that you, as a developer, will be able to grow your competency year by year, rather than feeling you are always relearning the same thing over and over when each new idea comes along. This will allow you to confidently move from one team to another or one company to another without worrying that your level of professionalism has been jeopardized just because you are changing jobs. You will no longer feel that you are starting all over just because a new methodology has come along. You will achieve your goal of working better, faster, and being happier, and nothing will ever again be able to take that away from you.

(1)  It is about finding a kernel.

It emphasizes that the RFP is not creating a new method; instead, it is to build a foundation that “consists of a kernel of software engineering domain concepts and relationships that is extensible (scalable), flexible and easy to use”.

(2)  Its target group is the practitioners, not the process engineers.

The kernel has to be agile and lightweight to be successful. It focuses on people who do the work: the practitioners (e.g., analysts, developers, testers). This foundation is created by practitioners, and serves the practitioners.

(3)  Its focus is on the usage of methods, not on the definition of them.

Methods are enactable. The enactment of a method can be defined as the carrying out of that method in the context of a specific project effort.

These are the critical features that separate this initiative from previous and existing efforts in this space.  These features cannot be achieved by simply extending previous and existing work. These are the aspects that will fundamentally change our understanding of how to work with methods and processes.

Please refer to an extract of the OMG RFP draft for more detailed information:

http://www.semat.org/pub/Main/WebHome/ADTF_SEMAT_RFP_Brief_version.pdf

You can also read the Highlights of the OMG RFP Draft at:

http://www.semat.org/pub/Main/SematDocuments/OMG_RFP_Highlights.pdf

Do you agree with the three key differentiators mentioned in this blog? We would love to hear your feedback and comments. Your involvements ensure we are doing the right things and keep us in the right direction.

Thank you!

Best regards,
Shihong Huang

Semat is a community that has come together to address some profound problems in software engineering with an agreed goal of refounding it as a rigorous discipline. It will establish a new platform allowing people to describe their current and future practices, and methods so that they can be composed, compared, evaluated, queried, used, adapted, measured, taught and researched.

Semat’s primary focus is the practitioners: the architects, analysts, developers, testers, and project leads, for example, and by implication the process engineers. Furthermore, in order to be wholly successful Semat focuses on two other important groups: industry and academia.

Semat will provide a kernel that will embody the essential elements of software engineering. One of Semat’s key objectives is to provide the basis for better-shared understanding across practitioners, industry, and academia. It will be scalable, extensible and easy to use, allowing people to describe the essentials of their existing and future methods and practices.

Semat emphasizes the use of method in practice, rather than the definition of method. We understand that the way we work in practice is not always the same as the approach we say we follow. Slavishly following a process can lead to poor results: it cuts out creativity and brings in inefficiency. Successful teams evolve and adapt their ways of working away from their original stated intent; they recognise that the method is there for support and guidance rather than to constrain and restrict. Semat believes that methods should allow teams to use them when needed, help them whatever their current progress, and guide them to their next objective. A method is not just a description of what is expected to be done, but also a description of what is actually done.

Semat is not creating a new method, instead, Semat is making it easier to deliver and access methods. Semat provides a different way of looking at methods, making them a composition of a number of practices. From a Semat perspective a practice can be thought of as systematic and verifiable way of addressing particular aspects of the work at hand. Practices are mini-methods with a clear beginning and end; the intention is that the practice will deliver something of value. Practitioners are free to mix and match the practices they feel are appropriate for their work.

Semat provides a common ground for teaching software engineering. By introducing Semat’s concepts (methods, practices and the kernel) into regular software engineering curriculum, instructors can focus on the fundamental common ground and teach widely accepted practices instead of the particular ideas that form every method or process.

New curricula can be developed by utilizing the basic building blocks from Semat. These curricula provide a way to compare and show students the pros and cons of difference methods and practices they encounter during their course study.

Semat makes research and its results more relevant to industry. Researchers can develop empirical assessment methods to compare software engineering outcome, practices and methods; foster the advancement of formal foundations of software engineering, and sharpen our understanding of software engineering ideas. Research can also provide an infrastructure that serves as a test-bed and fast deployment of new ideas in the community. By tightly coupling with industry, Semat makes research results more relevant with a faster transfer of research results into industry practices.

Semat sets you free. Practitioners will be free to select and compose practices into something useful for their company, team or current endeavour.

Moving forward we need competent people to actively participate in the different task groups.  We need people with the following expertise: user experience design to give the language a graphical, intuitive syntax; formal language designers to make sure the concrete syntax is mapped to meaningful semantics; identifying and defining kernel elements (modeling expertise); metrics and measurement experts to help measure the impact of Semat on the external world and to help measure each of its practices; open source tool support for language and kernel; requirement specification of what Semat should do, and more.

Ed Seymour
Fujitsu
Follow me on Twitter @edwaado

Time flies. Semat seems to have been quiet for a long time, but for very good reasons. Silence doesn’t imply inactive. On the contrary, activities have been going on quietly and have never stopped. In this blog, I give you a few glimpses of what’s new with Semat, and how you could get involved and contribute.

Basically we have been working on three things:

1. Moving the work on the kernel and the language to OMG.  A Request for Proposal (RFP) was submitted on Feb 21, 2011, and this proposal was discussed one month later in an OMG meeting.
2. Working on a Semat response to the RFP.  This is basically just continuing what we started before.  The intention is to have the first version of the response by early this summer.
3. Working on how to move Semat forward.  Even the kernel and the language have been moved to Object Management Group (OMG), there are still a lot to do within Semat to implement the Grand Vision.  As a first effort, we have been working on two papers: a) Semat – The Three Year Vision, and b) Towards The Vision.

Since mid November 2010 around 20 people have been working on these three subjects.  Bertrand and Richard have become part of the advisory board; many other people have stepped up to work on implementing concrete ideas.

The story so far

As you may recall, the Grand Vision of Semat was to refound software engineering based on a widely agreed upon kernel representing the essence of software engineering.  The kernel would include elements covering societal and technical needs that support the concerns of industry, academia and practitioners.

Within a short period of time, a dozen corporate and academic organizations, and some 3 dozen renowned individuals from the field of software engineering and computer science, have become signatories to support the vision.  In addition, more than 1500 other supporters agreed to the call.

In November 2010, the troika agreed that we would move the work on the kernel and the language to OMG to get the proper governance we needed.

Moving the development of the kernel to OMG

In order to move the work to OMG, we have to follow a standard procedure for all OMG related work.  First, a request for proposal (RFP) has to be worked out and presented to a couple of groups within OMG.  Upon approval, people can submit proposals as responses to the RFP.

In our case an RFP has been developed by a couple of people from Semat and a couple of OMG members.  The RFP is called ‘ESSENSE: A domain-specific language and a kernel of essentials for software engineering.’  This RFP draft was first presented in early December 2010 in Santa Clara, CA. It was positively received, and then followed up by a RFP document, which was discussed at an OMG meeting in Arlington, VA, in March this year. The feedback was very constructive and passionate.  The RFP will be improved to address the relationship between the current proposal and the previous OMG standardization of SPEM (Software Process Engineering Metamodel).  The RFP will also detail the requirements for the ESSENSE language and the kernel.

The next time when an updated RFP proposal will be presented will be in June 2011 in Salt Lake City.

Our proposal to a kernel

Since March last year a group of 12-15 people have been working on a proposal for a kernel.  This team is continuing with its work and plans to respond to the OMG RFP. Despite the work has been slowing down due to focusing on the OMG RFP, we will deliver some results of value that can be used by industry within the next few months.  Those who are involved in this effort are as enthusiasm as ever about Semat, and believe even more strongly today than before based on the ongoing work that a kernel of widely accepted essentials will successfully be achieved.  The kernel will not be large.  We are estimating the initial version will only have 7-8 elements, but we have already begun to see the potential value this kernel can bring to the practitioner, to industry and to academia.

The kernel is just a first step towards the Grand Vision of Semat; much more work needs to be done from different aspects by different teams.

Semat moving forward

With the development of the kernel moving under the OMG umbrella, Semat remains active with tackling the other challenges set out in the Grand Vision. For example, we need to:

• collaborate with OMG to make sure that the results delivered meet the community’s needs
• support the community in its effort to get reusable practices,
• transfer the work to the academic community to inspire the development of new curricula and practical research.

Therefore, a vision for the next couple of years is needed.  A team of eight people has been working for more than a month to develop a proposal for a Three Year Vision of Semat. It focuses on the anticipated impact on Semat’s three key user groups: the practitioners, industry and academia, and discusses how Semat’s benefits can be realized and assessed.

As a self-organized community, the governance of ourselves is important. However, first and foremost, we need to agree on the direction we would like to go.

Call for participation

Moving forward we need competent people to actively participate in the different task groups.  We need people with the following expertise: user experience design to give the language a graphical, intuitive syntax; formal language designers to make sure the concrete syntax is mapped to meaningful semantics; identifying and defining kernel elements (modeling expertise); metrics and measurement experts to help measure the impact of Semat on the external world and to help measure each of its practices; open source tool support for language and kernel; requirement specification of what Semat should do, and more.

Final words

Arguably, the goals of Semat have from its beginning been very ambitious: to refound software engineering as a rigorous discipline based on a kernel of widely agreed upon elements.  Such a kernel works as a common ground for everything we do when developing software, thus it is the essence of software engineering.   The work will reveal many differences to the past history of process engineering, but it aligns very well with new ideas such as now promoted by agile approaches.  In fact it prepares us better for even the next ‘big thing’ living harmoniously with what we already have and have had.

Please contact me if you feel you can and are willing to contribute.

Thank you!

Shihong Huang
Member of the executive committee

This is the second email on the way ahead for Semat:

1.    The Troika is delighted to announce that the Semat community will continue to implement the Grand Vision (formulated in the call for action) and that the work on implementing the first year vision (formulated in the vision statement) will move to OMG.  The Troika’s task is done and the Troika as leaders of Semat will retire (however, see point 4 in the following).

2.    OMG is beginning work now to adopt a standard based on the Semat first-year vision.  Anyone may participate in the standardization process, which we expect to begin formally in December, 2010.

3.    The Semat community that we have built will continue to implement the Grand Vision that we three proposed a year ago.  It will flourish, contributing joint development, theoretical work, experimentation and research not only to the OMG standardization process but to the community at large.  We expect that current Semat community members — and potentially others — will respond to the OMG Request for Proposals which is coming shortly.

4.    The SEMAT Community will be managed by an Executive Committee, members yet to be determined and with advice only (NOT oversight) by an Advisory Board, members initially being Bertrand, Ivar, Richard, corporate and academic signatories and a few outstanding individuals).

- o -

As already announced Semat has taken a first step in achieving its goals.  Its work on implementing its first year vision has matured and is being turned into an RFP of OMG.  The key idea of our first year vision can be summarized in one sentence: all existing methods can be described as compositions of practices, described on a lightweight kernel consisting of essential elements and a small language.  The idea has been supported by our signatories, most importantly by our ‘customers’, that is our corporate and academic signatories. This is a great step on our way to reach our grand vision.

However, Semat will continue.  It will as, we said in our previous email to you: “also respond to the longer-term aspects of the call for action.  It will provide the platform for addressing other aspects of software engineering such as a) integration with other efforts such as SME, SPEM, OPF, EPF, UP, SWEBOK and CMMI, b) creating a sound theoretical basis for the work, c) experimentally evaluating and validating the practices, d) developing a library of (potentially competing) practices, etc.”

Moving forward we hope to be able to engage our signatories and our supporters more actively. This will be the role of the self-organizing community itself. However, some work comes from the work done thus far:

1.                            Requirements.  Semat started to work on the overall requirements of Semat lead by Martin Naedele, ABB.  These requirements go beyond the first year vision. Requirements from research, education, industry, developer, methodologists, etc.

2.                            Definitions. Definitions related to engineering and software engineering will focus on what is common ground. We don’t want to reinvent the wheel, so we will rely on previous standards when fit for purpose.

3.                            Theory.  Semat will deal with different kinds of theories and research related to methods, practices, languages, etc., both human as well as technical.

4.                            Methods and Practices. Many supporters are interested in getting access to certified practices designed to be extended.  Semat has already started to describe some practices (scrum and iterative development).  Work on a prioritized list of practices has been started.  The list includes not just traditional development practices (from requirements to test, architecture, test), but also practices from other areas such as team building, business, user experience, metrics and measurement, personal improvement, instrumentation, etc.  Around each such practice area we envision that a self-organized group will be created.  Such a group may also include researchers, closing the gap between academics and industry.

5.                            Assessment.  Much of the work on assessment is generic and not specific for the vision implementation.  Semat can potentially assess different proposals coming out from OMG.  It could also assess the practices being developed within Semat.

In the next couple of weeks work on the implementation of the first year vision will be moved to OMG.  In the same period we will engage to set up the Semat community.

We look forward working with all of you.

Best regards

– The Troika

Note: The work presented here will continue within the responsibility of OMG.

Summary

Semat was started as an initiative to revolutionize software engineering “based on a solid theory, theory, proven principles and best practices, that includes a kernel of widely-agreed elements.”

We have developed software for more than 50 years.  Independent of what kind of software we build, the specific application we are developing, the size of the software, the organization tat develops it (in-house, outsourced, offshored), and the practices being used, there is a common ground, a kernel of elements always prevalent in any software methodology.  This kernel provides the essential elements of software engineering.

The Semat initiative is clear about the search for a kernel that is agnostic to any particular method or any practices.  It is a base to describe methods in use to deliver real software; it includes practices for developing software, extensible for new uses, new technologies, new experiences, and new innovations. Such a kernel, standing on a solid theoretical ground, once widely accepted will change the software engineering field.  A common “practice infrastructure” will enable software developers to more quickly understand, compose and compare individual practices and entire methods.  It will form the basis for the appropriate governance of software organizations; it will provide appropriate freedom to the developers to use their preferred practices, melded with those of their organizations; it will allow us to evaluate and validate comparable entities; it will guide research to useful results; it will change the way that software developers are educated; and it will reduce the fashions and fads prevalent in the software methodologies and decision-making of today.

The kernel is not a new unified methodology, it is not a new software process meta-model, it is not a new body of knowledge, it is not a new modeling language, and it is not a trick to get people to build or buy more tools. The kernel is as simple as what we already have (such as teams and projects), what we already do (such as specify and implement), what we already produce (such as software systems) when we develop software independent of whether we document it or not, independent of what approach we use when we do it, independent of whether the result is good or bad.  The kernel is concrete, focused and small. Never before has the software community built something like this kernel.  It is as fresh as tomorrow, but it stands on the experience of the knowledge we as a community have captured over many decades.

Thus a widely agreed upon kernel is the beginning of Semat.

1       Prerequisites

The following documents are heavily referenced in this report.  Reading these documents before reading this document will greatly facilitate understanding what we have been doing and where we are going.

1. Call for Action statement [1]
2. Vision statement [2]
3. Alpha definition: Modeling project state through Abstract-Level Project Health Attributes [3]

2       Background

Semat was founded a year ago by Ivar Jacobson, Bertrand Meyer and Richard Soley (the Troika) who presented a call for action statement [1] which essentially said that a) “Software engineering is gravely hampered today by immature practices …” and b) “We support a process to refound software engineering based on a solid theory, proven principles and best practices…”

The statement has been endorsed by 35 signatories, all well-known individuals in the field of software engineering, 12 corporate and academic signatories representing organizations such as IBM, Microsoft, Fujitsu, Ericsson, ABB, Samsung, Peking University, and supported by close to 1400 individuals.

In Feb 2010, the Troika also presented a Vision statement [2] which outlined a solution and which laid out the work for the next 12 months.  This vision statement was adopted at the first Semat workshop in Zurich in March 2010 [4], and is guiding our work.

Since March we have been working in a number of tracks and we have had two more workshops, one in Washington DC in July [5] and another one in Milan at the end of September [6].  Since March we have focused on results and reduced our time spent on outward communication.  This is of course necessary to deliver on our goals within a tight schedule and with limited time to spend on this initiative. People who work on the initiative are all volunteers with their “day work” to do as well.  Since we now have come quite some way on the road we want to share what we have done so far with everybody.  This document intends to give a picture of what has been done so far, where the work is heading, and references other documents to complete the picture.

3       The vision – in one sentence

We will create a platform (the kernel) allowing people to describe their current and future practices and methods so that they can be composed, simulated, applied, compared, evaluated, measured, taught and researched.

4       Problem addressed by Semat

The original Semat Call for Action gave a broad definition of the problems that the Semat initiative is poised to address (here abbreviated for simplicity, see [1]):  We look like a fashion industry; we lack a theoretical basis; we have a huge number of methods which are hard to compare; we need experimental evaluation and validation;, there exists a split between industry practice and academic research.

As a consequence, we as a society are not moving forward with reasonable speed.  Exaggerating a little: we follow a zig-zag path in evolution; we don’t stand on previous work; every new good idea is accompanied by a redefinition of many already existing terms; we don’t reuse what we have already learned; in many cases universities teach software engineering only by examples (UP, XP, Scrum) and not with a solid theoretical base; research projects are not anchored in problems identified by the industry; the industry is ignorant of research that could help them; we don’t know how to distinguish good practices from bad practices; we can’t compare practices.   For example, individual practitioners get trapped into one particular method adopted by one organization and cannot easily move to another organization which has adopted a different method.

Summarizing the situation, the current status of the software engineering society costs industry and academics dearly in lost opportunities. Industry seeks ways to develop software systems that are dramatically better, faster and cheaper. Academics look to provide significantly improved education and research aligned with the practical needs of Industry.   When the troika suggested the call for action, we were asking people to join us to revolutionize software engineering.  These are strong words, but they match our conviction.

5       Key solution

In our call for action we envisioned that a solution should be “based on a solid theory, proven principles and best practices that:

• Includes a kernel of widely-agreed elements, extensible for specific uses
• Addresses both technology and people issues
• Are supported by industry, academia, researchers and users
• Supports extension in the face of changing requirements and technology”

The vision statement and the work since the vision statement was presented have developed these ideas into something more tangible:

1. A method is a composition of practices as opposed to an interconnection of process components (discipline, or similar). Practices give value one by one, they are what users want to make lean, and they provide interesting measures, all of which is the critical differentiator.
2. All methods have something in common – a kernel (a common ground).  The kernel consists of two things: the universals, and the kernel language
3. The primary users of methods and practices are project participants (developers, testers, project leads, etc.). Process engineers are also users.
4. Methods need theory – our work must stand on a solid theoretical basis.  Methods being composed of practices, practices being described in terms of universals and elements such as activities and work products; all formalized into a language is the beginning of such a theory.  Individual practices being supported by theories take our work across the whole research in software engineering.
5. Methods are enactable and executable (or operational).  Methods can be queried to discover guidance aligned with where you think you are today and help you get to where you want to go tomorrow; they are not just descriptions for developers to read, they are dynamic, supporting your day to day activities. This turns the concept of a method upside down.  Method is not just a description, but what is actually done.

Our solution as outlined here is fundamentally different from earlier approaches, such as SME, SPEM, OPF, EPF, UP, SWEBOK and CMMI.  We still believe that earlier works will play a significant role in Semat. However, since Semat has a different basis, we will have to complement these earlier works in the context of this new basis.

To reiterate, it is important to make clear that Semat is not about developing a new unified methodology.  It is instead about finding the common ground of all methods and practices.  Semat is not about evaluating methods and practices, but about facilitating evaluations in the future.

The kernel is at the core of our solution

Our solution relies on the discovery that underneath all methods is a kernel including the essential elements (the universals) of software engineering. The kernel also includes a kernel language to be used to precisely describe the universals as well as practices and entire methods [2].

For the time being the following universals have been identified but more will come: Opportunity, Stakeholder Community, Requirement, Software System, Work, Team, Method, and Practice.  These are all what we call alphas [3].  In the months ahead other kinds of universals will be identified such as activity spaces, competences or skills, etc.

6       The products of Semat

Semat needs customers and users.  This is why we have diligently worked to get corporate signatories and supporters involved. We need to know that we are doing the right things and that we are doing them right.

Our requirements are the starting point. These must be anchored among our corporate signatories.  They need to be agreed upon by the developer community. The requirements of the Semat initiative as a whole are captured as use cases allowing us to understand the problem and to assess it.  From these requirements we have also derived the requirements on the kernel. Thus, we are also specifying a use case model for the kernel and a related domain model.

With the requirements for the kernel as a start we identify and describe the universals and we design the Kernel Language with two related models; a meta-model and a formal definition. The meta-model uses UML class diagrams to describe its abstract syntax and OCL (Object Constraint Language, also part of UML) to describe its well-formedness rules.  The operational semantics is described in narrative form.  The formal definition defines the language in a mathematical form using Graphs and Transformation rules.  The Kernel Language has also a concrete graphical syntax, which of course is what the users apply when specifying methods, practices and universals.  Finally, to assess the work an assessment framework is being developed.

Work is now occurring iteratively on all these work products.  From a list of 250+ practices identified from different forum we selected a prioritized list of six practices to drive our work: Scrum, Iterative Development, User stories, Use Cases, Agile Requirements Management, Risk-Value Life Cycle.

We also prioritized the use cases that would also drive our work: Define Practice, Compose Practice, and Enact Practice.

For our first iteration, just completed, we did an architectural spike through all our work products resulting in sketches of two practices: Scrum and Iterative Development. This exercised the use case Define Practice and the domain model, referenced the relevant universals, drawing on the Meta model and produced a sketch of the practices by utilizing the concrete syntax.  The goal of the iteration was to synchronize and drive through consistency between the work products.  The report of all this work will be part of the group’s response to OMG’s RFP:

Work on the next iteration, also an architectural spike, has started.  The goal has not changed compared to the goal set in the vision statement.  We work on a kernel with a set of universals and a kernel language to be used to describe methods, practices and universals.  At that time we also plan to have a user manual for the language and a few important practices defined.

References:

[1]    Ivar Jacobson, Bertrand Meyer, and Richard Soley: “Call for Action: The Semat Initiative” Dr. Dobb’s Journal December 10, 2009. Online at http://www.drdobbs.com/architecture-and-design/222001342

[2]    Ivar Jacobson, Bertrand Meyer, and Richard Soley: “The Semat Vision Statement” online at http://www.semat.org/pub/Main/WebHome/SEMAT-vision.pdf

[3]    Bertrand Meyer: “Modeling project state through Abstract-Level Project Health Attributes” July 10, 2010 online at xxxxxxx

[4]    The 1^st Semat Workshop Report March 17-19, 2010 Zurich. Online at: http://www.semat.org/pub/Main/SematZurichMarch2010/Zurich_meeting_report.pdf

[5]    The 2^nd Semat Workshop Report, July 13 – 14, 2010 Washington D.C. Online at: http://www.semat.org/pub/Main/WebHome/2nd_Semat_Workshop_Report.pdf

[6]    The 3^rd Semat Workshop Report. September 29 – October 1, 2010. Milan Italy. Online at:

http://www.semat.org/pub/Main/WebHome/3rd_Semat_Workshop_Report.pdf

[7]    Ivar Jacobson and Bertrand Meyer: “Methods need theory” Dr. Dobb’s Journal, August 06, 2009. Online at http://www.drdobbs.com/architecture-and-design/219100242

[8]    Ivar Jacobson and Ian Spence: “Why we need a theory for software engineering” Dr. Dobb’s Journal, October 02, 2009. Online at http://www.drdobbs.com/architecture-and-design/220300840