The Value of Models in Agile Ways of Working

Agile methodologies are based on the notion of embracing change over following detailed plans; to quote the Agile Manifesto [5]: working software over comprehensive documentation. The future is uncertain. Customer needs change and so do external conditions; any plans you draw up will need to be altered anyway. This means that the role of architecture in an Agile context is different from the "waterfall" approach, where you first draw up plans and then execute them. In an Agile context the plans themselves, and hence the architecture, are constantly being adapted to match changing circumstances and requirements. This does not mean that architecture is no longer relevant in Agile. On the contrary, architecture captures the shared vision needed by Agile teams.

In Agile methodologies, the architecture is supposed to be created or incremented "when needed". One of the principles behind the Agile Manifesto [5] states that the best architecture arises from Agile teams. This is a naturally "emerging" architecture. However, this is focused on software development and software architecture, not Enterprise Architecture.

Software or system architecture, which is closest to a daily changing reality, needs to be created by the experts directly in touch with that reality; i.e., the members of Agile teams realizing that system. Classical software architect roles are increasingly being performed by experienced Agile team members rather than by separate experts. This kind of architecture is often expressed in sketches on whiteboards or perhaps simple Unified Modeling Language™ (UML®) models. While this is not the target user group of ArchiMate models, using a subset of the ArchiMate language and notation may come in handy as it helps ensure that people understand each other’s diagrams, and enables a continuum in the architecture description, whether drawn or modeled.

Solution architectures for larger systems with lots of connections and interactions do require a more thorough architecture focus. Keeping track of these dependencies can no longer be done with sketches on whiteboards or pieces of string on your Agile planning board. It is just too easy to lose track of crucial dependencies, let alone of the changes in them.

It is important to stress the value of models over just pictures. Models can be analyzed, transformed, and visualized in different ways, so they can be the basis for much more than just communicating the essence of an architecture. Moreover, models are much easier to maintain and keep consistent than pictures. Imagine you express an architecture of a complex system in Microsoft® PowerPoint or Visio and some fundamental aspects are changed. You will then need to change that manually, and check and fix all inconsistencies arising from such a change. With models and appropriate modeling tools, this is far easier.

Moreover, for true business agility you need to take care of much more than "working software" as the sole measure of progress (as advocated by the Agile Manifesto). This starts from a solid understanding of the business at hand, the challenges that need to be answered, the various constraints ranging from resources, cost, and risk to regulatory compliance, competitive pressures, and much more.

In addition, you need to ensure that the results developed (the working software, but also the organizational structure, capabilities, business processes, and possibly down to physical infrastructure) are also flexible and not an "instant legacy" that will be hard to change in the future. These aspects of understanding the broader context and impact are where Enterprise Architecture and solution architecture add real value and can help increase the agility of an enterprise. Key in avoiding this "instant legacy" problem is deferring decisions for as long as possible, since deciding too early without sufficient information increases the risk of that decision being wrong. Moreover, you should explore alternatives and not rush into a specific solution too soon.

To understand this broader context and to evaluate alternative solutions to your business challenge, architecture models are extremely useful, and even more so in a fast-moving Agile organization. We see three key issues standing in the way of true business agility:

The ArchiMate language can help in modeling this architecture. A graphical visualization is a powerful way to convey ideas and concepts, and using an ArchiMate model is more efficient than producing a long textual explanation. Furthermore, since the ArchiMate language is intentionally relatively simple and not too detailed, the effort in creating and maintaining models is reasonable and will not slow down an Agile team. It also offers a mechanism for defining stakeholder-oriented viewpoints that capture specific parts of the architecture to address particular concerns of these stakeholders, something not offered by domain-oriented languages like UML or Business Process Modeling Notation™ (BPMN™). During the incremental phases of Agile, it can be quicker and easier to make a modification to an ArchiMate model than to a lengthy descriptive document. Moreover, ensuring the consistency of models is a lot easier than with text.

Note that such models are explicitly not intended as big, up-front designs created by a small team of experts. Rather, they should be built and owned collectively by everyone involved in change in the enterprise. Different roles contribute their parts and can see how these are connected to the parts of others. This shared understanding then greatly facilitates the agility and speed that any adaptive enterprise needs.

As mentioned, Agile methodologies are based on the notion of embracing change over following detailed plans, because the future is uncertain. This uncertainty increases the further into the future you look. This implies that the longer-term direction of the enterprise should be given in general terms rather than in detailed plans, since those will be overtaken by the world changing around you.

Architecture models, especially those with a longer-term outlook, should therefore express intent rather than detailed design. This intent is usually quite stable: the strategy of an organization does not change on a daily basis, and translating that into action is where intentional architecture comes into play. A key role of Enterprise Architects is to convey that intent.

In addition to emerging architecture, Agile methodology requires some intentional architecture – architecture that is created beforehand. Intentional architecture may consist of both enablers (e.g., a data storage system should be in place before data storage features can be implemented by application components) and constraints (e.g., data storage must comply with the European General Data Protection Regulations (GDPR)). In addition, and perhaps more importantly, intentional architecture should describe the intention: the vision, purpose, goals, and targets of development. Some elements of intentional architecture can be realized by running code and infrastructure; by creating an environment where enablers exist and constraints are followed by default. Other elements need to be designed and communicated in a way that they are easy to understand by all participants.

The ArchiMate language provides ways of communicating both intentional and emerging architecture. Although a modeling notation requires some initial learning, shared concepts and structures soon become essential parts of communication between people from different backgrounds.

In large multi-team projects, it is useful to start with some intentional (but limited) architecture. Here, again, an ArchiMate modeling strategy is the perfect way to have a global view across teams on the initial concepts and work/responsibility split between the teams.

For large projects, an ArchiMate modeling strategy and Business Layer architecture might clarify and guide multiple Agile teams. Architects need to express the desired effects of the architecture; such as the value you want to deliver, business capabilities you need to develop, fundamental principles you should apply, and constraints that must be fulfilled. The motivation and strategy concepts are very useful in expressing this intentionality. Concepts such as stakeholder, driver, value, goal, principle, capability, and course of action let you express what the enterprise wants to achieve, where it wants to go, and which steps it intends to take to get there. This is where you express the linkage between the strategy, business model, and the underlying core architecture of the enterprise. Perhaps the capability concept is the most important intentional notion that the ArchiMate language offers, and certainly one of the most popular. It expresses what an enterprise is able to do, or wants to be able to do, rather than how it does it. Modeling the Intent: Themes, Epics, Features, and Stories as ArchiMate Motivation Elements addresses how you can model this business intent in detail using ArchiMate concepts.

An important use-case for architecture models in an Agile context is evaluating alternatives and priorities. Modeling the business intent as mentioned in Using Models to Capture Architecture Intent, and linking that to key architectural choices, helps decision-makers to analyze the impact of those choices and obtain a high-level impression about the budgets that need to be allocated to the various Agile value streams.

Based on an analysis of the Enterprise Architecture, you can see how different epics and features are related to business processes, capabilities, business goals, and associated stakeholders. Tracing through your architecture models, you can find out which business goals or stakeholders an epic or feature supports and decide whether that is more important than other features. Objective information like this may help product owners put a stop to "decibel-driven" prioritization; for instance, where the users that complain the loudest get their wishes granted.

In creating Agile organizations, it is not enough to focus on Agile change processes and teams. The solutions themselves that these teams develop should also be easy to change. Microservice "spaghetti" is an example of the "instant legacy" problem you want to avoid. Using models can also help to uncover and avoid unwanted dependencies that will later on harm your agility. A key consideration in architecture, still valid in an Agile world, is the separation of concerns. This is often expressed in terms of coupling and cohesion: ensure that unrelated concerns are separated in loosely-coupled components so you end up with a set of relatively independent, internally cohesive components.

Architecture models are a great instrument to show and analyze this. If you see that two components communicate in 25 ways, perhaps these are so dependent on each other that they should be treated as one. Keeping in mind Conway’s Law,footnote:["Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure." [7] giving each of these components to a different component team will likely require a lot of coordination between these teams about all this communication between the components.

There are various metrics you can apply to your architecture to calculate coupling, cohesion, and other aspects of complexity. These are useful to help you partition systems (and whole enterprises) into parts that can evolve independently, thereby increasing your enterprise agility. An interesting perspective (including a method to help you partition your architecture) is given by Roger Sessions [6]. He advocates establishing autonomous business capabilities that own their own IT systems and communicate through services, rather than having them share the same IT landscape. Business-driven IT change can then be localized, and dependencies reduced. It may introduce some inefficiencies – for example, because data needs to be stored in and synchronized between different systems – but it greatly improves the speed of change. Since in today’s winner-takes-all economy, speed often trumps cost, such an approach could offer important advantages.

On a smaller scale, this thinking also applies to microservices: they should address specific and limited business requirements in order to minimize dependencies and impact of change. Analyzing your architecture models could uncover, for instance, services that have a scope that is too broad, because they are used by a great many different systems in different contexts.

Similarly, by analyzing models you can find single points of failure, communication bottlenecks, resource constraints, and other issues impeding agility and resilience. Many of these issues are not easily visible at code level (let alone in presentations or text documents), and simple, informal pictures (e.g., sketches on whiteboards) don’t lend themselves to any kind of formal analysis; you can follow the lines with your finger but that is about it. Moreover, managing these kinds of dependencies and constraints over time requires a solid foundation. This is where architecture models add great value.

Agile methodology includes the concept of technical debt. Sometimes, for example, in the interest of time-to-market, a short-term solution has to be implemented in the knowledge that this will need to be brought in line with the architecture with a better/cleaner/more elegant technical solution later on. This causes a "debt" that needs to be paid off later.

Similarly, and at a higher level of aggregation, there is the concept of architectural debt: short-term choices that have to be fixed later in order to avoid growing complexity that will hamper future changes and therefore reduce enterprise agility. Architecture models are a good way to record this technical and architectural debt and the eventual longer-term solution that needs to be put in place. The model can be re-visited during an Agile iteration and, once updated, modifications needed to the product can be detailed as new stories, features, and enablers.

Following the Agile philosophy, artifacts should be created only if they add value. An ArchiMate model can help and add value at each level of an Agile initiative. This includes, for instance:

These are just some of the benefits of using architecture models expressed in the ArchiMate language in an Agile context. In the next chapters, we will explore in more detail how models can help individual Agile teams (Architecture Models in an Agile Context) and support the collaboration between teams (Supporting Collaborating Agile Teams).