Ports, Components, and Connectors: the next great abstraction?

My friend George Fairbanks is trying to make the case that the abstractions provided by ports, connectors, and components are the next escalation in the war against complexity and scale.


First, the good news…

Unfortunately (or maybe fortunately for you), many start Scrum without good engineering practices. I’m planning a talk with Noopur Davis at Agile2009 about this. Excerpt: “Simon, Struggling Agilist. Simon and his team have been using SCRUM and some agile practices for several iterations. They started out with great enthusiasm, but are now struggling as estimates are not improving, quality problems persist, and technical debt accumulates. Simon and his team want specific guidance on what they need to change.” We have data on a team that started with Scrum and added engineering practices of design, design review and code review. Their velocity went up and their defects/KLOC numbers improved significantly.

There is a tidal wave of folks who started Scrum without good engineering practices and now realize that they need them. You might try to ride that wave. Your best shot is with the Scrum folks who are now struggling. Lots of other folks are advocating more requirements and design work (iteration zero). You have another take on that.

Now the bad news…

Agilest will argue that feedback from actual running code is the best way to battle complexity and scale. Tighten the feedback loop with actual code. You can make more rapid progress by building something, even the wrong thing, and re-building it, than you can by wasting time modeling it and thinking about it because the assumptions you make when thinking about it always miss something important. You’ll call that something an insignificant detail. They’ll say, it’s the details that end up getting you.

I think you will lose the argument if someone comes up with a story about how Google’s (substitute, amazon web services, apache, or some other well known “big” system) massively scalable (although arguably not complex) system was created iteratively without any serious architecture work. Then, your approach to architecture is no longer design; it’s archeology, a concise way to document a system after a real engineer built it. I suspect that Eclipse was built with a more structured approach. Do you know if they followed an architecture approach?

Another way to lose the argument is by frameworks. I know architects whose job is mostly done once they decide what framework to use. Can you make the case that your approach could be used for them to make this decision? Be careful, most folks make the choice based upon things like how little boilerplate it makes you write, and the syntax of the templating language (Exhibit A). I see tons of discussion and thought go into the difference between SQLAlchemy and Hibernate’s approach to ORM versus RoR’s or Django’s. Can you address these things with your approach?

Alternatively, can you make the case that your approach could help in designing these frameworks? If that’s your goal, you limit your audience terribly and I’m still not convinced. Because for most design issues, I think most developers will prefer to use object-oriented terms and supplement that with OO design pattern terms when necessary. For instance, I recently worked on a team where we had to expand our system to support a remote service for storage (Amazon S3), whereas the system currently supported a local embedded storage. The conversation we had went something like this:

  • Developer A: Let’s just put a remote proxy in front of the embedded storage interface with the same methods as the current embedded storage API. We can block on calls to S3.
  • Developer B: Yeah, and if that’s too slow, we can add pre-fetch to the remote proxy.
  • Developer C: I disagree. By doing that, any code calling this might not anticipate the delay. The more general model is the async one. Let’s bite the bullet now and implement a pub-sub “observer” eventing system and create our own internal async interface for storage. Let’s look at Dojo’s storage API for ideas on generality. Then when using the local storage, we’ll just immediately trigger the callback event.

We all agreed with Developer C and that’s what we did. How would you address this conversation with ports, connectors, and components? Is that approach any more concise or revealing? If your response is that you are really targeting higher level issues, then you’ve greatly limited your audience because (other than deciding what stack/framework to use) this scenario is and example of the highest level issues that most developers deal with.

What about massively parallel designs? Can you help there?

I’m not sure what the right answer is for you. I’ve never been much of a believer myself. I hate to say that without reading the other draft chapters. I’ll probably do that some time but I don’t have the time right now.

Maybe if you started with the patterns/styles and thought of the connectors, ports, and components as merely ways to express the styles, that would be more palatable to the average developer.

I don’t want to throw your own words back at you but do you remember writing this?

First, it had been assumed by many, including the architecture group,
that software architecture would be a readily teachable skill. This project
uncovered six competent, experienced developers who showed little aptitude
for creating architectural models.

Second, if this finding generalizes then it limits the way companies can use
software architects. The traditional approach to adopting a new technique is
simply to train people to use it, but this pilot shows that a software
architecture training program may be ineffective.

If you still believe what you wrote, then maybe you should be targeting architects not general developers. More bad news for you is that I think the use of the title “architect” is diminishing. Those folks still exist but they have new titles now. Or do you think you now have a better way to bring them along and make more “architects”?

Sorry to be so glum. I kept going because I was hoping to come up with a good angle for you. Instead, I just ended up with a long depressing discussion.

Your friend,


Posted in Software craftsmanship | Tagged , , , | 1 Comment

Visibility -> Retrospection -> Adaptation

This is too funny.

On the one hand, we have George Fairbanks (who comes from the architecture world) arguing that data models are too low level to be considered architecture. On the other we have, John Owens (a leader in business process modeling), arguing for them in the context of his integrated modeling method.

My point is that it doesn’t matter whether you consider them part of the architecture or not. It doesn’t matter whether they are in YOUR definition of a good process. The rise of Agile is a response to the insistence upon adopting someone else’s approach. This resulted in misapplication and the developers became cynical.

If you don’t want to be dismissed by the Agile world, you have to start selling in smaller chunks and let the developers decide which chunks to use. For me (a process, quality, and security guy), the easy sell is some form of peer review and the effective use of passively gathered data. For John, it might very well be data modeling. It’s a bridge between the business and development domains. I’ve read the excerpt of his ebook on this subject, and it is one of the clearest explanations of the topic I’ve ever read. I’m not sure what chunk is the easiest sell for George but his ability to extract abstractions (even from domains other than software) has impressed me greatly over the years. I just don’t know how he can best share that.

What I do know is that we should stop pushing integrated anything. We should stop defending our definitions. Rather, we need to target our efforts at the primary form of the Agile feedback loop (Visibility -> Retrospection -> Adaptation) and make our case in the retrospection phase after they’ve experienced trouble by not doing enough design, requirements, or quality activities. Don’t worry about the second form of the Agile feedback loop (Plan -> Build -> Inspect -> Refactor). It is more easily expanded as we’ve seen with the introduction of Iteration Zero proposals.

Posted in Software craftsmanship | Tagged , | Leave a comment

We have to get PEOPLE to build this stuff

I started to write a long comment in response to a post on my friend George Fairbanks blog, but when it got above three paragraphs, I decided to move it over here. The question was whether or not data modeling should be considered part of software architecture.

The short answer is, “Yes” but my reasons why may surprise you.

I’ve sat in many a team meeting where early design decisions are being made. Nothing helps engage the team better, than settling on a data model. At levels above that, where you just say, here’s an employee, here’s a customer, the team will just nod and agree to whatever the architect/design lead/etc. is saying. It’s only when you start to ask, “How are we going to represent these objects in the system?” (and really only after you discuss which objects go in which tables), do you start to get folks to actually think about (and commit to) building it. I’m not sure you have to go down to the data modeling level all the time, but for many situations you have to do it to get the team engaged.

Developers won’t require that you decide in the meeting whether or not it’s a varchar or a string(20), but they may not engage unless the conversion includes things like, “Are we going to use the same table to store customers and employees.” That’s when you start to ask questions like, “Is an employee ever going to also be a customer? If so, is it OK for them to have two separate entries in the system? How much data is in common between the two? How much is different?” This does get a little close to George’s fear about discussing N-th normal form but there it is.

But that wasn’t the original question. Nobody is saying that data modeling isn’t useful. We’re just asking whether or not data modeling should be considered part of software architecture.

Never forget, we have to get PEOPLE to build this stuff. The single most important driver for the success of a software project is the commitment of the people doing the work. If the coders think that the plan was pulled out of the air or the design is wasteful, pipe dreaming, or otherwise deficient, they will passive aggressively kill your project. If you’ve ever been part of an effort where the folks on keyboards think, “the PowerPoint architects who designed the systems are idiots”, you understand what I mean. So the number one priority is to get the developers to say, “our design”. If they ever think, “his design”, you are much more likely to fail.

The architect’s job is to get the team to commit to a good design. It’s not to deliver a good design to the team.

And the only way to be certain, that the team has committed to it, is to get them to find it themselves. You can tell them until you are blue in the face that you have analyzed this nine ways to Sunday and you are convinced… blah, blah, blah. I repeat, the only way to convince them, the design is good is for them to come up with it on their own.

As a great architect, you should have already walked through the issues yourself. Certainly, you’ll use the higher level abstractions that you are writing about, George, to think things through. But you can’t stop there. If you throw your design over the wall at that point, who knows how it will turn out. And don’t tell me that you sent it out for comment. Nobody reads that stuff. You have to host a meeting or a series of meetings, preferably with a good facilitator/coach (like me 🙂 or Noopur Davis, who does this better than anyone I know). You go into the meeting with an architecture in mind and probably even well documented, but don’t hand those documents out. Start with a blank whiteboard. Add a simple block diagram and proceed from there with the Socratic method. Someone will suggest something you’ve already rejected and you’ll say, “Sounds reasonable, but what about foo?” They will then either 1) reject it like you did; 2) suggest a creative way around foo; or 3) convince you why your assumption or priority is wrong. Every time, I’ve seen this done, the architecture is different coming out of the meeting than going in. Sometimes the improvements are minor and might have happened anyway during implementation. More often, the changes are major.

If the architecture is for a large system of systems, then you may only have sub-team leads/architects in the meeting and they will probably not want to talk at the data model level. But when architecting for a 2-pizza team you better be ready to go down to the data model level.

You should of course follow up the meeting by delivering back to the team, the architecture that they came up with in the meeting. If it’s close to what you had already documented before the meeting, so much the better. To do this effectively, you may need several meetings at different levels. Don’t worry about the cost/time. If you delivered a finished architecture, you’d have to spend time training them on it anyway. This is just a fun (and much more effective) way of accomplishing that goal of them understanding what they are going to build. Your biggest problem will be resisting the temptation to share your wonderful architecture with the world and get credit for it. Your best approach is to make sure the team gets all the credit.

If the book you are writing is for software architecture researchers, then you can ignore everything I’ve written. If on the other hand, you are writing something that can be used by practicing architects, then never forget, we have to get PEOPLE to build this stuff.

Posted in Software craftsmanship | Tagged , | 4 Comments

Adopting Agile doesn’t mean forgetting what you’ve learned

Agile is particularly attractive to two very different groups: 1) those whose organizations don’t already have evolved practices, and 2) those whose processes have grown to become burdensome. In both cases, there is a tendency to make your first Agile sprints have a bare minimum process. After all, that’s what Agile says to do, right?…

Not right! It’s a common misconception, but minimum weight is not the controlling function for Agile adoption.

The controlling idea of Agile is learning (visibility and inspection/retrospection) and applying that learning (adaptation). Agile practices are predicated on the idea that trying to apply someone else’s process template to your situation is rarely ideal and often counterproductive. Rather, the agile approach is principled-based, allowing you to adapt as you learn and your situation changes.

So what does this mean? Well, if you currently have over evolved processes, don’t throw the baby out with the bath water. If your organization has experienced problems in the past and the situation hasn’t changed in a way that invalidates that “learning”, don’t necessarily throw out all process elements that resulted from those experiences. Similarly, if you are just getting started, don’t be afraid to learn from others who have gone before. In fact, using what you’ve learned is the foundation of the Agile approach.

Let me give you a concrete example. It’s no coincidence that every strong team that I’ve ever seen has mandated some means to get another set of eyeballs on the code. In some organizations, this means “formal inspections”. You may believe that formal inspections cost more than it’s worth for your situation and I won’t disagree with you, but even XP advocates pair programming. Open source development has the “many eyeballs” effect built into their licensing and  commit practices. For closed source practitioners, what I’m starting to see more is some form of asynchronous peer review using tools like Google Code Reviews.

Why do all these strong practitioners utilize some form of peer review? Because we’ve shown time and again, in various and sundry quantitative research, as well as in qualitative studies, that it’s the single most efficient thing we can do to remove defects from the code. In general, it’s more efficient than testing at removing all kinds of defects PLUS it allows you to systematically address things that testing cannot like maintainability/evolvability (code smell issues), as well as things that are nearly impossible to find with testing including certain kinds of security and concurrency issues. Why then do I see Agile teams going through their first few sprints without any form of peer review?

Similarly, there is a tendency to swing too far on the issue of design. Sure, I’m a firm believer in YAGNI and that the best way to improve the design is to evolve working code. I’ve suffered the paralysis of analysis. I’ve even been the guilty party. However, refactoring is not a substitute for design. It’s very difficult to achieve certain non-functional requirements (scalability, security, etc.) without some architecture work up front. Similarly, depending upon your situation, appropriate requirements elicitation and documentation practices can save much more than they cost.

The good news is that Agile supports doing these things in its “definition of done”. Furthermore, if you fail to do them, it provides the feedback loops that will highlight the need for them later. However, do yourself a favor, when you are trying to settle on your FIRST “definition of done”, include at least some form of peer review for your code.

Posted in Software craftsmanship | Tagged , | Leave a comment

First post

Now that it’s the new year, I’m going to try this blogging thing again. Topics will include:

  • Software engineering:
    • Process
    • Agile methods
    • Measurement
    • Static and dynamic analysis
    • Test-first design and automated unit testing
    • Tools
  • The software development industry
  • Flex and Actionscript
  • Anything else that seems interesting
Posted in Administrivia | Tagged | Leave a comment