Ports, Components, and Connectors: the next great abstraction?


My friend George Fairbanks is trying to make the case that the abstractions provided by ports, connectors, and components are the next escalation in the war against complexity and scale.

George,

First, the good news…

Unfortunately (or maybe fortunately for you), many start Scrum without good engineering practices. I’m planning a talk with Noopur Davis at Agile2009 about this. Excerpt: “Simon, Struggling Agilist. Simon and his team have been using SCRUM and some agile practices for several iterations. They started out with great enthusiasm, but are now struggling as estimates are not improving, quality problems persist, and technical debt accumulates. Simon and his team want specific guidance on what they need to change.” We have data on a team that started with Scrum and added engineering practices of design, design review and code review. Their velocity went up and their defects/KLOC numbers improved significantly.

There is a tidal wave of folks who started Scrum without good engineering practices and now realize that they need them. You might try to ride that wave. Your best shot is with the Scrum folks who are now struggling. Lots of other folks are advocating more requirements and design work (iteration zero). You have another take on that.

Now the bad news…

Agilest will argue that feedback from actual running code is the best way to battle complexity and scale. Tighten the feedback loop with actual code. You can make more rapid progress by building something, even the wrong thing, and re-building it, than you can by wasting time modeling it and thinking about it because the assumptions you make when thinking about it always miss something important. You’ll call that something an insignificant detail. They’ll say, it’s the details that end up getting you.

I think you will lose the argument if someone comes up with a story about how Google’s (substitute, amazon web services, apache, or some other well known “big” system) massively scalable (although arguably not complex) system was created iteratively without any serious architecture work. Then, your approach to architecture is no longer design; it’s archeology, a concise way to document a system after a real engineer built it. I suspect that Eclipse was built with a more structured approach. Do you know if they followed an architecture approach?

Another way to lose the argument is by frameworks. I know architects whose job is mostly done once they decide what framework to use. Can you make the case that your approach could be used for them to make this decision? Be careful, most folks make the choice based upon things like how little boilerplate it makes you write, and the syntax of the templating language (Exhibit A). I see tons of discussion and thought go into the difference between SQLAlchemy and Hibernate’s approach to ORM versus RoR’s or Django’s. Can you address these things with your approach?

Alternatively, can you make the case that your approach could help in designing these frameworks? If that’s your goal, you limit your audience terribly and I’m still not convinced. Because for most design issues, I think most developers will prefer to use object-oriented terms and supplement that with OO design pattern terms when necessary. For instance, I recently worked on a team where we had to expand our system to support a remote service for storage (Amazon S3), whereas the system currently supported a local embedded storage. The conversation we had went something like this:

  • Developer A: Let’s just put a remote proxy in front of the embedded storage interface with the same methods as the current embedded storage API. We can block on calls to S3.
  • Developer B: Yeah, and if that’s too slow, we can add pre-fetch to the remote proxy.
  • Developer C: I disagree. By doing that, any code calling this might not anticipate the delay. The more general model is the async one. Let’s bite the bullet now and implement a pub-sub “observer” eventing system and create our own internal async interface for storage. Let’s look at Dojo’s storage API for ideas on generality. Then when using the local storage, we’ll just immediately trigger the callback event.

We all agreed with Developer C and that’s what we did. How would you address this conversation with ports, connectors, and components? Is that approach any more concise or revealing? If your response is that you are really targeting higher level issues, then you’ve greatly limited your audience because (other than deciding what stack/framework to use) this scenario is and example of the highest level issues that most developers deal with.

What about massively parallel designs? Can you help there?

I’m not sure what the right answer is for you. I’ve never been much of a believer myself. I hate to say that without reading the other draft chapters. I’ll probably do that some time but I don’t have the time right now.

Maybe if you started with the patterns/styles and thought of the connectors, ports, and components as merely ways to express the styles, that would be more palatable to the average developer.

I don’t want to throw your own words back at you but do you remember writing this?

First, it had been assumed by many, including the architecture group,
that software architecture would be a readily teachable skill. This project
uncovered six competent, experienced developers who showed little aptitude
for creating architectural models.

Second, if this finding generalizes then it limits the way companies can use
software architects. The traditional approach to adopting a new technique is
simply to train people to use it, but this pilot shows that a software
architecture training program may be ineffective.

If you still believe what you wrote, then maybe you should be targeting architects not general developers. More bad news for you is that I think the use of the title “architect” is diminishing. Those folks still exist but they have new titles now. Or do you think you now have a better way to bring them along and make more “architects”?

Sorry to be so glum. I kept going because I was hoping to come up with a good angle for you. Instead, I just ended up with a long depressing discussion.

Your friend,

Larry

 

This entry was posted in Software craftsmanship and tagged , , , . Bookmark the permalink.

One Response to Ports, Components, and Connectors: the next great abstraction?

  1. Hi Larry,

    Thanks for chatting with me about this. It’s helping me to understand and articulate my ideas, and it’s stretching my thinking.

    It struck me when you said this:

    Agilest will argue that feedback from actual running code is the best way to battle complexity and scale. Tighten the feedback loop with actual code. You can make more rapid progress by building something, even the wrong thing, and re-building it, than you can by wasting time modeling it and thinking about it because the assumptions you make when thinking about it always miss something important. You’ll call that something an insignificant detail. They’ll say, it’s the details that end up getting you.

    I’m a strong supporter of the spiral model and iterative development. Many folks who propose doing architecture do indeed suggest you do Big Design Up Front. I disagree (in most cases). When an agile developer sits at the keyboard, he’s thinking of what he’s building before he types. He’s not a monkey getting lucky typing Shakespeare. “Am I violating a design principle here? Is it OK for the front end to talk directly to the DB?” I know agile developers think about these things, and these are things I consider part of architecture.

    One of the points I’m trying to hammer home in the book is that your architecture effort should match the risk. Small risks can be resolved at the keyboard in a coding session. Bigger risks may need prototyping or modeling. Really serious risks might require formalisms and analysis (e.g., avoiding priority inversion as happened on the Mars Rover).

    I may have hinted at something that makes you think I’d disagree with “Tighten the feedback loop with actual code”, and I want to find where that is and not only eliminate it but actually reinforce that I agree with writing actual code. Here are some excerpts from Chapter 2: “We could keep adding models that explained more about our system, but spending effort building and maintaining models must be traded off against building the system” … “We must prototype our design to ensure it is viable before we commit more time and effort.” … “Prototyping can accelerate learning. Problems that seem easy at 10,000 feet often have unforeseen difficulties. Use research and prototyping to discover assumptions baked into COTS components, and express those assumptions in your models so you do not have to re-discover them later.” and then in the summary “A point that should be clear is that architecture modeling by itself is insufficient. Before modeling, we usually need to learn about our domain, which takes many forms. Prototyping is essential before proceeding too far with modeling, because continuing to work on a model with unvalidated assumptions is itself risky.”

    Re-reading what I wrote (and I’m not sure if you’ve looked at Ch2 anyway), I’m concerned that people could see “prototyping” and think “throw-away”, but what I’m trying to convey is that Yes, there will be details and wrinkles that your models won’t see and you must write code to get feedback. Those are most certainly not insignificant details. I do not want tether this idea (write code!) to Agile specifically, but maybe it would be clearer to discuss that this need to reduce risk by writing code is one of the Agile core principles. Thoughts?

    I think you will lose the argument if someone comes up with a story about how Google’s (substitute, amazon web services, apache, or some other well known “big” system) massively scalable (although arguably not complex) system was created iteratively without any serious architecture work. Then, your approach to architecture is no longer design; it’s archeology, a concise way to document a system after a real engineer built it. I suspect that Eclipse was built with a more structured approach. Do you know if they followed an architecture approach?

    I’m pretty sure that engineers/developers on large systems have some way of thinking about the big chunks, and they analyze that thinking to ensure the design gets them where they want to go, then they write some code to be really sure. Of course, once they write code they learn it’s more complex than they thought or that their design had bugs in it, etc., and they iterate. The point I’m making is that they neither 1) randomly write code, nor 2) fail to have the big picture in mind before they write code. Of course you’re not suggesting that either, but I want to make sure we agree that there’s thinking about the design before coding. So let’s agree to call that big picture a model, even if they do not write it down, and call that thinking “analysis”.

    Perhaps I need to make the point more clearly that I’m not advocating the following: Developers should think hard about problems, fire up a CASE tool and design a solution, then code it up. Iterate as necessary.

    Closer to what I advocate is: When your system is big and complex you won’t be able to build it unless you can fit the problem and solution in your head. This means constructing models, which by definition elide details. These models might be in your head, on a whiteboard, or in a CASE tool. You will use these models to convince yourself that the code you are about to write (ie the design) will probably work (ie achieve functions and qualities). Serious risks might force you to write out detailed models that you analyze carefully, but small risks might entail whiteboard sketches and quick prototyping. NASA is probably going to build more models than MyStartup.com, but perhaps friendster.com would not have failed because of performance problems if they had done more design work.

    (BTW, I have no inside knowledge of the Friendster case, but my reading of the post-mortems says that they wrote a system that didn’t scale with users, were convinced to rewrite it in a different programming language, which also did not scale, then changed programming languages again but also tweaked mySQL’s implementation to make their queries more efficient, which finally gave them acceptable performance but too late. The paper I read had the flavor of “my favorite programming language rocks” but it seemed clear that the DB tweaks were the solution, and the PL was not relevant to the scaling problem.)

    Another way to lose the argument is by frameworks. I know architects whose job is mostly done once they decide what framework to use. Can you make the case that your approach could be used for them to make this decision? Be careful, most folks make the choice based upon things like how little boilerplate it makes you write, and the syntax of the templating language (Exhibit A). I see tons of discussion and thought go into the difference between SQLAlchemy and Hibernate’s approach to ORM versus RoR’s or Django’s. Can you address these things with your approach?

    Frameworks are a great example of where almost every client program is a big ball of mud, with only a few exceptions — like Eclipse since it encourages building sub-frameworks using OSGi plugins/components.

    I’ve just deleted a huge rathole I was starting about what appears to be desirable vs. what is actually what the project needs. We can go in to that later if you like. I bet we can agree that there are many cases where developers might prefer to use X because it’s fun, elegant, new, etc. but not the best choice for the project. Of course other times you’d be foolish to use unsharp or inefficient tools. I think it’s worth questioning if my liking to use language/tool X translates into project success.

    Currently fwks and architecture models are oil and vinegar. If I did another dissertation it might be on this topic. It’s on my list of things that are still hard about architecture.

    I’m not sure what the right answer is for you. I’ve never been much of a believer myself. I hate to say that without reading the other draft chapters. I’ll probably do that some time but I don’t have the time right now.

    It’s not clear what you are expressing concern about: is it the idea of using architecture abstractions, or spending time drawing out models? One of the novel (I think) parts of the book is that you should do just enough modeling to discharge your risks, then start building. Other books tell you how to draw out all kinds of models about everything. This one says no, just model when you are worried, and stop modeling when you stop worrying. It could be that for most projects that Agile is applied to (small to medium IT projects are its bread and butter, or at least they used to be) that there are relatively few risks worth spending much time on. They copy the standard styles (usually N-tier or J2EE) and standard solutions (eg use DB to handle concurrency), so most of the time it works out.

    I don’t want to throw your own words back at you but do you remember writing this? …

    I’m almost ready to say and mean “software architecture should be taught immediately after data structures”. Of course we first have to figure out how to teach it. I can’t remember if this idea made it into the paper you’re quoting, but I thought of the math analogy — you used to have to mentor with a mathematician and even then by your 20’s you were only doing what we now consider elementary school math. We got better at teaching math so now we do long division in 3rd grade (though your average bar patron will struggle with it). You might not realize that the quote above was challenging an earlier paper by someone I highly respect, David Garlan, who had a few years earlier written a paper concluding that it was relatively easy to teach architecture. I will point out that he may be able to teach it better than I can, but I did have trouble.

    Can you imagine how much more interesting an OS class could be after taking an architecture class?

    Yes, getting general developers to read and embrace the book will be difficult. Perhaps the best entrance is as you suggest, with “architects”. (You probably haven’t read far enough to see my criticism of the term and explanation of why I don’t use the term in the book). I really think this kind of dialog with you is helping me to express myself more clearly, and to address concerns that Agile and other reasonable developers will have, so maybe there’s hope yet for the general developer audience).

    Sorry to be so glum. I kept going because I was hoping to come up with a good angle for you. Instead, I just ended up with a long depressing discussion.

    One thing I’m concerned about is that you (or others) will read my answers and feel like it’s judo — I’m avoiding disagreement by saying “oh, I can do that too, architecture can be agile”. Judo is seen in another architecture book I recently read, which is actually a thorough description of how to apply up-front design, when it describes how to reformat itself to be Agile, Scrum, etc. But that reformatting was unconvincing, because it said things to the effect of “Use iteration 0 for six months to develop a product line architecture”, which would be contorting things too much.

    I would be delighted if Agile developers read my book and never bought a CASE tool, and didn’t change the code they write, just so long as they agreed that they need words to describe big chunks of code and communication channels, need to distinguish the module, runtime, and allocation perspectives, and need to deliberately choose styles that promote the quality attributes they want.

    And I’d be happy to be wrong about my ideas, so long as I can understand that and fix them. I hope I’ve been clear to you that if you have misinterpreted my work so far that it’s my fault for not being clearer, or for hinting the wrong direction. And boy am I thankful for your time and ideas in this discussion!

    Regards,

    -George
    http://rhinoresearch.com

Leave a Reply

Your email address will not be published. Required fields are marked *

Spam Protection by WP-SpamFree