Sunday, April 1, 2007

Testable by Design?!?

A few years back I was at a Rational Conference in Philly and sat in on a presentation by Sam Guckenheimer titled "Quality by Design". We all have heard of Performance, Scalability, Robustness, Configurability, and even CA's "QWAN" being factored in during the design phase, but what about testability. We all have seen so many times in our career where testing gets "trimmed" in the project schedule once the rubber meets the road. Are there things that we architects can do to help preserve some dignity in this ever-shrinking phase? Modularity, Use Case driven, Unit Tests (like NUnit or the likes) and of course our lip service to the cause can all be elements to help preserve this phase, but I wonder what else can we do. With the rise of SOA architecture we see even more of a need to "prove" out a single service/layer/slice (you pick the term). Everyone's canned response is to just have the developer build the tests as they build the main code. We know that it doesn't always play out like that. The way I see it, many of the challenges with the "automated" testing support systems, is that is still requires MORE effort than management is willing/able to invest. The crux is the lack of context. We still need humans to add the context to most of the tests in order to be useful. What if, I am just thinking aloud here, MS could add some additional Attributes that a tool could reflect one, have a user enter some additional context to (in addition to message signature & the likes) so we could easily piece together a string of Unit Tests with setup & tear down to create useable system tests. These additional metadata would need to be centrally located and workable by the non-technical business analysts in order to contain any business value. Not free just cheaper. Sorry, I digressed, back to the Architecture. Any thoughts on what we could build into the core design to make testing less of a chopping block victim?

2 comments:

Anonymous said...

I've been thinking about this for a few days now and I think the best I can come up with is to maybe look at other industries where their software is running in mission critical, life-or-death environments and see how they handle testing.

If the software in cars or planes was tested like most other software out there, we'd all be screwed. I have to think they'd have testability down to a science, but who knows.

M. Wilson said...

I would agree that realtime systems are great to study and that many things can be learned from them. One observation though, is that realtime systems with the TRUE 24/7 availability requirement are VERY expensive to build. Remind me later to include a particular white paper link I have related to some architectural concept used for one of the Mar's rover robots. It was very interesting and got me wondering how we could use it in our quest for "cheaper" testing. In SOA web services, WSDL enable most systems to discover the mechanics of a function call but not context. What if we could apply a AI like concept to some testing framework that learned the context?!? Maybe even a Neural Net could be leveraged?