The I/O of it all. Another abstraction layer example.

Recently I have been working on some "one-hit-wonders" programs that are needed just once. Working in the "COTS" like systems for years have me always thinking long term this & long term that. There is a reason the term "legacy" is used to describe existing systems; they have a VERY VERY long life, well beyond the original developers. So, my program using the latest pieces of the .Net XML Navigator and scans through a tons of flat files reacting to various things discovered in these files. My huge backlog of programming experience made me first think "Oh man, this program will take a long time to finish." Man I was so wrong.

Then I started to remember an article (ftp://ftp.research.microsoft.com/pub/tr/TR-2004-136.pdf) read a few years ago written by Jim Gray (currently in the MS R & D division). He has been fundamental in the RDBMS arena for years. This article was all about "fast" file reading and writing. For a moment I was thinking, what the heck is a RDBMS guy care about file I/O. Duh, everything is ultimately written to disk dummy! For simplicity sakes, all "good" computers must be turned off every now & then & therefore will lose all in-memory data. This made me realize that Relational Databases aren’t necessarily "faster" in the pure sense of performance (with all other things being equal - which isn't ever the case). But it is easier. That is one of the main goals of abstraction.

As I see it, lower level solutions attempt to handle the "ugly" details of it's layer leaving only exposed the common utilization scenarios for the consumer. Databases are no exception. Given the time constraints, learning curves, lack of skill sets (sorry to say), and elaborate business problems to solve; leaving the file I/O of the data to someone like SQL Server or Oracle isn't a bad idea. Why re-invent the wheel right? I agree with that, I guess I just get a little annoyed from time to time with the mystical impression people have of RDBMS. Please note, I am not saying I could do any better for a "commercial" product or that I don’t have respect for the development that went into these kinds of products. I am just not a fan of the "magical" black box mentality that exists in many development shops related to this topic.

So do you think I offended every DBA in the world? I hope I didn't offend all of them. J

The Rat Race, Leaders, and Labor

There is a saying that great leaders are born and not made but I am not too sure. I would agree that some people are predisposed with certain characteristics and/or instinctive talents that improve their personal probability to successful leadership but it is not guaranteed. I compare this to muscle development (yes, I am a weight lifter). Some people are genetically more inclined to responding to weight training & thus develop muscle mass easier but it is not free. These "blessed" people still need to invest time in the gym lifting; it does not just magically happen. They may be more efficient at muscle development but they still have to lift to grow (as does everyone else). So back to leadership, it must be developed. I have begun my reading of Adam Smith's Wealth of Nations and in his first book he is focusing on specialized labor and it's benefit to society. Yet I have also read how specialization is more like a prison for "super-stars" and can grossly limit their ability to excel. Although in conflict, I see truth in both. Another saying is that a good leader can do extraordinary things with ordinary people (his/her team). With the hope of not sounding condescending, it is true that many people are not interested in improving beyond the masses and more interested in just getting through the day & it's routine. Good leaders need to effectively and efficiently leverage these people and specialization of labor is, IMHO, the least resistant path to doing this. Specialized labor has done spectacular things for society, bringing us well past the hunting & gathering era. Many benefits, abundances, and advancements have only been possible due to this direction. It is my, un-qualified, opinion that it is the creator of the middle class. This is where the ordinary masses reach their goal of living. This is the demographic I think of when the "Rat Race" complaint is raised. But what about the leaders that want more? How do you do go beyond that? This is the ultimate challenge I think. Excluding fluke events, and extreme situations, the majority of people wishing for more don't escape. Global thinkers must find the right combination of broad ideas and practical daily actions within the specialized labor division to break into the higher arena. Again excluding dumb luck, it is the aspiration of such people to find the right formula. Perhaps this not that much different than a LARGE complex software system where the best approach is to divide it into several smaller problems and systematically solve each one (yet not forgetting the integration requirement all of these smaller pieces mandate). Obviously I haven't solved this yet ... but I am working on it! :)

Testable by Design?!?

A few years back I was at a Rational Conference in Philly and sat in on a presentation by Sam Guckenheimer titled "Quality by Design". We all have heard of Performance, Scalability, Robustness, Configurability, and even CA's "QWAN" being factored in during the design phase, but what about testability. We all have seen so many times in our career where testing gets "trimmed" in the project schedule once the rubber meets the road. Are there things that we architects can do to help preserve some dignity in this ever-shrinking phase? Modularity, Use Case driven, Unit Tests (like NUnit or the likes) and of course our lip service to the cause can all be elements to help preserve this phase, but I wonder what else can we do. With the rise of SOA architecture we see even more of a need to "prove" out a single service/layer/slice (you pick the term). Everyone's canned response is to just have the developer build the tests as they build the main code. We know that it doesn't always play out like that. The way I see it, many of the challenges with the "automated" testing support systems, is that is still requires MORE effort than management is willing/able to invest. The crux is the lack of context. We still need humans to add the context to most of the tests in order to be useful. What if, I am just thinking aloud here, MS could add some additional Attributes that a tool could reflect one, have a user enter some additional context to (in addition to message signature & the likes) so we could easily piece together a string of Unit Tests with setup & tear down to create useable system tests. These additional metadata would need to be centrally located and workable by the non-technical business analysts in order to contain any business value. Not free just cheaper. Sorry, I digressed, back to the Architecture. Any thoughts on what we could build into the core design to make testing less of a chopping block victim?