Saturday, December 30, 2006
Developing Enterprise Systems - Some Guidelines
Edited: 11/20/2005 and 12/28/2006
Table of Contents
1. Overview
2. Architectural Goals for Enterprise Level Systems
..2.1 High Level Concepts
....2.1.1 THE BASICS - Blueprint, Framework, and Standards
....2.1.2 Business Domain\Functional Area Database Model
..2.2 Patterns
....2.2.1 Modular Architectural Pattern\Business Domain Modeling
....2.2.2 Layers Architectural Pattern
....2.2.3 Software\Business Patterns In Genera
..2.3 Techniques
....2.3.1 Components Versus Objects – A System’s Building Blocks
....2.3.2 To Components And Beyond -- Interfaces
....2.3.3 Configurable Systems Through Use Of Meta-Data
....2.3.4 Code Generation
....2.4 Being Forward Looking
....2.4.1 Scalability Versus Performance
....2.4.2 Technology Usage
....2.4.3 Long System Life
..2.5 Development Methodology
....2.5.1 Iterative Development With Regular Reviews
....2.5.2 Test Driven Programming and Continuous Integration
..2.6 Low Level Concepts
....2.6.1 Flexible Deployment
....2.6.2 Provide Instrumentation Of Process And Data Pathways
....2.6.3 Components - Further Considerations
....2.6.4 Simplify Format Of Data Storage And Manipulation
3. Conclusion
4. References
5. Additional Notes
1 Overview
This document provides a short selection of important topics, which should be considered by an architect of an Enterprise System.
This document looks at things primarily from a development\architectural perspective, rather then an overall project management view. Thus, many other important things that contribute to the success of any enterprise system development effort, such as having strong sponsors, active participation by managers and key business people, are not discussed here.
2 Architectural Goals For Enterprise Level Systems
The table of contents of this document identifies the list of architectural\design issues that I discuss below, all of which are attributes of adaptable enterprise systems.
2.1 High Level Concepts
This section of the document “High Level Concepts” presents some concepts that should be definitely followed.
2.1.1 THE BASICS - Blueprint, Framework, and Standards
Large software systems should be implemented according to an overall guiding blueprint. A practice that is present in most other engineering disciplines developing any kind of large or complex system.
Can you imagine trying to build a large office building without a blueprint? Who would be able to review or approve the proposed design? How would other engineers (plumbers, electricians, and telecommunication experts) coordinate their activity so they didn’t end up working at odds with each other? How would they know if their proposed effort will be adequate to support other trades people efforts?
With large enterprise systems, which can easily end up with million plus lines of code, everyone needs to be working with the same overall plan, if good re-use is to be made of each other’s efforts. Without a guiding blueprint, components and services designed by different individuals will not necessarily work together well, if at all. Following a good blueprint can also help developers build components and services within a vision of how the entire system needs to scale to handle future growth. Thus, part of any guiding architectural blueprint should be the specification of a scaleable application framework or bus architecture, into which many of the generic services such as data access, security, authentication, authorization, etc., can be plugged-in.
Furthermore, all development work should also be carried out according to agreed upon standards, which should cover conventions for naming, coding, formatting, and organization of code. Also, the company’s development activity should have a fulltime hands on software architect\technical lead to guide it and coordinate standards setting.
What use are standards?
If individual items within a large system that perform similar activities are consistently named, people can guess what something does just by its name. Naming can also be used to organize all the stored procedures or tables that cover one business domain together, making it easier to identify and work within that functional area.
Real life problem!
Over the last three years I’ve worked at a manufacturing company as a consultant, the topic of using a common set of standards for development has repeatedly come up, but no single set of coding standards has been adopted by all developers. The group doesn’t even have something as simple as a consistent way of naming new items. For example, the original developer of their enterprise system had no observable standard at all for naming stored procedures, and he created 200 of them. Of the three newer developers, one used begin all his procedures with a ‘p_’ prefix and another with a ‘proc’ prefix, but neither of them followed any other naming rules for their procedures. Between those two developers, 100 new procedures have been added in those two separate groupings. The third developer instituted his own more formalized standard, adding procedures with the following format domain_prefix + main_table_name + action_verb + optional_qualifier. Whose collection of procedures do you want to be working on? For me it's definitely the last guy mentioned, and even though he's added 200 stored procedures to the database, I can easily identify what each of them is likely to be doing before I even examine their code. His EDI_Order_ins stored procedure does what it says. In fact, his EDI_Order_del, EDI_Order_sel, and EDI_Order_upd procedures all do what they say and can be found grouped together in any alphabetically listing. However, trying to find the set of procedures written by any of the other developer on any particular topic is not so easy.
So, as you can likely guess, with so little agreement on even the smallest of standards the development of their enterprise system does not follow most of the standard guidelines outlined below for developing adaptable enterprise systems, which has made working with their enterprise system increasingly difficult as it grew. While management was made aware of these issues, they chose to ignore them, as the system as it was, supported an annual average growth rate of 13% during that period. However, in my view, the limitations of the system were actually constraining factors that prevented the company from growing even more that 13% per year. Moreover, they are now kicking off a project to examine whether they should by a commercial package are start writing a replacement for the current system as they don't believe it will support their growth and business direction in the next couple of years, and is no longer worth the cost of modifying it to do so.
2.1.2 Business Domain\Functional Area Database Models
In regards to design, the next level of detail that enterprise system development should get into is the development of business domain\functional area database models with the goal of producing a common language and understanding between business and IT personnel for those business activities. And, subsequently offer guidance for the actual development.
Divide and conquer should be the mantra for any large system. Humans are only capable of juggling so much detail and interactions at one time. So scoping deliverables at an appropriate level makes it easier to work on large solutions successfully.
2.2 Patterns
Patterns offer the opportunity to learn from other people’s successes and avoid some mistakes.
2.2.1 Modular Architectural Pattern\Business Domain Modeling
Modularization is one of the fundamental software architectural patterns that’s been successfully applied to build all large enterprise systems over the last twenty years. It involves the break-up of the enterprise system into smaller relatively self contained subsystems that handle major functional areas of the business, such as, Product Development\Merchandising, Ordering, Logistics, and Finance. These smaller subsystems can then be developed and installed separately or even substituted with best of breed applications.
This reduction in the reliance on direct linkages between the different areas of a system is commonly referred to as “loose coupling.” Because of the use of indirect links\data exchanges, modules can for the most part be individually enhanced and tested without risking cascading side affects to other parts of the system. It is also the reason why many vendors of commercial enterprise systems can supply individual companies with paired down versions of their products containing just the modules necessary for running their particular business activities. One additional nice aspect of loose coupling is that it dovetails well with the use of Business Domain\Functional Area Database Models.
2.2.2 Layers Architectural Pattern
Layering is another one of the fundamental software architectural patterns that’s been successfully applied to build large systems of all sorts. It’s a tried and true technique for packaging cohesive elements of a system in their own layer, to provide some core service to the rest of the application and even between applications. If the communication protocols across the boundaries of a layer are respected by all parts of the system the classes within that layer can be significantly changed without interfering with the rest of the application as long as the layer continues to service its original interfaces.
This is one of the most challenging areas of building enterprise systems as it requires that people be able to think abstractly about what services need to be made widely available and how they might be generically implemented to facilitate their reuse. And which services are similar enough in nature that they should be packaged together in a larger cohesive part of the system. Often the best guidance on this topic is to review other publicly available frameworks to see what types of things they do and what parts of them could be adapted for ones own use.
A common use of the layering pattern in enterprise systems is to inoculate the rest of a system from any undue influence of a particular data access technological. Such technologies tend to change every two to three years and systems in which it has not been isolated in a single layer get caught in a situation where there is no easy path to the new data access technology and they end up tending to use the older technology well beyond it best use date.
An excellent use of layering that is not used commonly enough is to put business logic\rules inside business components that live within a business layer that is accessible by the remaining upper layers of the system. This would provide (at the very least) the presentation layers direct access to that business logic, reducing the amount of code that needs to be written within them. For example, this would permit both a Windows and Web application to make use of the same business component to check when a customers next delivery can be expected or his account balance And report that back to them.
Layering also facilitates the application of individual developer skills to parts of an enterprise system to which they are best suited. Those that are better with databases can work on the data access layer, for example, those who are better thinking along abstract lines can spend more time working on the core reusable services, and those who have web experience can work on browser base forms.
2.2.3 Software\Business Patterns In General
Many great efforts have been made over the last ten plus years to document software and business domain patterns that offer solutions which can be reused within other systems. These patterns provide tried and true techniques for solving problems encountered in many systems. They give software architects and developers immediate access to tools that they can used to solve problems they are faced with, without requiring them to go through all the trial and error the original authors went through in finding that optimal solution. They also facilitate the ability by architects and developers to describe how major areas of functionality within the system works by just naming the underlying patterns.
Documented patterns can be found for many levels of software development. There are patterns called “design patterns” that solve many smaller problems within an application. Some commonly known design patterns are: singleton, bridge, facade, strategy, and chain of command. There probably isn’t a programmer who has built a large system who hasn’t used all of those patterns whether he was aware of there names are not. Details on when and where to use these particular patterns can be found in the classical reference, "Design Patterns," by Gamma et al. There are more substantial patterns that can be used for the overall architecture of an enterprise’s systems. Some commonly used architectural patterns besides the ones mentioned earlier are pipes and filters, broker, microkernel, and model-view-controller. Additional information on when and where to use these patterns can be found in the book "Pattern Oriented Software Architecture – A System Of Patterns," by Buschmann et al. There are even collections of patterns for individual business domains such as accounting and finance, observation and measurement, and trading. Martin Fowler’s book "Analysis Patterns – Reusable Object Models" has more to say on that subject.
Any new enterprise development effort should try and benefit from the knowledge of others in order to deliver a successful solution with the least effort that will withstand the test of time. Patterns offer a quick means of doing that. But if the mentality remains that everything needs to be “invented here” the whole project is going to take much longer. Because of the importance of patterns, the one question I asked all development candidates is “Have you heard about, or do you know about software patterns.” Their answer allows me gauge whether they keep up with best practices in the industry and are open to learning from others. At the end of the day, the enterprise is its people and what they know.
2.3 Techniques
The following sections discuss some very specific things that enterprise system builders should be doing.
2.3.1 Components Versus Objects – A System’s Building Blocks
Components involve the packaging of smaller objects into a container capable of managing them to provide a more robust implementation of their combined functionality. By packaging smaller objects into larger containers capable of handling bigger units of work, other parts of the system (and developers) need know less about the specifics of what the individual objects do, or the details of what it takes to make them work together, making it much easier to re-use them.
It is much easier to construct new system modules quicker working with components rather than objects, as the units of work that are cobbled together are bigger. They also permit the development team to more quickly adapt a system to changes in business needs, as functionality is packaged at a more comprehensible and manageable level.
At this juncture in time, Object Oriented Development (OOP) would probably be better referred to as Component Oriented Development (COD), as that is where it has progressed.
2.3.2 To Components And Beyond -- Interfaces
As significant as the movement to emphasizing components over objects has been, so has been the movement to emphasizing that people focus on programming to interfaces versus class hierarchies.
In the past, systems that relied heavily on class hierarchies to get there work done were found to be brittle. Changes to one level of the hierarchy would often break classes in many other levels. The primary problem is that class hierarchies tend to create tightly couple items, which fly’s in the face of our advocacy of loose coupling between parts. Today, system development work tends to focus more on whether a class can perform a particular set of behaviors as identified by a specific interface (contract). Interfaces represent a set of methods with particular signatures that define a collection of behaviors. Any individual class can implement a number of different interfaces to represent a broader range of functionality. As long as an object or component implements a particular interface, it does not matter where in the object hierarchy it is, other parts of the system can make use of it through its interface.
2.3.3 Configurable Systems Through Use Of Meta-Data
Greater emphasis is being placed on meta-data to configure and drive many areas of new systems. In the past, this has been an area that commercial software developers paid most attention to, because of the needs of their packages to run different types of businesses. However, it does also offer individual enterprises the opportunity to do such things as easily change deployment, reporting, and data access options. Or provide the means to produce highly configurable business rules for systems that can be adapted quickly to changing business needs. And probably most importantly of all, it can provide inputs for a code generator capable of producing 50% plus of all code needed by a new enterprise system.
A balance does need to be established between the complexity of the meta-data gathered and its utilization in regards to the size of the development team. If the development team is large enough, it’s easier to have developers dedicated to this activity to produce a sophisticated implementation that provides an excellent return on investment. It’s also easier to provided continuity amongst programmers to keep the solution productive overtime. However, in a smaller programming shop it’s often much harder to get permission to be able to allocate even a single developer part time to such an activity, because each developer has such a wide set of demands to meet already. And even if one does do so successfully, it’s harder to keep productive continuity on this activity overtime, as when an individual programmer leaves such smaller teams they take a huge block of knowledge with them, sometimes the entire knowledge about a particular subsystem. Especially, in the case of companies who have the same programmers work on the same subsystem every time it needs to be enhanced.
2.3.4 Code Generation
The idea of code generation (code that writes code) has become a hotter topic of late, with many companies having great success using it to cut development costs and delivery schedules, especially, in large new development efforts where they are generating between 50 to 80 percent of the actual products code.
Today’s code generation engines are a better breed then those of years ago. They tend to offer more flexibility through the creation and use of simple templates that instruct the code generator on how to produce all the repetitive and highly structured code that they would otherwise need to write.
In a world class operation, the ultimately goal should be the ability to re-generate code as needed when things change. For example, be able to recreate the properties for business components when new fields are added to tables and any associated stored procedures.
2.4 Being Forward Looking
The topics under this section discuss some items that any forward-looking system designer should take into account.
2.4.1 Scalability Versus Performance
The scalability of a system defines how responsive it remains overall while greater work loads (more users\transactions) are placed on it. A well-designed scalable enterprise system is capable of handling hundreds to thousands of people, while still being capable of maintaining a good response to each individual.
Generally, enterprise systems are made scaleable by splitting them across tiers with each tier specializing in a specific activity. Inserting additional tiers does significantly and adversely affect the performance of any single request because of network latency and other issues. However, because of specialization in what they do, tiers can be designed to handle additional requests by more users in such an efficient manner, that any additional performance hit per user is negligible. For example, if everyone logging on to your application requires a list of active stores, that information could be gathered from the database by an enterprise tier when the first person makes such a request, it could then be cached on that tier, so it can be subsequently passed along immediately to all others making the same request. The .NET framework itself also implements pools of shared re-usable resources that users can be given access to as needed, the resources are returned to the shared pool once the user no longer needs them, from where they can be distributed to the next person without having to recreate the resource from scratch, often a costly activity.
The Modularization and Layering architectural patterns mentioned above can be made use of, in the creation of tiers, permitting the offloading of one or more system services to a separate computer. For example, reporting could be run as its own tier on a separate computer, allowing the user to continue with the full use of his computer after dispatching a report request.
Why focus on scalability and performance?
While a good architecture can help a system scale to hundreds and thousands of users, simple bad practices can bring it to its knees, even with only a limited number of users. Even small systems grow overtime, so it pays to give appropriate attention to architecting and following best practices from the beginning in regards to scalability and performance, else, there's a much larger cost to be paid down the road when the troubles begin with increased usage of the system.
Real life problem!
When I arrived as a consultant at my most recent assignment three years ago the company had an enterprise system that had a number of particular bad behaviors that constrained its scalability, such as: 1) Just about all significant activities (functional areas) in the system either read or updated just a few common tables, with many of them having an even greater dependency on a single table called OrderItems. (With such a concentration of access a lot of effort has to be expended to avoid contention issues when the system is busy.) 2) Large complex reports and summary processes were being run against the online-transaction database during normal working hours. 3) Worse of all, tasks in both the above activities that really only need to be executed with dirty reads were not being executed as such.
Luckily enough, the fix to many of the bottlenecks that have been encountered over the last three years, as the work load on the database has increased, has been as simple as adding the ‘with NoLock’ option I recommended to many of the problem queries. However, even with such simple changes preventing many of the problems that prevent the system from coming to a grinding halt, the running of complex reports and activity summaries against the production online-transaction database can still significantly slow down the entire system. Moreover, there are still a couple of major processes, which if run at the same time in the busy season are capable of bringing certain system activities to a halt. Fixing these problems for the fundamental shortcomings in the systems original architecture (or lack there of it) requires addressing some major issues. Such as, providing a separate reporting database, whose data is periodically updated from the production database for complex reports and activity summaries, and rewriting some major business activities not to be so interdependent on the same data, or to better schedule their access to that shared data. All items, that should have been considered in the beginning when building the system when it would have been much cheaper to do so.
2.4.2 Technology Usage
In building an enterprise level application, people should always be looking to the future, not only in regards to where the company might be going, but also as to the likely direction technology might follow. Older technology offers less functionality, and is harder and more costly to maintain as time progresses. Working with current technology makes its easier to find developers, free code, and productivity tools in a larger more active development community.
For example, if one was using the current technology of C# 2.0 and .NET 2.0 she would have access to freely available frameworks that would take care of much of the plumbing of any new system, a major cost and time saver. And the execution of smart business components that are easily re-used across Windows and Web Applications, and Window Services.
2.4.3 Long System Life
Systems always seem to live longer then most people expect. In fact, prior to focusing on being an enterprise developer, I completed a number of departmental level systems that were only expected to be run for six months where just about all of them ended up lasting 3 to 5+ years. Had I actually programmed against the original estimate of their lifetime usefulness, the client would have incurred additional cost to extend their life. However, the cost of programming for a much longer life was so small while writing the application, that I did it, which in the end turned out to be a major time saving rather then having to go back through another design, analysis, and testing cycle six months down the road when I was no longer as familiar with those requirements.
It’s even more important to plan for long life with a custom enterprise system built for a growing company, as the costs of retrofitting it later to extend its life is quite costly. Just as with small system its durability should be built with a long-term horizon in mind, not just a company’s present needs. It should be many, many years before a business starts to stress the capacity of a newly developed system. It certainly shouldn’t be necessary to make apologies for a system’s shortcomings shortly after it’s been delivered, (or even as soon as it’s delivered).
Real life problem!
My current client, a manufacturing company, whom I've been with for the last three years, had their custom in-house enterprise system designed and built by a couple of mid level programmers who worked away hard trying to produce a system that would meet their current needs. Four years ago, after two years of development, management because of there existing system crashing regularly everyday said, enough done, go live as soon as you can with what you have. Almost as soon as this system was released it showed signs of premature ageing, and had numerous issues handling heavy demands, many of which over the years were capable of bringing the entire system to a grinding halt. In the three years, I've been there I've seen much effort and money spent analyzing those problem areas and fixing them. The saddest thing of it all, is that when they were only six months into the project, they discussed it with me, and I'd advised them to make a course correction and add a software architect to the project, who would be better able to ensure that a more robust system was built that would enjoy a longer life, but they chose to proceed along the path they were already traveling.
2.5 Development Methodology
2.5.1 Iterative Development With Regular Reviews
Completing a detailed enterprise system design up front is an impossible task. Don’t let anyone tell you otherwise. Commonly, when interviewing business people, things are not always recorded correctly, or even interpreted properly. Not to mention, that people forget to tell you about important things because their not part of their everyday activity, or because they assume you know such information already. And also because they can’t easily visualize how a new improved process might be put together on a computer, (there tool users not builders). In addition, as volumes of detailed information are gathered on a large system, designers tend to lose their ability to keep everything straight in their heads and plan out a perfect solution. Another issue with spending a lot of time designing large systems up front is that parts of the design are made obsolete overtime as an organization gets into new business areas.
The only really practical way to develop a large system efficiently is by gathering some high level requirements initially, and then moving on quickly to focus on individual functional areas\modules in an iterative development-cycle. With most importantly, regular feedback being sought from actual business people, so that regular adjustments can be made in a timely manner while the cost of doing them is still small. With a constant and regular feedback cycle the module that is finally released is more likely to be what business user needs.
2.5.2 Test Driven Programming and Continuous Integration
Test Driven Programming is a popular technique for ensuring the delivery of robust systems and maintaining them as such. It involves the construction of tests at the same time requirements are specified, where those tests are coded before the business functionality itself. As working code start passing those tests, one gains increasing confidence that the new functionality meets requirements. In addition, those tests can be executed in a testing harness with automated executions which allow them to be run at anytime during the initial development process or any subsequent enhancement cycle to make sure that the current version continues to pass all existing tests.
A complementary activity to Test Driven Programming is the concept of Continuous Integration, which set-ups automatic builds of systems from frequently checked in code, upon which tests are automatically run, after which a final exception report is emailed to interested parties.
The loose coupling of applications and components described elsewhere in this document permits simpler tests to be created as fewer interdependencies need to be handled.
Some advocates of Test Drive Programming even view the tests as a means of providing some additional documentation as to what a system’s business rules are. However, I personally feel that such information would not be widely used and tests should instead likely make reference to particular rules in some better and more widely accessible classification of business rules.
Test Driven Programming and Continuous Integration can provide pragmatic tools for maintaining a robust system and vendor tool support and company usage of these techniques is growing strongly.
2.6 Low Level Concepts
2.6.1 Flexible Deployment
The ability to deploy the system in a single tier (on a single PC) or in multiple tiers (individual user PCs plus one or more application, database, or web servers) should be easily configurable from an application’s file settings.
For example, if a remote sales office only has one user, then it might make best sense to run the module(s) he needs on a single PC. Other larger remote offices or facilities might need to run their modules across individual PCs and at least one server for better performance. The main office with hundreds and thousands of employees may want to make use of separate servers for individual tiers to get good performance for everyone. All of this should be configurable from an easily modified application settings file.
It would also be highly desirable that the application itself be deployable over the web with a single action on the part of the user.
2.6.2 Provide Instrumentation Of Process And Data Pathways
Using a single framework (or bus architecture) though-out a system provides the opportunity to provide instrumentation of individual processes and data pathways to identify who used what, when, how and where. Such activity can help in trouble shooting performance problems and in identifying what parts of a system are still being actively used.
2.6.3 Components - Further Considerations
Some additional considerations that should be made when constructing components are described below.
2.6.3.1 Components – Promotion Of Abstraction In Design
Promoting abstraction while designing components can make them more flexible. If components are not unduly specific to any current set of steps or datum, they can be more easily enhanced, as business needs change.
2.6.3.2 Components - Design For Behavior Not Data
Many developers new to Object Oriented Programming (OOP) tend to make the rookie mistake of creating too many simple classes that are largely only record set representations of all the tables in their database, a behavior that is encouraged by vendors of object-mapping-relation products, which will automate the creation of such objects. The biggest problem with having a data-centric view in performing OOP is that it tends to result in light weight objects that contain little intelligence. Which requires, that the smarts they should have contained, be coded elsewhere in the system where its not easily reused and often ends up getting reproduced everywhere those objects are used.
Rather then simple data objects, it is better to focus on building business components that might represent data from a collection of tables and all the behavior that needs to be wrapped around them to perform some business task to promote code reuse. For example, an Order business component in its most simple form should be capable of holding customer summary data, such as their name, phone number and address, and a separate data line for each product they ordered, along with methods (behavior) to check that the customer has a good credit rating and calculate the time it will take to deliver their goods. If all this data and behavior is wrapped up in a single business component, then it is easier for Windows and Web programmers to build ordering tools that reference that single item to allow people place orders from both sources. And while the company needs to incur the cost of building two different types of interfaces they save money by only building one business component that does a good deal of the work.
2.6.3.3 Components - Normalize Behavior
Just as relational databases should focus on storing data in a normalized form (without repeating data groups) the practice of object oriented programming should focus on the normalization of behavior.
Taking our example of the Order component described in the section “Components - Design For Behavior Not Data.” it’s not necessary that the component actually know shipping times for products to a customer if the shipping system can already calculate that information. It would be better that such behavior only be implemented in one place, most likely the shipping system, and that the Order component would request a shipping schedule after passing the shipping system a list of products and their destination. The goal being that actual behavior gets coded in only one place, and therefore, there is only place it ever needs to be updated when changes are needed. And while the actual calculation of a shipping date\schedule would be done elsewhere, users of the Order component do not need to be aware of that. They would just ask the Order component directly for the estimated ship date, whether the component calculates that information itself, or passes the request along elsewhere, should not be of concern to anyone using the component.
2.6.4 Simplify Format Of Data Storage And Manipulation
It’s desirable that data be passed around the system in a fairly rudimentary format that is largely independent of any vendor specific data management technology that is likely to change in the short-to-medium term. This helps reduce an application’s dependence on any software vendor’s more specialized data management technology, which they seem to implement a new model for every two to three years.
For example, large amounts of table like data in .NET components can be kept in simple structures such as Binding Lists, which are largely, just simple table organizations of data into rows and columns that also support data binding which provides for bidirectional data change. Rather then the much more sophisticated data typed xml datasets which are much more likely to be subjected to changes in each new release of .NET and eventual replacement within two to three years when the vendor creates a new data management model. The simpler data structures may change too, but generally those changes will be small and manageable, if they affect the business components at all. Even better, if the behavior around those simple structures is coded in a common ancestor for business components changes only need to be made in one place for all components, which is not the case for the more complex data structures, as the get implemented as independent entities with no common ancestor in the enterprise’s custom code base. Another advantage of the simpler structure is that it’s easier and faster to pass data from one component to another for any additional processing, as the data’s storage container carries much less baggage with it.
3 Conclusion
Building enterprise systems is an inherently complex business. Getting all the requirements and design right at the beginning of a large system never happens. If what was presented in this document didn’t persuade you of that, consider the fact, that no one has the rights to a working crystal ball to predict the future, and stuff happens. So by necessity, system development has to be done while trying to hit a moving target. Change is the only constant in our lives, small changes happen all the time, more fundamental changes occur as time progresses.
In accepting the above facts, one can be proactive about planning to handle change when building large systems. The development effort will be much easier and more successful if one selects methodologies, tools, and techniques to craft an adaptable development process that is responsive to changes. Be such change, a result of new knowledge that is uncovered in regular business reviews, or new business practices that occur as the system’s development is underway. Having an adaptive development process, allows the development personnel to make necessary changes while development is underway, while it’s still cheaper to do so, and to produce a system that is better geared to what the business’s needs are at the time of delivery.
Some of the most important guidelines discussed in this document for providing an adaptable were: 1) Following architectural patterns such as modularization and layering. 2) Focusing on development of business components rather than simpler data-centric objects. 3) Programming to interfaces rather then physical implementations of classes, and normalizing behavior across classes. 4) Using the Agile methods of Interactive Development, Test Driven Programming, and Continuous Integration. And 5) Performing abstract thinking while designing.
While this document covered a number of areas of knowledge that any enterprise software architect\developer should be familiar with; it’s not intended to be a comprehensive list. Overall, software architecture is a pretty comprehensive and complex subject area offering many challenges.
In larger companies, the software architectural load can be split amongst architects by allowing them develop specialties from which they can contribute unique knowledge to the team effort, while only being required to have a general knowledge about other areas of the field of software architecture.
In smaller companies, the software architect generally needs to cover even more ground, as the company isn’t large enough to have someone dedicated to work on that activity full time. In such cases, the architect also needs to manage the development staff, do project management, and actually help write the software. This can result in them having less then an ideal level of knowledge in particular areas, so to compensate, certain responsibilities need to be pushed further down the line to other development personnel who are willing to accept those responsibilities and learn new skills to perform.
4 References
The following books have provided me with much of the schooling which helped me write this document. They are definitely books that should be in any software architect's library.
Analysis Patterns – Reusable Object Models by Martin Fowler, ISBN 0201895420. Provides a catalog of “business domain” patterns.
Applying UML and Patterns—An Introduction to Object-Oriented Analysis and Design by Larman, ISBN 013748807. Provides a foundational discussion for building robust, scalable, maintainable systems, using object technology.
Building Business Intelligence Applications with .NET by Robert Ericsson, ISBN 1584502711. A practical guide to using .NET tools to build applications for OLAP and data mining.
Code Generation in Microsoft .NET by Kathleen Dollard, ISBN 1590591372. Teaches code generation as a scriptable and repeatable process using templates, so one is not tied to a particular framework or style. Kathleen is one of the industry leaders and a major advocate of code generation. Her book also has a couple of chapters on code generation for the CSLA framework (refer to the “Expert C# Business Objects” book below).
Component Software – Beyond Object-Oriented Programming by Clemens Szyperski, ISBN 020117885. It’s about developing re-usable off-the-shelf components that handle a significant amount of functionality. It emphasizes that one should not focus on trying to build each new piece of functionality from collections of small independent objects while developing larger systems. One should instead, focus on developing more robust components that wrap up smaller objects into bigger containers that are capable of performing bigger units of work, which in turn, require less effort by other developers to reuse them to build even bigger functional units.
Design Patterns – Elements of Reusable Object-Oriented Software by Erich Gamma, Richard helm, Ralph Johnson, and John Vlissides, ISBN 0210633612. This book is the original classic on software design patterns, and still the best reference on the subject.
Domain Driven Design by Eric Evans ISBN 0321125215. This book makes a persuasive argument for the use of domain models to provide a ubiquitous language that ties the domain experts and technologists together. Also, it emphasizes that really powerful domain models only evolve overtime and that some of the best ideas appear after the initial release of a system.
Enterprise Integration—An Architecture of Enterprise Application Systems Integration by Fred A Cummins, ISBN 0471400106. Excellent book that reviews the technology landscape defining enterprise integration objectives, and which provides generalized enterprise integration architecture.
Enterprise Integration Patterns—Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf, ISBN 0321200683. Describes using asynchronous message as a proven strategy for enterprise integration, and offers 65 patterns for building such systems.
Expert C# Business Objects by Rockford Lhotka, ISBN 1590593448. A book about application architecture, design, development, and using object-oriented concepts in .NET. Book describes the building of a “Component-Based, Scaleable, Logical Architecture” (CSLA) in C# .NET, which has been used as the basis of system development efforts in many companies with good success. Also, the company Rocky Lhotka works for relies on his framework and code generation to produce upwards of 80% of the code in new systems they develop for clients.
Object Technology – A Manager’s Guide, 2nd Edition by David A. Taylor, PH.D., ISBN 0210309947. This book is an easily read classic on the subject. One of its central themes is that objects are not about the data they contain, but the abstracted behaviors they represent.
Patterns of Enterprise Application Architecture by Martin Fowler, ISBN 0321127420. A book written by a well known industry expert on the topic of suitable patterns for enterprise architecture. The book is written in two parts, the first part is a short tutorial on developing enterprise applications. The second section and bulk of the book is a detailed reference to patterns that assist in building enterprise applications.
Pattern Oriented Software Architecture—A System of Patterns by Frank Buschmann, Regine Meunier, Hans Sommerland, and Michael Stal. An excellent book concentrating on Architectural Software Patterns such as: Layers, Pipes and Filters, Distributed System Broker, Model-View-Controller, and Microkernel.
Strategic Data-Planning Methodologies by James Martin, ISBN 0138511136. The first good book I ever read on the topic of strategic data planning, and still one of favorites. Additional information about managing enterprise system development can also be found in the list of references provided at the end of this document.
5 Additional Notes
Loose coupling can be achieved by linking the parts of an application through the transfer of information relating to business transactions in an asynchronous (“direct real time”) mode of communication or in a more flexible manner by using an intermediary system that specializes in the transfer of information, which receives information on transactions from subsystem\module of an application, which are placed in a storage queue after acknowledging receipt, those messages can then be either broadcasted or polled by the subsystems of other applications as needed.
It’s not necessary that a layer be an actual physical separation within the application as long as the functionality that defines that layer is well scoped and does not vary. For example, the basic CRUD (Create, Read, update, and Delete) operations of business components could be grouped together within a component and named the same across all business components. When it comes time to update those routines for a new data access technology they can then all be easily located and dealt with together. Current features in .NET 2.0 languages such as code folding regions and partial classes provide some nice mechanisms to facilitate such activity. While these techniques may suffice for smaller individual companies who will only use one database management product, large companies or commercial software vendors may go further and create a separate Data Access Layer (DAL) that abstracts the access and manipulation of many database management systems to produce a single API for their development staff, who through that API can request or manipulate data in a variety of database products such as Oracle, Microsoft SQL Server, or IBM’s DB2, without being aware of local variations in the SQL language of those products.
Even being the smaller elements of a system, objects should in turn package individual behaviors with minimal interdependencies on other objects, so that larger components can more easily make use of them.
An application’s tiers can be set to run on one or more similar computers in a matrix (farm) like setup when necessary to provide scalability to thousands of users.
Edited: 11/20/2005 and 12/28/2006
Table of Contents
1. Overview
2. Architectural Goals for Enterprise Level Systems
..2.1 High Level Concepts
....2.1.1 THE BASICS - Blueprint, Framework, and Standards
....2.1.2 Business Domain\Functional Area Database Model
..2.2 Patterns
....2.2.1 Modular Architectural Pattern\Business Domain Modeling
....2.2.2 Layers Architectural Pattern
....2.2.3 Software\Business Patterns In Genera
..2.3 Techniques
....2.3.1 Components Versus Objects – A System’s Building Blocks
....2.3.2 To Components And Beyond -- Interfaces
....2.3.3 Configurable Systems Through Use Of Meta-Data
....2.3.4 Code Generation
....2.4 Being Forward Looking
....2.4.1 Scalability Versus Performance
....2.4.2 Technology Usage
....2.4.3 Long System Life
..2.5 Development Methodology
....2.5.1 Iterative Development With Regular Reviews
....2.5.2 Test Driven Programming and Continuous Integration
..2.6 Low Level Concepts
....2.6.1 Flexible Deployment
....2.6.2 Provide Instrumentation Of Process And Data Pathways
....2.6.3 Components - Further Considerations
....2.6.4 Simplify Format Of Data Storage And Manipulation
3. Conclusion
4. References
5. Additional Notes
1 Overview
This document provides a short selection of important topics, which should be considered by an architect of an Enterprise System.
This document looks at things primarily from a development\architectural perspective, rather then an overall project management view. Thus, many other important things that contribute to the success of any enterprise system development effort, such as having strong sponsors, active participation by managers and key business people, are not discussed here.
2 Architectural Goals For Enterprise Level Systems
The table of contents of this document identifies the list of architectural\design issues that I discuss below, all of which are attributes of adaptable enterprise systems.
2.1 High Level Concepts
This section of the document “High Level Concepts” presents some concepts that should be definitely followed.
2.1.1 THE BASICS - Blueprint, Framework, and Standards
Large software systems should be implemented according to an overall guiding blueprint. A practice that is present in most other engineering disciplines developing any kind of large or complex system.
Can you imagine trying to build a large office building without a blueprint? Who would be able to review or approve the proposed design? How would other engineers (plumbers, electricians, and telecommunication experts) coordinate their activity so they didn’t end up working at odds with each other? How would they know if their proposed effort will be adequate to support other trades people efforts?
With large enterprise systems, which can easily end up with million plus lines of code, everyone needs to be working with the same overall plan, if good re-use is to be made of each other’s efforts. Without a guiding blueprint, components and services designed by different individuals will not necessarily work together well, if at all. Following a good blueprint can also help developers build components and services within a vision of how the entire system needs to scale to handle future growth. Thus, part of any guiding architectural blueprint should be the specification of a scaleable application framework or bus architecture, into which many of the generic services such as data access, security, authentication, authorization, etc., can be plugged-in.
Furthermore, all development work should also be carried out according to agreed upon standards, which should cover conventions for naming, coding, formatting, and organization of code. Also, the company’s development activity should have a fulltime hands on software architect\technical lead to guide it and coordinate standards setting.
What use are standards?
If individual items within a large system that perform similar activities are consistently named, people can guess what something does just by its name. Naming can also be used to organize all the stored procedures or tables that cover one business domain together, making it easier to identify and work within that functional area.
Real life problem!
Over the last three years I’ve worked at a manufacturing company as a consultant, the topic of using a common set of standards for development has repeatedly come up, but no single set of coding standards has been adopted by all developers. The group doesn’t even have something as simple as a consistent way of naming new items. For example, the original developer of their enterprise system had no observable standard at all for naming stored procedures, and he created 200 of them. Of the three newer developers, one used begin all his procedures with a ‘p_’ prefix and another with a ‘proc’ prefix, but neither of them followed any other naming rules for their procedures. Between those two developers, 100 new procedures have been added in those two separate groupings. The third developer instituted his own more formalized standard, adding procedures with the following format domain_prefix + main_table_name + action_verb + optional_qualifier. Whose collection of procedures do you want to be working on? For me it's definitely the last guy mentioned, and even though he's added 200 stored procedures to the database, I can easily identify what each of them is likely to be doing before I even examine their code. His EDI_Order_ins stored procedure does what it says. In fact, his EDI_Order_del, EDI_Order_sel, and EDI_Order_upd procedures all do what they say and can be found grouped together in any alphabetically listing. However, trying to find the set of procedures written by any of the other developer on any particular topic is not so easy.
So, as you can likely guess, with so little agreement on even the smallest of standards the development of their enterprise system does not follow most of the standard guidelines outlined below for developing adaptable enterprise systems, which has made working with their enterprise system increasingly difficult as it grew. While management was made aware of these issues, they chose to ignore them, as the system as it was, supported an annual average growth rate of 13% during that period. However, in my view, the limitations of the system were actually constraining factors that prevented the company from growing even more that 13% per year. Moreover, they are now kicking off a project to examine whether they should by a commercial package are start writing a replacement for the current system as they don't believe it will support their growth and business direction in the next couple of years, and is no longer worth the cost of modifying it to do so.
2.1.2 Business Domain\Functional Area Database Models
In regards to design, the next level of detail that enterprise system development should get into is the development of business domain\functional area database models with the goal of producing a common language and understanding between business and IT personnel for those business activities. And, subsequently offer guidance for the actual development.
Divide and conquer should be the mantra for any large system. Humans are only capable of juggling so much detail and interactions at one time. So scoping deliverables at an appropriate level makes it easier to work on large solutions successfully.
2.2 Patterns
Patterns offer the opportunity to learn from other people’s successes and avoid some mistakes.
2.2.1 Modular Architectural Pattern\Business Domain Modeling
Modularization is one of the fundamental software architectural patterns that’s been successfully applied to build all large enterprise systems over the last twenty years. It involves the break-up of the enterprise system into smaller relatively self contained subsystems that handle major functional areas of the business, such as, Product Development\Merchandising, Ordering, Logistics, and Finance. These smaller subsystems can then be developed and installed separately or even substituted with best of breed applications.
This reduction in the reliance on direct linkages between the different areas of a system is commonly referred to as “loose coupling.” Because of the use of indirect links\data exchanges, modules can for the most part be individually enhanced and tested without risking cascading side affects to other parts of the system. It is also the reason why many vendors of commercial enterprise systems can supply individual companies with paired down versions of their products containing just the modules necessary for running their particular business activities. One additional nice aspect of loose coupling is that it dovetails well with the use of Business Domain\Functional Area Database Models.
2.2.2 Layers Architectural Pattern
Layering is another one of the fundamental software architectural patterns that’s been successfully applied to build large systems of all sorts. It’s a tried and true technique for packaging cohesive elements of a system in their own layer, to provide some core service to the rest of the application and even between applications. If the communication protocols across the boundaries of a layer are respected by all parts of the system the classes within that layer can be significantly changed without interfering with the rest of the application as long as the layer continues to service its original interfaces.
This is one of the most challenging areas of building enterprise systems as it requires that people be able to think abstractly about what services need to be made widely available and how they might be generically implemented to facilitate their reuse. And which services are similar enough in nature that they should be packaged together in a larger cohesive part of the system. Often the best guidance on this topic is to review other publicly available frameworks to see what types of things they do and what parts of them could be adapted for ones own use.
A common use of the layering pattern in enterprise systems is to inoculate the rest of a system from any undue influence of a particular data access technological. Such technologies tend to change every two to three years and systems in which it has not been isolated in a single layer get caught in a situation where there is no easy path to the new data access technology and they end up tending to use the older technology well beyond it best use date.
An excellent use of layering that is not used commonly enough is to put business logic\rules inside business components that live within a business layer that is accessible by the remaining upper layers of the system. This would provide (at the very least) the presentation layers direct access to that business logic, reducing the amount of code that needs to be written within them. For example, this would permit both a Windows and Web application to make use of the same business component to check when a customers next delivery can be expected or his account balance And report that back to them.
Layering also facilitates the application of individual developer skills to parts of an enterprise system to which they are best suited. Those that are better with databases can work on the data access layer, for example, those who are better thinking along abstract lines can spend more time working on the core reusable services, and those who have web experience can work on browser base forms.
2.2.3 Software\Business Patterns In General
Many great efforts have been made over the last ten plus years to document software and business domain patterns that offer solutions which can be reused within other systems. These patterns provide tried and true techniques for solving problems encountered in many systems. They give software architects and developers immediate access to tools that they can used to solve problems they are faced with, without requiring them to go through all the trial and error the original authors went through in finding that optimal solution. They also facilitate the ability by architects and developers to describe how major areas of functionality within the system works by just naming the underlying patterns.
Documented patterns can be found for many levels of software development. There are patterns called “design patterns” that solve many smaller problems within an application. Some commonly known design patterns are: singleton, bridge, facade, strategy, and chain of command. There probably isn’t a programmer who has built a large system who hasn’t used all of those patterns whether he was aware of there names are not. Details on when and where to use these particular patterns can be found in the classical reference, "Design Patterns," by Gamma et al. There are more substantial patterns that can be used for the overall architecture of an enterprise’s systems. Some commonly used architectural patterns besides the ones mentioned earlier are pipes and filters, broker, microkernel, and model-view-controller. Additional information on when and where to use these patterns can be found in the book "Pattern Oriented Software Architecture – A System Of Patterns," by Buschmann et al. There are even collections of patterns for individual business domains such as accounting and finance, observation and measurement, and trading. Martin Fowler’s book "Analysis Patterns – Reusable Object Models" has more to say on that subject.
Any new enterprise development effort should try and benefit from the knowledge of others in order to deliver a successful solution with the least effort that will withstand the test of time. Patterns offer a quick means of doing that. But if the mentality remains that everything needs to be “invented here” the whole project is going to take much longer. Because of the importance of patterns, the one question I asked all development candidates is “Have you heard about, or do you know about software patterns.” Their answer allows me gauge whether they keep up with best practices in the industry and are open to learning from others. At the end of the day, the enterprise is its people and what they know.
2.3 Techniques
The following sections discuss some very specific things that enterprise system builders should be doing.
2.3.1 Components Versus Objects – A System’s Building Blocks
Components involve the packaging of smaller objects into a container capable of managing them to provide a more robust implementation of their combined functionality. By packaging smaller objects into larger containers capable of handling bigger units of work, other parts of the system (and developers) need know less about the specifics of what the individual objects do, or the details of what it takes to make them work together, making it much easier to re-use them.
It is much easier to construct new system modules quicker working with components rather than objects, as the units of work that are cobbled together are bigger. They also permit the development team to more quickly adapt a system to changes in business needs, as functionality is packaged at a more comprehensible and manageable level.
At this juncture in time, Object Oriented Development (OOP) would probably be better referred to as Component Oriented Development (COD), as that is where it has progressed.
2.3.2 To Components And Beyond -- Interfaces
As significant as the movement to emphasizing components over objects has been, so has been the movement to emphasizing that people focus on programming to interfaces versus class hierarchies.
In the past, systems that relied heavily on class hierarchies to get there work done were found to be brittle. Changes to one level of the hierarchy would often break classes in many other levels. The primary problem is that class hierarchies tend to create tightly couple items, which fly’s in the face of our advocacy of loose coupling between parts. Today, system development work tends to focus more on whether a class can perform a particular set of behaviors as identified by a specific interface (contract). Interfaces represent a set of methods with particular signatures that define a collection of behaviors. Any individual class can implement a number of different interfaces to represent a broader range of functionality. As long as an object or component implements a particular interface, it does not matter where in the object hierarchy it is, other parts of the system can make use of it through its interface.
2.3.3 Configurable Systems Through Use Of Meta-Data
Greater emphasis is being placed on meta-data to configure and drive many areas of new systems. In the past, this has been an area that commercial software developers paid most attention to, because of the needs of their packages to run different types of businesses. However, it does also offer individual enterprises the opportunity to do such things as easily change deployment, reporting, and data access options. Or provide the means to produce highly configurable business rules for systems that can be adapted quickly to changing business needs. And probably most importantly of all, it can provide inputs for a code generator capable of producing 50% plus of all code needed by a new enterprise system.
A balance does need to be established between the complexity of the meta-data gathered and its utilization in regards to the size of the development team. If the development team is large enough, it’s easier to have developers dedicated to this activity to produce a sophisticated implementation that provides an excellent return on investment. It’s also easier to provided continuity amongst programmers to keep the solution productive overtime. However, in a smaller programming shop it’s often much harder to get permission to be able to allocate even a single developer part time to such an activity, because each developer has such a wide set of demands to meet already. And even if one does do so successfully, it’s harder to keep productive continuity on this activity overtime, as when an individual programmer leaves such smaller teams they take a huge block of knowledge with them, sometimes the entire knowledge about a particular subsystem. Especially, in the case of companies who have the same programmers work on the same subsystem every time it needs to be enhanced.
2.3.4 Code Generation
The idea of code generation (code that writes code) has become a hotter topic of late, with many companies having great success using it to cut development costs and delivery schedules, especially, in large new development efforts where they are generating between 50 to 80 percent of the actual products code.
Today’s code generation engines are a better breed then those of years ago. They tend to offer more flexibility through the creation and use of simple templates that instruct the code generator on how to produce all the repetitive and highly structured code that they would otherwise need to write.
In a world class operation, the ultimately goal should be the ability to re-generate code as needed when things change. For example, be able to recreate the properties for business components when new fields are added to tables and any associated stored procedures.
2.4 Being Forward Looking
The topics under this section discuss some items that any forward-looking system designer should take into account.
2.4.1 Scalability Versus Performance
The scalability of a system defines how responsive it remains overall while greater work loads (more users\transactions) are placed on it. A well-designed scalable enterprise system is capable of handling hundreds to thousands of people, while still being capable of maintaining a good response to each individual.
Generally, enterprise systems are made scaleable by splitting them across tiers with each tier specializing in a specific activity. Inserting additional tiers does significantly and adversely affect the performance of any single request because of network latency and other issues. However, because of specialization in what they do, tiers can be designed to handle additional requests by more users in such an efficient manner, that any additional performance hit per user is negligible. For example, if everyone logging on to your application requires a list of active stores, that information could be gathered from the database by an enterprise tier when the first person makes such a request, it could then be cached on that tier, so it can be subsequently passed along immediately to all others making the same request. The .NET framework itself also implements pools of shared re-usable resources that users can be given access to as needed, the resources are returned to the shared pool once the user no longer needs them, from where they can be distributed to the next person without having to recreate the resource from scratch, often a costly activity.
The Modularization and Layering architectural patterns mentioned above can be made use of, in the creation of tiers, permitting the offloading of one or more system services to a separate computer. For example, reporting could be run as its own tier on a separate computer, allowing the user to continue with the full use of his computer after dispatching a report request.
Why focus on scalability and performance?
While a good architecture can help a system scale to hundreds and thousands of users, simple bad practices can bring it to its knees, even with only a limited number of users. Even small systems grow overtime, so it pays to give appropriate attention to architecting and following best practices from the beginning in regards to scalability and performance, else, there's a much larger cost to be paid down the road when the troubles begin with increased usage of the system.
Real life problem!
When I arrived as a consultant at my most recent assignment three years ago the company had an enterprise system that had a number of particular bad behaviors that constrained its scalability, such as: 1) Just about all significant activities (functional areas) in the system either read or updated just a few common tables, with many of them having an even greater dependency on a single table called OrderItems. (With such a concentration of access a lot of effort has to be expended to avoid contention issues when the system is busy.) 2) Large complex reports and summary processes were being run against the online-transaction database during normal working hours. 3) Worse of all, tasks in both the above activities that really only need to be executed with dirty reads were not being executed as such.
Luckily enough, the fix to many of the bottlenecks that have been encountered over the last three years, as the work load on the database has increased, has been as simple as adding the ‘with NoLock’ option I recommended to many of the problem queries. However, even with such simple changes preventing many of the problems that prevent the system from coming to a grinding halt, the running of complex reports and activity summaries against the production online-transaction database can still significantly slow down the entire system. Moreover, there are still a couple of major processes, which if run at the same time in the busy season are capable of bringing certain system activities to a halt. Fixing these problems for the fundamental shortcomings in the systems original architecture (or lack there of it) requires addressing some major issues. Such as, providing a separate reporting database, whose data is periodically updated from the production database for complex reports and activity summaries, and rewriting some major business activities not to be so interdependent on the same data, or to better schedule their access to that shared data. All items, that should have been considered in the beginning when building the system when it would have been much cheaper to do so.
2.4.2 Technology Usage
In building an enterprise level application, people should always be looking to the future, not only in regards to where the company might be going, but also as to the likely direction technology might follow. Older technology offers less functionality, and is harder and more costly to maintain as time progresses. Working with current technology makes its easier to find developers, free code, and productivity tools in a larger more active development community.
For example, if one was using the current technology of C# 2.0 and .NET 2.0 she would have access to freely available frameworks that would take care of much of the plumbing of any new system, a major cost and time saver. And the execution of smart business components that are easily re-used across Windows and Web Applications, and Window Services.
2.4.3 Long System Life
Systems always seem to live longer then most people expect. In fact, prior to focusing on being an enterprise developer, I completed a number of departmental level systems that were only expected to be run for six months where just about all of them ended up lasting 3 to 5+ years. Had I actually programmed against the original estimate of their lifetime usefulness, the client would have incurred additional cost to extend their life. However, the cost of programming for a much longer life was so small while writing the application, that I did it, which in the end turned out to be a major time saving rather then having to go back through another design, analysis, and testing cycle six months down the road when I was no longer as familiar with those requirements.
It’s even more important to plan for long life with a custom enterprise system built for a growing company, as the costs of retrofitting it later to extend its life is quite costly. Just as with small system its durability should be built with a long-term horizon in mind, not just a company’s present needs. It should be many, many years before a business starts to stress the capacity of a newly developed system. It certainly shouldn’t be necessary to make apologies for a system’s shortcomings shortly after it’s been delivered, (or even as soon as it’s delivered).
Real life problem!
My current client, a manufacturing company, whom I've been with for the last three years, had their custom in-house enterprise system designed and built by a couple of mid level programmers who worked away hard trying to produce a system that would meet their current needs. Four years ago, after two years of development, management because of there existing system crashing regularly everyday said, enough done, go live as soon as you can with what you have. Almost as soon as this system was released it showed signs of premature ageing, and had numerous issues handling heavy demands, many of which over the years were capable of bringing the entire system to a grinding halt. In the three years, I've been there I've seen much effort and money spent analyzing those problem areas and fixing them. The saddest thing of it all, is that when they were only six months into the project, they discussed it with me, and I'd advised them to make a course correction and add a software architect to the project, who would be better able to ensure that a more robust system was built that would enjoy a longer life, but they chose to proceed along the path they were already traveling.
2.5 Development Methodology
2.5.1 Iterative Development With Regular Reviews
Completing a detailed enterprise system design up front is an impossible task. Don’t let anyone tell you otherwise. Commonly, when interviewing business people, things are not always recorded correctly, or even interpreted properly. Not to mention, that people forget to tell you about important things because their not part of their everyday activity, or because they assume you know such information already. And also because they can’t easily visualize how a new improved process might be put together on a computer, (there tool users not builders). In addition, as volumes of detailed information are gathered on a large system, designers tend to lose their ability to keep everything straight in their heads and plan out a perfect solution. Another issue with spending a lot of time designing large systems up front is that parts of the design are made obsolete overtime as an organization gets into new business areas.
The only really practical way to develop a large system efficiently is by gathering some high level requirements initially, and then moving on quickly to focus on individual functional areas\modules in an iterative development-cycle. With most importantly, regular feedback being sought from actual business people, so that regular adjustments can be made in a timely manner while the cost of doing them is still small. With a constant and regular feedback cycle the module that is finally released is more likely to be what business user needs.
2.5.2 Test Driven Programming and Continuous Integration
Test Driven Programming is a popular technique for ensuring the delivery of robust systems and maintaining them as such. It involves the construction of tests at the same time requirements are specified, where those tests are coded before the business functionality itself. As working code start passing those tests, one gains increasing confidence that the new functionality meets requirements. In addition, those tests can be executed in a testing harness with automated executions which allow them to be run at anytime during the initial development process or any subsequent enhancement cycle to make sure that the current version continues to pass all existing tests.
A complementary activity to Test Driven Programming is the concept of Continuous Integration, which set-ups automatic builds of systems from frequently checked in code, upon which tests are automatically run, after which a final exception report is emailed to interested parties.
The loose coupling of applications and components described elsewhere in this document permits simpler tests to be created as fewer interdependencies need to be handled.
Some advocates of Test Drive Programming even view the tests as a means of providing some additional documentation as to what a system’s business rules are. However, I personally feel that such information would not be widely used and tests should instead likely make reference to particular rules in some better and more widely accessible classification of business rules.
Test Driven Programming and Continuous Integration can provide pragmatic tools for maintaining a robust system and vendor tool support and company usage of these techniques is growing strongly.
2.6 Low Level Concepts
2.6.1 Flexible Deployment
The ability to deploy the system in a single tier (on a single PC) or in multiple tiers (individual user PCs plus one or more application, database, or web servers) should be easily configurable from an application’s file settings.
For example, if a remote sales office only has one user, then it might make best sense to run the module(s) he needs on a single PC. Other larger remote offices or facilities might need to run their modules across individual PCs and at least one server for better performance. The main office with hundreds and thousands of employees may want to make use of separate servers for individual tiers to get good performance for everyone. All of this should be configurable from an easily modified application settings file.
It would also be highly desirable that the application itself be deployable over the web with a single action on the part of the user.
2.6.2 Provide Instrumentation Of Process And Data Pathways
Using a single framework (or bus architecture) though-out a system provides the opportunity to provide instrumentation of individual processes and data pathways to identify who used what, when, how and where. Such activity can help in trouble shooting performance problems and in identifying what parts of a system are still being actively used.
2.6.3 Components - Further Considerations
Some additional considerations that should be made when constructing components are described below.
2.6.3.1 Components – Promotion Of Abstraction In Design
Promoting abstraction while designing components can make them more flexible. If components are not unduly specific to any current set of steps or datum, they can be more easily enhanced, as business needs change.
2.6.3.2 Components - Design For Behavior Not Data
Many developers new to Object Oriented Programming (OOP) tend to make the rookie mistake of creating too many simple classes that are largely only record set representations of all the tables in their database, a behavior that is encouraged by vendors of object-mapping-relation products, which will automate the creation of such objects. The biggest problem with having a data-centric view in performing OOP is that it tends to result in light weight objects that contain little intelligence. Which requires, that the smarts they should have contained, be coded elsewhere in the system where its not easily reused and often ends up getting reproduced everywhere those objects are used.
Rather then simple data objects, it is better to focus on building business components that might represent data from a collection of tables and all the behavior that needs to be wrapped around them to perform some business task to promote code reuse. For example, an Order business component in its most simple form should be capable of holding customer summary data, such as their name, phone number and address, and a separate data line for each product they ordered, along with methods (behavior) to check that the customer has a good credit rating and calculate the time it will take to deliver their goods. If all this data and behavior is wrapped up in a single business component, then it is easier for Windows and Web programmers to build ordering tools that reference that single item to allow people place orders from both sources. And while the company needs to incur the cost of building two different types of interfaces they save money by only building one business component that does a good deal of the work.
2.6.3.3 Components - Normalize Behavior
Just as relational databases should focus on storing data in a normalized form (without repeating data groups) the practice of object oriented programming should focus on the normalization of behavior.
Taking our example of the Order component described in the section “Components - Design For Behavior Not Data.” it’s not necessary that the component actually know shipping times for products to a customer if the shipping system can already calculate that information. It would be better that such behavior only be implemented in one place, most likely the shipping system, and that the Order component would request a shipping schedule after passing the shipping system a list of products and their destination. The goal being that actual behavior gets coded in only one place, and therefore, there is only place it ever needs to be updated when changes are needed. And while the actual calculation of a shipping date\schedule would be done elsewhere, users of the Order component do not need to be aware of that. They would just ask the Order component directly for the estimated ship date, whether the component calculates that information itself, or passes the request along elsewhere, should not be of concern to anyone using the component.
2.6.4 Simplify Format Of Data Storage And Manipulation
It’s desirable that data be passed around the system in a fairly rudimentary format that is largely independent of any vendor specific data management technology that is likely to change in the short-to-medium term. This helps reduce an application’s dependence on any software vendor’s more specialized data management technology, which they seem to implement a new model for every two to three years.
For example, large amounts of table like data in .NET components can be kept in simple structures such as Binding Lists, which are largely, just simple table organizations of data into rows and columns that also support data binding which provides for bidirectional data change. Rather then the much more sophisticated data typed xml datasets which are much more likely to be subjected to changes in each new release of .NET and eventual replacement within two to three years when the vendor creates a new data management model. The simpler data structures may change too, but generally those changes will be small and manageable, if they affect the business components at all. Even better, if the behavior around those simple structures is coded in a common ancestor for business components changes only need to be made in one place for all components, which is not the case for the more complex data structures, as the get implemented as independent entities with no common ancestor in the enterprise’s custom code base. Another advantage of the simpler structure is that it’s easier and faster to pass data from one component to another for any additional processing, as the data’s storage container carries much less baggage with it.
3 Conclusion
Building enterprise systems is an inherently complex business. Getting all the requirements and design right at the beginning of a large system never happens. If what was presented in this document didn’t persuade you of that, consider the fact, that no one has the rights to a working crystal ball to predict the future, and stuff happens. So by necessity, system development has to be done while trying to hit a moving target. Change is the only constant in our lives, small changes happen all the time, more fundamental changes occur as time progresses.
In accepting the above facts, one can be proactive about planning to handle change when building large systems. The development effort will be much easier and more successful if one selects methodologies, tools, and techniques to craft an adaptable development process that is responsive to changes. Be such change, a result of new knowledge that is uncovered in regular business reviews, or new business practices that occur as the system’s development is underway. Having an adaptive development process, allows the development personnel to make necessary changes while development is underway, while it’s still cheaper to do so, and to produce a system that is better geared to what the business’s needs are at the time of delivery.
Some of the most important guidelines discussed in this document for providing an adaptable were: 1) Following architectural patterns such as modularization and layering. 2) Focusing on development of business components rather than simpler data-centric objects. 3) Programming to interfaces rather then physical implementations of classes, and normalizing behavior across classes. 4) Using the Agile methods of Interactive Development, Test Driven Programming, and Continuous Integration. And 5) Performing abstract thinking while designing.
While this document covered a number of areas of knowledge that any enterprise software architect\developer should be familiar with; it’s not intended to be a comprehensive list. Overall, software architecture is a pretty comprehensive and complex subject area offering many challenges.
In larger companies, the software architectural load can be split amongst architects by allowing them develop specialties from which they can contribute unique knowledge to the team effort, while only being required to have a general knowledge about other areas of the field of software architecture.
In smaller companies, the software architect generally needs to cover even more ground, as the company isn’t large enough to have someone dedicated to work on that activity full time. In such cases, the architect also needs to manage the development staff, do project management, and actually help write the software. This can result in them having less then an ideal level of knowledge in particular areas, so to compensate, certain responsibilities need to be pushed further down the line to other development personnel who are willing to accept those responsibilities and learn new skills to perform.
4 References
The following books have provided me with much of the schooling which helped me write this document. They are definitely books that should be in any software architect's library.
Analysis Patterns – Reusable Object Models by Martin Fowler, ISBN 0201895420. Provides a catalog of “business domain” patterns.
Applying UML and Patterns—An Introduction to Object-Oriented Analysis and Design by Larman, ISBN 013748807. Provides a foundational discussion for building robust, scalable, maintainable systems, using object technology.
Building Business Intelligence Applications with .NET by Robert Ericsson, ISBN 1584502711. A practical guide to using .NET tools to build applications for OLAP and data mining.
Code Generation in Microsoft .NET by Kathleen Dollard, ISBN 1590591372. Teaches code generation as a scriptable and repeatable process using templates, so one is not tied to a particular framework or style. Kathleen is one of the industry leaders and a major advocate of code generation. Her book also has a couple of chapters on code generation for the CSLA framework (refer to the “Expert C# Business Objects” book below).
Component Software – Beyond Object-Oriented Programming by Clemens Szyperski, ISBN 020117885. It’s about developing re-usable off-the-shelf components that handle a significant amount of functionality. It emphasizes that one should not focus on trying to build each new piece of functionality from collections of small independent objects while developing larger systems. One should instead, focus on developing more robust components that wrap up smaller objects into bigger containers that are capable of performing bigger units of work, which in turn, require less effort by other developers to reuse them to build even bigger functional units.
Design Patterns – Elements of Reusable Object-Oriented Software by Erich Gamma, Richard helm, Ralph Johnson, and John Vlissides, ISBN 0210633612. This book is the original classic on software design patterns, and still the best reference on the subject.
Domain Driven Design by Eric Evans ISBN 0321125215. This book makes a persuasive argument for the use of domain models to provide a ubiquitous language that ties the domain experts and technologists together. Also, it emphasizes that really powerful domain models only evolve overtime and that some of the best ideas appear after the initial release of a system.
Enterprise Integration—An Architecture of Enterprise Application Systems Integration by Fred A Cummins, ISBN 0471400106. Excellent book that reviews the technology landscape defining enterprise integration objectives, and which provides generalized enterprise integration architecture.
Enterprise Integration Patterns—Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf, ISBN 0321200683. Describes using asynchronous message as a proven strategy for enterprise integration, and offers 65 patterns for building such systems.
Expert C# Business Objects by Rockford Lhotka, ISBN 1590593448. A book about application architecture, design, development, and using object-oriented concepts in .NET. Book describes the building of a “Component-Based, Scaleable, Logical Architecture” (CSLA) in C# .NET, which has been used as the basis of system development efforts in many companies with good success. Also, the company Rocky Lhotka works for relies on his framework and code generation to produce upwards of 80% of the code in new systems they develop for clients.
Object Technology – A Manager’s Guide, 2nd Edition by David A. Taylor, PH.D., ISBN 0210309947. This book is an easily read classic on the subject. One of its central themes is that objects are not about the data they contain, but the abstracted behaviors they represent.
Patterns of Enterprise Application Architecture by Martin Fowler, ISBN 0321127420. A book written by a well known industry expert on the topic of suitable patterns for enterprise architecture. The book is written in two parts, the first part is a short tutorial on developing enterprise applications. The second section and bulk of the book is a detailed reference to patterns that assist in building enterprise applications.
Pattern Oriented Software Architecture—A System of Patterns by Frank Buschmann, Regine Meunier, Hans Sommerland, and Michael Stal. An excellent book concentrating on Architectural Software Patterns such as: Layers, Pipes and Filters, Distributed System Broker, Model-View-Controller, and Microkernel.
Strategic Data-Planning Methodologies by James Martin, ISBN 0138511136. The first good book I ever read on the topic of strategic data planning, and still one of favorites. Additional information about managing enterprise system development can also be found in the list of references provided at the end of this document.
5 Additional Notes
Loose coupling can be achieved by linking the parts of an application through the transfer of information relating to business transactions in an asynchronous (“direct real time”) mode of communication or in a more flexible manner by using an intermediary system that specializes in the transfer of information, which receives information on transactions from subsystem\module of an application, which are placed in a storage queue after acknowledging receipt, those messages can then be either broadcasted or polled by the subsystems of other applications as needed.
It’s not necessary that a layer be an actual physical separation within the application as long as the functionality that defines that layer is well scoped and does not vary. For example, the basic CRUD (Create, Read, update, and Delete) operations of business components could be grouped together within a component and named the same across all business components. When it comes time to update those routines for a new data access technology they can then all be easily located and dealt with together. Current features in .NET 2.0 languages such as code folding regions and partial classes provide some nice mechanisms to facilitate such activity. While these techniques may suffice for smaller individual companies who will only use one database management product, large companies or commercial software vendors may go further and create a separate Data Access Layer (DAL) that abstracts the access and manipulation of many database management systems to produce a single API for their development staff, who through that API can request or manipulate data in a variety of database products such as Oracle, Microsoft SQL Server, or IBM’s DB2, without being aware of local variations in the SQL language of those products.
Even being the smaller elements of a system, objects should in turn package individual behaviors with minimal interdependencies on other objects, so that larger components can more easily make use of them.
An application’s tiers can be set to run on one or more similar computers in a matrix (farm) like setup when necessary to provide scalability to thousands of users.
