Enterprise Software Architect\Developer

Selling More
By Having An Adaptive Enterprise System
An XYZ Company Story

This blog post is the beginning of a series of 9 posts representing a package I assembled for a recent client. There is a separate post on this blog for each section (document) in the original package. The material was developed over a year and half while working for that client in response to various questions about how to build a custom adaptive enterprise system to replace their existing one that had developed performance issues due to architectural shortcomings. Because the original questions came from various people on similar subject areas, there is overlap in the posted material. The material is a good overview of things that an organization might consider in deciding on how to acquire an adaptive enterprise system that is capable of responding to changing buiness needs. Or the knowledge that a software architect should know in building such a system. If you find any of the topics discussed interesting and would like more information on the subject, you might try one of the books I recommend on that material.

Sections

Introduction

1. Origins Of This Package

Part One: Enterprise Development Guidelines for XYZ Company

2. Building A Custom Enterprise System For XYZ Company So It Can Sell More (Intro Memo)
3. Selling More By Having An Adaptive Enterprise System - An XYZ Company Story
4. Diagram of ‘Suggested Enterprise Architecture For XYZ Company’ (A Simplified View)

Part Two: Generic Enterprise Development Guidelines by ABC System’s Inc.

5. Diagram of ‘ABC Systems Inc. Development Process’ (IT Activities And Class Levels)
6. Developing Enterprise Systems – Some General Guidelines
7. Development Steps – Some General Guidelines
8. Preferred Development Tools

Appendix

9. Recommended Book Lists for Enterprise Software Developers\Architects

(Selling More, By Having An Adaptive System, An XYZ Company Story)

Section 1: Origins Of This Package

An earlier version of this package was originally prepared more than a year ago as a compilation of articles I’d written over a period of a year and half for the owners and managers of a local company where I’d taken a consulting assignment. All the articles sought to provide guidelines and answers to questions I was asked about how to build the company a new more scaleable custom enterprise system to replace their existing one that was having performance issues.

I'm now making them available to a wider audience. With that in mind, I reworked all the package’s documents to give them a broader appeal. At this point, I feel they are likely to be of interest to anyone interested in building a scalable enterprise system. One word of warning, as the documents were written at different times covering largely the same general topic there is quite a bit of overlap in them.

I have removed all references to the original company and its enterprise system. The only company name you will find though-out this package is the generic name of “XYZ Company,” and the only reference to a custom enterprise system is the generic name “123 Enterprise”. Documents in section one of this package use those references, and the story lines presented there have been reworked to be more representative of organizations in general.

(Selling More, By Having An Adaptive System, An XYZ Company Story)

Section 2: Building A Custom Enterprise System For XYZ Company So It Can Sell More (Introductory Memo)

(Selling More, By Having An Adaptive Enterprise System, An XYZ Company Story)

Section 4: Selling More, By Having An Adaptive Enterprise System, An XYZ Company Story

Section Contents

1 Overview
2 What’s An Adaptive Enterprise System?
3 Characteristics Of An Adaptive System
4 Why Isn’t XYZ Company’s Monolithic System Adaptive?
5 XYZ Adaptive System Acquisition Choices
5.1 First Steps, Regardless Of Acquisition Choice
5.2 Buy A Canned System
5.3 Refactor The Existing 123 Enterprise System
4. Build A New custom Enterprise System
6 Restructuring IT Activities For In-house Development
7 Building A New Adaptive Enterprise System
7.1 Tool Selection
7.2 Code Standards And Smart Business Components
7.3 Architectural Guidance
7.4 Code Generation
8 Application Of Code Generation To XYZ Company
8.1 Using A Code Generator For The Database Stored Procedure Layer
8.2 Using A Code Generator For Business Components
9 Six month test case—proof of concept
10 In Summary

1 Overview

This document describes the advantages of having an adaptive system, and what characteristics makeup such systems. It also describes what considerations XYZ Company might take in going from an existing in-house custom enterprise system that has poor adaptive characteristics and signs of premature aging, to a state, where its business is running on a highly adaptive system that will permit it to sell much more, and grow at a greater rate in the future. It ends, by exploring how such an objective might be achieved through the recommended approach of building a new adaptive system from scratch.

2 What’s An Adaptive Enterprise System?

An adaptive enterprise system is one that was built with a guiding architectural principal that it be easy to modify to accommodate changing business needs, and which was built with a forward looking view as to what changes might be desirable to provide plumbing for, from the very beginning.

What Good Is It Having An Adaptive System?

Having an adaptive system permits a company to grow much faster, and sell more product, in more channels, to more customers, in more countries, while operating with greater efficiency.

On the other hand, when an enterprise system is not designed to be adaptive it’s much harder to change to accommodate the organization’s growth, or help it take advantage of new business opportunities.

XYX Company’s EDI Example Of Its Non-Adaptive System

When an effort was made to re-use small parts of XYZ Company’s main system 123 Enterprise in a new custom EDI system it was found to be too costly and all the EDI code eventually had to be written from scratch. Why? Because it turned out that the individual pieces of the original system were never designed for reuse outside of that system. The effort that was started to try and reuse them for EDI non-visual services had to be abandoned after modifications had been made to dozens of the system’s units, as many more changes remained, and the cost of trying to reuse the code was finally estimated as greater than the cost of building from scratch. Changes to one unit only lead to changes that needed to be made in other units, which in turn, lead to yet other unit changes that needed to be made. At one point, so much of the system had been checked out for changes, it was impossible for other developers to work on any remaining parts of the original system unencumbered.

Besides the units having hard references to each other in almost a never ending chain, they also had no clear separation of responsibilities. Lower level data-centric business objects were passed visual form controls from the presentation layer, to be populated with data that required them to incorporate support for the visual surface of the application, which made them not viable for use in the EDI non-visual services in their original form. To use them in non-visual EDI would have required pulling out all such behavior into entirely new units, before the original code to be reused, a fairly costly exercise

Basically, the issue was that none of the 123 Enterprise’s responsibilities were packaged into units with autonomy from each other so that they can be easily recombined in new configurations to solve other business problems.

3 Characteristics Of An Adaptive System

Adaptable systems sound wonderful! But what makes all the magic happen? What are the characteristics of systems that are adaptive?

The following list provides a basic selection of some suitable key characteristics for adaptable systems:

o Be built using a blueprint, framework, and standards (the basics)
o Constructed with proven software patterns, some keys ones being:
- Abstraction
- Layering
- Modularization
- Loose Coupling
- Smart Business Components and\or Services
o Actively adopt and adapt other useful software business patterns
o Use light documentation that’s easier to develop, re-use, and evolve, such as:
- Business domain descriptions and functional area data models
- Bulleted functional points and rule classifications, (maintained in online help)
o Use a flexible and self-checking iterative development methodology, that involves:
- Coordination of iterative development with regular reviews
- Automatic and ongoing validation of the system via Test Driven Programming
o Make use of successful development tools and techniques to:
- Focus on smart business components versus dumb data-centric objects
- Achieve configurable systems through use of meta-data.
- Make good use of code generation and even re-generation
o Be forward looking in regards to:
- The system’s lifetime scalability and associated performance tradeoffs
- Providing scalability via configuration changes versus programming
- Selecting technology with a good future and long life
- System life and what changes might be required for a growing business
o Reduce data specific access\storage dependencies
o Provide for instrumentation of process and data pathways.
o Offer flexible deployment

An adaptable system is so, because it uses flexible blue prints to build a collection of core services and components using good architectural and development techniques that have been proven successful in other systems found to be easily modifiable to accommodate changing business needs. A number of those techniques can be found in the above list, but I only describe some of those items briefly in this document, just enough to give you an idea of what adaptable systems are like and what they can offer, additional details on all of those items and more, can be found in my document Developing Enterprise Systems - Some Guidelines.

Five fundamental architectural software development patterns that help make a system adaptable that I discuss briefly below are: A) Abstraction, B) Layering, C) Modularization, D) Loose Coupling, and E) Smart Business Components. Most of these development patterns have been around for decades, (with the possible exception of Smart Business Components), and have been used successfully in many large flexible systems. They have primarily been used to organize systems into more manageable pieces that can be easily constructed and evolved by separate developers or development teams. After discussing each of these features briefly, I round out this section of the document with a discussion of why system monolithicness (no separation of responsibilities) constrains ones ability to keep a system adaptable.

A) Abstraction

The principal of abstraction in system design involves an effort to step back and look at what you are trying to solve, to see if there is a more generic pattern that can be more widely applied to a broader problem space with the assistance of parameterization to handle similar tasks with the same code base. Abstracted concepts have a much better chance of being re-used then non-generic solutions, and for a little up-front investment in considering such matters, a better long-term return on your investment can be achieved, and ultimately faster delivery of new functionality.

An example of abstraction in an enterprise system might be the handling of business components where instead of forcing the system to recognize each component as a unique entity, it is instead encouraged it to recognize that each one has a set of common behaviors e.g. they can fetch data from a database, allow form data controls to bind to their properties for bidirectional data exchange, and finally, save any changes back to the database. Now instead of all parts of the system having to be intimately aware of different business components such as Orders, Shipments, Invoices, etc., they only need to be aware that all such components have a set of common behaviors that can be used in exactly the same manner. More importantly, that behavior can be put in a separate base class from which the custom business components can inherit that common behavior through inheritance, (without any new code having to be written by any of them).

XYZ Company’s Aversion To Abstraction

Aversion may be much too strong a term here, but nevertheless, there is no consistent focus on thinking abstractedly about the functionality in XYZ Company’s 123 Enterprise system. As in the past, current development of new data objects is started from scratch, and each one is developed for a specific purpose and handled uniquely. This eliminates any form of predictability when working with them, and requires programmers to review their intent and internal working before making any changes. Ultimately, there is no reuse of code and no efficiencies developed over time.

B) Layering

The layering architectural software pattern provides the ability to isolate roles and responsibilities to individual strata in an application, where lower levels are largely ignorant of higher levels. Without getting into too much technical detail, I will try to outline what benefits this offers. Each upper layer of an application can be built, so that it manipulates the lower application layers via well-defined communication protocols (interfaces). This prevents the lower layers from having to know anything about the upper layers of the application. In fact, the lower layers only perform behaviors requested of them. This simplified means of communication and coordination of work reduces dependencies between the layers making it easier to replace a single layer or re-work it extensively without worrying too much about its affect on other layers.

For example, in a well defined system, the lower business component layer would be ignorant of how its data is presented to users in the upper presentation layer, (in separate fields, or as a grid, or tree view, etc.), and because of that, its components can be safely revamped without worrying about any affect that might have on the presentation layer, as long as they continue to service the same basic interfaces they previously did while being called by that upper layer.

Another advantage of using the layering pattern is that developers can be assigned to work on whatever layer they are most suited to: User Interface, Reporting, Business Domain Component Development, or Framework Building.

Sidebar: Currently, a number of the lower layers required in any enterprise system can be had free from multiple sources for .NET. Microsoft publishes a collection of free Enterprise Libraries that implement various common software patterns to provide many services required of all applications. Three other sources I like can be found in the following books: I) Expert C# 2005 Business Object by Rockford Lhotka appeals to me because of his use of smart business objects. II) Developing Application Frameworks in .NET by Xin Chen appeals to me because of his focus on supplying common services such as Class factories, Caching, Configuration, Event Notification, Windows, Message Queues, Authorization, Authentication, Cryptography, and Transactions. Services required in nearly all enterprise applications. And III) .NET Enterprise Development in C# by Matthew Reynolds and Karli Watson appeals to me because of their focus on deploying applications across the internet and building custom application menus based on peoples access rights to particular pieces of the application.

XYZ Company And The Problem Of Non-Layered Data Access

Because the current use of the long deprecated Borland Database Engine (BDE) in XYZ Company’s 123 Enterprise system isn’t isolated in its own later, it can’s be easily replaced. Had all data access via the BDE been isolated in its own layer, the technological shift that made the BDE data access technology obsolete could have been corrected by replacing just that layer, without unduly affecting the rest of the application.

C) Modularization

The Modularization of a system generally involves dividing an application into business functional areas (domains) where collections of components have high cohesion (similar responsibilities) with each other, but are only setup to communicate with other functional areas through well specified interfaces that can be defined separately. Because communication between modules is through separately specified interfaces the general development of individual functional areas can be done without any undue disturbance of other functional areas, as long as all areas continue to respect the communication interfaces they original used or supported.

Physically, all the code units relating to a module or business functional area are typically stored together in a single folder specific to that module, making it easier for someone to identify and work with them. Modularization also permits individual developers to be assigned to work on business domains in which they have the greatest expertise.

The biggest benefit of using modularization is that systems can be grown organically one module at a time. Using this architectural pattern also permits parallel system building efforts by letting separate business domain development teams build multiple modules simultaneously. And is the main means by which an adaptable system acquires plug-and-play functionality, allowing the integration of best of breed solutions when necessary to supply better functionality

XYZ Company – Modularization

In regards to XYZ Company a manufacturing organization, a basic modular division of the enterprise system might be at a minimum the five modules mentioned by the VICS Architectural Guide: I) Product Development\ Merchandising, II) Sales\Ordering, III) Logistics, IV) Finance, and V) Business Support. However, XYZ Company’s 123 Enterprise system has no modules, which makes it difficult for separate developers to work on it together without interfering with each others work.

D) Loose Coupling

Loose coupling refers to the concept of reducing the direct inter-dependencies between the pieces of a system and between systems themselves. Its goal is to permit the construction of such items where each knows nothing about the internals of any other. Thus, there are no interdependencies on how they each perform their own responsibilities. Without such interdependencies, each piece can be developed and enhanced in largely an autonomous manner relative to other pieces. Loose coupling is primarily achieved through the techniques of Abstraction, Layering, and Modularization, as discussed in the preceding sections.
Without loose coupling it’s difficult for developers to work on a system without interfering with overlapping work that other developers are trying to do in other functional areas.

Sidebar: Loose coupling between enterprise applications is now more commonly being plumbed along a message queuing model rather than any direct application-to-application method, to further reduce interdependencies between them. A separate specialized system is used to provide a facility by which messages can be exchanged, that allows all other systems who need to exchange data to only have to know about one single interface to be able to communicate with a number of different systems, simplifying the delivery of such functionality. Better fault tolerance can also be provided by redundantly operating the messaging system, because if one version goes down, the other one can step in immediately and take over.

XYZ Company’s Loose Coupling Problems

Just about everything in XYZ Company’s 123 Enterprise system is tightly coupled, and any effort to create controlled zones of data exchange around individual modules would require substantial work. At its worse, nearly every major process in 123 Enterprise reads and\or updates a single table (Order Items) that holds information on product originally ordered by customers. Major processes such as Sales Ordering, Credit Reviews\Order Releasing, Inventory Allocation\Reservation, Shipping, Freight Logging, Invoicing, Three Year Customer Sales History loaded each time a Customer is viewed, and the YTD Sales Performance Dashboard that is updated every five minutes, all access the same data area causing serious competition for it. That competition has often resulted in concurrency issues such as blocked processes and system lockouts. And over the years, as the system has grown, quite a number of those issues became so bad, they had to be corrected with additional development effort. Had the individual modules of the system been built with looser coupling there would have been much less contention and concurrency problems.

E) Smart Business Objects

What many frameworks and enterprise applications currently fail to do is emphasize the implementation of business components with a good deal of their own smarts. In fact, most commercial enterprise system frameworks deliver smaller light weight data-centric business objects that aren’t much more than direct object-relation-mapping exercises to data stored in the backend database tables. With such light weight business objects, business logic often needs to be repeated though-out the system to guarantee that all business rules are obeyed at all time. Often the business logic is repeated in stored procedures, graphical user interfaces, helper classes, and even separate rule engines. All of this, results in multiple places that business logic needs to be maintained anytime changes are required. A better solution would be to package more of the business logic in more functional complete business components though which all business activity could be processed, and which could be used in multiple platforms.

Use of such smart business components are one of the primary goals of the CSLA (Component Scaleable Logical Architecture) framework built by Rockford Lhotka and distributed with his book Expert C# 2005 Business Objects. His smart business components can do the typical things of other framework business objects e.g. retrieve and cache their data, and subsequently update the database with any changes when requested. However, they also typical contain a lot more intelligence to perform such additional work as:

· Validate business rules for data entered and report back to the user interface the current state of those rules.

· Provide their data and business logic directly to both Windows and Web forms through the data binding of individual properties with a bi-directional exchange of updated data.

· Transfer themselves across networks to servers to actual fetch data, back to client PC to provide rich feedback to users as they edit the data, and back to server again when updates need to be saved.

· Determine a user’s rights for the business object as a whole, or individual properties and then
synchronize such information directly with the display controls on the user interface.

I hope it’s apparent from this list, that with smart business objects a lot less coding needs to be done by the developer working in any presentation layer, regardless of whether its windows, web, or web services.

XYZ Company’s Lost Opportunities Due To Lack Of Smart Business Components

Because XYZ Company’s 123 Enterprise system was developed with only a limited number of light weight data objects, nearly all of its business logic ended up being programmed directly into individual Window forms, therefore, there is no opportunity to reuse any of that code to produce a web form or web service front-end to its business. In fact, it would need to duplicate all existing business logic in the new web forms, or on the web server, or start the construction of new business components containing that logic from scratch. The company would also have to go through a similar process if it bought a new ERP system but decided it wanted to implement a number of core functions itself. Yet worse again, because the business logic is coded in specific Window forms its not even reusable in other configurations within 123 Enterprise.

4 Why Isn’t XYZ Company’s Monolithic System Adaptive?

People’s initial attraction to monolithic systems stems from the perception that “having everything in one place means there is only one place to look for things and that should make it easier to work with it.” That’s true and a huge advantage to any application while it remains small, but as it grows, that advantage quickly erodes, as it has for XYZ Company’s 123 Enterprise.

If you stop and consider any large volume of information in your life, you will likely find you have organized it by categories to make it easier to work with. For example, you probably have assembled documents by subject matter in hanging folders in your file cabinets. And you have organized names in your address book alphabetically, all to make that information easier to access and work with. Thus, it should not be any great leap of logic to consider that organizing a large system into smaller sections is a beneficial thing to do. Large systems contain thousands of classes and millions of lines of code, and it should go without saying, that when organized into cohesive collections make it much easier to find the particular ones you need when working on a given functional area.

Another analogy that might be helpful in illustrating this point is to consider if everyone at XYZ Company had their seating randomly rearranged throughout the building, now, how easy to you think it might be for each department to perform its duties efficiently. This is the type of confusion that any large system can dissolve into overtime, especially, if it’s built without guidelines as to its growth and organization. Eventually such systems grow into what is referred to in the industry as spaghetti code, not a good thing, as it makes it very difficult to just pullout one piece and work on it, and put it back again. This is the path that XYZ Company’s 123 Enterprise system is currently on.

Sidebar: While many modern applications may give the appearance of being large monolithic systems to users, their not. Instead, they appear to be a single application by stitching separate relatively autonomous subsystems (modules) together by building a menu on the fly based on the access rights of the person logging on, giving that person access to just the pieces they are allowed see.

Now that you know why having an adaptive system is good for the company and what the key architectural attributes of such systems are, the next section moves onto consider how XYZ Company could arrive at a place where it had such a system.

5 XYZ Adaptive System Acquisition Choices

The three primary choices to acquire an adaptive enterprise system for XYZ Company are:

0 Buy A Canned System
o Refactor 123 Enterprise
o Write A New System

I discuss these acquisition choices below, but before I do that, I first examine some steps that are common to each choice.

5.1 First Steps, Regardless Of Acquisition Choice

Regardless of what acquisition choice you make there are some initial steps that need to be performed.

A Role For Everyone – Module\Activity Identification

Regardless of what acquisition choice is made, a core first step in any of the processes will be deciding on what desired functional areas of the business need automation and what specific individual activities need to be handled in each of those areas, and how they should be handled. This should be an exercise not so much in examining how things have been done in the past at XYZ Company, but how they could be done better and what additional things might be needed in the future.

For example, should orders be restricted to single stores as they are now, or instead allow a store to be specified for each line item (repeating products as needed), or even contain a separate relationship to allow the specification of allocations of each product quantity to multiple stores. Either of the latter two methods would allow the representation of Purchase Orders by large EDI customers as a single order within the system, making them easier to handle then 123 Enterprises current restriction that forces such orders to be broken down into separate individual orders by store. However, if one goes with the final more abstract method for handling orders one can cover all order types without having to go back and make any changes to the system.

Before making such detailed considerations as outlined in our example above, much broader considerations need to be taken into account such as what business functional areas\modules might be suitable for XYZ company.

For example, are the functional areas outlined by the VICS business model and which XYZ Company is currently structured along (listed below) suitable for the company’s future growth plans. And are the major activities that are performed in each of those areas (as shown below) enough to accommodate future needs. Do certain areas need to be operated better? Are there any missing pieces?

o Product Development\Merchandizing
- Product Design
- Product Demand\Supply Forecasting
- Vendor Contract Negotiation and Purchases
- Vendor Purchases And Scheduling
o Ordering
- Sales
- Call Center\Customer Relationship Management Support
- Collections
o Logistics
- Order fulfillment
- Warehouse Management
- Shipping
o Finance
- Credit Review
- Accounts Payable
- Invoicing\Accounts Receivable
- Collections
- Cost Control
- General Ledger
o Business Support
- Customer Relationship Setup
- Organizational Structure Setup
- Business Rules Setup
- Reporting
- Third Party Product Integration

This process of identifying required activities offers a role that many existing XYZ Company people can play and by which they can make a valuable contribution.

The results of the identification process can help in deciding between a canned package, refactoring, or a new custom development effort.

Once a final selection of functional areas is identified and the specific activities to be performed in each of them are listed, some broad stroke process modeling comparisons of those areas can be made across the three acquisition choices to see which offers the most suitable path to the future.

5.2 Buy A Canned System

For canned packages, the natural transition from the identification of the company’s future business needs might be to get demo installations of those packages and move onto evaluating how they actually perform the identified activities.

Using the word ‘canned’ to describe third-party commercial systems is a bit of a misnomer, as they are not exactly ready to use as soon as you open them. Purchasing a commercial package gets you access to a slew of features many of which you might not need, and those that you do, will require them to be configured to work best for you company. Actually, most commercial systems have more switches then an old fashioned jumbo jet that need to be set correctly for them to work well for you particular business. In fact, there are consulting firms making a business from selling information filtered from their databases on the best practices that should be chosen from the ERP vendor’s package that might work best with any particular business.

Sidebar: The medium sized Axapta ERP package from Microsoft that was demoed at the XYZ Company had almost 10 times as many tables as the company’s own 123 Enterprise system, and as many more functional areas, all of which require configuration before use. Oracle’s popular and larger ERP called SAP is many times more complex then that, it has upwards of 40,000 tables, and over 1,000 best practices to choose from.

Just about any third party commercial system will address more functional areas then XYZ Company’s 123 Enterprise currently does. And, while they will also offer many options for configuring each of those areas, none of the packages are likely to address any one area in exactly the same manner as XYZ Company’s own 123 Enterprise, with or without configuration. It’s also probably worth noting that the more streamlined a company runs its business, the fewer features it will need to use in any particular commercial package.

I believe the extra baggage commercial packages bring along with them is not aligned well with XYZ Company’s philosophy of running an exceptionally streamlined operation. In fact, the XYZ Company will be even more challenged then other companies using any such package in its attempt to try and identify what parts of the application it needs to use, and what it can ignore, so as not to incur any unnecessary overhead in its effort to continue to run such a streamlined operation.

While freely admitting commercial packages can be adaptive, I still believe you will regularly bump up against the boundaries of what they can do, and therefore, another of XYZ Company’s philosophies of saying yes to what the customer wants, instead of telling them “no our system can’t do that,” will result in it incurring significant custom programming costs to modify the commercial system. Work that will need to be performed on a bigger system then it needed anyways, and kept in sync with subsequent releases of that package.

It’s possible much of the above two concerns could be mitigated by purchasing a package that is highly configurable, and which permits a company to run streamlined operations. Such aspects should likely be key selection criteria for XYZ Company considering a package. Especially, as when you buy an ERP package you are buying into the vision of that company. How they grow their package will determine how you can grow your business. You are in fact outsourcing your entire development effort to that company, and trusting your future to them.

Besides the original purchase cost of an ERP, the learning and consulting costs associated with one aren’t insignificant. In fact, from various other people at the XYZ Company, I’ve heard estimates for purchasing, configuring, and implementing an ERP package of between $0.75 and $1.25 million, (average $1 million). In addition, annual maintenance costs paid directly to the vendor would normally run between 18% and 25% of the original purchase cost. If those figures are right, it sets the backdrop against which to consider the costs of writing a new system, or refactoring 123 Enterprise. Obviously, one to two million dollars is a lot less then the estimated $4.5 million dollars for building your own custom system that I’ve gave, so the big question becomes whether a custom system can offer enough competitive advantage to justify its additional cost. I believe it can, as going forward you will have many more opportunities that you’ve had in the past, to do things your competitors can’t, and profit all the more from those activities. However, things are likely to be even better then I painted above, as I actually believe the figures offered by others for implementing a commercial ERP system are grossly on the low side. Nor do I actually believe that the XYZ Company would approach the average cost for implementing an ERP quoted by the META group of $15 million. But the true cost of doing so, probably lies somewhere closer to $4.5 million then $1 million, so the actual cost difference between buying and building to acquire a competitive is not that great.

As a final word, its worth mentioning that in selecting a commercial package there’s a risk that an incorrect choice might be made, resulting in significant interruptions to the company’s business activities. I wasn’t around when XYZ Company paid $250,000 for the warehouse system prior to my arrival that was never implemented, so I can not speak to any of the details about that transaction, but I would guess that XYZ Company is not only aware of the possibility of making an incorrect choice, but is very wary about doing so again.

5.3 Refactor The Existing 123 Enterprise System

The results of the identification process could be used to estimate what it would take to refactor the current system, and then compare that cost against the other acquisition options.

Cost estimates would need to cover the re-organization of the existing system’s code units into subdirectories representing individual functional areas (modules), and an effort to make those modules loosely coupled. The estimates would also need to cover the re-working of data access code and other services into separate layers with a priority on getting rid of the Borland Database Engine. Followed by, the creation of smart business components and the movement of business logic code to them, to perform the various activities required by the business, and possibly making those business components available to both Web Forms and Web Services to be used in other applications. An estimate should also be provided for building or implementing a framework to facilitate future growth and services not currently found within 123 Enterprise, such as, encryption, caching, event notification, and messaging.

Having seen and worked with much of XYZ Company’s 123 Enterprise system, it’s my opinion that refactoring it to accommodate all the above, would actually require more effort then rewriting those modules from scratch after acquiring a good framework that permitted the use of a code generation to produce much of the code of a new system.

5.4 Build A New custom Enterprise System

There’s not much to be said about this acquisition decision here, as the rest of this package deals with that. However, I will say that this package would not exist at all, if I did not believe that building a custom system was a worthwhile activity for XYZ Company.

A new adaptive custom system will eventually be needed to permit the company to continue to pursue its current path for operating and growing its business, in order to maintain its current competitive advantage. I emphasize eventually, because at its current growth rate the company can get by with its existing system by fixing its bottlenecks as they are exposed by the company’s growth.

Should XYZ Company have an interest in accelerating its growth rate and build a system that will scale with it as it heads one billion dollars in sales and beyond, with operations across many countries, with much more product and customer variety, it needs to make building such a system, as much a core competency as any other activity it performs.

When it does come time to get a new adaptive system, and if XYZ Company chooses to build one again, it would be wise to focus more on the fact that building large systems is an inherently complex activity. Not paying close enough attention to this fact in the past has resulted in two systems that didn’t scale well with the company. But maybe that was the expectation, and those systems served their purpose and provided a good return on their investment, However, the next system that will be needed, will need to be much larger than either of those systems, and will likely need to be much more adaptable if the company wants it to have a longer life. Much more time will need to be spent up front in its design and more money will need to be spent on the people who will be designing it, than any of its predecessors.

6 Restructuring IT Activities For In-house Development

It’s not that restructuring IT development activities isn’t a popular idea with everyone in IT at XYZ Company. They all realize things could be done better. But no one else has yet put forth a plan to realign all IT activities. Why? Because even though a company’ personnel need to constantly respond to changes in the business environment, its easier to respond when something has happened, or is imminent and well defined, then, be visionary on how to setup the business so it can handle a host of possible situations in the future. While it’s easier to respond to change then be visionary, not all companies are even successful at responding to change, and many go out of business. However, those that are good at handling change, can often grow at the expense of their competition, as XYZ Company has. On the other hand, those that successfully execute a unique visionary plan can come to dominate an industry, as UPS and FedEx did in the United States parcel carrier industry, and as the low priced airline carrier Ryan Air did in European airline business. A common thread of all such companies is a focus on executing IT operations as a core competency. In order to come up with an overall development plan to meet such objectives, one has to be passionate about the work. But besides being passionate, one needs time to dedicate to the task, as there’s a wealth of information constantly evolving external to the company that needs to processed, analyzed, and synthesized to craft the best solution. As everyone at XYZ Company is currently 100% occupied with day-to-day operations, nobody currently has time to do the necessary planning and research between 8 AM and 6 PM, that leaves two choices; someone needs to either do the work on their own time, or wait until the business recognizes it as a priority and makes such an activity part of their work duties. In my case, my passion runs higher then most, so I did it my own time. However, I believe most others in XYZ Company’s IT department stand ready to do it at the direction of the company. I do feel that if one wants to have an adaptive system in place in a couple of years, the time has already past to start making changes, and there is some catch-up to be done.

Being proactive about change, being able to respond quicker, plan better, and even support a visionary direction is what ABC Systems, Inc. Development Process seeks to do. However, those in XYZ Company’s IT department who have read about the process have returned the documents without comment, so it’s not possible to say everyone in the department agrees with it. That’s unfortunate, as the most efficient and successful development in any IT shop will be achieved when everyone is working with the same vision. I don’t actually consider the ABC Systems, Inc. Development Process the final word, it’s also meant to be adaptable to the realities of the work place in which it would be implemented. Many of its activities might need to have a delayed implementation, or be implemented with increasing granularity overtime. It’s also a living process that tries to organically adapt by absorbing and leveraging industry best practices as they become public. I also realize that the development process I put forth here, does not describe the company’s individual business domains in-depth, which might have made it more receptive to management, but please realize such information is already held by many company personnel, and while I may have gathered much of that myself, I’ve not sought to record it here, as I believe the final form of such information for basing a new system design on, is best formulated by XYZ Company’s management\key users working in close collaboration with its IT department, in a forward looking manner.

Sidebar: It’s never productive in the long-run, for IT members to come-up with and implement their own individual designs, standards, etc. when building a large system that is meant to be easily adapted over a long-life. If each person did his own thing, that would be akin to trying to construct a large building by independently hiring framers, plumbers, and electricians, then telling them, you want a big building and letting them all do whatever they thought best. Oh, the chaos that would ensue! Big buildings are actually designed with requirements in mind, they have architectural plans drawn up for them based on engineering standards to describe how many elements of it should be built, and those plans are executed by a general contractor who hires the different trades people per a development schedule he works out, and then each type of trade person would perform their work to the best of their abilities within the overall framework of the architectural plans.

To summarize ABC Systems Inc. Development Process, it advocates that IT staff operate in an agile like manner directed by strong technical leads working in close partnership with the business users. That everyone follows a system blue print covering the system’s overall direction and development, so they have strong guidance and are working toward common goals. System development is done on a framework that provides common system services and a bus architecture within which smart business components can be executed. Code generation be used to write most of the simple code for the system. Development be performed in interactive development cycles with regular reviews and releases of working software. During development, the software should be compiled and tested automatically. Documentation should be light, maintained online in a central repository, with a good portion if not all of it being online in the system’s help, from where it can be made available to all developers and eventually all end-users. Such agile methods as these are becoming the norm across industries, and reflect a codification of things consultants have found down through the years that have let them successfully build adaptive systems in an efficient manner.

The key individual who takes the overall technical lead for a rebuild\rewrite at XYZ Company should be equally comfortable with hands on operations as well as abstract ideas and concepts. This person needs to be able to participate and lead on a daily basis. Be able to craft technology plans, stay current with technology changes, and have an in-depth knowledge of business issues. When problems arise, he needs to be able to deal with them in a pragmatic and measured manner. And he also needs to be able to clearly communicate his ideas, while being able to incorporate improvements suggested by others.

Actually, the more people who can address the overall development process, the better things would be for XYZ Company. At the very least, the delivered solution would greatly benefit from having at least two other people that could rise to the position of technical lead with a little training. This requires education for existing members of XYZ Company’s IT department, or the hiring of outside people by the company to acquire those skills.

7 Building A New Adaptive Enterprise System

In this section, I briefly describe some pragmatic actions to be taken to build a new adaptive system. I will also make a number of references to other documents in this package that provide additional information.

7.1 Tool Selection

The second step after defining business functionality in building a new system should be the selection of appropriate tools to build it as a loosely coupled modular system built with various strata (layers) of responsibilities. A key tool in achieving that would be the use of a framework to provide most of the system’s basic services, along with a platform for executing smart business components that make it easy to compose additional functionality. For details on actual candidate tools and a suitable framework, and the reasons why I selected them, see my document Preferred Development Tools.

7.2 Code Standards And Smart Business Components

The next steps in an enterprise system rewrite would involve setting code standards and laying out the basic structure of business components so the entire system has consistency, and much of the simple repetitive and tedious code for smart business components could be produced by a code generator. The candidate development tools I’ve selected support this. Once the simpler and more mundane code has been generated for the business components, developers can get to work on adding custom code to those components to reflect the business rules they must implement.

7.3 Architectural Guidance

Going forward the long-term successful development of a new custom enterprise system including its framework and core services are dependent on having a chief software architect at the outset. Each of the two previous custom enterprise efforts developed or XYZ Company both suffered from the lack of such a resource. This time around the development effort will be even more complex and thus the need is even greater for someone to fill this role. For an entire road map for running IT development efforts, see the diagram ABC Systems Inc. Development Process.

7.4 Code Generation

Code generation is a very powerful tool in any programmer’s toolbox for generating new functionality, yet it amazes me how many times it’s overlooked. I’m not sure why programmers either ignore or actively dismiss the ability to have their code write additional code for them, especially because it is probably the most powerful tool available to reduce the cost and delivery time of new functionality. Using a code generator in large new systems can produce up to 80% of the required code. And because the code is identically in nature, the overall learning and maintenance cost curves for the system are reduced.

Without the use of code generation to automate the production of tedious and detailed code, developers often feel the pressure to take short-cuts to deliver things more quickly in the short-term, that they know are detrimental in the long-run. Code generation by giving them a means by which to get the extra work done during the lifetime of the system would remove much of that pressure.

XYZ Company’s Disinterest In Code Generators (And Other Useful Tools)

At XYZ Company when I mentioned I had used a code generator to produce 5,000 lines of code for the EDI system, reactions ranged from non-interest to warnings against actually using such a tool. When I pointed out, that code generators are currently part of everyday development efforts in the form of screen and report designers, those reactions were somewhat muted, but still no one actively asked to see how they could make use of such a tool in their own work. There was also similar disinterest in the publicly available test harness I used to automate the testing of the complex logging service I produced at the collective request of the department.

However, on a better note, over my time with XYZ Company there has been broader acceptance of other ideas I advocate. Ideas such as, using a shared development database, coding standards, better source control, object programming\component building, and code reviews. Even if some of these ideas have still to be totally adopted, people are seeing more and more value in them as time progresses. Yet, disappointingly, it seems the adoption of these ideas is rather slow, which has delayed the benefit to be obtained from them in XYZ Company’s ongoing development effort. It would appear that the company stands to benefit by having someone empowered so he can make sure these ideas are followed through on.

8 Application Of Code Generation To XYZ Company

I consider code generation one of the most powerful tools available for developers, but as there appears to be disinterest in code generation at XYZ Company, I’m going to take some time to try to make a more persuasive argument for it. In fact, I will illustrate just how much it could have been applied at XYZ Company to its great benefit.

The specific applications of code generation that I will discuss are:

o Using code generation to produce the SQL code in the data stored procedure layer
o Using code generation to produce the bulk of code for business components
o Specific code generation examples using 123 Enterprise database

You’ll see from the example below that a code generator could have produced upwards of 200,000 lines of code for a system like 123 Enterprise, which represents about 40% of its code base, and it could have done so in only a matter of days. Admittedly, while it’s true that this is the mundane and more easily written code of any system its still code that needs to be written.

8.1 Using A Code Generator For The Database Stored Procedure Layer

I advocate having a data access layer that removes direct access to tables, because having such a layer gives you an additional opportunity to implement any key business rules before the data gets to the database tables, and also allows you hide the actual storage structure of data from an application, which allows you to make performance optimizations that might not otherwise appear intuitive to end users of those procedures. Probably one of the most common uses of code generators is to write stored procedures for such a data access layer, and more than 80% of that code could be created in such a manner.

What would have generating the data access layer meant for XYZ Company?

Taking a quick look at XYZ Company’s own 123 Enterprise database using code a generator to produce Select, (Select All,) Insert, Update, and Delete procedures for each table would have saved someone from having to write at least 50,000 lines of code. Producing such code by hand would take approximately 100 business days or 5 work months, but with a code generator just a matter of a few days at most. Also, all code generated would be error free, which would save on subsequent testing times. And it would be 100% standardized, which would make the learning curve smaller for the next programmer to do maintenance. The benefits to be obtained when building a new enterprise database for future growth would be proportionally greater for any increase in size of that system.

8.2 Using A Code Generator For Business Components

With the use of commonly structured business components to represent tables\functionality in a system, more than 50% of the code relating to such activity could be code generated.

What would have generating business components meant for XYZ Company?

Taking a quick look at XYZ Company’s own 123 Enterprise database, using code a generator to produce basic Parent, Child, and Collection objects for each table in the 123 Enterprise database would have saved someone from having to write 150,000 lines of code. Producing such code by hand would take approximately 300 business days or 1 1/2 work years, but with a code generator just a matter of a few weeks. The same comments made above in regard to generating stored procedures about error free code, testing times, and future benefits apply here too.

In a ten-minute exercise, I ran some commonly identifiable 123 Enterprise tables through my code generator and I got the following results:

0 Addresses Editable Master Component = 1103 lines of code.
0 Carriers Editable Master Component = 596 lines of code.
0 Constants Editable Master Component = 470 lines of code.
0 Contacts Editable Master Component = 592 lines of code.
0 Communications Editable Master Component = 492 lines of code.
0 Countries Editable Master Component = 492 lines of code.
0 Global Notes Editable Master Component = 669.
0 Hot List Editable Collection = 610.
0 Hot List Companion Editable Collection = 481.
0 Orders Editable Master Component = 943 lines of code.
0 Order Item Editable Detail Component = 981 lines of code.
0 Order Items Editable Detail Collections Object = 197
0 Products Editable Master Component = 1339.
0 Stores Editable Master Component = 1919

That’s over 10,000 lines of code, or approximately 770 lines of code per 123 Enterprise table referenced, which worked out to be 1,000 lines of code a minute.

As many 123 Enterprise tables would require multiple related components in a production system, I would expect collections of components relating to each table would on average require at least 1,000 lines of automatically generated code, an activity that would take just days with a code generator for the entire database.

Some additional details behind this estimate are as follows: The 123 Enterprise database contains about 300 hundred tables. Approximately, 50 of those tables are EDI related, and 100 are Shadow tables, leaving 150 tables comprising the XYZ Company’s current enterprise system. All those tables can be divided into three sets of tables such as Master, Detail, and Code tables. For each table, business components would have to exist to permit editing of individual records, collections of those records, or lookup lists to them.

9 Six month test case—proof of concept

If one decides to write a new system, one does not have to commit millions of dollars or even years to the effort before being able to get a fairly reliable prediction of its eventual outcome.

In the case of writing a new modular system, one can start with just one or two of the easier modules and less than half the development team with a six month completion schedule, with even more frequent delivery cycles of working software. At the end of that period the XYZ Company would have a couple of working modules and in-depth knowledge as to how successful the overall effort might be.

10 In Summary

All-in-all, a new system that is more functionally complete and adaptable then the company’s current one could be operational in less then 2 to 2.5 half years. However, the company only needs such a system in that timeframe if it wants to accelerate it growth and sell more product to a wider variety of customers over a wider variety of channels. Otherwise, it can get by with its current system for many years to come.

When it is time for the company to acquire a more adaptable system, such can be done by evaluation and purchase of a new system, refactoring of the existing system, or the development of an entirely new system.

When that time comes, in order to keep XYZ Company’s competitive edge, I advocate the development of a new custom system. Such a development could be completed with the ABC Systems, Inc. Development Process in less then 2.5 years. As the development effort ramps up and the team gains experience working together, there’s a possibility that individual modules would not have to be developed in a linear process, and the entire length of the project could be shortened to some period less then two years. Also, there’s the possibility that the implementation of the system could be phased in over time, and parts of it could be put into production earlier then other parts. Regardless of the final delivery schedule ABC Systems, Inc. Development Process ensures that along the way there would be plenty of opportunities to verify that the project is progressing well.

(Selling More, By Having An Adaptive System, An XYZ Company Story)

Section 5: Suggested Enterprise Architecture for XYZ Company

(Selling More, By Having An Adaptive System, An XYZ Company Story)

Section 6: ABC Systems, Inc. Development Process (IT Target Activities and Classification Levels) Diagram

NOTES

1) Central Repository of Code, System Design Documents, Stored Procedures, and Installation and Setup Directions. One should be able to checkout and compile current versions of all applications at anytime.

2) Standards should cover more than code, they should also cover such things as the general handling of errors, user environment and application settings.

3) Some characteristics of the desired architecture would be

a. Flexible modularization
i. Loose coupling (via messaging)
ii. Plug-and-play (via interfaces)
iii. Flexible configuration
b. Proven Practices
i. Patterns
ii. Components
iii. Interfaces
c. Efficiency
i. Performance versus Scalability
ii. Influence of Layers and Tiers

4) Sample brief requirement documents would be Use Cases, High Level Data Flow Diagrams, Bulleted Requirements, Class Diagrams for Components, Classified Business Rules, and a Business Glossary.

5) The building of online help should flow directly from brief requirement and rule classification documents. It should also be editable at any point by developers or quality assurance people to correct observed errors.

6) Code generation of business components, stored procedures, screens, reports, etc.

7) Code re-generation (i.e. the re-creation of previously generated code) to pick-up latest changes can be accomplished by careful separation of generated and custom code into separate classes (or in .NET into partial classes—think, include files for Delphi)

8) Advanced education can take the form of degrees, certificates, books, software betas, blogs, industry software reviews and lectures. Basically, any quality source that keeps people up-to-date with successful practices in this evolving industry we call IT.

(Selling More, By Having An Adaptive System, An XYZ Company Story)

Section 6) Developing Enterprise Systems - Some Guidelines

Section Contents

1 Overview
2 Architectural Goals for Enterprise Level Systems
2.1 High Level Concepts
2.1.1 THE BASICS - Blueprint, Framework, and Standards
2.1.2 Business Domain\Functional Area Database Model
2.2 Patterns
2.2.1 Modular Architectural Pattern\Business Domain Modeling
2.2.2 Layers Architectural Pattern
2.2.3 Software\Business Patterns In General
2.3 Techniques
2.3.1 Components Versus Objects – A System’s Building Blocks
2.3.2 To Components And Beyond -- Interfaces
2.3.3 Configurable Systems Through Use Of Meta-Dat
2.3.4 Code Generation
2.4 Being Forward Looking
2.4.1 Scalability Versus Performance
2.4.2 Technology Usage
2.4.3 Long System Life
2.5 Development Methodology
2.5.1 Iterative Development With Regular Review
2.5.2 Test Driven Programming and Continuous Integration
2.6 Low Level Concepts
2.6.1 Flexible Deployment
2.6.2 Provide Instrumentation Of Process And Data Pathways
2.6.3 Components - Further Considerations
2.6.4 Simplify Format Of Data Storage And Manipulation
4 Conclusion
5 References

1 Overview

This document provides a short selection of important topics, which should be considered by an architect of an enterprise system. Additional information about managing enterprise system development can be found in the list of excellent references provided at the end of this document.

This document looks at things primarily from a development\architectural perspective, rather then an overall project management view. Thus, many other important things that contribute to the success of any enterprise system development effort, such as having strong sponsors, active participation by managers and key business people, are not discussed here.

2 Architectural Goals for Enterprise Level Systems

The table of contents of this document identifies the list of architectural\design issues that I discuss below, all of which I consider attributes of adaptable enterprise systems.

2.1 High Level Concepts

This section of the document presents some high level concepts that should be definitely followed.

2.1.1 THE BASICS - Blueprint, Framework, and Standards

Large software systems should be implemented according to an overall guiding blueprint. A practice that is present in most other engineering disciplines developing any kind of large or complex system.

Can you imagine trying to build a large office building without a blueprint? Who would be able to review or approve the proposed design? How would other engineers (plumbers, electricians, and telecommunication experts) coordinate their activity so they didn’t end up working at odds with each other? How would they know if their proposed effort will be adequate to support other trades people efforts?

With large enterprise systems that can easily end up with a million plus lines of code, everyone needs to be working with the same overall plan if good re-use is to be made of each other’s efforts. Without a guiding blueprint, components and services designed by different individuals will not necessarily work together well, if at all. Following a good blueprint can also help developers build components and services within a consistent vision of how the entire system needs to scale to handle future growth.

Furthermore, all development work should be carried out according to agreed upon standards, which cover conventions for naming, coding, formatting, and organization of code units into modules.

The company’s development activity should have a fulltime hands on software architect and technical lead to coordinate the use of standards, the blueprint, and framework.

What use are standards?

If individual items within a large system that perform similar activities are consistently named, people can guess what something does just by its name. Naming can also be used to organize all the stored procedures or tables that cover one business domain together, making it easier to identify and work on that functional area.

2.1.2 Business Domain\Functional Area Database Models

In regards to design, the next level of detail that enterprise system development should proceed to, is the development of business domain\functional area database models with the goal of producing a common language and understanding between business and IT personnel for those business activities. And, subsequently offer guidance for the actual development.

Divide and conquer should be the mantra for any large system. Humans are only capable of juggling so much detail and interactions at one time. There’s an often quoted rule that the maximum number of items a person can visualize at one time, is between five and seven. So scoping deliverables at an appropriate level makes it easier to work on large solutions successfully.

2.2 Patterns

Software Patterns offer the opportunity to learn from other people’s successes and avoid mistakes.

2.2.1 Modular Architectural Pattern\Business Domain Modeling

Modularization is one of the fundamental software architectural patterns that’s been successfully applied to build all large enterprise systems over the last twenty years. It involves the break-up of the enterprise system into smaller relatively self contained subsystems that handle major functional areas of the business. For example, in the case of a manufacturing company, suitable modules might be: Product Development\ Merchandising, Ordering, Logistics, and Finance. These smaller subsystems can then be developed and installed separately or even substituted with best of breed applications.

This reduction in the reliance on direct linkages between the different areas of a system is commonly referred to as “loose coupling.” Because of the use of indirect links\data exchanges, modules can for the most part be individually enhanced and tested without risking cascading side affects to other parts of the system. It is also the reason why many vendors of commercial enterprise systems can supply individual companies with paired down versions of their products containing just the modules necessary for running a particular company’s activities. One additional benefit of loose coupling is that it dovetails well with the use of Business Domain\Functional Area Database Models, (core system design documents).

2.2.2 Layers Architectural Pattern

Layering is another one of the fundamental software architectural patterns that’s been successfully applied to build large systems of all sorts. It’s a tried and true technique for packaging cohesive elements of a system in their own layer to provide some core service to the rest of the application and even between applications. If the communication protocols across the boundaries of each layer are respected by all parts of the system, the classes within those layer scan be significantly changed without interfering with the rest of the application as long as each layer continues to service its original communication interfaces.

This is one of the most challenging areas of building enterprise systems, as it requires that people be able to think abstractly about what services need to be made widely available, and how they might be generically implemented to facilitate their reuse. And which services are similar enough in nature, that they should be packaged as a larger cohesive part of the system. Often the best guidance on this topic is to review other publicly available frameworks to see what types of things they do, and what parts of them could be adapted for ones own use.

A common use of the layering pattern in enterprise systems is to inoculate the rest of a system from any undue influence of a particular data access technological. Such technologies tend to change every two to three years and systems in which it has not been isolated in a single layer get caught in a situation where there is no easy path to the new data access technology and they end up tending to use the older technology well beyond it best use date.

It’s not necessary that a layer be an actual physical separation within the application as long as the functionality that layer represents is well scoped and does not vary across its implementations. For example, the basic CRUD (Create, Read, update, and Delete) operations of business components could be grouped together within a component and named the same across all business components. When it comes time to update those routines for a new data access technology they can all be easily located and dealt with together. Current features in .NET 2.0 languages such as code folding regions and partial classes provide some nice mechanisms to facilitate such activity.

An excellent use of layering that is not used commonly enough is to put business logic\rules inside business components that live within a business layer that is accessible by the remaining upper layers of the system. This would provide the different presentation layers the same direct access to that business logic, reducing the amount of code that needs to be written within them. For example, this would permit both a Windows and Web application to make use of the same business component to check when a customer’s next delivery can be expected or whether his account balance is in the black.

Layering also facilitates the application of individual developer skills to parts of an enterprise system to which they are best suited. Those that are better with databases can work on the data access layer, those who are better thinking along abstract lines can spend more time working on the core reusable services, and those who have web experience can work on browser base forms.

2.2.3 Software\Business Patterns In General

Many great efforts have been made over the last ten years to document software and business domain patterns that offer solutions which can be reused within other systems. These patterns provide tried and true techniques for solving problems commonly encountered in building systems. They give software architects and developers immediate access to techniques they can used to solve problems they are faced with, without requiring them to go through all the trial and error the original authors did. They also provide the ability to architects and developers to describe how major areas of functionality within the system works, by just naming the underlying patterns with which they were constructed.

Documented patterns can be found for many levels of software development. There are patterns called “design patterns” that solve many smaller problems within an application. Some commonly known design patterns are: singleton, bridge, facade, strategy, and chain of command. There probably isn’t a programmer who has built a large system who hasn’t used all of those patterns whether he was aware of there names are not. Details on when and where to use particular design patterns can be found in the classical reference on the subject, Design Patterns by Gamma et al. There are more substantial patterns that can be used for the overall architecture of an enterprise system. Some commonly used architectural patterns besides the ones mentioned earlier are: pipes and filters, broker, microkernel, and model-view-controller. Additional information on when and where to use these patterns can be found in the book Pattern Oriented Software Architecture – A System Of Patterns by Buschmann et al. There are even collections of patterns for individual business domains such as accounting and finance, observation and measurement, and trading, Martin Fowler’s book Analysis Patterns – Reusable Object Models has more to say on that subject.

Any new enterprise development effort should try and benefit from the knowledge of others in order to deliver a successful solution with the least effort that will withstand the test of time. Patterns offer a quick means of doing that. The one question I asked all developer candidates is, “Have you heard about, or do you know about software patterns,” their answer generally allows me gauge whether they keep up with best practices in the industry and are open to learning from others. At the end of the day, the enterprise development effort is its developers and what they know.

2.3 Techniques

The following sections discuss some very specific things that enterprise system builders should be doing.

2.3.1 Components Versus Objects – A System’s Building Blocks

Components involve the packaging of smaller objects into a container capable of managing them to provide a more robust implementation of their combined functionality. By packaging smaller objects into larger containers capable of handling bigger units of work, other parts of the system (and developers) need know less about the specifics of what the individual objects do, or the details of what it takes to make them work together, making it much easier to re-use them. Actually, even though objects are the smaller elements of a system they too should be constructed with well defined cohesive behaviors with minimal interdependencies so that larger system items such as components can more easily make use of them.

It is much easier to construct entirely new system modules working with components, rather than objects, as the units of work that can be cobbled together are bigger. They also permit the development team to more quickly adapt a system to changes in business needs, as functionality is packaged at a more comprehensible and manageable level.

2.3.2 To Components And Beyond -- Interfaces

As significant as the movement to emphasizing components over objects has been, so has been the movement to emphasizing that people focus on programming to interfaces versus class hierarchies.

In the past, systems that relied heavily on class hierarchies to get there work done were found to be brittle. Changes to one level of the hierarchy would often break classes in many other levels. The primary problem is that class hierarchies tend to create tightly couple items, which flies in the face of advocating loose coupling between parts. Today, system development work tends to focus more on whether a class can perform a particular set of behaviors as identified by a specific interface (contract). Interfaces represent a set of methods with particular signatures that define a collection of behaviors. Any individual class can implement a number of different interfaces to represent a broader range of behaviors, and new interfaces can be added at anytime. Also, as long as an object or component implements a particular interface, it does not matter where in the object hierarchy it is, other parts of the system can make use of it through its interface.

2.3.3 Configurable Systems Through Use Of Meta-Data

Greater emphasis is being placed on meta-data to configure and drive many areas of new systems. In the past, this has been an area that commercial software developers have paid more attention to, then individual organizations have, because of the needs of their packages to be able to accommodate different types of businesses. However, it does also offer individual enterprises the opportunity to do such things as easily change deployment, reporting, and data access options. Or provide the means to produce highly configurable business rules for systems that can be adapted quickly to changing business needs. And probably most importantly of all, it can provide inputs for a code generator capable of producing 50% plus of all code needed by a new enterprise system.

A balance does need to be established between the complexity of the meta-data gathered and its utilization in regards to the size of the development team. If the development team is large enough, it’s easier to have developers dedicated to this activity to produce a sophisticated implementation that provides an excellent return on investment. It’s also easier to provided continuity amongst programmers to keep the solution productive overtime. However, in a smaller programming shop it’s often much harder to get permission to be able to allocate even a single developer part time to such an activity, because each developer has such a wide set of demands to meet already. And even if one does do so successfully, it’s harder to keep productive continuity on this activity overtime, as when an individual programmer leaves such smaller teams, they often take a huge block of unique knowledge with them, (sometimes the entire knowledge about a particular subsystem).

2.3.4 Code Generation

The idea of code generation (code that writes code) has become a hotter topic of late, with many companies having great success using it to cut development costs and delivery schedules, especially, in large new development efforts where they are generating up to 80 percent of the system’s code.

Today’s code generation engines are a better breed then those of years ago. They tend to offer more flexibility through the creation and use of simple templates that instruct the code generator on how to produce all the repetitive and highly structured code that developers would otherwise need to write.

Taking code generation to the next level, where code regeneration can also be productively used to maintain a system as it evolves, is a very challenging task that is normally only achievable if everyone buys into the use of code generation in the first place. However, such achievements are assisted by the current generation of code generators that use templates to provide more flexible output then their predecessors and .NET partial classes which store generated code for a class in separate files to that of its custom written code. Because the templates are parameter driven, small changes can be accommodated by rerunning templates though the code generator with a slightly different set of parameter values and replacing the custom generated code in the partial class files without touching any of the custom code for the class.

2.4 Being Forward Looking

The topics under this section discuss some items that any forward-looking system designer should take into account.

2.4.1 Scalability Versus Performance

The scalability of a system defines how responsive it remains overall while greater work loads (more users\transactions) are placed on it. A well-designed scalable enterprise system is capable of handling hundreds to thousands of people while still being capable of maintaining a good response to each individual.

Generally, enterprise systems are made scaleable by splitting them across tiers, which basically means running parts of the system on different computers dedicated to providing some specialized service or sets of data. While inserting additional tiers does tend to significantly and adversely affect the performance of any single request because of network latency and other issues, the ability to handle additional requests causes little additional system load because of the specialization in what they do. For example, if everyone logging on to your application requires a list of active stores, that information could be gathered from the database by an enterprise tier when the first person makes such a request, and it could then be cached on that tier, so it can be subsequently passed along to all others making the same request. The .NET framework itself also implements pools of shared re-usable resources such as database connections that users can be given access to as needed. When the user is done with the database connection it is returned to the shared pool where it remains ready to be used by the next person needing it. This removes the need for the system to incur the costly activity of creating a database connection every time someone needs one.

An application tier can be run on one or more computers in a load balance matrix (farm) like setup when necessary, to provide scalability to thousands of users. Such a setup can also provide fault tolerance for system, as the loss of any one instance of a tier does not stop the overall system from working.

The Modularization and Layering architectural patterns mentioned above are used to package tiers for system scalability.

2.4.2 Technology Usage

In building an enterprise level application, people should always be looking to the future, not only in regards to where the company might be going, but also as to the likely direction technology might take. Older technology offers less functionality, and is harder and more costly to maintain as time progresses.

Working with current technology makes its easier to find developers, free code, and productivity tools in a larger more active development community.

For example, if one was using the current technology of C# 2.0 and .NET 2.0 one would have access to freely available frameworks that would take care of much of the plumbing of any new system, a major cost and time saver. And the execution of smart business components that are easily re-used across Windows and Web Applications, and Window Services.

2.4.3 Long System Life

Systems always seem to live longer then most people expect. In fact, prior to focusing on being an enterprise developer, I completed a number of departmental level applications that were only expected to be run for six months, where just about all of them ended up lasting 3 to 5+ years. Had I actually programmed against the original estimate of their useful life, the client would have incurred additional cost to extend their life. However, the cost of programming for a much longer life was so small while writing the application, that I did it, which in the end turned out to be a major time savings, as opposed to me having to go back through another design, analysis, and testing cycle six months down the road, when I was no longer as familiar with the original requirements.

It’s even more important to plan for long life with a custom enterprise system built for a growing company, as the costs of retrofitting it later to extend its life is quite costly. Just as with small systems, its durability should be built with a long-term horizon in mind, not just a company’s present needs. It should be many, many years, before a business starts to stress the capacity of a newly developed system. It certainly shouldn’t be necessary to make apologies for a system’s shortcomings shortly after it’s been delivered, (or even as soon as it’s delivered).

2.5 Development Methodology

2.5.1 Iterative Development With Regular Reviews

Completing a detailed enterprise system design up front is an impossible task. Don’t let anyone tell you otherwise. Commonly, when interviewing business people, things are not always recorded correctly, or even interpreted properly. Not to mention, that people forget to tell you about important things because their not part of their everyday activity, or because they assume you know such information already. And also because they can’t easily visualize how a new improved process might be put together on a computer, (they are tool users not builders). In addition, as volumes of detailed information are gathered on a large system, designers tend to lose their ability to keep everything straight in their heads, and plan the perfect design. And its not uncommon for designs put together over a long period to be obsolete before they are implemented as the organization continues to adapt to changing business needs during the design phase.

The only really practical way to develop a large system efficiently is by gathering some high level requirements initially, and then moving on quickly to focus on individual functional areas\modules in an iterative development-cycle, soliciting regular feedback and guidance from actual business people so that regular adjustments can be made in a timely manner while the cost of doing them is still small. With regular feedback cycles and releases of working modules the final system is more likely to have what business users need.

2.5.2 Test Driven Programming and Continuous Integration

Test Driven Programming is a popular technique for ensuring the delivery of robust systems and maintaining them as such. It involves the construction of tests at the same time requirements are specified, where those tests are coded before the business functionality itself. As working code starts passing the tests, one gets assurance that the new functionality meets requirements. In addition, those tests can be executed in a testing harness with automated executions, which allow them to be run at anytime during the initial development process or any subsequent enhancement cycle to make sure that all versions of the system continue to pass all tests.

A complementary activity to Test Driven Programming is the concept of Continuous Integration, which set-ups automatic builds of systems from frequently checked in code, upon which tests are automatically run, and after which a final exception report is emailed to interested parties.

The loose coupling of applications and components described elsewhere in this document permits simpler tests to be created as fewer interdependencies need to be handled.

Some advocates of Test Drive Programming even view the tests as a means of providing some additional documentation as to what a system’s business rules are. However, I personally feel that such information would not be widely used and tests should instead likely make reference to particular rules in some better and more widely accessible classification of business rules.

Test Driven Programming and Continuous Integration can provide pragmatic tools for maintaining a robust system. Vendor tool support and company usage of these techniques is growing strongly.

2.6 Low Level Concepts

2.6.1 Flexible Deployment

The ability to deploy the system in a single tier (on a single PC) or in multiple tiers (individual user PCs plus one or more application, database, or web servers) should be easily configurable from an application’s file settings.

For example, if a remote sales office only has one user, then it might make best sense to run the module(s) he needs on a single PC. Other larger remote offices or facilities might need to run their modules across individual PCs, and at least one server for better performance. The main office with hundreds or thousands of employees may want to make use of separate servers for individual tiers to get good performance for everyone. All of this should be configurable from an easily modified application settings file.

It would also be highly desirable that the application itself be deployable over the web with a single action on the part of the user.

2.6.2 Provide Instrumentation Of Process And Data Pathways.

Using a single framework (or bus architecture) though-out a system provides the opportunity to provide instrumentation of individual processes and data pathways to identify who used what, when, how, and where. Such activity can help in trouble shooting performance problems and in identifying what parts of a system are no longer being used and can be retired

2.6.3 Components - Further Considerations

Some additional considerations that should be made when constructing components are described below.

Components – Promotion Of Abstraction In Design

Promoting abstraction while designing components can make them more flexible. If components are not unduly specific to any current set of steps or datum, they can be more easily enhanced as business needs change.

Components - Design For Behavior Not Data

Many developers new to Object Oriented Programming (OOP) tend to make the rookie mistake of creating too many simple classes that are largely only record set representations of all the tables in their database, a behavior that is encouraged by vendors of object-mapping-relation products, which will automate the creation of such objects. The biggest problem with having a data-centric view in performing OOP is that it tends to result in light weight objects that contain little intelligence. Which requires, that the smarts they should have contained be coded elsewhere in the system where its not easily reused, and therefore, it often ends up getting reproduced everywhere those objects are used.

Rather then simple data objects, it is better to focus on building business components that might represent data from a collection of tables and all the behavior that needs to be wrapped around them to perform some business task to promote code reuse. For example, an Order business component in its most simple form should be capable of holding customer summary data, such as their name, phone number and address, and a separate data line for each product they ordered, along with methods (behavior) to check that the customer has a good credit rating and calculate the time it will take to deliver their goods. If all this data and behavior is wrapped up in a single business component, then it is easier for Windows and Web programmers to build ordering tools that reference that single item to allow people place orders from both sources. And while the company needs to incur the cost of building two different types of interfaces they save money by only building one business component that does a good deal of the real business work.

Components - Normalize Behavior

Just as relational databases should focus on storing data in a normalized form (without repeating data groups) the practice of object oriented programming should focus on the normalization of behavior.

Taking our example of the Order component described in the section “Components - Design For Behavior Not Data.” it’s not necessary that the component actually know shipping times for products to a customer if the shipping system can already calculate that information. It would be better that such behavior only be implemented in one place, most likely the shipping system, and that the Order component would request a shipping schedule after passing the shipping system a list of products and their destination. The goal being, that actual behavior gets coded in only one place, and therefore, there is only one place it ever needs to be updated when changes are needed. And while the actual calculation of a shipping date\schedule could be done elsewhere, users of the Order component do not need to be aware of that. They would just ask the Order component directly for the estimated ship date, whether the component calculates that information itself, or passes the request along elsewhere, should not be of concern to anyone using that component.

2.6.4 Simplify Format Of Data Storage And Manipulation

It’s desirable that data be passed around the system in a fairly rudimentary format that is largely independent of any vendor specific data management technology that is likely to change every few years, as it has done for Microsoft and Borland over the last 15 years. This helps reduce an application’s dependence on any software vendor’s more specialized data management technology, making it easier to replace in the future.

For example, large amounts of table like data in .NET components can be kept in simple tabular structures with rows and columns that support the .NET Binding List interface that provides data binding for forms permitting bidirectional data exchanges between both. This can be done in preference to the much more sophisticated data typed xml datasets, which are much more likely to be subject to change in each new release of .NET and eventual replacement within two to three years when the vendor creates a new data access technology. The simpler data structures may change too, but generally those changes will be small and manageable, and possibly not affect the business components at all. Even better, if the behavior around those simple structures is coded in a common ancestor for all business components, then any adjustments that need to be made only need to be adjusted for in one place for all components, which would not be the case for more complex data structures, as they get implemented as independent entities with no common ancestor. Another advantage of the simpler structure is that it’s easier and faster to pass data from one component to another for any additional processing, as the data’s storage container carries much less baggage with it.

3 Conclusion

Building enterprise systems is an inherently complex business. Getting all the requirements and design right at the beginning of a large system never happens. If what was presented in this document didn’t persuade you of that, consider the fact, that no one has the rights to a working crystal ball to predict the future, and stuff happens. So by necessity, system development has to be done while trying to hit a moving target. Change is the only constant in our lives, small changes happen all the time, more fundamental changes occur as time progresses.

In accepting the above facts, this document has shown that one can be proactive about planning to handle change when building large systems. It explained how the development effort will be much easier and ultimately more successful if one selects methodologies, tools, and techniques to craft an adaptable development process that is responsive to changes. Having an adaptive development process, allows the development personnel to make necessary changes while development is underway, while it’s still cheaper to do so, and to produce a system that is better geared to what the business’s needs are at the time of delivery.

Some of the most important guidelines discussed in this document for providing an adaptable system were: 1) Making use of blueprints, domain models, and standards. 2) Adopting proven software patterns. 3) Using coding techniques such as component building, interfaces, meta-data, and code generation. 3) Being forward looking with system scalability, performance tradeoffs, and technology usage 4) Using the Agile methods of Interactive Development, Test Driven Programming, and Continuous Integration. 5) Providing for flexible deployments, instrumentation to monitor system usage, and being careful to adopt a fairly neutral method for data storage and handling of data.

While this document did cover a number of major areas of knowledge that any enterprise software architect\developer should be familiar with, it’s not intended to be a comprehensive list. Overall, software architecture is a pretty large and complex subject area offering many challenges. For those interested in the subject, the References section below provides additional sources of information on it.

In larger companies, the software architectural load can be split amongst architects by allowing them develop specialties from which they can contribute unique knowledge to the overall team effort, while only being required to have a general knowledge about other areas. In smaller companies, the software architect generally needs to cover broader ground in less depth, as the company isn’t often large enough to have someone dedicated to work on that activity full time. In such cases, the architect often also needs to manage the development staff, do project management, and actually help write the software. To compensate for being spread so thin, certain responsibilities need to be pushed further down the line to other development personnel who are willing to accept and learn new skills to help make the enterprise development effort a reality. Skills such as those outlined in this document.

4 References

Analysis Patterns – Reusable Object Models by Martin Fowler, ISBN 0201895420. Provides a catalog of “business domain” patterns.

Applying UML and Patterns—An Introduction to Object-Oriented Analysis and Design by Larman, ISBN 013748807. Provides a foundational discussion for building robust, scalable, maintainable systems, using object technology.

Building Business Intelligence Applications with .NET by Robert Ericsson, ISBN 1584502711. A practical guide to using .NET tools to build applications for OLAP and data mining.

Code Generation in Microsoft .NET by Kathleen Dollard, ISBN 1590591372. Teaches code generation as a scriptable and repeatable process using templates so one is not tied to a particular framework or style. Kathleen is one of the industry leaders and a major advocate of code generation. Her book also has a couple of chapters on code generation for the CSLA framework (refer to the “Expert C# Business Objects” book below).

Component Software – Beyond Object-Oriented Programming by Clemens Szyperski, ISBN 020117885. It’s about developing re-usable off-the-shelf components that handle a significant amount of functionality. It emphasizes that one should not focus on trying to build each new piece of functionality from collections of small independent objects while developing larger systems. One should instead, focus on developing more robust components that wrap up smaller objects into bigger containers that are capable of performing bigger units of work, which in turn, require less effort by other developers to reuse them to build even bigger functional units.

Design Patterns – Elements of Reusable Object-Oriented Software by Erich Gamma, Richard helm, Ralph Johnson, and John Vlissides, ISBN 0210633612. This book is the original classic on software design patterns, and still the best reference on the subject.

Domain Driven Design by Eric Evans ISBN 0321125215. This book makes a persuasive argument for the use of domain models to provide a ubiquitous language that ties the domain experts and technologists together. Also, it emphasizes that really powerful domain models only evolve overtime and that some of the best ideas appear after the initial release of a system.

Enterprise Integration—An Architecture of Enterprise Application Systems Integration by Fred A Cummins, ISBN 0471400106. An excellent book, that reviews the technology landscape defining enterprise integration objectives, and which provides generalized enterprise integration architecture.

Enterprise Integration Patterns—Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf, ISBN 0321200683. This book describes using asynchronous message as a proven strategy for enterprise integration, and offers 65 patterns for building such systems.

Expert C# Business Objects by Rockford Lhotka, ISBN 1590593448. A book about application architecture, design, development, and using object-oriented concepts in .NET. Book describes the building of a “Component-Based, Scaleable, Logical Architecture” (CSLA) in C# .NET, which has been used as the basis of system development efforts in many companies with good success. Also, the company Rocky Lhotka works for relies on his framework and code generation to produce upwards of 80% of the code in new systems they develop for clients.

Object Technology – A Manager’s Guide, 2nd Edition by David A. Taylor, PH.D., ISBN 0210309947. This book is an easily read classic on the subject. One of its central themes is that objects are not about the data they contain, but the abstracted behaviors they represent.

Patterns of Enterprise Application Architecture by Martin Fowler, ISBN 0321127420. A book written by a well known industry expert on the topic of suitable patterns for enterprise architecture. The book is written in two parts, the first part is a short tutorial on developing enterprise applications. The second section and bulk of the book is a detailed reference to patterns that assist in building enterprise applications.

Pattern Oriented Software Architecture—A System of Patterns by Frank Buschmann, Regine Meunier, Hans Sommerland, and Michael Stal. An excellent book concentrating on Architectural Software Patterns such as: Layers, Pipes and Filters, Distributed System Broker, Model-View-Controller, and Microkernel.

Strategic Data-Planning Methodologies by James Martin, ISBN 0138511136. The first good book I ever read on the topic of strategic data planning and still one of my favorites.