AIIM Enterprise Content Bus (ECB) Proposal
Last week AIIM announced that the iECM consortium was dissolved. As you can see from my 2005 posts, I was quite enthusiastic about working on this effort at its beginning, but the fee-based consortium structure and the standardized Web services technical approach didn’t work for me.
My clients and customers for decades have shown themselves unwilling to modify legacy systems except when absolutely necessary, such as for Y2K, so they’re not going to do so for the sake of interoperability.
Also, the differences between various repositories are just too complicated to accept a new standard API without making significant internal changes like adding new columns and tables to their relational persistence. This may be feasible for vendor products undergoing yearly new feature additions, but doesn’t work for home-grown or legacy repositories and these installed repositories may run for a decade or more before being upgraded or replaced.
The industry application integration solution is to use middleware. With iECM now gone, I’m proposing an SOA ESB solution to address the interoperability problem that iECM was trying to solve — plus it offers many business benefits to invoke services for processing, routing, and transforming content and for interacting with business processes — which were not part of the iECM scope.
Here is my proposal posted to the AIIM iECM mailing list today. I hoping to get time on the agenda at the Boston AIIM meeting this week to discuss it.The proposal is for an “Enterprise Content Bus (ECB)” project as the logical technology successor to iECM — recognizing the very rapid advances and evolution in SOA methodology and products made over the last two years.
——————-
While the iECM initiative represented the best thinking of the AIIM community two years ago, practitioners now know that the interoperability benefits expected by defining standardized Web services can indeed be achieved at a much lower cost and in shorter time frames — along with greater business agility and flexibility — using proven SOA middleware approaches and products instead.
For interoperability to work, it still remains critical that the requisite data is available, and that the data formats, relationships, and information semantics are all clearly defined and understood. Given this knowledge, content management and workflow applications can be connected using advanced SOA software to achieve interoperability and federate requests. These solutions can both map and transform metadata and content passing between applications without requiring that each participating system or application adopt the same Web service message APIs.
This is equivalent to the everyday miracle of two cell phones — each using different frequency, modulation, and signaling approaches — being able to communicate effectively because “middleware” in the radio towers, switches, and networks transforms and routes the caller’s voice. This middleware-enabled interoperability spans time and technologies to extend even to legacy landline phones and yet-to-be-invented devices.
SOA now provides equivalent middleware tools for transforming and routing structured data and this AIIM initiative will expand upon this successful approach to establish best practices and standards for handling unstructured data for enterprises and organizations that need to access, process, and manage content from varied legacy, contemporary, and future systems.
Here is some background and a synopsis of the proposal for both those who might attend as well as those who can’t:
Backgrounder and Primer
Many of the participants in iECM were interested in standardized Web service interfaces to directly interconnect ECM systems as a means to achieving interoperability. I understood the iECM goal to be that each system would implement essentially the same services stack in order to exchange searches, results, and content. As the group discovered, gaining agreement among vendors and users on requirements, crafting a detailed consensus solution, and then achieving traction for new and retrofit implementations are indeed all very difficult tasks.
Interconnecting disparate information systems is a common enterprise need, and a large body of practice and supporting products has been created over the years for Enterprise Application Integration (EAI). The standard EAI solution is to introduce “middleware” in the form of a Message Broker server that transforms the requests from one system into a format acceptable to another system, and it does the same with the responses. Information is successfully exchanged, but the systems themselves do not necessarily understand each other’s messages directly.
With the growth of SOA, this concept has morphed into the Enterprise Service Bus (ESB) which can support many diverse system interactions by transforming, enriching, routing, filtering, and orchestrating services using adapters written specifically to connect with each system. The ESB is more resilient and flexible than a central EAI Message Broker.
The ESB has made huge progress in the last two years having evolved from a concept that was implemented by a few vendors to become the standard enterprise SOA solution platform now.
At the AIIM May 2005 conference in Philadelphia, I presented a session that discussed the concept of the “Enterprise Content Bus (ECB)” as an extension of the ESB to solve similar information integration problems in the ECM domain.
While an ESB typically transforms XML, EDI, or legacy structured business data formats, an ECB would transform unstructured data, such as converting office documents and images into PDFs, performing OCR processing, invoking virus checking, etc. It would also perform federation and virtualization of repositories.
Actually, an ECB would need to do both — perform content transforms on unstructured content, and also perform structured data transforms on the accompanying transaction and content metadata. Each requires its own set of distinct transformations, routes, processes, and enrichments. By recognizing this and handling them separately, there are performance and scalability gains to be had.
Bruce Silver wrote an excellent article on the same theme of using content integration middleware subsequently that year in the October 2005 issue of Intelligent Enterprise:
http://www.intelligententerprise.com/showArticle.jhtml?articleID=171000641
Two years have passed since I presented the ECB concept at AIIM, and in the meantime ESBs have proved to be a practical and effective solution so that they now represent essential and core elements of most enterprise SOA installations. SOA architects uniformly embrace the concept that direct consumer-to-provider (point-to-point) services have connectivity value, but the major business and agility benefits of SOA come from the introduction of message queuing and ESB middleware to allow reconfiguration of services and the injection of new services into a process flow.
I presented ECB concepts last year to the Philadelphia AIIM Chapter and have been introducing them again in my monthly SOA online column for E-DOC. My April column discusses Document Governance via SOA, and my May column, which is already in Bryant’s hands, talks about ECM middleware for SOA.
The time has come to put the pieces together.
Observations on SOA/ESB/ECB
Enterprise Service Bus products are now available from many SOA vendors, as well as from open-source projects. ESB offerings are also provided by ERP vendors.
ESBs exist for J2EE and .NET environments on a broad range of platforms. Because ESBs are message and services-based, an ESB can handle requests integrating a mix of J2EE, .NET and clients/services using other languages and platforms.
These are relatively complex software products, but I would expect them to be customized and packaged for a organization, much like the complexities of SQL database table and index definitions are hidden when installing customized and packaged business applications.
Direct point-to-point SOA Web services are still quite appropriate for departmental or SMB use, but for larger applications and the enterprise, the ECB (ESB) would be the appropriate choice.
An ESB, and by extension a ECB, represent very flexible tools for creating a processing flow for structured and unstructured data. For example, an ECB could recognize events, such as the arrival of certain types of documents, and place them into an appropriate workflow or send a subscribed notification even if the repository itself lacked such a feature set.
An ECB could achieve most, if not all, of the goals of ECM interoperability by performing syntax-, format-, semantic-, and language-based transformations on requests and transform results into a form acceptable or desired to the requestor.
Transformations vs. Standards
The iECM project followed the traditional interoperability approach of defining a common exchange format — like what would be needed to interconnect two fax machines using fixed hardware technologies.
Now that products are software based and easily updated, we can introduce new software transformations and we don’t care as much about format differences. My Web browser is quite happy to display a gif, or jpg, or png — or many other graphic formats — by doing the transformation for me on the fly. If a new or different format is used, the old ones still work and a user installs a new transform.
Standards, of course, remain highly useful and necessary, and they reduce costs and speed progress in this environment, but in the end it is not required that all connected systems use exactly the same data exchange standard as it was with earlier technologies such a represented by my example of a hardware-based fax machine.
Adapters
A significant difference vis-à-vis iECM is that the this integration is achieved by using adapters, rather than seeking agreement on a common API or retrofitting systems.
The adapters are customized software components running on the bus, on the target system, or on an intermediary server or device between the bus and the target. The adapters could be written by vendors, by users, or by third-parties and would represent new product and services opportunities in their own right.
Legacy repositories would not need to have their existing software modified — an adapter would utilize their legacy APIs and provide a presence on the bus for them.
The adapter concept is also quite different from implementing a common standard. A bus adapter mediating messages between IBM CM and EMC Documentum may provide different behaviors than one between FileNet P8 and EMC Documentum.
Audience
The target audience for this solution would be organizations with more than one repository technology or vendor, and/or with multiple departments having varying content management requirements that could be best met by sharing configurable services.
That is, single department installations and small businesses would continue to be well served by single-vendor solutions without requiring an ECB. However, they might want to use an ECB-hosted compatible enterprise content services or external SaaS provider services for transformations and added-value processing.
Obviously a departmental-scale product that already supports Web services would be an ideal participant in a larger enterprise SOA environment.
Benefits
The ECB approach using adapters facilitates and enhances interoperability of information repositories without necessitating modifications or enhancements to the repositories themselves.
An ECB would reduce the incremental cost of adding ECM services to business applications by allowing sharing and reuse of services. It would simultaneous open new market opportunities for software vendors.
Because an ESB would allow content services to be shared, reconfigured, and new services injected in a content processing flow, a department could utilize shared services that are not available from the departmental solution vendor or that are too expensive to install in each departmental system.
The primary business drivers are the same as those for SOA: business agility, flexibility and responsiveness, and the capability to integrate, utilize, and reuse expensive software assets. Most of the cost savings in SOA come from subsequent projects that can reuse services that have already been built and deployed.
An ECM can facilitate repository upgrades and replacements by providing virtualization of interfaces so that changes to old applications and systems can be deferred by emulating these interfaces on the bus to access new systems. This same service virtualization would allow for consolidation of repositories due to technology evolution or M&A events.
Federated and inter-organizational searching would be natural capabilities stemming from the ECB’s ability to orchestrate multiple services to satisfy a single request and its ability to selectively apply specific transformations and security approaches to each service provider. For ECM this would mean additional applications, departments, or business partner connections would be accomplished more quickly and at lower costs and these flows could be rapidly changed to meet new business requirements or opportunities.
Key Takeaway
The key concept with the ECB is that there would always be a message transformation and processing engine “in the middle” between service consumers (clients) and providers (servers) to mediate the differences between systems and applications.
From a practical standpoint this allows the bus to mediate or fix discrepancies between implementations until future conformance with a standard is achieved, or the standard itself evolves, or handle the case for integration of a function for which no standard exists at all.
Enterprise Content Bus Proposal and Deliverables
The Enterprise Content Bus (ECB) proposal is for AIIM to create a standards-related project to explore and coordinate the use of SOA ESB middleware technologies for enterprise content management services.
The scope would include facilitating SOA services supporting the whole gamut of AIIM’s ECM world: capture, processing, storage, search, retrieval, archiving, and life-cycle management.
The participants would be AIIM end-user and vendor members. One would expect end-user organization enterprise architects and SOA infrastructure staff, as well as non-ECM vendors in the SOA space, to join AIIM and participate in this effort.
Deliverables would include edited collaborative discussions (Wiki), best practices recommendations, and subsequently standards for:
- Message Exchange Patterns (MEPs) for moving content and metadata between the consumer and provider systems and the bus.
- Service functional definitions and requirements for filing, retrieving, searching, transforming content, and value-added services like virus checking and records management.
- Detailed message definitions and processing patterns with variations for Web services (SOAP), RESTful XML services, WebDAV, FTP, MTOM/MIME/SwA/DIME, portal and AJAX, MQ and JMS messages as well as event and email notification messages to manage content, metadata, and associated transaction data. However, if agreements on detailed standards cannot be achieved, then transformations would be used so that progress and participation continue.
- Message and service definitions that are extendable to large collections of documents to support high-volume capture and conversions.
- Mappings of service messages to fully support the JSR-170 and in-process JSR-283 models (including the JSR-283 SPI) for ease of implementation for both J2EE and .NET environments. The goal would be that if a repository supported these standards (or the .NET equivalent interfaces), then the ECB would be layered on top with little or no change.
- Event and workflow APIs to provide a common approach to signaling notifications and invoking human workflows from an ECB and providing flexible references to the related documents.
- Subsystem models defining responsibilities for major architectural components such as capture, repository, RM, tracking and logging services, metric and metering services, temporary document store and URN/URL reference resolution.
- Metadata, mapping, semantic definitions, and other representational and transformational recommendations and best practices for supporting and facilitating interoperability.
- Service programming models (e.g. Service Component Architecture - SCA) examples for working with content and content metadata in various software languages and platforms.
- Reference implementations for some selected services on one or more ESB platforms for demonstrations, education, performance modeling, and conformance verification.
- Catalog of vendor, service provider, open-source, and other bus adapters that support content services.
There is also specialized work needed for addressing security, semantics, search languages, metadata, and other critical integration areas.
Approach
This would be a long-term project for AIIM, but can product short-term results since the bus can integrate systems without having to fully define a common interoperability standard.
One could envision in a few years a whole “track” during the AIIM conference on SOA ECB topics with new vendor, service, and educational offerings in this space, and ongoing refinement and standardization of best practices and interoperability.
As a vendor-neutral technologist, SOA architect, and manager, I would be willing to coordinate this ECB project for AIIM assuming I maintain a appropriate consulting workload. I hold a BS in engineering, MS in computer science, and an MBA as well as an AIIM MIT designation and middleware certifications.
My recommendation would be to structure the project with a number of sub-groups with their own experts, leadership, coordinated deliverables, and schedules for dovetailing their work products within a common framework.
An incremental and iterative delivery approach would be used to gain design consensus, as well as implementation experience and feedback in relatively short cycles.
Since many AIIM users and vendors are not necessarily familiar with enterprise SOA and ESB technologies, there is a need for initial research, education, and collaborative discussions to establish a common base of understanding and a shared vision.
It would be key if one or more of the SOA vendors could provide software and group training and education to facilitate the ECM vendors quick uptake on ESB programming and design approaches.
Notes
Google can provide definitions and details on any of the acronyms used above.
If anyone on the iECM list has any immediate thoughts on this — pro, con, questions, suggestions, or contributions — I’d appreciate hearing from you in advance of Wednesday. I believe in Fred Brooks’ Mythical Man-Month observation: “How does a large software project get to be one year late? Answer: One day at a time!”