Building the=
By
Panta rhei =
211;
"Nothing is as stable as change”
The purpose of an SOA is to provide a fer=
tile
ground to allow for restructuring the IT landscape in light of the Internet,
and the future of pervasive computing. SOA is not a goal but rather a strat=
egic
master plan toward greater, more powerful, and more efficient IT. SOA will
easily free up to ninety percent of current inefficiently-used resources, a=
nd
use them in other ways to gain greater business benefits. The savings are m=
ade
through technical means like conversion utilities, a common enterprise serv=
ice
bus and central repositories that make flexible and agile recombination and
reuse of existing programs as easy as possible. But that is not all: SOA wi=
ll
literally introduce a new dimension in computing by presenting the ubiquito=
us
ground for parallel and asynchronous processing as the standard way of
assembling algorithms. This new gain in complexity will lead to an
outburst of new possibilities in computing.
Pizza Pronto?
If you’ve recently ordered a pizza =
to be
delivered from a local pizzeria, then you may well have unwittingly become =
part
of a new flavor of both Globalization and Service Oriented Organization. Le=
t us
assume you live in Pizzburg and you dial the =
number
of the pizzeria. A nice, polite person picks up the phone and says “P=
izza
Pronto, Giovanni speaking, how can I help you? ”. This person helps y=
ou
make your choice and notes down your order, delivery address, and payment
information. Half of an hour later you call the same number again to ask why
the pizza has still not been delivered. “Pizza pronto, Carla speaking,
how can I help you? “. Now it is Carla and she quickly checks with the
pizza driver, and then advises you that the delivery guy has been held up i=
n a
traffic jam but is now only ten minutes away. So far, so conventional. But what does hap=
pen
behind the scenes? Did you ever question yourself who the person on the oth=
er
end of the phone line might be, where she or he is actually sitting and how=
the
communication with the pizza driver takes place? If you are very misanthrop=
ic,
you may think that your polite helper does not really talk with the driver =
but
simply makes up a story and just treats your case as any other case. The tr=
uth
may be much more hi-tech than you would imagine. The person that picks up t=
he
phone is sitting in one of the better call-centers in
Globalization Need=
s SOA
This scenario is a result of Globalizatio=
n. In
our case,
What does this example have to do with a
Service-Oriented Architecture? Mainly the fact that this scenario is so
dependent on an agile electronic and computer assisted communication struct=
ure
that it could not live without a Service Architecture. Let’s do a bri=
ef
anatomy of the technical landscape of a global call center.
·
·
·
·
Making Money with SOA: 90% Savings Today
Figure 1 shows a brief calculation of how=
much
money the SOA development approach would have saved us. This is a typical
calculation that we made from the estimates in man-days, to compare costs of
developing the project traditionally or with an SOA approach. Traditional
development would mean that we develop most components end-to-end, on our o=
wn.
Figure1:
Comparing Development Costs Between Traditional And SOA
|
What to do |
Traditional (man days) |
SOA (man days) |
|
Building the master data link |
10 |
25 |
|
Recognizing telephone caller ids |
120 |
5 |
|
Display town map |
120 |
2 |
|
Inject SMS messages into CRM |
5 |
|
|
* This component was never
developed as the necessary intellectual and technical capacity to build an
SMS bridge to SAP could not be found. |
Figure1:
Comparing Development Costs Between Traditional and SOA Projects
The remarkable point is the fact that the
scenario here had so many technical specialities that it would have been
difficult (if not impossible) to bring all the necessary know-how into the
project. We also did not consider (in the calculations) the gain in quality
with an SOA by sheer reuse of the components. The more frequent a component=
is
used, the less likely it is to contain substantial unknown errors. Traditio=
nal
components are developed individually for a dedicated task. They are develo=
ped,
tested, and used exclusively by a single consumer.
Virtualization
Let'ss&n=
bsp;
look at the component that determines, in real-time, if a certain pi=
zza
can be baked or not. We find that we are dealing here with a sort of master
data management task for a restaurant. We, in fact, have the most topical
master data stored on the restaurant’s computer (material data manage=
ment
system) in Pizzburg, and it needs to be synchronized with the CRM system in=
While this appears to be relatively easy,=
we
should not forget that there is hardly a modern ERP system that can handle =
such
a scenario. Just think of SAP with two independent instances. How nice woul=
d it
be to keep the material data in one instance and just use them ad hoc in the
other instance when they are needed? Figure 2 shows the process as it is to=
day
contrasted with a possible future scenario.
Figure
2: Master Data Management in A Future SOA Landscape
This concept of virtualization is an esse=
ntial
building block of SOA. The purpose of an SOA is to restructure the IT lands=
cape
in consideration of the Internet and the future of pervasive computing. An =
SOA
requires breaking down existing enterprise applications into small and atom=
ic
components that can be consumed by other programs as services. The SOA prov=
ides
the technical structure for conversion utilities, a common enterprise servi=
ce
bus, and central repositories. This will allow flexible and agile recombina=
tion
and reuse of existing programs, as easily as possible.
However, allowing for reuse requires the
serving applications to have a sufficient filigree texture of their
applications. If (and only if) the older applications can expose the essent=
ial
logical steps of their behavior as an API, then they are ready and well-sui=
ted
for the SOA structure. That brings us to one of the premises of an SOA.
An SOA requires breaking down existing
enterprise applications into small and atomic components that can be consum=
ed
by other programs as services. In our example of the virtual MARA, it is
important that SAP introduces a virtualization layer for database table acc=
ess
in the ABAP language.
Today you code:
SELECT
SINGLE * FROM mara WHERE matnr =3D ‘M4711’.
This call reads the table MARA from the currently connected database. You could now read data from a remotely insta= lled SAP instance with an RFC call (e.g., through the function module RFC_TABLE_READ) or a suitable BAPI. But this is not satisfactory; after all, wouldn’t it require changing the code wherever you accessed MARA dire= ctly with a "Select" statement? In addition, coding a "Select&quo= t; is not only easier, but is also much more flexible than a function module.<= o:p>
To introduce a virtualization layer into =
SAP,
the correct place would be the ABAP database abstraction layer. This is the
place where the ABAP Open SQL statements are translated into real database
commands, and all communication with the DB takes place. It would be relati=
vely
easy and safe to build functionality that checks whether a table like MARA =
is
located in the current instance, or hosted centrally on another instance as=
a
common shared table. Let’s wait and see if SAP comes up with a soluti=
on;
only recently has Walldorf made a proper U-turn in strategy, with ABAP
development once again following the real practical needs.
Services
When building services for an SOA, we concentrate on building wrappers and proxies to access existing enterprise functionality, and on commonly used utilities, like accessing the ESB elementary components such as email, databases, message queues, and user interfaces. Rewriting an application for SOA may be an option in rare cases, but SOA can only be the trigger to do so. The reason for rewriting a working application can only be that the existing code is insufficient, either in t= he sense of a lack of functionality or with respect to poor quality. If, howev= er, the program works satisfactorily, it is wiser to build wrappers around it.<= o:p>
Figure
3: Wrap Rather Than Rewrite The Application
SAP-literate people have used services in=
the
form of BAPIs for quite a long time. They have been designed as a wrapper
around the business functionality and the business objects within R/3 that
allow access from anywhere. They can be accessed via the RFC protocol, or in
later versions of the ABAP kernel (also as web services) with ease. The SAP
ABAP database abstraction layer is also a wrapper that simply acts like a
middleman between an existing application and a requesting program. General=
ly
ABAP is a fairly mature SOA application framework, far more mature than mos=
t of
the competing solutions that are mostly incomplete, or can only act as a
bolt-on to an application suite, Software Development in the
Thinking in terms of the SOA structure for
software development will require a radical change. While we traditionally
develop hybrid applications that literally “compile” all the
necessary functionality into one single program, SOA requires letting go of=
this
quick and dirty way of writing a program.
Instead, programs need to go and componen=
ts
need to come in. Applications will then be a pure assembly of existing
components. The difficulty here is to find a well-balanced compromise of the
features a component should posess as a service. If a component is incomple=
te,
it is of no use. If there are too many features, they will not be used.
An example of an incomplete component wou=
ld be
a BAPI that does allow creation of a business object, with no way to modify=
or
read it. Unfortunately SAP neglected this crucial component of the SOA
framework in the recent years, so there are still many BAPIs that are
incomplete (e.g., the BAPI for sales orders that allows creating sales orde=
rs
with no compatible function to read the sales order again).
Examples of feature-overloaded components=
can
be found everywhere. Just look again at the sales order BAPI (Figure 4), wh=
ich
has so many parameters that it takes days to understand which ones are
mandatory and which are not.
FUNCTION
BAPI_SALESORDER_CREATEFROMDAT2.
*"-------------------------------------------------------------=
---------
*"*"Lokale Schnittstelle:
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"
*"---=
-------------------------------------------------------------------
Figure 4: The Sales Order BAPI Suffers Both From Affluence And Missing Functionali= ty
The function module BAPISDORDER_GETDETAIL=
EDLIST
does the job partly, but with an incompatible interface, as you can see in
Figure 5.
Figure 5: Sales Order BAPI (To Provide A Read Method)
Staging
The focal point of every component develo=
pment
is the interface. Client and server programs need to have a way to exchange
data so that both understand the data. While this problem can be solved wit=
h mappings,
there is a more complex problem when working in a distributed environment l=
ike
the Internet. Sender and receiver of the data packets may work at different
speeds.
If a pizza in the oven is ready, but the =
waiter
is still busy with another client, we cannot leave the pizza in the oven. We
need to take the pizza out and put it in a place where the waiter can pick =
it
up as soon as he passes by.
Such a place where we can temporarily sto=
re an
object (the pizza) is called a staging area. These staging areas are the
essential difference between SOA development in a distributed environment, =
and
classical programming. A calling program will no longer talk directly to a
receiving component, but simply drop the data container to a staging area w=
here
the calling program picks it up (Figure 6).
Figure 6: Traditional API Programming Versus Message Based Development
The Magic of Queues
The implementation of a staging area is
covered in detail in the documentation for message queues. These queues ser=
ve
as temporary and reliable data storage. There are many variations of queue
implementation. From the name, we immediately think of the very sophisticat=
ed,
database-driven, professional queuing systems like Microsoft MSMQ, WebSpher=
e MQ
(formerly known as MQ-Series), or the Open Source Apache ActiveMQ. However, any file system can be vi=
ewed
as a queue; picture a scenario in which we interpret a folder as the queue =
and
the files as the messages. Email boxes are also classical queues that recei=
ve
and store messages, and allow for asynchronous handling.
=
;Let
us now take a little journey through the SOA garden and demonstrate the real
magic of queues by discussing and practically demonstrating a real-life RFID
(“Radio-frequency identification”) scenario.
Practice: Making RFID Work With Queues=
In order to shape your understanding o=
n how
queue processing works, we have picked one simple example out of some curre=
nt
SOA projects. We have an RFID reader from Siemens that is already intellige=
nt
enough to deliver reader data in a convenient XML data format. The data from
the reader is transported to an SAP instance, where it can be used by an
application.
Let’s start with the difficulties
we’ll encounter.
While the reader can send XML, it does=
it
via a simple plain TCP/IP Telnet protocol; we need to have an agent that
triggers the data stream and listens to the incoming reader data.
Problems with this scenario:
- The
reader fires several times per second, but may not immediately interpr=
et
the reader data content.
- The
reader message format may change as the reader model changes.
- The
data should not be volatile (not be lost when the SAP system is down).=
- There
should only be one record per tag sent to SAP, even if the tag is read
multiple times (in fact, a single tag may fire a thousand times).
We have not found a single person who =
can
solve all these issues right away. A lot of experimenting and studying is
necessary. So, we took the issue list and tried to solve every issue in an
isolated environment, independent from each other.
For instance, when we receive duplicate
data, we need an agent to review the data stream and aggregate it to a sing=
le
result record. Therefore we defined a message queue as a staging area where=
we
expect the reader handler to deliver the messages. In our case, we decided =
to
simply write the messages to a folder in the file system with the intention=
to
replace it with an MSMQ implementation eventually. Our aggregation routine =
will
now find all the messages in a folder, and can review them to check if there
are duplicates.
The basic Telnet protocol has been kept
really basic. It reads the data stream as it comes in and puts it in an
in-memory queue. A second asynchronous task empties the memory queue and co=
pies
the messages, message-by-message, into the staging area. In Figure 7, we sh=
ow
the overall landscape of our scenario, with a lot of intermediate queuing
involved.
Figure 7: Software Development with Queues
RFID Technical Process Flow
So are you ready to see this RFID scenari=
o in
action? Figures 8 through 14 show the process running, and how the messages=
are
“staged” at each step in the scenario.
Before we start, it is perhaps important =
to see
how communication with the RFID reader is achieved using the traditional
protocol TELNET. Figure 8 shows the “host greeting” XML message
being sent to the reader and the huge array of messages that stream in after
that.
Figure
8: RFID Tag Messages Streaming Back Using Telnet
The technical process consists of a numbe=
r of
“Python[1=
]
scripts” that provide the necessary services for the scenario to run =
from
end-to-end. The Python components have been written in a very “event
driven” nature whereby the communicating services react to certain ev=
ents
that tell them, for example, that it is their turn to do something with the
incoming RFID tag messages. The “event-driven architecture” (ED=
A)
provides a truly asynchronous / staging sequence of events that is extremely
easy to understand and troubleshoot.
Figure 9 shows the start of the technical
process, whereby the “RFID receiver service” creates a socket
connection to the RFID reader, passes in the “host greeting” XML
message (just like the TELNET session above), and then sits back and receiv=
es
the RFID tag messages.
Figure
9: Real-time RFID Tag Reads
As soon as the RFID tag messages are
received, they are immediately written to a “memory queue”, and
once there, they are given a unique “timestamp” and then are
written to the file (queue) system. Figure 10 depicts the real-time RFID tag
reads being written to the first “staging area”…the file
system.
Figure
10: RFID XML Tag Messages Written to the Filesystem (Staging Area)
From here, another Python service (tha=
t is
running) immediately recognizes that new files have arrived, and sends the
messages off to Microsoft Message Queue (MSMQ). Figure 11 illustrates the
running code, and the debug messages showing that new files arrived and were
immediately transported to MSMQ.
Figure
11: RFID XML Tag Messages Transported to MSMQ
Figure 12 simply shows the RFID tag me=
ssages
sitting in MSMQ. Queues bring about such a great asynchronous flavor to
integration, and MSMQ fits in perfectly into this scenario.
Figure
12: RFID XML Tag Messages on MSMQ
Once the messages have arrived on MSMQ=
, an
event is raised to tell another Python service to pick up the messages and =
move
them into an “RFID table” on MYSQL. Figure 13 shows the Python
service recognizing that new messages have arrived on MSMQ, and immediately
transporting them to MYSQL.
Figure
13: RFID XML Tag Messages Transport from MSMQ to MYSQL
Finally, the messages end up in an
“RFID” table in MYSQL. This is depicted in Figure 14.
Figure 14: RFID XML Tag Messages in MYSQL Table
<=
/span>
How Much Time Does This T=
ake?
Our task was to connect an arbitrary R=
FID
reader to an SAP system. The challenge is that the reader type may change
easily and that the SAP system is not available at all times. In addition, =
we
had no resources familiar with all of the involved technologies. As a pract=
ical
matter, this resulted in a sequence of components that could be developed by
different people. In order to guarantee the data communication between the
components we agreed on simple data storage like the filesystem or an ODBC
database. We chose these as we knew that any data store can essentially be =
seen
as a message queue, so that we could assume that at least one of the IO
mechanisms could be read (and written) by the developers.
Writing the actual components typicall=
y took
less than a couple of hours. Testing could be accomplished in one day, on
average. In our case, it was two people testing for half a day. Putting the
components together depended on the number of links; we had four of them.
Altogether the whole scenario was developed in approximately 10 man-days. We
have to add an overhead of another 20 man-days of training and research, to=
get
acquainted with the somewhat new technologies like Python and MSMQ, but this
will not occur when we build the next interfaces.
Less is more: Saving m=
oney
and time and win quality
In this installment, we demonstrated t=
hat
its possible to save money and time when developing for an SOA. The savings=
are
so impressive that we can even allow for "playing", research, and
training within the same budget limits of a traditional development project=
. But
that is not all; the holistic way of development in SOA also adds to the
quality of the product (system), and makes solutions possible that could not
have been achieved otherwise.
About the Authors:
Lynton
Grice, logosworld.com is an SAP integration specialist
contracting in
[1] P=
ython is a
dynamic object-oriented programming language that is a true winner when it
comes to agile SOA development. (http://www.python.org/)