Complex Event Processing (CEP), a term coined by Stanford professor David Luckham, describes a set of concepts and design patterns related to processing "events". The goal of CEP is to extract and use information from events as they pass through the enterprise. Particularly, CEP focuses on detecting and using patterns of events. CEP does not promote a particular method of detecting event patterns and is compatible with a wide variety of pattern recgnition, predictive analytics and pattern matching techniques.
Per Luckham: "CEP emphasizes certain aspects of event processing such as event patterns, event abstraction and event hierarchies. And CEP is about the technology for this kind of processing."
The acronym CEP is also used to describe certain software products that are designed to process events. Typically, CEP software provides both a programming language and a run-time server that simplify the task of working with events.
History of the term
The term Complex Event Processing was popularized by Professor Luckham in his book "The Power of Events" somtimes abbreviated PoE. He derived it from Event Processing, a series of technologies and concepts first used in the 1950's[2]. Over time, it has incorporated concepts from computer networking, active databases, middleware, SOA and other areas that deal with events. To the broad range of research on processing events, CEP introduces the concept of combining several simple events to form an event that represents a more complicated scenario, situation or real-world event[3].
CEP goes a step further to define a "virtual event" as "an event that does not happen in the physical world but appears to signify a real world event; an event that is imagined or modeled or simulated."[4] A virtual event is treated as any other event in CEP.
Clearly, almost anything in the real world or in the computer can be considered as an event for use with CEP. This definition is deliberately broad, since CEP intends to express relationships between as well as design patterns for processing these events, in a way that does not limit their semantics, storage or transmission. CEP discusses high level and broad concepts that apply to the use of computers to handle events in general.
In particular, this definition is compatible with (but more broad than) the definition of event from probability theory.
Event abstractions: events, event relationships and context
Working with events in the computer means, at most, working with a good abstraction of a real event; it means never working with the actual event. For example, the computer representation of a robbery is not the same as the actual event of the robbery. The computer works with representations and abstractions (or more accurately, humans use representations and abstractions when they program the computer).
Here are a few of the commonly used patterns in CEP. There are many more of these both in the PoE[1] and in documents such as Luckham's overview of CEP concepts[5].
Event Hierarchies
As described by David Luckham[5]:
Example of complex event, expanded from the PoE section 3.7.1[1]:
A distributed transaction will eventually commit, rollback, become stuck or enter a situation where participants have inconsistent views of the state. Each of these can be modeled as a complex event:
State, context and situation
In many cases, the exact meaning of an event depends on some state, context or situation.
Per Luckham: "CEP emphasizes certain aspects of event processing such as event patterns, event abstraction and event hierarchies. And CEP is about the technology for this kind of processing."
The acronym CEP is also used to describe certain software products that are designed to process events. Typically, CEP software provides both a programming language and a run-time server that simplify the task of working with events.
History of the term
The term Complex Event Processing was popularized by Professor Luckham in his book "The Power of Events" somtimes abbreviated PoE. He derived it from Event Processing, a series of technologies and concepts first used in the 1950's[2]. Over time, it has incorporated concepts from computer networking, active databases, middleware, SOA and other areas that deal with events. To the broad range of research on processing events, CEP introduces the concept of combining several simple events to form an event that represents a more complicated scenario, situation or real-world event[3].CEP concepts
CEP describes high level concepts and design patterns that deal with events and patterns of events. Some of these terms overload uses from other areas of computing and mathematics. Although this overloading has caused some debate in the CEP community, it is generally compatible with existing literature and the meaning can usually be inferred from the context.What is an event?
CEP overloads the term event as follows[4]:- Anything that happens, or is contemplated as happening
- An object that represents, encodes or records an event, generally for the purpose of computer processing.
CEP goes a step further to define a "virtual event" as "an event that does not happen in the physical world but appears to signify a real world event; an event that is imagined or modeled or simulated."[4] A virtual event is treated as any other event in CEP.
Clearly, almost anything in the real world or in the computer can be considered as an event for use with CEP. This definition is deliberately broad, since CEP intends to express relationships between as well as design patterns for processing these events, in a way that does not limit their semantics, storage or transmission. CEP discusses high level and broad concepts that apply to the use of computers to handle events in general.
In particular, this definition is compatible with (but more broad than) the definition of event from probability theory.
Abstractions and design patterns
CEP is primarily concerned with tying together existing and new ideas and design patterns for processing events. There are many, many disciplines that can be used under the CEP framework, ranging from algorithms to semantic models to domain specific languages to methods and mathematics for understanding and detecting patterns and probability. It should be noted that this does not make any of these areas "subsets" or "subfields" of CEP. CEP simply helps to tie together this broad range of research and practice.Event abstractions: events, event relationships and context
Working with events in the computer means, at most, working with a good abstraction of a real event; it means never working with the actual event. For example, the computer representation of a robbery is not the same as the actual event of the robbery. The computer works with representations and abstractions (or more accurately, humans use representations and abstractions when they program the computer).Here are a few of the commonly used patterns in CEP. There are many more of these both in the PoE[1] and in documents such as Luckham's overview of CEP concepts[5].
Event Hierarchies
As described by David Luckham[5]:An event abstraction hierarchy consists of the following elements.
- A sequence of levels of activities and associated event types. Each level consists of a set of descriptions of system activities and, for each activity, a specification of the types of events that signify instances of that activity. Level 1 is the lowest level.
- A set of event abstraction rules for each level. For each level (except level 1), there must be a rule for creating each type of event at that level as an abstraction of patterns of events at levels below. So an event hierarchy defines a set of levels of activities and a set of rules for computing events at each level as abstractions of patterns of events from the levels below.
The crucial aspect of this definition is the set of rules specifying how each event at a higher level is an abstraction of events at levels below it. This is where a CEP event hierarchy goes beyond prior uses of hierarchies in system organization. A hierarchy in CEP gives a constructive method - e.g., rules - for computing abstract events from events that can be observed in a system.
Derived events
From the CEP glossary[4]:Derived event (also synthesized event): an event that is generated as a result of applying a method or process to one or more other events.Note that in the second example, we have an event derived from a timer event. Since the computer deals with the discrete and can not detect the continuous passage of time, even time must be modeled as an event. And an event that exists only to communicate the current time or an elapsed time is a timer event.Examples:
- An event reporting that “company B has entered the bidding to take over A with probability 0.9″ might be derived from an event reporting that the price of company A’s stock has jumped 10% in 5 minutes.
- The absence of an event, say in a given time interval, can lead to a derived event reporting that the first event did not happen.
Complex Events
Now we get to the "complex" part of Complex Event Processing. A complex event is "an event that is an abstraction of other events called its members."[4] In this case, membership specifically does not imply that the complex event carry with it a collection of member events. A complex event is an abstraction. It might carry with it the collection of members, it might not.Example of complex event, expanded from the PoE section 3.7.1[1]:
A distributed transaction will eventually commit, rollback, become stuck or enter a situation where participants have inconsistent views of the state. Each of these can be modeled as a complex event:
- The member events are the underlying transaction protocol messages.
- The commit and rollback events consist of sets of messages that properly terminate a transaction such that it is clear that all parties are in synch.
- A stuck transaction might include a timer event where if, after a certain period of time, nothing has happened, the transaction can be classified as stuck. In this case, the complex event is "transaction got stuck".
- A transaction entering an inconsistent state is more complicated to model. However, without going into the technical details, one can imagine various ways to detect or derive this event.
State, context and situation
In many cases, the exact meaning of an event depends on some state, context or situation. This topic is summarized in this blog entry and this blog entry by Opher Etzion, which includes the following image:
More content to be added here soon.
Event relationships, causality
Per Luckham[5]:There are many differing ideas about patterns of events. For some, an event
pattern is simply a Boolean combination of events, like A and B. For others it is
an SQL query. But in its most general form in CEP, an event pattern can be a
timing or causal relationship between patterns of events as well as a Boolean
combination. A pattern contains variables which are replaced by values to form
instances of the pattern. A pattern can have many different instances.
Examples are
• Shopping (?Customer, ?Time, ?DollarAmount)
• A -> B
• A at t and B at t’
• A at t -> B at t’
• A within [t, t’]
In the first example of an event pattern the variables are preceded by “?”. Its
instances are Shopping events with actual customer, time and dollar values. In
other examples “->” means “causes”, [t, t’] is an interval between two times t, t’,
and A and B are themselves patterns of events. Relationships between events
such as causality and time are an integral part of patterns of events in CEP.
Note here that the "causality" described by the -> operator is not necessarily the commonly accepted definition of causality. It could mean "direct and sole cause" or "direct and partial cause" or "contributes to" or "probably contributes to". CEP discusses the pattern of causality, but leaves many specifics up to an implementation.
Luckham notes that causality can be used as an algebraic operator similar to a greater than or less than[1]. In other words, if event A is caused by event B, then we can write A->B and this -> operator works in much the same way as less-than (A<B).
Given a set of events, if some of those events cause other events, this forms a partially ordered set or Poset on the events, where the comparison operator is -> (rather than the usual < or >).
Note that not every system needs to use causality. There are many cases where useful information can be determined without assigning causality of any kind. Causality is just one concept discussed in CEP.
If causality is used, Luckham defines the Cause-Time Axiom: "If event A caused event B in system S, then no clock in S gives B an earlier timestamp than it gives A." That's from PoE 5.3.1[1]. Causality may be easier to model within a system that obeys this axiom.
Luckham notes that causality can be used as an algebraic operator similar to a greater than or less than[1]. In other words, if event A is caused by event B, then we can write A->B and this -> operator works in much the same way as less-than (A<B).
Given a set of events, if some of those events cause other events, this forms a partially ordered set or Poset on the events, where the comparison operator is -> (rather than the usual < or >).
Note that not every system needs to use causality. There are many cases where useful information can be determined without assigning causality of any kind. Causality is just one concept discussed in CEP.
If causality is used, Luckham defines the Cause-Time Axiom: "If event A caused event B in system S, then no clock in S gives B an earlier timestamp than it gives A." That's from PoE 5.3.1[1]. Causality may be easier to model within a system that obeys this axiom.
Causality, clouds and streams
Some CEP literature refers to the causality Poset (described above) as an event cloud. However, this use of cloud to mean Poset is not consistent and often the word "cloud" simply refers to all events.It has been said that if a Poset is a cloud of events, then a Toset must be a stream of events. Note, however, that Toset in this case is not with respect to the causality operator, but to some kind of ordering such as by time. A Toset of causality is unlikely in real life because it would mean that every event in the set either causes or is caused by every other event of the set. Usually, this is far from true.
This definition makes the cloud and stream discussions confusing (author's opinion). Half of the time, the discussion will be about Posets formed by causality, the other half about Tosets formed by ordering by time. In the end, these two definitions of cloud an stream do not meet up in the area of causality, although with respect to ordering by time they do meet up (again, author's opinion)
Design patterns: EPL, rules, agents, event processing networks
More content will be added here, hopefully in the near future.Event Processing Language (EPL)
A programming language designed specifically to process events. There are many kinds of EPL. EPLs range from languages that look much like C++ but have language constructs specific to event processing, to SQL-like languages (see streaming SQL below) to drag-and-drop graphical languages that express the semantic relationships between events.There are many opinions on which EPL is "the right one". Opher Etzion has a good summary of that debate in this post on his blog.
Examples of CEP
Content to be added here (soon?).CEP software
The term CEP has been adopted by various software vendors as a broad classification of thier capabilities. This lends itself to ideas like "implementing CEP" in the same sense as "implementing an SOA" or "implementing an EDA."These has been much debate about what kind of software is "truly CEP" versus some other designation. David Luckham has been liberal in allowing the use of the term CEP to be applied to pretty much any software that processes events, network messages or does soft or hard real-time computing. He is carful to note that in every case, the software should be considered to implement concepts or patterns included in CEP, but that no software "is CEP" or defines CEP.
Generalities about "CEP software"
Calling something "CEP software" is like calling something "event-driven architecture software" in that the term does not tell you much about what the software does. Most of the software that bills itself as "CEP" provides a language (EPL) that is customized to work to work with events, in soft real-time. This language is usually compiled and run by a server that receives events and does the actual work.The EPTS
Many vendors of "CEP software" have joined together along with academia and customers to form the Event Processing Technical Society. From there, they coordinate the interaction with various research projects as well as meetings on CEP language standards, reference architectures and more.
Streaming SQL
There are many languages and types of languages offered by "CEP software", but the one that most people land on first is streaming SQL. People land on streaming SQL because it is neat, new and intuitively useful. But it should be said that not everyone agrees on whether it is good or "the best choice" for processing events. As a rule, vendors of streaming SQL products think it's great, while vendors of competing products that do not use streaming SQL think it's bad.Streaming SQL is loosely based on traditional SQL. Different vendors diverge from SQL in different ways. The best way to learn about this topic is to take a look at all the available products (which, thankfully, are available for download).
A few streaming SQL vendors:
- StreamBase
- Coral8
- Aleri
- EsperTech - the only open source product
Other CEP vendors
This list is taken directly from the blog of Marco Seiriö , who may choose to maintain a more up to date list.- Agent Logic
- Aleri
- Amit
- AptSoft (Now IBM)
- Coral8
- EsperTech
- Event Zero
- Gemfire
- Herald Logic
- iSpheres - RIP
- Kaskad Technology - RIP
- Progress Apama
- Red Rabbit Software
- RiverGlass
- RTM Realtime Monitoring
- ruleCore CEP Server
- SeeWhy
- Senactive
- Skyler Technology, Inc.
- StreamBase
- Syncron
- Syndera
- Tibco BusinessEvents
- TriggerWare
- WebSphere Business Events
- WestGlobal
CEP on the web
Articles
- RFID Pill Monitors Body Temperature at Walking Race
- Plenty more coming
Sites
- Complex Event Processing web site
- CEP forums
- The Event Processing Technical Society
- Plenty more coming here too
Definitions of CEP
Part of my reason for writing this Knol was to combine the various definitions of CEP from around the web.- The Event Processing Glossary (contains an entry for Complex Event Processing)
- CEP on Wikipedia
- CEP on the IT toolbox Wiki
- What is CEP? on The CEP Blog
Open Source Initiatives
Esper
Intelligent Event Processor
Extremely Reusable Monitoring API (ERMA) and Graphite
- Press: Orbitz Open Sources Monitoring Tools ERMA and Graphite
- Press: Orbitz paves the way to enterprise open-source contributions
- https://launchpad.net/erma
http://erma.wikidot.com/ - https://launchpad.net/graphite
http://graphite.wikidot.com/
Disclaimer
Many of my (the author's) academic and other colleagues complain that the terms used in CEP are a hodge-podge from other disciplines. Yes, this is partially true, CEP pulls together many fields of study.I (the author) wrote this knol as an excercise in writing knols, I didn't think any of it up and I don't endorse any of it. So complaining to me about how you don't like the ideas in CEP is mostly useless. Feel free to head over to the CEP forum and give them an ear full.
But if you love this concept, then I personally invented all of it and you may feel free to heap praise on me for my genius. Kidding. Still, head over to the forums and interact with the community.
Help edit this knol
Please feel free to make constructive changes to this knol, I will be liberal with approving all constructive changes.
Comments
Write New Comment ▼
Write New Comment
Sorry! This knol's owner(s) have blocked you from editing, making suggestions, or commenting here.