Integrating services with Apache Camel

One of our clients came not long ago with a problem: a complex system with lots of information exchange between remote services… We simply did not know how to handle such situation in a neat and simple manner. We had to consider the different potential APIs and their protocols, what would be easier to integrate into existing blocks, clients implementations and their development status, maintenance… What was worse is that probably nobody would even understand the resulting architecture nor be able to maintain it easily. But after some head-scratching we found a lovely theoretical solution: messaging patterns, or, more precisely, Enterprise Integration Patterns (or EIPs for short).

Why are EIPs useful?

Usually enterprise environments are made of a bunch of services that must interact between them to provide another service, let’s say an integrated service. Thinking through this integration used to take lots of time and effort from engineers. But someone came up with an idea.

Yep, a great idea: well thought patterns to address these exact integration problems. EIP had just born.

EIPs are useful to avoid re-inventing the wheel by sticking to existing standards to produce better software in an simpler manner. Software that is more readable, more intuitive, more flexible, more documented.

More about EIP?

For EIP, information between different services has the shape of messages. If there are messages in the picture, there must be also producers and consumers. Along their way to the consumers, messages pass through different elements: queues, processors, routers, among others. These transform messages, route them, creates additional messages, etc. For instance this is how a route looks like.

This is a very simple example. There is a producer of messages, that will go through a translator element, and a router that will set their way to reach either Queue A or Queue B.

EIP establishes very intuitive patterns that allow interconnection of different services, from message producers until message consumers. Some popular known patterns are publish-subscribe, pipes and filters (or pipeline). There are some other lesser known patterns, for instance the very helpful Dead Letter Channel that specifies a way to treat messages that for some reason failed to be delivered.

Too theoretical, how do I proceed with my integration?

If you like making your own way through the jungle, you just need to read Enterprise Integration Patterns by Gregor Hohpe (2004) and implement whatever EIPs you need. Sticking to existing patterns is always a great idea for many reasons.

However, for those who like highways, there are is a very good framework I would like you to befriend. Camel: the reader. The reader: Camel.

How does Camel fit with my existing software? How could we use all this if our system already uses existent technologies? Adapting Camel to your scenario might not be as painful as you think.

What is really Camel?

Camel is an open source integration framework based on EIPs. Apart from providing the possibility to implement lots of EIP patterns, you can benefit from their more than hundred connector components, which makes it really easy for developers to integrate it to existing software.

Technologies supported by Camel?

More concretely, a component allows you to connect your EIP to specific endpoints technologies. For instance you use a local storage File component to generate one message per file in a given directory of your filesystem. It usually works two-ways for most of the components, which means that for instance using the same File component you can also dump messages into files in your filesystem.

Similarly, there are components for remote storage: FTP, FTPS, Dropbox and even S3 (Simple Storage Service) Amazon Services!

If planning to use queueing technologies, you’re not out of luck because Camel supports JMS and AMQP queues, and also AWS-SQS (Amazon Simple Queue Service).

Messages could be also generated from / dumped to items on different databases: SQL, Cassandra, Elasticsearch and CouchDB.

Some management connectors include JMX, SSH. Also there are some crazy connectors you might want to take a look at: Facebook (to access Facebook API), Docker (to communicate with Docker), Exec (for executing system commands).

What else?

Camel is an open source Apache project, developed in Java over Spring framework. Since its first commits in 2007, it is actively being improved in this repository by a growing community (now including me!). Don’t miss the chance to give it a try!

Happy integration!