Minikube on VirtualBox Ubuntu

My primary goal was to try Kubernetes on VMs the simplest way. I have chosen minimal version of k8s called minikube and used a Virtualbox VM with Ubuntu 16.04. minikube creates a single-node cluster.

There is a problem with minikube on VirtualBox VM: minikube uses VirtualBox hypervisor for virtualization and thus requires VT-X support on host. However VirtualBox doesn’t support VT-X/AMD-v. That could be solved by using vm-driver=none option and Docker on host.

Firstly we need to install Docker following this guide. Install kubectl:

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.9.0/bin/linux/amd64/kubectl

Make kubectl executable and move it to system path:

sudo chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

Now install minikube:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.1/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

minikube version

Start minikube with vm-driver=none:

sudo minikube start -vm-driver=none

Here is an issue: root privileges are required to use minikube and kubectl (‘minikube start’ command outputs “When using the none driver, the kubectl config and credentials generated will be root owned and will appear in the root home directory”). To fix it run:

export CHANGE_MINIKUBE_NONE_USER=true

sudo -E minikube start -vm-driver=none

Let’s make the fix persistent so it will apply on system bootload:

echo ‘export CHANGE_MINIKUBE_NONE_USER=true’ >> ~/.bashrc

Check that everything works properly:

kubectl cluster-info

Microservices using Spring Netflix stack

Here is the presentation on Microservices.
Below blog covers basic concepts of Monolithic vs Microservices.

Microservice Presentation

Github Code
Below Git code has below stuff



Github Code

My Favorite Technical books on Designing Data intensive applications

I mostly follow Martin Kleppman and  Adrian Cockroft.

Below are my favorite books and blogs

Designing Data-Intensive Applications

(Download pdf )

https://github.com/bjut-hz/E-Books/blob/master/distributed%20system/Martin%20Kleppmann-Designing%20Data-Intensive%20Applications_%20The%20Big%20Ideas%20Behind%20Reliable%2C%20Scalable%2C%20and%20Maintainable%20Systems-O%E2%80%99Reilly%20Media%20(2017).pdf

Martin Kleppman Blog

https://martin.kleppmann.com/archive.html

Introduction to Reliable and Secure Distributed Programming

http://www.beck-shop.de/fachbuch/leseprobe/9783642152597_Excerpt_001.pdf

https://books.google.co.in/books?hl=en&lr=&id=Y8lHVFeJ6EQC&oi=fnd&pg=PR3&dq=introduction+to+reliable+and+distributed+secure+programming&ots=uE-MgsMPM7&sig=KJj7-3Frep4eq4uxJ0a3d8cI2iA#v=onepage&q=introduction%20to%20reliable%20and%20distributed%20secure%20programming&f=false

Cassandra Indexing

How Cassandra works?

Cassandra all about indexing.  Good for write heavy.

Cassandra can be thought of as a key-value database it actually contains a lookup key for every data in the form of a primary key

Primary Key:    (mandatory for create/read)

• Primary Key is mandatory
• Primary key is immutable.

  PRIMARY KEY (partition_key):

  A partition key will always belong to one node in a cluster and that partition’s data will always be found on that node.

We can never access the second-level data (for instance, the email of a user) without accessing the primary username key first.

CQL (Cassandra Query Language) to get email would be

SELECT "email" FROM "user_tweets" WHERE "username" = 'jochasinga';      //Primary key mandatory to pass along with query

Different other form of Primary Key:

PRIMARY KEY ((partition_key), primary_key1, primary_key2)

Primary keys 1 & 2   are also called cluster columns

Where the partition key is important for data locality (points to a node in a cluster), the clustering column specifies the order that the data is arranged inside the partition

Association looks like this in a map    PARTITION KEY -> PRIMARY KEY -> DATA

Map [String, Map[String, Data]].   //keys of a map are unique

More information

https://www.datastax.com/dev/blog/the-most-important-thing-to-know-in-cassandra-data-modeling-the-primary-key

https://medium.com/@jochasinga/cassandra-schemas-for-beginners-like-me-9714cee9236a

SELECT * FROM "user_tweets" WHERE "email" = 'jo.chasinga@gmail.com';     // Throws Error. Reason no primary key passed along with query. 

Cassandra introduced secondary index in order to address above query get email without primary key.

Query using secondary index (with no primary key) will span across all nodes in a cluster.  (FanOut)

Cassandra supports secondary indexes on column families where the column name is known (Not on dynamic columns)

More Information on Secondary Index:https://pantheon.io/blog/cassandra-scale-problem-secondary-indexes 

Important links:

https://medium.com/@jochasinga/cassandra-schemas-for-beginners-like-me-9714cee9236a

http://outworkers.com/blog/post/a-series-on-cassandra-part-2-indexes-and-keys

Advantages with Cassandra:

•   Supports write operations at massive scale 
•   Highly Available
•   Elastic scalability  (new nodes can be added to cluster seamlessly)

Key information on Cassandra

•  Schema has to define as per Cassandra documentation
•  Choosing right primary key and clustered keys will be a challenge for MO data set. 
•  Values of Primary keys in cassandra has to be immutable.  ( update not allowed on primary key )
•  Mandatory to provide partition key and cluster key for read operation
•  Secondary indexes wont scale with Cassandra as they work with out partition key (Fan out pattern)
•  Secondary indexes applicable only on known columns (static columns)
•  Aggregations in Cassandra are not supported by the Cassandra nodes

Scan ports of hosted applications

Below command can run to get all open ports on a host

This will help to understand which port is open and why. Compare with other websites to get more insights.

Kafka Performance Tuning

We all know the power and advantages of KAFKA. Apache Kafka is publish-subscribe messaging system which basically has three major components

KAFKA CONSUMER

KAFKA PRODUCER and

KAFKA BROKER

                                                                                                                                                      

Broker Side Configuration

Producer side 

Compression

Batch size

Sync or Async

Consumer Side configuration

Fetch Size

Applicable for all consumer instance in a consumer group.

Broker Side Configuration

num.replica.fetchers

This configuration parameter defines the number of threads which will be replicating data from leader to the follower. Value of this parameter can be modified as per availability of thread. If we have threads available we should have more number of replica fetchers to complete replication in parallel.

replica.fetch.max.bytes

This parameter is all about how much data you want to fetch from any partition in each fetch request. It’s good to increase value for this parameter so that it helps to create replica fast in the followers

replica.socket.receive.buffer.bytes

In case of less thread available for creating replica, we can increase the size of buffer. It will help to hold more data if replication thread is slow as compared to the incoming message rate.

num.partitions

This is the very important configuration which we should be taken care while having Kafka in live. As many partitions are there, we can have that level of parallelism and write data in parallel which will automatically increase the throughput

Having more partitions slows down performance and throughput if the system OS configuration can’t capable of handle it.

Creating more partitions for a topic depends on available threads and disk.

num.io.threads

Setting value for I/O threads directly depends on how much disk you have in your cluster. These threads are used by server for executing request. We should have at least as many threads as we have disks.

Producer:

Compression

compression.codec

Compression reduces  disk footprint leading to faster reads and writes.Currently Kafka supports - Values are 'none', 'gzip' and 'snappy'

Property values are none, gzip and snappy.

Batch Size

Batch.size measures batch size in total bytes instead of the number of messages. It controls how many bytes of data to collect before sending messages to the Kafka broker. Set this as high as possible, without exceeding available memory. The default value is 16384.

If you increase the size of your buffer, it might never get full. The Producer sends the information eventually, based on other triggers, such as linger time in milliseconds.

Batch size always confusing what batch size will be optimal. Large batch size may be great to have high throughput but you might feel latency issue in that. So, we can conclude that latency and throughput is inversely proportional to each other.

Async Producers

Publish the message and get the callback to get the acknowledgement of send data status.

'producer.type=1' to make producer async

'queue.buffer.max.ms =duration of match window

'batch.num.messages" = number of messages to be sent in batch.

Large Messages  

Consider placing large files on the standard storage and using Kafka to send a message with the file location. In many cases this can be much faster than using Kafka 

to send the large file itself.

Monitoring & Alerting for Micro Services

I got opportunity to design and develop Monitoring and Alerting framework for all the micro services deployed in the organisations. Monitoring majorly classified into 3

1.   System Monitoring
2.   Application Monitoring
3.   Server Monitoring 

Any abnormolity on any of the above 3 will raise an alert. Alert can be   "Slack Notification",  "Email",  "Pager" and "Call".

NOTE: This is not for micro services tracing. For micro services tracing sophisticated open source tools available like Jaeger and  zipkin

 Monitoring & Alerting Tools used:

   Sensu    

  Uchiwa  

  Pagerduty 

Technologies:

 Fluentd           -  Used for data/log forwarding

OpenTSDB      - Used for Time Series Data. Replaces old RRD tools
                          ( competitors would be InfluxDB, Druid, Cassandra)

ElasticSearch   -  Used for log search (Ex:  "ERROR"  count > 1  on app1 raise an alert)

Monitoring Types:

System Monitoring:  
        includes CPU, Disk, I/O, Processes,Virtual Memory, DHCP, Network etc.
Application Monitoring : 
        includes Failed Services, Batch/Cron jobs, Cache monitoring,
        DB monitoring, transaction, 3rd party interactions etc.
Server Monitoring:  
       Apache Tomcat, Ngnix, HAProxy , Request/Response latency, Server health etc


Scripts has to write for all of the above monitoring modules. Scripts can be written in Ruby where Sensu has many sensu plugins which will help to have less number of lines in scripts.

Deployment Topology:


All scripts has to upload to chef server with version. Project configuration and other artefacts will be uploaded to chef-server.

Chef-client can be run from development machine/laptop.

Micro service will be up and running with all monitoring scripts, configuration files, application jar,fluentd, opentsdb agents.
Below shows final Java micro services which will be up and running.

Detail explanation:

Flow - (a)
Sensu agent runs on Micro service. Agent periodically executes scripts(& server monitoring) and sends output to sensu server.
Sensu Server  aggregates data from the micro services.  Sensu forwards alerts to PagerDuty if any threshold breach by a sensu metric.
Uchiwa is the dashboard for Sensu. It gives nice alerts view in order Data centre, VMs, Metrics.

Use Cases:  CPU, Disk, File I/O, Server health etc.

Flow - (b)
Fluentd agent runs on micro service is a data forwarder. Fluentd listens on a application log file path and forwards data to Elastic Search.
ElasticSearch does indexing for the log data. Error count cron job runs on elastic search which does search on "error" count on logs group by application. If there is any error on the log script forwards to sensu server which in turn converts to Pager Duty alert.

Use Cases:  Server access logs( tells how many 401 Rest codes group by Region and application,   Application logs error count group by Region and application)

Flow - (c)
This will be very interesting use case. I have used lot of time series metrics forwarded to OpenTSDB. But I would like to mention metric which helped a lot. REST call requests are recorded as time series metrics.  Example:   (Rest EndPoint, timestamp,  hitCount) .

Use Cases:
For every rest call OpenTSDB client on micro-service sends data to OpenTSDB server.   OpenTSDB graph shows traffic group by Region and Date. Which helps to understand how HTTP traffic on each data centre.


Example:
 How to use sensu checks.

Below few checks on RAM & Disk

sensu_check 'ram' do
  command 'check-ram.rb -w 20 -c 10'
  interval node['monitor']['check_interval']
  subscribers %w[all]
end

sensu_check 'disk' do
  command 'check-disk.rb'
  interval node['monitor']['check_interval']
  subscribers %w[all]
end

Above ruby script raises alert if Ram breaches threshold.  File  "check-ram.rb" will be available at Checf cookbooks as default file
https://github.com/sensu-plugins/sensu-plugins-memory-checks/blob/master/bin/check-ram.rb

Subscribe: is for notification.

Data pipeline design for Mobile Data Traffic using AWS

I worked for a Mobile Operating System Company which has 50 Million + user base across the globe. All these mobile generates a billion data per day.

What kind of Mobile data ?
Every user activity on the mobile is treated as a data.  Below are few examples
a) Install/Uninstall app
b) Open an app
c) Closing an app
d) Total time spend on an app
e) Network connectivity details
f) Heart beat  (Contains  OS build no, single sim/dual sim, network operator names)


Problem:
Organisation needs data collection, data pipeline and analytics for all the mobile data traffic.


Solution:
I have leveraged AWS services(EC2, Route 53, ELB, EBS, S3, Lambda, Redshift) for implementation. Not just pipe line design and analytics also implemented robust monitoring and alerting system for the entire pipeline. Also took approaches which will minimise operational cost of AWS. Lot of factors tested(Apache Benchmark testing) and optimised while implementation.

Overall Architecture

STAGE - 1   (DATA Collection & Data Enriching)

Data Collection (Shopvac):  Code name is Shopvac
  Shopvac service should be front facing, low latency and high throughput.  Shopvac Service is hosted on a EC2. Below is the network topology diagram before i go deep about service.

Mobiles:
    All mobiles post data using  HTTP POST on a host where DNS resolves host to a AWS Route 53.

AWS Route 53:
   Route 53 does simple redirection to AWS ELB.

AWS ELB:
     ELB has list of services where java process is running for data collection. In Non Amazon world the same has to be done using  Zookeeper/Etcd for Service discovery.

Shopvac Service: (Data Collection)
    Its a light weight java process running. It has a REST END Point which will be listening for mobile POST data.  End point stores data to local file system ( /var/app/shopvac/metric/<eventname.json>) .

This file system acts as a buffer. FluentD will listen on this path and forward data to Amazon S3.
FluentD forwards when ever file size reaches 100MB size or 5 secs.

Fluentd is an Open Source log forwarder like Logstash, SysD, CollectD, RSyslog etc.
Fluentd parses data for event name and creates a file with event name if not exist otherwise append data to the file.

Below Shopvac Service insight

Every service developed in organisation has to be bootstrapped with chef with above process running.

Java Process(Vertx) : Used Vertx Async programming. Vertex has Rest End point which will get invoked on mobile POST. Data will be enriched for example stores City, State and Country information(gets from Latitude and Longitude) to the existing metrics.

Java process stores data on local file system.

FluentD:  Fluentd listens on local files system where metrics are stored. Fluentd forwards mobile data to AWS S3.  Also forwards logs to ElasticSearch.

OpenTSDB Client:  All time series data are written to OpenTSDB. One most important data we store in OpenTSDB is REST End point call.  When ever REST point is invoked it is stored in time series data.  This will give insight how many times Rest end pint is invoked per day, per week, per hour this will give traffic insight. What duration has peak traffic on the server. 

Sensu Client :  Sensu scripts will be deployed along with chef. Basic scripts like Disk utilisation, RAM utilisation, Server health a lot other metrics sense will report to sense server. On Abnormality or threshold breach Sense  Server will alert through Pager Duty.

STAGE - 2   (Data Processing)

AWS-S3 will be source of truth. All types of metrics will be forwarded blindly to S3.

AWS Lambda (Serverless Architecture)
will listen on S3 bucket and filters for interested metrics. Forwards interested metrics to RedShift. Instead of directly forwarding to Redshift forwards to CMET service.(developed as a proxy service internally).

Note: Lambda will be charged based on cpu cycles it spent with the code. So code on lambda should be as less as possible and also should be error free. It is difficult to debug Lambda.

Note(Non-Amazon World):
S3 has to replace with Kafka
Lambda replace with Kafka Consumers/Storm

STAGE-3  (DATA Storing)

Selected Redshift as Storage because of its advantages.

Lambda can directly forward metrics to Redshift but this will not be a right approach. As Data Warehouse will be exposed to events which are abnormal in behaviour and also every DB comes with concurrent DB connections at a time. There is a need for intermittent service which acts as a proxy to Redhisft. So CMET is a service which acts as a proxy and do connection management and dropping long holding connections.

One good use case of CMET:

1.  Long running queries will be stopped and will be marked as failure. So that lambda will retry. This helps when Redshift can't respond nor process at that point of time. Also helps reduce load on Redshift.
2.  Exposing read connections(GUI Visualization) for set of users and write connections for users like AWS like Lambda and Admin.

 

HTTPS

Below content contains
1) What is HTTPs
2) Server Authentication
3) Important check list to follow for having secure website with good performance.
(web site links to check your server security online. Moazilla standards for configuring TLS on server)
4) Reasons for insecure communications over https.


Even static data should be encrypted. That's the best way to keep website secure.

In reality all the below 3 give secure website.
1)Authentication
2)Data Integrity (Data doesn't change between client and server)
3) Encryption  (can any one see my conversation)


All the above 3 are taken care by Transport Layer Security(TLS)

HTTPS ==>    HTTP running on top of TLS

    HTTP   (http running on top of TLS)
    TLS  
    TCP
     IP

Do we need to encrypt of all of the web data ?
My answer would be yes.

why we need to ?
For example casual surfing  at restaurants with out https can give information to
hackers that what sites being visited if it is financ.yahoo.com. what shares
you are interested etc.

Hackers can change text, password etc if client doesn't connects to right Server. That's the reason we need Server Authentication.
1) If client wants to connect to the right server. Client has to connect over
https. Over https browser downloads server signed certificate(public key) this gives guarantee that client is connected
to right server.


Advantage of TLS
1) Passive and Active attackers cant listen in because we are encrypting the data.
2) Attacker cant tamper as data is check suming.
3) attackers cant impersonate.


Configuring TLS
1) Arent Certificate expensive
2) wont it make server and site slow ?
3) what are the configuration best practices

Important check list need to follow in the order
1) Get a 2048-bit TLS Certificate
2) Configure TLS on your servers.
3) Verify TLS server configuration
4) Monitor performance: resumption rate etc.
5) Tune Server configuration. Cache etc.
6) Investigate SPDY & HTTP2.0

1)Get a 2048-bit TLS Certificate
 If there is any 1024-bit certificate on server better to migrate to 2048-bit.
Certificates are below types.
a) Free certificates  ( which are for non commerfical use from StartSSL)
b) Single host     ( google.com)
c) Multi-domain   (google.co.in, google.co.us, google.co.uk)
d) Wildcard       (*.mysite.com)

2) Configure TLS on your servers
More about Server Side TLS configuration in the blow link
https://wiki.mozilla.org/Security/Server_Side_TLS


3) Verify TLS server configuration
 How to verify TLS Server configuration (Qualys provides online to test Server, browser etc)
https://www.ssllabs.com/ssltest/
It gives score and useful tips. Before you access any website you can check that site security aspects using this tool.

4)   Monitor performance: resumption rate etc
Usually cryptography stuff consumes more CPU. Modern CPUs are designed to handle huge data traffic over TLS
Assymetric cryptography - verify the public certificate and do public crypto  (This one is expensive)
Symetric Cryptography - how we encrypt the application data

5) Tune Server configuration
Using HTTP Keep alive and session resumption doesn't require full handshake. So handshake doesn't dominate CPU Usage.
.

6) Google developed a protocal SPDY which gives better page load performance over regular https connections.
SPDY1&2 not only improves client performance also does on Server. SPDY allows single connection to server instead of many
connections to server. Single connection means few handshakes, fewers sockets, few buffers to allocate. SPDY consumes less memory but
more CPU and also fewer worker threads.

Few more reasons for insecure communication.

Few reasons for broken cert between client and server. Developer  points
1) Incorrect host name return by server in the cert.
2) Incomplete Certificate Chain
3) Expired Certificates.

Insecure references 
Some secure websites having javascript/css code like below.
<script src="http://aaa.com/script.js"></script>
Some browsers wont allow(http:)  type of communication. This script is blocked will not
execute. If browser allows also it is secruity leak.

Use Protocol relative URI's. Protocol relative uri's will be
<script src="//aaa.com/script.js"></script>

Even secure website can have insecure hrefs
<a href="http://abc.com"/>
use Protocol relative urls
<a href="//abc.com"/>

Insecure re directions are expensive
1) https -->redirect to --> http --> again redirect to https


HSTS (HTTP strict transport security) eliminates  HTTP--> HTTPs redirects (costly operations)

Server can return with this header when returns a page.
Strict-Transport-Security: max-age=20491234; includeSubDomains

max-age in seconds. Remember this policy(HSTS) for this many seconds.
includeSubDomains is optional. says remember this policy for all the sub domains.

Java7 NIO WatchService

Java 7: NIO WatchService

In my previous post I have written many of the Java 7 features for developing Java Telnet Server.
Here I want to go deep into one such feature WatchService.

The WatchService is a very interesting feature of the new java.nio.file package in Java 7.
One of the more interesting is the WatchService, adding the capability to watch a directory for changes.

   

The WatchService maps directly to the native file event notification mechanism, if available. If a native event notification mechanism is not available, then the default implementation will use polling. As a result, the responsiveness, ordering of events and details available are implementation specific.

 

Source Code:

https://bitbucket.org/nkancharla/javaexamples/src/611caf00aed3/Examples/src/java7NIO/MonitorDir.java?at=master

Watching Directory:
We have to create a File Path for the specific directory we want to watch.
Path:
    Path faxFolder = Paths.get("C:\\foo");

The Path interface implements the register method that takes a WatchService object and varargs of type WatchEvent.Kind as arguments. There are 4 events to watch for:

ENTRY_CREATE

ENTRY_DELETE

ENTRY_MODIFY

OVERFLOW

While the first 3 types are easy to understand, OVERFLOW indicates that events may been lost or discarded. A WatchService is created by calling FileSystem.newWatchService().

Watching a directory is accomplished by registering a Path object with the WatchService:

WatchService watchService = FileSystems.getDefault().newWatchService();

 

faxFolder.register(watchService,   StandardWatchEventKinds.ENTRY_CREATE,  StandardWatchEventKinds.ENTRY_DELETE, 
StandardWatchEventKinds.ENTRY_MODIFY);

WatchKey watchKey = watchService.take();

As you can see above code register method returns a watchKey.  The WatchKey is a token that represents the registration of the Path with the WatchService.

The WatchKey

As a result of the registration process, the WatchKey is in a ‘ready’ state and is considered valid. A WatchKey remains valid until one of the following occurs:

1. WatchKey.cancel() is called.
2. The directory being watched is no longer available.
3. The WatchService object is closed.

Checking For Changes
When a change is detected, the WatchKey state is set to ‘signaled’ and it is placed in a queue for processing.  Getting WatchKeys off the queue involves calling WatchService.poll() or
WatchService.take().

Here is a basic example:

WatchKey watchKey = watchService.poll(60,TimeUnit.SECONDS);
 //this will retrieve all the events for this watch key

List<WatchEvent.Kind<?>> events = watchKey.pollEvents(); 

boolean value = watchKey.reset()

The "reset" method sets the WatchKey state back to 'ready'(meaning listen for events) and returns a boolean indicating if the WatchKey is still valid. If there are any pending events, then the WatchKey will be re-queued, otherwise it will remain in the ready state until new events are detected.

How to process events

Now that we have detected an event, how do we determine:

1. On which directory did the event happen? (assuming more than one directory registered)
2. What was the actual event? (assuming listening for more than one event)
3. What was the target of the event, i.e what Path object was created,deleted or updated?

//WatchKey watchable returns the calling Path object of Path.register
 Path watchedPath = (Path) watchKey.watchable();
for (WatchEvent<?> event : watchKey.pollEvents()) {
   
   //returns the event type
     StandardWatchEventKinds eventKind = event.kind();
  
   //returns the context of the event
     Path target = (Path)event.context();
}

There are two things that about the WatchService.
The WatchService does not pick up events for sub-directories of a watched directory.
We still need to poll the WatchService for events, rather than receive asynchronous notification.

 Here is the sample output:

RabbitMQ using Java

RabbitMQ is an open source message broker which implements AMQP messaging protocol. It also has multiple client libraries (Java, .NET, Erlang) which can be used to send/recieve AMQP messages to/from an AMQP broker.

It's based on Erlang/OTP which gives very good scalability. Finally, you should also look at QPID, an implementation by the Apache Foundation which is supported by Red Hat and Microsoft, and OpenAMQ.

In this post I’m going to explain how to use the RabbitMQ Java Client Library to send and Receive messages. Since RabbitMQ also can act as an AMQP server, I’ll be using it in this post.

It doesnt require any server.

What is AMQP?AMQP (Advanced Message Queuing Protocol) is a networking protocol that enables conforming client applications to communicate with conforming messaging middleware brokers.
Brokers and Their RoleMessaging brokers receive messages from producers (applications that publish them) and route them to consumers (applications that process them).
Since AMQP is a network protocol, the producers, consumers and the broker can all reside on different machines.

It implements a broker architecture, meaning that messages are queued on a central node before being sent to clients. This approach makes RabbitMQ very easy to use and deploy, because advanced scenarios like routing, load balancing or persistent message queuing are supported in just a few lines of code. However, it also makes it less scalable and slower because the central node adds latency and message envelopes are quite big.

Message Queue Servers
Message queue servers are available in various languages, Erlang (RabbitMQ), C (beanstalkd), Ruby (Starling or Sparrow), Scala (Kestrel, Kafka) or Java (ActiveMQ).

Sparrow

• Sparrow is a lightweight queue written in Ruby that “speaks memcache”

Starling

• Starling is a Message Queue Server based on MemCached
• written in Ruby
• stores jobs in memory (message queue)

Kestrel

• Starling clone written in Scala (a port of Starling from Ruby to Scala)
• Queues are stored in memory, but logged on disk

RabbitMQ

• RabbitMQ is a Message Queue Server in Erlang
• stores jobs in memory (message queue)

Apache ActiveMQ

• ActiveMQ is an open source message broker in Java

Beanstalkd

• written by Philotic, Inc. to improve the response time of a Facebook application
• in-memory workqueue service mostly written in C
• Docu: http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue

Kafka

• Written at LinkedIn in Scala
• Used by LinkedIn to offload processing of all page and other views
• Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)
• Supports both on-line as off-line processing

 HornetQ is also an option you can look into, it is JMS Complaint, a better option than ActiveMQ.

Nice working Example is given in below link

http://keyholesoftware.com/2013/04/01/rabbitmq-with-spring-tutorial/#comment-3663

Source Code Link

https://bitbucket.org/nkancharla/javaexamples/src/8a0a66ab66092c468524d20c9004bdc67bfcf04a/RabbitMQ-Proj?at=master

Core Java to create a RabbitMQ connection. It is documented very well in RabbitMQ site.

create a connection to the server:

       ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();The connection abstracts the socket connection, and takes care of protocol version negotiation and authentication and so on for us.

we must declare a queue for us to send to; then we can publish a message to the queue:

        private final static String QUEUE_NAME = "hello";

       channel.queueDeclare(QUEUE_NAME, false, false, false, null);
    String message = "Hello World!";
    channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
    System.out.println(" [x] Sent '" + message + "'"); it will only be created if it doesn't exist already.

close the channel and the connection;

        channel.close();
    connection.close();

 Reciever

        ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();

    channel.queueDeclare(QUEUE_NAME, false, false, false, null);
    System.out.println(" [*] Waiting for messages.
   
    QueueingConsumer consumer = new QueueingConsumer(channel);
    channel.basicConsume(QUEUE_NAME, true, consumer);

    while (true) {
      QueueingConsumer.Delivery delivery = consumer.nextDelivery();
      String message = new String(delivery.getBody());
      System.out.println(" [x] Received '" + message + "'");
    }