Installing Kafka on Mac OSX

Apache Kafka is a highly-scalable publish-subscribe messaging system that can serve as the data backbone in distributed applications. With Kafka’s Producer-Consumer model it becomes easy to implement multiple data consumers that do live monitoring as well persistent data storage for later analysis.

STEP 1: Installation, the best way to install the latest version of the Kafka server on OS X and to keep it up to date is via Homebrew.

1
$ brew install kafka

this will also install all dependencies, like zookeeper which is required to run the server. (more…)

Read More

Install Rabbit MQ 3.3 on CentOS 7

This is a step by step guide to install the Rabbit MQ 3.3.5.1 for the series of topics about AMQP messaging. Rabbit MQ support most of the Linux distributions, Mac OS and MS Windows. I would demonstrate it on CentOS 7 as an example.

1. Install compiler and related if  necessary

# sudo yum install gcc glibc-devel make ncurses-devel openssl-devel autoconf

2. Update latest EPEL

# wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-1.noarch.rpm
# wget http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
# sudo rpm -Uvh remi-release-7*.rpm epel-release-7*.rpm

3. Install Erlang

# wget http://packages.erlang-solutions.com/erlang-solutions-1.0-1.noarch.rpm
# yum install -y erlang

Type “erl” to verfiy if Erlang is installed correctly.

4. Install Rabbit MQ

# wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.3.5/rabbitmq-server-3.3.5-1.noarch.rpm

Add the necessary keys for verification by

# rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc

Install with command

# yum install rabbitmq-server-3.3.5-1.noarch.rpm

Issue command to turn on the web UI plugin

# sudo rabbitmq-plugins enable rabbitmq_management

Change permission

# chown -R rabbitmq:rabbitmq /var/lib/rabbitmq/

Issue following command to start the server (*1)

# /usr/sbin/rabbitmq-server

5. Setup admin user account
In /usr/sbin, create new user “mqadmin” by

# rabbitmqctl add_user mqadmin mqadmin

Issue following command to assign administrator role

 

# rabbitmqctl set_user_tags mqadmin administrator

Issue command ‘rabbitmqctl set_permissions -p / mqadmin “.*” “.*” “.*” ‘ to grant permission

# rabbitmqctl set_permissions -p / mqadmin ".*" ".*" ".*"

Now you can access the web admin by http://host:15672/ (*2)
Remark:

(*1) You may not start the server with command “service rabbitmq-server start” successfully as documented in official manual, please read this link to resolve https://groups.google.com/forum/#!topic/rabbitmq-users/iK3q4GLpHXY

(*2) The default guest/guest user account can only be accessed via localhost

Read More

Installing Python scikit-learn package

scikit-learn is a Python module integrating classic machine learning algorithms in the tightly-knit world of scientific Python packages (numpy, scipy, matplotlib). It aims to provide simple and efficient solutions to learning problems that are accessible to everybody and reusable in various contexts: machine-learning as a versatile tool for science and engineering.

Step 1: Dependencies

First, we need to install all dependencies for it:

# yum update && yum install scipy.x86_64 numpy.x86_64 python-devel.x86_64 python-matplotlib.x86_64 python-pip.noarch gcc-c++.x86_64

Step 2: Installing scikit-learn using pip

Now, I will use the fastest way to install the package using pip:

# pip-python install scikit-learn

This command should give a output very similar to this:

# warning: no files found matching 'test.py' # warning: no files found matching '*.TXT' under directory 'sklearn/datasets' # Installing /usr/lib64/python2.7/site-packages/scikit_learn-0.9-py2.7- # Successfully installed scikit-learn Cleaning up...

Step 3: Runing the examples

The final step is to run several examples provided here

 

Read More

Mesos Installation in CentOS 7.0

This post will walk through setting up a cluster which includes Apache Mesos.

Highly-available clusters will typically have multiple master nodes and any number of slave nodes. Each master node runs Apache Mesos and ZooKeeper (to provide leader election). Running three ZooKeeper nodes will allow one to fail and for the service to still be available (see ZooKeeper reliability for more information). We recommend running at least three master nodes for a highly-available configuration. Run the steps below on each master node.

Master Node Setup:

The easiest way to install Mesos is via the GitHub repositories. Alternatively, you can download the latest deb or rpm directly from the Mesosphere downloads page and install it manually.

Mesosphere has official package repositories which connect directly to the native package management tools of your favorite Linux distribution — namely apt-get and yum — to install Mesos on top of the most common Linux distributions (RedHat, CentOS, Ubuntu and Debian).

Step 1: Install mesos and zookeeper in master machines:

# Add the repository

$sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
$sudo yum -y install mesos marathon
$sudo yum -y install mesosphere-zookeeper

Step 2: Configuration
ZooKeeper

Set /var/lib/zookeeper/myid to a unique integer between 1 and 255 on each node.

$sudo nano /var/lib/zookeeper/myid
#set 1

 

Start Zookeeper

$sudo systemctl start zookeeper

 

Step 3: Set mesos master  address in mesos zookeeper setting:

On each node, replacing the IP addresses below with each master’s IP address, set/etc/mesos/zk to:

$sudo nano /etc/mesos/zk
zk://1.1.1.1:2181,2.2.2.2:2181,3.3.3.3:2181/mesos

Step 4: Disable mesos-slave service on each master servers.

$sudo systemctl stop mesos-slave.service
$sudo systemctl disable mesos-slave.service

Step 5:  Restart all the master node using following cmd.

$sudo systemctl restart mesos-master

(more…)

Read More