Installing Apache Kafka and Zookeeper on Windows

In this article, I’ll talk about how to Install, configure and start Apache Zookeeper and Apache Kafka Server on Windows OS.

Prerequisite

• JRE running on your machine and path must set to Environment Variable
• Any Zip tool like 7-zip, WinZip or WinRAR.
• Download and Extract Apache Zookeeper using 7-zip.
• Download and Extract Apache Kafka using 7-zip

ZooKeeper Installation Instructions:

• Go to the conf directory of your Zookeeper. For me its under D:\Softwares\apache-zookeeper-3.5.5
• Copy and rename zoo_sample.cfg to zoo.cfg file.
• Open and Edit dataDIr=/tmp/zookeeper to dataDir=D:\Softwares\apache-zookeeper-3.5.5

• Add entries in System Environment Variables
• ZOOKEEPER_HOME=D:\Softwares\apache-zookeeper-3.5.5
• Append D:\Softwares\apache-zookeeper-3.5.5\bin to PATH system variable.
• Open command prompt and type zkserver to start the Zookeeper application.
• You can easily edit the default port ( 2181) in zoo.cfg file.

Congratulations, ZooKeeper is up and running on port 2181.

Installing Apache Kafka Instructions

• Please make sure that the Zookeeper instance is up and running before starting any kafka server.
• Go to the Kafka Installation directory (D:\Softwares\apache-tomcat-8.5.37) and run below command.
• .\bin\windows\kafka-server-start.bat .\config\server.properties

Congratulations, Apache kafka is up and running on port 9092

  

Create Topic

To create a topic, we need to run the kafka-topics script

• Go to Apache kafka installation directory and navigate to bin\windows folder (D:\Softwares\kafka_2.11-2.2.0\bin\windows).
• Execute below command to create topic where <TOPIC_NAME> should be name of your topic
• kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic <TOPIC_NAME>

Create Producer

Now the topic has been created, Lets start a producer so that we can published some message against sales topic.

·       Go to Apache kafka installation directory and navigate to bin\windows folder (D:\Softwares\kafka_2.11-2.2.0\bin\windows).

·       Execute: kafka-console-producer.bat --broker-list localhost:9092 --topic sales

Create Consumer

Let’s create a consumer which should subscribe to sales topic to get all the messages.

·       Go to Apache kafka installation directory and navigate to bin\windows folder (D:\Softwares\kafka_2.11-2.2.0\bin\windows).

·       Execute: kafka-console-consumer.bat --bootstrap-server localhost:2181 --topic sales

By running all four components (zookeeper, kafka, producer, and consumer) in different terminals, we will be able to enter messages from the producer’s terminal and can see them appearing in the consumer’s terminal and If everything works fine, you will be able to push and see messages.

Bonus Installation:

There are multiple UI applications which can be use for monitoring apache kafka. It will display the information like total brokers, topics, partition and even lets you view messages as well.

One of the tools is Kafdrop which we will install right now and see all those details on UI.

Requirements:

• Java 8
• Kafka (0.8.1 or 0.8.2 is known to work)
• Zookeeper (3.4.5 or later)
• Download and Extract Kafdrop using 7-zip.

Installation

·       Go to the Kafdrop directory and run mvn clean package which will generate an executable JAR in target directory.

·       Go to target directory and run

o   java -jar ./target/kafdrop-<version>.jar --zookeeper.connect=localhost:2181

·       Open a browser and navigate to http://localhost:9000 to view the Kafdrop UI.  

How to enable multiple domains in Apache Server using Name-Based VirtualHosts and SSL

Scenario: I have an Apache Server(SSL enabled) and tomcat running on my machine and there is one application (app1) hosted on tomcat which is only accessible from Apache Server. You cannot access it directly from tomcat.

Now you want to access app1 using multiple domains i.e. domain1.waheedtechblog.com annd domain2.waheedtechblog.com should point to the same application which is hosted on tomcat.

(I want to implement different Authentication mechanism based on different domains)
Solution: The above case can be achieved using NameBased VirtualHosts and SSL.
First you need to uncomment following lines form ${apache}/conf/httpd.conf file

• LoadModule ssl_module modules/mod_ssl.so
• LoadModule proxy_module modules/mod_proxy.so
• LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
• LoadModule rewrite_module modules/mod_rewrite.so
• LoadModule proxy_http_module modules/mod_proxy_http.so
• Include conf/extra/httpd-ssl.conf

then goto ${apache}/conf/extra/httpd-ssl.conf file add Virtualhost tag for each domain.

Listen 443

NameVirtualHost *:443

<VirtualHost *:443>

DocumentRoot "c:/Apache2/htdocs"

ServerName domain1.waheedtechblog.com

ServerAdmin admin@example.com

ErrorLog "c:/Apache2/logs/error_domain1.log"

TransferLog "c:/Apache2/logs/access_domain1.log"

SSLEngine on

SSLCertificateFile "C:\Apache2\certificate\domain1.crt"

SSLCertificateKeyFile "C:\Apache2\certificate\domain1.key"

ProxyPass / ajp://sp.domain.com:8009/app1/

BrowserMatch "MSIE [2-5]" \

nokeepalive ssl-unclean-shutdown \

downgrade-1.0 force-response-1.0

</VirtualHost>

<VirtualHost *:443>

DocumentRoot "c:/Apache2/htdocs"

ServerName domain2.waheedtechblog.com

ServerAdmin admin@example.com

ErrorLog "c:/Apache2/logs/error_domain2.log"

TransferLog "c:/Apache2/logs/access_domain2.log"

SSLEngine on

SSLCertificateFile "C:\Apache2\certificate\domain2.crt"

SSLCertificateKeyFile "C:\Apache2\certificate\domain2.key"

ProxyPass / ajp://sp.domain.com:8009/app1/

BrowserMatch "MSIE [2-5]" \

nokeepalive ssl-unclean-shutdown \

downgrade-1.0 force-response-1.0

</VirtualHost>

Points to remember:

• Add all DNS entry in system hosts file Eg: for windows (Add “127.0.0.1 domain1.waheedtechblog.com” to “C:\Windows\System32\drivers\etc\hosts”)
• Make sure you are using different certificates and Key for each domain or else it will always point to default VirtualHost i.e. first one that you have defined. Click here on how to generate key and certificate.
• In proxyPass, always ends with '/' or else you will see redirect issue.

In case of any issue, You can drop your comment in the comment box.

Reference:

https://wiki.apache.org/httpd/NameBasedSSLVHosts

How to install and configure Apache Tomcat 7 on Cent OS

Here are the steps to install and configure Apache Tomcat 7 on Cent OS :

• Make sure you have JAVA 6 or later version installed on your machine.
  • Java --version
• Download Tomcat7 Archive
  • wget http://www.us.apache.org/dist/tomcat/tomcat-7/v7.0.64/bin/apache-tomcat-7.0.64.tar.gz
• Untar it at some location (eg: /opt/tomcat7)
  • tar xzf apache-tomcat-7.0.64.tar.gz
• Start the tomcat (bydefault it will start at port 8080)
  • sh /opt/tomcat7/bin/startup.sh
• Verify Apache Tomcat Server
• Goto your browser and check http://localhost:8080 or http://127.0.0.1:8080

• Stop the tomcat
• sh /opt/tomcat7/bin/shutdown.sh

• [Optional] To access admin|manager pages, you need to create user accounts. Add below lines inside <tomcat-users></tomcat-users> tags in ${tomcat7}/conf/tomcat-users.xml file.
  <role rolename="manager-gui"/>
  <user username="manager" password="root123" roles="manager-gui"/>
  <role rolename="admin-gui"/>
  <user username="admin" password="root123" roles="manager-gui,admin-gui"/>

How to install and Configure Apache Http Server on Cent OS

Here are the steps to install and configure Apache WebServer on Cent OS :

• Install Apache HTTP Server (By default Cent OS comes with Apache)
  • yum install httpd
• Set to Chkconfig to start on boot
  • chkconfig --level 235 httpd on
• Uncomment below line from configuration(/etc/httpd/conf/httpd.conf) file
  • NameVirtualHost *:80
• Restart Apache Http Server
  • service httpd restart
• Verify Apache Server
  • Goto your browser and check localhost or localhost.localdomain It will display Apache Test Page

How to install mod_jk.so on Cent OS

The Basics - What is mod_jk?

The mod_jk connector is an Apache HTTPD module that allows HTTPD to communicate with Apache Tomcat instances over the AJP protocol.

Steps:

1. Download the latest apache connector from http://tomcat.apache.org/download-connectors.cgi.
2. Untar the download by
         tar zxvf <filename>
3. Goto native directory of connector
         cd <connector dir>/native/
4. Run the buildconf.sh scripts
       ./buildconf.sh
Note: If you get any issue like "autocong" not installed then install following things:
       yum install autoconf
       yum install libtool
5. You need "httpd-devel" tools to build it. So make sure you have already installed it by
         yum list installed | grep httpd-devel    else install it "yum install httpd-devel"
6. From the native directory Run
      ./configure --with-apxs=/usr/sbin/apxs
      make
 

Congrats!! Now you can see the mod_jk.so module under your native directory.

Apache Ant - Tutorial

Introduction:
Ant (originally an acronym for Another Neat Tool), is a build tool with special support for the Java programming language but can be used for just about everything.Ant is platform-independent; it is written purely in Java. Ant is particularly good at automating complicated repetitive tasks e.g. compiling source code, running software tests, creating jar files, javadocs, etc. and thus is well suited for automating standardised build processes.
A build process typically includes:
     -the compilation of the Java source code into Java bytecode
     -creation of the .jar file for the distribution of the code
     -creation of the Javadoc documentation
Ant accepts instructions in the form of XML documents("build.xml") thus is extensible and easy to maintain.

Installation:

Linux (Ubuntu / Debian)

On Debian /Ubuntu use "apt-get install ant" to install it.

Windows
Download Apache Ant from http://ant.apache.org/ .
Extract the zip file into a directory structure of your choice. Set the "ANT_HOME" environment variable to this location and include the "ANT_HOME/bin" directory in your path.
Make sure that also the JAVA_HOME environment variable is set to the JDK. This is required for running Ant.
Check your installation by opening a command line and typing "ant -version" into the commend line. The system should find the command ant and show the version number of your installed ant.
                                                                     OR
The documents entitled Configuring A Windows Working Environment andConfiguring A Unix Working Environment are of use to people who need to know more.

Basics:
A simple Ant example is shown below followed by a set of instructions indicating how to use Ant.

<?xml version="1.0"?>  

<project name="test" default="compile" basedir=".">  

        <property name="src" value="."/>  

        <property name="build" value="build"/>

        <target name="init">  

                <mkdir dir="${build}"/> 

        </target>

        <target name="compile" depends="init">  

                 <!-- Compile the java code -->

                 <javac srcdir="${src}" destdir="${build}"/>   

         </target> 

</project>

1) The document begins with an XML declaration which specifies which version of XML is in use.

2) The root element of an Ant build file is the project element, it has three attributes.

• name: The name of the project, it can be any combination of alphanumeric characters that constitute valid XML.
• default: The default target to use when no target is specified, out of these three attributes default is the only required attribute.
• basedir: The base directory from which any relative directories used within the Ant build file are referenced from. If this is omitted the parent directory of the build file will be used.

3) The property element allows the declaration of properties which are like user-definable variables available for use within an Ant build file. The name attribute specifies the name of the property and the value attribute specifies the desired value of the property. 

4) The target element is used as a wrapper for a sequences of actions. A target has a name, so that it can be referenced from elsewhere, either externally from the command line, or internally via the depends keyword, or through a direct call. The target in the example is called "init" (initiate), it makes a directory using the mkdir element with the name specified by the build property

5) depends allows one to specify other targets that must be executed prior to the execution of this target. In the listing abovedepends="init" is used to indicate that the compile target requires that the target named init be executed prior to executing the body of compile.

6) The example above causes javac to be executed, compiling all the.java files in the directory specified by the src property and placing the resultant .class files in the directory specified by the build property.

Example:

Copy the source code into a text editor and save the file as build.xml.Create a test directory and place the file in it. Create some arbitrary .javafile and place it in the same directory as build.xml. For convenience, here is an example .java file: test.java. Place the java file in the same directory asbuild.xml.

public class test { 

    public static void main(String[] args) { 

        System.out.println("Hello World!"); 

    }

}

Type the following at the commandline in the test directory:
ant -v
This will create a directory called build, compile test.java and place the .class file created in the build directory. The -v directs ant to be verbose. This verbosity causes the command to echo lots of information, information that is not really necessary for most normal purposes. Execute the command sequence again. An example output message is shown below:

[javac] test.java omitted as /path/to/temp/build/test.class is up todate

A nice feature of Ant is that by default, only those .java input files that have a more recent timestamp than their corresponding .class output files will be compiled.

For more info:

http://ant.apache.org/manual/tutorial-HelloWorldWithAnt.html

http://ant.apache.org/manual/tutorial-writing-tasks.html

http://ant.apache.org/manual/using.html