We have already seen how we write a Dockerfile that will create an image for us. This image can then be used to make instances of containers that run the image. We have successfully started the RabbitMQ container that will act as an input to our system, but this tutorial is called ‘Docker and Flume with Grandad‘ so we should build our Apache Flume image next and get a container up and running with that.
Note that this section is built upon the tutorial I found at (http://probablyfine.co.uk/2014/05/05/using-docker-with-apache-flume-1/), and the ‘probablyfine/flume‘ base image seems to be one of the more popular flume images available on Docker Hub. If you want more details check out the link.
To create a simple Flume image that we’ll build on and modify, do the following:
- Open a terminal
- Navigate to the ‘docker-poc’ folder we created earlier, it should already contain our ‘rabbit’ folder we created earlier.
- Create a new folder to hold our Dockerfile and associated assets for our Apache Flume container with `mkdir flume`
- Navigate into this new folder with `cd flume`.
- Create an empty Dockerfile with `touch Dockerfile`.
- Edit the newly created file with `nano Dockerfile`.
- Add the following content to the Dockerfile we just created:
FROM probablyfine/flume
ADD flume.conf /var/tmp/flume.conf
EXPOSE 44444
ENTRYPOINT [ "flume-ng", "agent", "-c", "/opt/flume/conf", "-f",
"/var/tmp/flume.conf", "-n", "docker",
"-Dflume.root.logger=INFO,console" ]
Remember to save the file.
What does this file do?
Like the last Dockerfile we created, this file contains some commands that create the base of our image and perform some operations.
As before, rhe `FROM` command sets the base image that we want to use in our container. This is basically using the ‘probablyfine/flume‘ image as a starting point. You can always go and check out its contents. This particular image has a basic OS and all the software needed to get Flume working.
The `ADD` command is one we have not used before. This command takes a named file from the same directory of the Dockerfile and copies to a specified location and filename in the image. This is an easy way for us to add files to the image. I guess the command name gives that away! Since Flume uses a config file to tell it where data is coming from, how it should process that data and where it should send it to, we will need to create this file before we build this image.
As before, the `EXPOSE‘ command makes a port, 44444 in this case, accessible to other docker machines and allows them to be mapped to ports on the host machine.
The `ENTRYPONT` command allows us to pass commands to the running docker container. Crucially, it allows us to specify what program to use to run those commands. Docker has a default entrypoint of ‘/bin/sh -c’ which means in our rabbitmq image Dockerfile, when we used ‘CMD rabbitmq-server start‘ what docker was doing behind the scenes was ‘/bin/sh -c rabbitmq-server start‘. Docker was passing our commands to the command line.
In this Dockerfile we actually want to pass commands to a different program called ‘flume-ng‘. ‘flume-ng‘ is the program we want to run, everything else are arguments to be passed to ‘flume-ng‘. Specifically, the arguments that Flume expects are: the configuration directory, the name of the configuration file and the name to be given to the running agent. (A Flume instance can run more than one data processing pipeline and each pipeline is managed by an agent).
So, if you want to run things on the command line inside your docker container, use CMD, if you want to pass commands to a specific program inside your docker container, use ENTRYPOINT. See http://blog.stapps.io/docker-cmd-vs-entrypoint/ for a good overview of CMD and ENTRYPOINT.
These four commands pull an image from the Docker Repository, add a config file, expose a port to the outside world and run flume as a service. Voila, one flume server.
Creating our Flume config file
As noted above, our Dockerfile contains a command to ADD a file to the container. The file being added is ‘flume.conf’ and it is used to tell flume where it will get data from, how it will process that data and where it should send that data once it has been processed. To make this ‘flume.conf‘ file perform the following steps:
- Navigate to ‘/docker-poc/flume’
- Create the file with `touch flume.conf`.
- Edit the file using `nano flume.conf`.
- Add the following to the file contents:
docker.sources = netcatSource
docker.channels = inMemoryChannel
docker.sinks = logSink
docker.sources.netcatSource.type = netcat
docker.sources.netcatSource.bind = 0.0.0.0
docker.sources.netcatSource.port = 44444
docker.sources.netcatSource.channels = inMemoryChannel
docker.channels.inMemoryChannel.type = memory
docker.channels.inMemoryChannel.capacity = 1000
docker.channels.inMemoryChannel.transactionCapacity = 100
docker.sinks.logSink.type = logger
docker.sinks.logSink.channel = inMemoryChannel
Remember to save the file.
Flume Configuration Overview
While it’s not strictly related to Docker, having a broad overview of how Flume is configured might help us to get things up and running. Flume has three main concepts:
Sources – Places where data comes into the system. It contains a large range of adapter to get data from external systems.
Channels – Pathways data travels down within the system, being processed by various components along the way.
Sinks – Places where data exits the system. Again, a large range of adapters exist to help to connect to external systems.
A Flume instance can run more than one data processing pipeline. Each pipeline is managed by an ‘agent‘. Each agent has a name. When we launch Flume inside our docker container, one of the arguments we pass to Flume in our ENTRYPOINT arguments is the name we want to give the agent. In our case, the name of the agent is ‘docker’.
We use the same name as the starting element in the flume configuration. This is how each agent knows how to identify its configuration elements within the configuration file.
In our configuration we setup a pretty simple pipeline that has a source to get data in, a channel to process that data and a sink to get the data out. The key elements in the configuration are:
- A NetcatSource that uses a built in adapter to read data from a port 44444 on the localhost and turn them into Flume events.
- An InMemoryChannel that buffers the incoming events in memory to give us some flexibility if events arrive faster than we can process.
- A LogSink, which just logs the events it receives as a form of output.
Firing up the Flume Container
Now that we have our Dockerfile in place we can actually execute it and get our container up and running. Assuming you are running on a Mac:
- Ensure you are running the `Docker Quickstart Terminal`.
- Navigate to the `docker-poc/flume` folder created earlier.
- This folder should contain the Dockerfile we just created.
- Type `docker build -t flume .` to use the dockerfile in the current directory to create a docker image with the name ‘flume’.
- The output should look something like:
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM probablyfine/flume
---> dcc1d3a26f17
Step 2 : ADD flume.conf /var/tmp/flume.conf
---> f2081cfbe5d5
Removing intermediate container e41cd0d0cfa9
Step 3 : EXPOSE 44444
---> Running in 93f57649da85
---> 2154236de5e3
Removing intermediate container 93f57649da85
Step 4 : ENTRYPOINT flume-ng agent -c /opt/flume/conf -f /var/tmp/flume.conf -n docker -Dflume.root.logger=INFO,console
---> Running in 2cc534aaaece
---> 0521c0ecf2b0
Removing intermediate container 2cc534aaaece
Successfully built 0521c0ecf2b0
7. Type `docker images` to list all of the images that exist on host machine. The output will look like:
REPOSITORY TAG IMAGE ID CREATED SIZE
flume latest 0521c0ecf2b0 41 seconds ago 418.9 MB
rabbit latest 68102a356d7d 20 hours ago 300.8 MB
...........
As you can see, the ‘flume’ image that we just built is shown at the top of the list along with the rabbit image we built earlier. Now that the image is in place we can use the image to create create a running Docker container, which is an instance of the image.
8. To start a container using the `flume` image we have created, type:
docker run -e FLUME_AGENT_NAME=docker -e
FLUME_CONF_FILE=/var/tmp/flume.conf -p 44444:44444 -t flume
The log output will be displayed on your terminal. Something similar to:
Info: Including Hive libraries found via () for Hive access
+ exec /opt/java/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/opt/flume/conf:/opt/flume/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /var/tmp/flume.conf -n docker
2016-05-25 09:07:31,016 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2016-05-25 09:07:31,024 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/var/tmp/flume.conf
2016-05-25 09:07:31,034 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:logSink
2016-05-25 09:07:31,039 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:931)] Added sinks: logSink Agent: docker
2016-05-25 09:07:31,040 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:logSink
2016-05-25 09:07:31,071 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:141)] Post-validation flume configuration contains configuration for agents: [docker]
2016-05-25 09:07:31,071 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:145)] Creating channels
2016-05-25 09:07:31,083 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel inMemoryChannel type memory
2016-05-25 09:07:31,087 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel inMemoryChannel
2016-05-25 09:07:31,088 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source netcatSource, type netcat
2016-05-25 09:07:31,111 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: logSink, type: logger
2016-05-25 09:07:31,127 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel inMemoryChannel connected to [netcatSource, logSink]
2016-05-25 09:07:31,145 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{netcatSource=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:netcatSource,state:IDLE} }} sinkRunners:{logSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@302477bc counterGroup:{ name:null counters:{} } }} channels:{inMemoryChannel=org.apache.flume.channel.MemoryChannel{name: inMemoryChannel}} }
2016-05-25 09:07:31,155 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel inMemoryChannel
2016-05-25 09:07:31,259 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: CHANNEL, name: inMemoryChannel: Successfully registered new MBean.
2016-05-25 09:07:31,259 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: inMemoryChannel started
2016-05-25 09:07:31,261 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink logSink
2016-05-25 09:07:31,262 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source netcatSource
2016-05-25 09:07:31,263 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:150)] Source starting
2016-05-25 09:07:31,271 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:44444]
8. Open a new terminal and type `echo foo bar baz | nc 192.168.99.100 44444`
9. You should see ‘OK‘ as a response.
10. Switch back to your original docker terminal and check the last line of the output. It should contain:
2016-05-25 09:14:43,009 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)]
Event: { headers:{} body: 66 6F 6F 20 62 61 72 20 62 61 7A foo bar baz }
What did we just do?
Firing data into the input of our Flume container instance and seeing it appear in the output log proves that we have our flume container up and running. The container is listening on a defined port for input, processing any received messages and then outputting them to it defined output. We have another piece of our system up and running. Next we’ll move on to get our last component, Kafka, into a container.
