Adding multiple flows in an agent:

We can have multiple sources, channels and sinks in a single Flume configuration file. A single agent can contain several independent flows. Let us see example of multiple flows:

# list the sources, sinks and channels for the agent

<Agent>.sources = <Source1> <Source2>

<Agent>.sinks = <Sink1> <Sink2>

<Agent>.channels = <Channel1> <Channel2>

Here we have setup 2 different flows in single agent. First flow is from external avro client to external HDFS and second flow is from output of an exec to avro sink:

# list the sources, sinks and channels in the agent

Agent1.sources = avro-source1 exec-source2

Agent1.sinks = hdfs-sink1 avro-sink2

Agent1.channels = mem-channel-1 file-channel-2



# flow #1 configuration

Agent1.sources.avro-soure1.channels = mem-channel-1

Agent1.sinks.hdfs-sink1.channel = mem-channel-1



# flow #2 configuration

Agent1.sources.exec-source2.channels = file-channel-2

Agent1.sinks.avro-sink2.channel = file-channel-2