flume+kafka+hdfs详解

flume架构图

单节点flume配置

flume-1.4.0 启动flume

bin/flume-ng agent --conf ./conf  -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -n agent

-n表示配置文件中agent的名字

agent.sources = r1agent.sinks = s1agent.channels = c1agent.sources.r1.channels = c1agent.sinks.s1.channel = c1#Describe/configure the sourcesagent.sources.r1.type = execagent.sources.r1.command = tail -F /home/flume/loginfo#Use a channel which buffers events in memoryagent.channels.c1.type = memoryagent.channels.c1.capacity = 1000  #Eventagent.channels.c1.transactionCapacity = 100agent.sinks.s1.type = logger

flume-1.4.0 + kafka-0.7.2+hdfs flume配置

agent.sources = r1agent.sinks = s_kafka s_hdfsagent.channels = c_kafka c_hdfsagent.sources.r1.channels = c_kafka c_hdfsagent.sources.r1.type = exec#下面这个脚本tail某个日志agent.sources.r1.command = tail -F /home/flume/loginfoagent.channels.c_kafka.type = memoryagent.channels.c_hdfs.type = memoryagent.sinks.s_kafka.type = com.sink.FirstkafkaSinkagent.sinks.s_kafka.channel = c_kafka#kafka需要连接zk，写入broker数据agent.sinks.s_kafka.zkconnect = localhost:2181agent.sinks.s_kafka.topic = testagent.sinks.s_kafka.serializer.class = kafka.serializer.StringEncoderagent.sinks.s_kafka.metadata.broker.list = localhost:9092 #配置文件server.propertiesagent.sinks.s_kafka.custom.encoding = UTF-8agent.sinks.s_hdfs.type = hdfsagent.sinks.s_hdfs.channel = c_hdfs#默认端口8020agent.sinks.s_hdfs.hdfs.path = hdfs://localhost:9000/root/sourceagent.sinks.s_hdfs.hdfs.filePrefix = events-agent.sinks.s_hdfs.hdfs.fileType = DataStreamagent.sinks.s_hdfs.hdfs.writeFormat = Textagent.sinks.s_hdfs.hdfs.rollCount = 30 #达到某一数值记录生成文件agent.sinks.s_hdfs.hdfs.rollSize = 0agent.sinks.s_hdfs.hdfs.rollInterval = 0agent.sinks.s_hdfs.hdfs.useLocalTimeStamp = trueagent.sinks.s_hdfs.hdfs.idleTimeout = 51agent.sinks.s_hdfs.hdfs.threadsPoolSize = 2

Flume内置channel,source,sink汇总

http://www.iteblog.com/archives/948