水槽不适用于 hdfs答案

【问题标题】：flume doesn't work with hdfs水槽不适用于 hdfs
【发布时间】：2014-05-10 19:19:02
【问题描述】：

我用两个节点设置了flume。我想从slave01加载数据到hdfs。 slave01:example-conf.properties

 agent.sources = baksrc
agent.channels = memoryChannel
agent.sinks =avro-forward-sink 
agent.sources.baksrc.type = exec
agent.sources.baksrc.command = tail -F /root/hadoo/test/data.txt
agent.sources.baksrc.checkperiodic = 1000
agent.sources.baksrc.channels =memoryChannel 
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 10000
agent.sinks.avro-forward-sink.type = avro
agent.sinks.avro-forward-sink.hostname = master
agent.sinks.avro-forward-sink.port = 23004
agent.sinks.avro-forward-sink.channel = memoryChannel

master：example-conf.properties

agent.sources = avrosrc
agent.sinks =hdfs-write 
agent.channels = memoryChannel
agent.sources.avrosrc.type =avro
agent.sources.avrosrc.bind =master
agent.sources.avrosrc.port =23004
agent.sources.avrosrc.channels=memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity =10000
agent.sinks.hdfs-write.type = hdfs
agent.sinks.hdfs-write.hdfs.path =hdfs://172.16.86.38:9000/flume/webdata
agent.sinks.hdfs-write.hdfs.rollInterval = 0   
agent.sinks.hdfs-write.hdfs.rollSize = 4000000  
agent.sinks.hdfs-write.hdfs.rollCount = 0   
agent.sinks.hdfs-write.hdfs.writeFormat = Text  
agent.sinks.hdfs-write.hdfs.fileType = DataStream  
agent.sinks.hdfs-write.hdfs.batchSize = 10  
agent.sinks.hdfs-write.channel=memoryChannel

然后我运行一个shell脚本：像这样：

#!/bin/sh
for i in {1..1000000}; do
    echo "test flume to Hbase $i" >>/root/hadoop/test/data.txt; 
    sleep 0.1; 
done

启动水槽： flume-ng agent --conf conf --conf-file example-conf.properties --name agent -Dflume.root.logger=DEBUG,console 我在控制台上没有错误。

14/05/06 16:38:44 INFO source.AvroSource: Avro source avrosrc stopping: Avro source avrosrc: { bindAddress: master, port: 23004 }
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] DISCONNECTED
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] UNBOUND
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] CLOSED

但我在 hdfs 中看不到文件，我的配置有问题吗？我在master上测试过，效果很好。

【问题讨论】：

标签： hadoop hdfs flume

【解决方案1】：

您使用哪个版本的水槽？
你设置HADOOP_HOME了吗？
Flume 是否从 HADOOP_HOME 输出带有 hadoop jar 的类路径？
如果你使用的是 apache flume，那么一步一步来： 1.设置HADOOP_HOME
2.编辑hadoop core-site.xml，确保namenode ip正确
3.使用hdfs路径：agent.sinks.hdfs-write.hdfs.path =/flume/webdata
4. 启动水槽

【讨论】：