配置具有多个通道的水槽时数据通道锁定错误

【问题标题】：data channel lock error while configuring flume with multiple channels配置具有多个通道的水槽时数据通道锁定错误
【发布时间】：2018-08-23 15:03:02
【问题描述】：

我尝试将流从一个源分散到两个通道。此外，我在channel lock error while configuring flume's multiple sources using FILE channels 问题中为每个通道指定了不同的 dataDirs 和 checkpointDirs 属性。我使用了多路复用通道选择器。我收到以下错误。

18/08/23 16:21:37 **ERROR file.FileChannel: Failed to start the file channel** [channel=fileChannel1_2]
java.io.IOException: Cannot lock /root/.flume/file-channel/data. The directory is already locked. [channel=fileChannel1_2]
    at org.apache.flume.channel.file.Log.lock(Log.java:1169)
    at org.apache.flume.channel.file.Log.<init>(Log.java:336)
    at org.apache.flume.channel.file.Log.<init>(Log.java:76)
    at org.apache.flume.channel.file.Log$Builder.build(Log.java:276)
    at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:281)
    at unAndReset(FutureTask.java:308) .....

我的配置文件如下。

agent1.sinks=hdfs-sink1_1 hdfs-sink1_2 

agent1.sources=source1_1

agent1.channels=fileChannel1_1 fileChannel1_2

agent1.channels.fileChannel1_1.type=file

agent1.channels.fileChannel1_1.checkpointDir=/home/Flume/alpha/001
agent1.channels.fileChannel1_1.dataDir=/mnt/alpha_data/
agent1.channels.fileChannel1_1.checkpointOnClose=true

agent1.channels.fileChannel1_1.dataOnClose=true

agent1.sources.source1_1.type=spooldir

agent1.sources.source1_1.spoolDir=/home/ABC/
agent1.sources.source1_1.recursiveDirectorySearch=true
agent1.sources.source1_1.fileSuffix=.COMPLETED

agent1.sources.source1_1.basenameHeader = true

agent1.sinks.hdfs-sink1_1.type=hdfs

agent1.sinks.hdfs-sink1_1.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_1.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/CA

agent1.sinks.hdfs-sink1_1.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_1.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_1.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_1.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_1.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_1.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_1.hdfs.useLocalTimeStamp=false

agent1.channels.fileChannel1_2.type=file

agent1.channels.fileChannel1_2.capacity=200000
agent1.channels.fileChannel1_2.transactionCapacity=1000
agent1.channels.fileChannel1_2.checkpointDir=/home/Flume/beta/001
agent1.channels.fileChannel1_2.dataDir=/mnt/beta_data/
agent1.channels.fileChannel1_2.checkpointOnClose=true
agent1.channels.fileChannel1_2.dataOnClose=true

agent1.sinks.hdfs-sink1_2.type=hdfs

agent1.sinks.hdfs-sink1_2.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_2.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/AZ

agent1.sinks.hdfs-sink1_2.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_2.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_2.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_2.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_2.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_2.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_2.hdfs.useLocalTimeStamp=false

agent1.sources.source1_1.channels=fileChannel1_1 fileChannel1_2

agent1.sinks.hdfs-sink1_1.channel=fileChannel1_1

agent1.sinks.hdfs-sink1_2.channel=fileChannel1_2

agent1.sources.source1_1.selector.type=multiplexing
agent1.sources.source1_1.selector.header=basenameHeader
agent1.sources.source1_1.selector.mapping.CA=fileChannel1_1
agent1.sources.source1_1.selector.mapping.AZ=fileChannel1_2

有人可以为此提供任何解决方案吗？

【问题讨论】：

    标签： flume-ng


    【解决方案1】：

    尝试为复用选择器中的默认属性设置通道

    agent1.sources.source1_1.selector.default=fileChannel1_1

    【讨论】：

    • 谢谢。选择器默认情况下，它将所有事件复制到两个通道。我需要的是多路复用数据流到两个通道。根据 Apache Flume 文档，Flume 根据事件头多路复用事件。为此，我应该将文件名添加到事件标头。我怎样才能做到这一点？我应该使用 Flume 拦截器吗？ Flume 中有内置的文件名拦截器吗？
    • fileHeader 和 fileHeaderKey 属性应将文件名添加到事件标头
    • 我使用了 fileHeaderKey 和 fileHeader 属性，如下所示。 agent1.sources.source1_1.fileHeader=true agent1.sources.source1_1.fileHeaderKey=file 那么应该是agent1.sources.source1_1.selector.header=???
    • agent1.sources.source1_1.selector.header=file 但是header会包含绝对文件路径所以映射要指定文件路径agent1.sources.source1_1.selector.mapping.ABSOLUTE_FILE_PATH=fileChannel1_1跨度>
    • 我想使用文件名而不是文件路径进行映射。是否可以使用上面的 agent1.sources.source1_1.selector.mapping.ABSOLUTE_FILE_PATH=fileChannel1_1 命令进行文件名映射？？？
    【解决方案2】：

    数据通道锁定错误已得到纠正。但仍然无法进行多路复用。代码如下。

    agent1.sinks=hdfs-sink1_1 hdfs-sink1_2 hdfs-sink1_3

agent1.sources=source1_1

agent1.channels=fileChannel1_1 fileChannel1_2 fileChannel1_3

agent1.channels.fileChannel1_1.type=file

agent1.channels.fileChannel1_1.capacity=200000

agent1.channels.fileChannel1_1.transactionCapacity=1000

agent1.channels.fileChannel1_1.checkpointDir=/home/Flume/alpha/001

agent1.channels.fileChannel1_1.dataDirs=/home/Flume/alpha_data

agent1.channels.fileChannel1_1.checkpointOnClose=true

agent1.channels.fileChannel1_1.dataOnClose=true



agent1.sources.source1_1.type=spooldir

agent1.sources.source1_1.spoolDir=/home/ABC/

agent1.sources.source1_1.recursiveDirectorySearch=true

agent1.sources.source1_1.fileSuffix=.COMPLETED

agent1.sources.source1_1.basenameHeader = true

agent1.sources.source1_1.basenameHeaderKey = basename



agent1.sinks.hdfs-sink1_1.type=hdfs

agent1.sinks.hdfs-sink1_1.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_1.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/CA

agent1.sinks.hdfs-sink1_1.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_1.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_1.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_1.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_1.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_1.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_1.hdfs.useLocalTimeStamp=false


agent1.channels.fileChannel1_2.type=file

agent1.channels.fileChannel1_2.capacity=200000

agent1.channels.fileChannel1_2.transactionCapacity=1000

agent1.channels.fileChannel1_2.checkpointDir=/home/Flume/beta/001

agent1.channels.fileChannel1_2.dataDirs=/home/Flume/beta_data

agent1.channels.fileChannel1_2.checkpointOnClose=true

agent1.channels.fileChannel1_2.dataOnClose=true



agent1.sinks.hdfs-sink1_2.type=hdfs

agent1.sinks.hdfs-sink1_2.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_2.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/AZ

agent1.sinks.hdfs-sink1_2.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_2.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_2.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_2.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_2.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_2.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_2.hdfs.useLocalTimeStamp=false

agent1.channels.fileChannel1_3.type=file

agent1.channels.fileChannel1_3.capacity=200000

agent1.channels.fileChannel1_3.transactionCapacity=10

agent1.channels.fileChannel1_3.checkpointDir=/home/Flume/gamma/001

agent1.channels.fileChannel1_3.dataDirs=/home/Flume/gamma_data

agent1.channels.fileChannel1_3.checkpointOnClose=true

agent1.channels.fileChannel1_3.dataOnClose=true


agent1.sinks.hdfs-sink1_3.type=hdfs

agent1.sinks.hdfs-sink1_3.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_3.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/KT

agent1.sinks.hdfs-sink1_3.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_3.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_3.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_3.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_3.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_3.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_3.hdfs.useLocalTimeStamp=false



agent1.sources.source1_1.channels=fileChannel1_1 fileChannel1_2 fileChannel1_3

agent1.sinks.hdfs-sink1_1.channel=fileChannel1_1

agent1.sinks.hdfs-sink1_2.channel=fileChannel1_2

agent1.sinks.hdfs-sink1_3.channel=fileChannel1_3



agent1.sources.source1_1.selector.type=replicating

agent1.sources.source1_1.selector.header=basename

agent1.sources.source1_1.selector.mapping.CA=fileChannel1_1

agent1.sources.source1_1.selector.mapping.AZ=fileChannel1_2

agent1.sources.source1_1.selector.default=fileChannel1_3

    【讨论】：

      猜你喜欢
      • 1970-01-01
      • 2019-04-08
      • 2017-03-17
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2019-07-20
      • 2018-10-04
      • 1970-01-01
      相关资源
      最近更新 更多
      热门标签
      Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode