Kafka 无法删除 Windows 上的旧日志段

Question

我在 Windows 上遇到了 Kafka 的问题，它尝试删除日志段，但由于另一个进程可以访问这些文件而无法删除。这是由于 Kafka 持有对文件本身的访问权限并试图删除它打开的文件。错误如下，供参考。

我发现两个 JIRA 错误已在此问题上提交https://issues.apache.org/jira/browse/KAFKA-1194 和 https://issues.apache.org/jira/browse/KAFKA-2170。第一个记录在 0.8.1 版下，第二个记录在 0.10.1 版下。

我个人尝试过 0.10.1 和 0.10.2 版本。他们都没有修复他们的错误。

我的问题是，有没有人知道可以修复这个问题的补丁，或者知道 Kafka 人员是否有一个即将推出的修复程序。

谢谢。

kafka.common.KafkaStorageException: Failed to change the log file suffix from  to .deleted for log segment 6711351
    at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:340)
    at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:342)
    at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:981)
    at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:971)
    at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:673)
    at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:673)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at kafka.log.Log.deleteOldSegments(Log.scala:673)
    at kafka.log.Log.deleteRetentionSizeBreachedSegments(Log.scala:717)
    at kafka.log.Log.deleteOldSegments(Log.scala:697)
    at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:474)
    at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:472)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at kafka.log.LogManager.cleanupLogs(LogManager.scala:472)
    at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:200)
    at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
    at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: c:\kafka-logs\kafka-logs\metric-values-0\00000000000006711351.log -> c:\kafka-logs\kafka-logs\metric-values-0\00000000000006711351.log.deleted: The process cannot access the file because it is being used by another process.

    at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
    at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
    at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:711)
    at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:210)
    ... 28 more
    Suppressed: java.nio.file.FileSystemException: c:\kafka-logs\kafka-logs\metric-values-0\00000000000006711351.log -> c:\kafka-logs\kafka-logs\metric-values-0\00000000000006711351.log.deleted: The process cannot access the file because it is being used by another process.

            at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
            at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
            at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
            at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
            at java.nio.file.Files.move(Files.java:1395)
            at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:708)
            ... 29 more

Answer 1

我在本地运行 kafka 时遇到了类似的问题，kafka 服务器似乎在无法删除日志文件时停止执行。为了避免这种情况发生的解决方案，我必须增加日志的日志保留时间以避免自动删除。

# The minimum age of a log file to be eligible for deletion due to age log.retention.hours=500

将日志设置为 xxx 小时将避免在本地运行时出现这种情况，但我认为对于基于 linux 的系统的生产来说，这不应该发生。

如果您需要删除日志文件，请手动将其删除到您的日志所在的位置，然后重新启动 kafka。