简单使用 Solr 时如何解决“锁定获取超时”？答案

【问题标题】：How to solve the 'Lock obtain timed out' when using Solr plainly?简单使用 Solr 时如何解决“锁定获取超时”？
【发布时间】：2013-12-29 07:41:23
【问题描述】：

我的 Solr 系统有两个内核（Solr 版本 3.6.1）。当我在我们的专用 Solr 服务器上调用以下命令行来添加并索引文件时：

java -Durl=http://solrprod:8080/solr/original/update -jar /home/solr/solr3/biomina/solr/post.jar /home/solr/tmp/2008/c2m-dump-01.noDEID_clean.xml

/usr/share/tomcat7/logs/solr.2013-12-11.log 文件中出现异常（大约等待 6 分钟后）：

SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/solr/solr3/biomina/solr/original/data/index/write.lock

（您可以在此消息的末尾看到它的详细输出）。

我尝试修改锁的超时时间（通过将writeLockTimeout 设置为300000），但这并没有解决问题。我没有使用任何自定义脚本，只是使用 Solr 3.1.6 附带的 post.jar 来添加和索引。

关于需要更改哪些内容以消除此错误并成功将 XML 文件添加到 Solr 并对其进行索引有什么想法吗？

/home/solr/solr3/biomina/solr/solr.xml的内容：

<?xml version="1.0" encoding="UTF-8" ?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->

<!--
 All (relative) paths are relative to the installation path

  persistent: Save changes made via the API to this file
  sharedLib: path to a lib directory that will be shared across all cores
-->
<solr persistent="true">

  <!--
  adminPath: RequestHandler path to manage cores.  
    If 'null' (or absent), cores will not be manageable via request handler
  -->
  <cores adminPath="/admin/cores">
    <core name="original" instanceDir="original" />
    <core name="deidentified" instanceDir="deidentified" />
  </cores>
</solr>

solrconfig.xml 的相关部分（对于名为original 的核心）：

<indexConfig>
    <!-- maxFieldLength specifies max number of *tokens* indexed per 
field. Default: 10000 -->
    <!-- <maxFieldLength>10000</maxFieldLength>  -->
    <!-- Maximum time to wait for a write lock (ms) for an IndexWriter. 
Default: 1000 -->
    <writeLockTimeout>300000</writeLockTimeout>

solrconfig.xml 的相关部分（对于名为deidentified 的核心）：

<indexConfig>
    <!-- maxFieldLength specifies max number of *tokens* indexed per 
field. Default: 10000 -->
    <!-- <maxFieldLength>10000</maxFieldLength>  -->
    <!-- Maximum time to wait for a write lock (ms) for an IndexWriter. 
Default: 1000 -->
    <writeLockTimeout>300000</writeLockTimeout>

异常的详细输出

Dec 11, 2013 11:27:25 AM org.apache.solr.core.SolrCore execute
INFO: [original] webapp=/solr path=/update params={} status=500 QTime=300070 
Dec 11, 2013 11:32:25 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/solr/solr3/biomina/solr/original/data/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:84)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
    at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:84)
    at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:101)
    at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:171)
    at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:219)
    at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
    at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
    at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:157)
    at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
    at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626)
    at java.lang.Thread.run(Thread.java:804)

Dec 11, 2013 11:32:25 AM org.apache.solr.core.SolrCore execute
INFO: [original] webapp=/solr path=/update params={} status=500 QTime=556916

系统详情：

uname -a

Linux solrprod 3.0.93-0.8-default #1 SMP Tue Aug 27 08:44:18 UTC 2013 (70ed288) x86_64 x86_64 x86_64 GNU/Linux

java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr6-20131015_01(SR6))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20131013_170512 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR6_20131013_1510_B170512
JIT  - r11.b05_20131003_47443
GC   - R26_Java726_SR6_20131013_1510_B170512_CMPRSS
J9CL - 20131013_170512)
JCL - 20131011_01 based on Oracle 7u45-b18

【问题讨论】：

标签： java xml linux solr lucene

【解决方案1】：

以下修改解决了这个问题：

应用了https://stackoverflow.com/a/3035916/236007中描述的更改
切换到 Oracle Java 运行时（它是 IBM Java 运行时）。
将ulimit -v unlimited 放入/etc/init.d/tomcat7。
修改/usr/share/tomcat7/bin/setenv.sh文件如下（给它大约4 GB内存）：
export JAVA_OPTS="$JAVA_OPTS -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/data/tomcat_dump"

【讨论】：

【解决方案2】：

Lock obtain timed out 的 write.lock 文件出现问题。它发生在一个相当硬的重置之后，结果我在重新启动后有两个进程正在运行，因为第一个进程之前被不优雅地杀死了。

运行ps aux | grep solr并杀死损坏的进程并让另一个启动然后解决问题。

【讨论】：

是的。如果一切都失败了，也可以停止 solr 进程（如果 solr 在 tomcat 下运行，则可以停止 tomcat），手动删除 write.lock 文件，然后启动 solr 进程。
求你的建议。我刚重启了tomcat，就生效了。

【解决方案3】：

我在使用一般 Lucene 库时遇到了这些错误，问题是文件系统错误，即在 fsck 修复后可重现的错误消失了。我在这个问题中添加了这个答案，因为我首先找到了这个问题。

【讨论】：