Nexus Repository 版本 3.2.0-01 中的 OrientDB 损坏状态答案

【问题标题】：OrientDB corruption state in Nexus Repository version 3.2.0-01Nexus Repository 版本 3.2.0-01 中的 OrientDB 损坏状态
【发布时间】：2017-08-14 13:42:09
【问题描述】：

我们使用的是 Nexus Repository 版本 3.2.0-01，并开始在 nexus.log 中收到以下错误。 OrientDB 已损坏。 Nexus 不再启动。

2017-03-21 13:00:36,329+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - OrientDB version: 2.2.13
2017-03-21 13:00:36,348+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.13 is starting up...
2017-03-21 13:00:36,355+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Databases directory: /c9/setup/apps/nexus/sonatype-work/nexus3/db
2017-03-21 13:00:36,647+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies - Not enough physical memory available for DISKCACHE: 1,873MB (heap=1,161MB direct=2,048MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB
2017-03-21 13:00:36,648+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies - OrientDB config DISKCACHE=256MB (heap=1,161MB direct=2,048MB os=1,873MB)
2017-03-21 13:00:36,735+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Found ORIENTDB_ROOT_PASSWORD variable, using this value as root's password
2017-03-21 13:00:36,899+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.handler.OJMXPlugin - JMX plugin installed and active: 

profilerManaged=true
2017-03-21 13:00:36,901+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Studio available at $ANSI{blue 

http://localhost:2480/studio/index.html}
2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - $ANSI{green:italic OrientDB Server is active} v2.2.13.
2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Activated
2017-03-21 13:00:36,904+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
2017-03-21 13:00:37,632+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed
com.orientechnologies.orient.core.exception.OSecurityAccessException: User or password not valid for database: 'config'^M
        DB name="config"
        at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityExternal.authenticate(OSecurityExternal.java:62) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityProxy.authenticate(OSecurityProxy.java:107) [na:na]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:274) [na:na]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:142) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56) [na:na]
        at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:258) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:232) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:61) [org.sonatype.nexus.base:3.2.0.01]
        at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:1) [org.sonatype.nexus.base:3.2.0.01]
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
        at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:133) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [com.google.inject:4.1.0]
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
         at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.BeanProviders.firstOf(BeanProviders.java:179) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.PlaceholderBeanProvider.lookup(PlaceholderBeanProvider.java:122) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.PlaceholderBeanProvider.get(PlaceholderBeanProvider.java:90) [org.eclipse.sisu.inject:0.3.2]
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
        at org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176) [org.eclipse.sisu.inject:0.3.2]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore.doStart(ModelVersionStore.java:82) [org.sonatype.nexus.upgrade:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.CGLIB$start$5(<generated>) [3.2.0-01:na]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f$$FastClassByGuice$$94ba58c4.invoke(<generated>) [3.2.0-01:na]
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [com.google.inject:4.1.0]
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) [com.google.inject:4.1.0]
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.2.0.01]
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) [com.google.inject:4.1.0]
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [com.google.inject:4.1.0]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.start(<generated>) [3.2.0-01:na]
        at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:80) [org.sonatype.nexus.upgrade:3.2.0.01]
        at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:154) [org.sonatype.nexus.extender:3.2.0.01]
        at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:94) [org.sonatype.nexus.extender:3.2.0.01]
        at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:185) [org.sonatype.nexus.extender:3.2.0.01]
        at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.4.0.jar:na]
        at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.4.0.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

根本原因 发现此问题的根本原因是由于磁盘空间已满，Nexus 未正常关闭，这导致了数据库损坏。

2017-03-17 07:09:49,480+0000 WARN  [elasticsearch[A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4][local_transport][T#1]] *SYSTEM org.elasticsearch.cluster.action.shard - [A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4] [cab22f445c1892acef64d17ac2da6d0086023e21][0] received shard failed for [cab22f445c1892acef64d17ac2da6d0086023e21][0], node[HBjARpI7RyqqqruVqEJ8Ig], [P], v[11], s[INITIALIZING], a[id=FmxOHS7nQcGNz_93VlTpKg], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-03-17T07:09:49.101Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [component#36e3dec8de528c9b41c5c6ac3b18e3f7]]; nested: NotSerializableExceptionWrapper[No space left on device]; ]], indexUUID [fLN4hdE8QPKH7Pj5RpLtmw], message [engine failure, reason [index]], failure [NotSerializableExceptionWrapper[No space left on device]]
org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [na:1.8.0_121]
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) [na:1.8.0_121]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [na:1.8.0_121]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) [na:1.8.0_121]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) [na:1.8.0_121]
at org.elasticsearch.common.io.Channels.writeToChannel(Channels.java:211) [na:na]
at org.elasticsearch.common.bytes.PagedBytesReference.writeTo(PagedBytesReference.java:132) [na:na]
at org.elasticsearch.index.translog.BufferingTranslogWriter.add(BufferingTranslogWriter.java:57) [na:na]

无备份 我们还没有备份 Nexus VM，因此我们无法恢复它。什么是可能的出路 - 重新创建数据库 - 损害最小。感谢任何帮助。

【问题讨论】：

标签： nexus orientdb nexus3

【解决方案1】：

我们在 Nexus OSS 版本 3.2.0-01 和 OrientDB 损坏状态中遇到了类似的问题，因为不干净的关闭。

在我们的例子中，损坏的存储是组件位于

$install-dir/sonatype-work/nexus3/db/component

2017-04-14 15:09:20,007+0200 ERROR [qtp18266400-74] *UNKNOWN com.google.common.eventbus.EventBus.reentrant - Could not dispatch event AssetUpdatedEvent{metadata=AttachedEntityMetadata{schema=asset, document=asset#18:91856{bucket:#13:3,format:maven2,last_updated:Fri Apr 14 15:09:19 CEST 2017,attributes:[5],component:null,name:org/apache/maven/plugins/maven-jar-plugin/maven-metadata.xml,size:743,content_type:application/xml,blob_ref:default@ECF4439E-D7F184F8-578BAB14-ACFC0FB8-5589B163:3fb95745-4aed-460f-b711-b5f5b74b837c,last_accessed:Fri Apr 14 15:09:19 CEST 2017} v466}, remoteNodeId=null} to subscriber org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet$$EnhancerByGuice$$a88d7327@83eca7 method [public void org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.on(org.sonatype.nexus.repository.storage.AssetEvent)]
com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit
        DB name="component"
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeRollback(OAbstractPaginatedStorage.java:2210) [na:na]
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1433) [na:na]
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:539) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:104) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2756) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2725) [com.orientechnologies.orientdb-core:2.2.13]
        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:172) [org.sonatype.nexus.repository:3.2.0.01]
        at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source) [na:na]

我们设法通过以下策略解决了我们的问题

调度 Nexus 任务，特别是：
- 重建 Maven 存储库元数据。
- 重建存储库索引。
- 紧凑的 blob 存储。
东方数据库操作：
- 重建索引。
- 正在修复数据库。

根据 Nexus 支持，您可以使用 Nexus OSS 版本 3.2.1-01 访问 Orient DB 控制台。一切都解释了here。在对数据库应用任何操作之前，请确保您的 Nexus 应用程序已正确关闭。

在 Orient DB 控制台中，您连接到本地数据库

> CONNECT PLOCAL:${your_install_dir}/sonatype-work/nexus3/db/${database} admin admin

然后你执行以下操作：

> REBUILD INDEX *
> REPAIR DATABASE --fix-graph
> REPAIR DATABASE --fix-links
> REPAIR DATABASE --fix-ridbags
> REPAIR DATABASE --fix-bonsai
> DISCONNECT

我绝对知道我的回答可能无法解决您的具体问题，但我仍然想给您一些指导，因为您没有任何备份。

【讨论】：

在我们的案例中，当空间不足时，wal 文件已损坏，我们无法从 orient DB 控制台连接到数据库。
对我来说，起初这似乎可行，但推送仍然给了 500。最后不得不从备份中恢复

【解决方案2】：

以防万一这对其他人来说真的很重要 - 以上内容对我没有帮助，但 json 导出/导入之一。

在我的例子中，nexus 在官方 docker 镜像中运行，所以

docker exec -it nexus bash

或者无论你如何访问你的 nexus shell

cd /tmp
/opt/jdk1.8.0_141/bin/java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar

CONNECT plocal:/nexus-data/db/component admin admin

export database component-export
drop database
create database plocal:/nexus-data/db/component
import database component-export.json.gz

在您的情况下，java 路径可能会有所不同，我们用于连接/创建数据库的数据路径（关联数据）也可能会有所不同。

之后，重新启动服务，您的组件数据库应该会重新启动并运行

【讨论】：

我必须使用CREATE DATABASE PLOCAL:/usr/local/orientdb/databases/demo 来创建数据库
非常感谢分享，也为我工作。解决了由完整磁盘引起的组件相同的损坏问题。也 +1 @Markus，必须前缀 plocal:，否则我得到 incorrelt url 异常

【解决方案3】：

数据库预写日志已损坏。

备份$install-dir/sonatype-work/nexus3/db/component，然后从该目录中删除扩展名为.wal 的文件。然后再次尝试启动

如果问题仍然存在，请尝试清除“DB”目录下的所有.wal文件，即可解决问题，

我确实尝试过这种方法，效果很好。

问候维奈

【讨论】：

【解决方案4】：

使用 Nexus 3.16 现在有一个任务： 创建修复 - 从 blob 存储协调组件数据库

超级慢，但它有助于重新创建仍在磁盘上的意外删除的 blobstore。

【讨论】：

【解决方案5】：

删除缓存文件夹 (/opt/sonatype-work/nexus3/cache) 为我解决了这个问题

【讨论】：

如果你和我一样，只是重新安装，我想这就是解决方案。

【解决方案6】：

我也犯了同样的错误：

无法使用 mode=rw 打开本地存储“/nexus-data/db/config”

在启动 nexus 控制台 (java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar) 时出现以下消息：

原因是 Nexus 内存不足，并在文件夹“/nexus-data/db/config/”中创建了一些 0 字节的“.wal”文件。我删除它们后，一切都恢复了。

也非常感谢 yobert https://gist.github.com/marcelmaatkamp/123e8793e07a72a382d8d0e8d66bbd8f

【讨论】：