Amazon DynamoDB Local - 未知错误、异常或失败

【问题标题】：Amazon DynamoDB Local - unknown error, exception or failureAmazon DynamoDB Local - 未知错误、异常或失败
【发布时间】：2016-03-12 06:15:27
【问题描述】：

我必须测试一个严重依赖亚马逊 DynamoDB 的应用程序。我希望测试能够单独运行，这就是我选择 DynamoDB Local .jar 的原因。我知道最近的更新，使我们能够在没有外部 bash 命令调用的情况下运行它。但是，当我尝试运行指定为 here 的示例时，我得到以下堆栈跟踪：

Exception in thread "main" com.amazonaws.AmazonServiceException: The request processing has failed because of an unknown error, exception or failure. (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalFailure; Request ID: cab7a550-aaa6-4bfe-a591-0b255481cc14)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1805)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1223)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1235)

这是我要运行的代码：

public static void main( String[] args ) throws Exception
    {
        AmazonDynamoDB dynamodb = null;

        DynamoDBProxyServer server = null;
        final String[] localArgs = { "-inMemory", "-port", "13005" };
        server = ServerRunner.createServerFromCommandLineArgs(localArgs);
        server.start();

        BasicAWSCredentials auth = new BasicAWSCredentials("key", "secret");
        dynamodb = new AmazonDynamoDBClient(auth);
        dynamodb.setEndpoint("http://127.0.0.1:13005");

        // use the DynamoDB API over HTTP
        System.out.println(dynamodb.listTables());

        // Stop the DynamoDB Local endpoint
        if(server != null) {
            server.stop();
        }
    }

我观察到，如果我尝试完全从 Java 程序本身运行它，则会引发异常并且指定的端口不再可用（引发错误，说明该端口已被占用）。但是，如果我从命令提示符启动 DynamoDB Local 并将 Java 程序仅用作访问客户端，那么一切运行正常。 有什么建议吗？

【问题讨论】：

  • 这是在 Mac OS X 上吗？如果是这样，那可能有一个特定的问题。
  • 无论如何，我认为我现在在下面的答案中拥有成功的完整途径。呃，那很痛苦。
  • 您可能需要在根目录下启动本地 dynamodb 或使用 sudo

标签： amazon-web-services amazon-dynamodb dynamo-local


【解决方案1】：

DynamoDBLocal 至少存在 2 个问题。解决这两个问题，您将运行嵌入式 DynamoDB。

首先，-port 参数不能正常工作。所以 Jetty 没有设置在您期望的端口上。而是将端口 51205（或随机？）之类的端口设置为 Jetty 侦听器作为默认端口。

这是我如何启动服务器的代码，避免了内置的命令行解析，无论如何这更好......以这种方式启动服务器后，http://localhost:19444/shell 工作，所以 Jetty 很好。但是你可能对 Sqlite4java 有另一个问题（见代码块之后）。

注意： 代码是 Kotlin，但 Java 会非常相似。此外，我还配置了 SLF4j，并且不会让 ServerRunner 类破坏日志记录，因此这是启动服务器和 ServerRunner 在内部执行的双重更好的方法。

class TestAccountManager {
    companion object {
        private val localDbPort = 19444

        private lateinit var localDb: DynamoDBProxyServer
        private lateinit var dbClient: AmazonDynamoDBClient

        @BeforeClass @JvmStatic fun setup() {
            System.setProperty("org.eclipse.jetty.util.log.class", "org.eclipse.jetty.util.log.Slf4jLog")
            localDb = DynamoDBProxyServer(localDbPort, LocalDynamoDBServerHandler(
                    LocalDynamoDBRequestHandler(0, true, null, true, true), null)
            )
            localDb.start()

            val auth = BasicAWSCredentials("fakeKey", "fakeSecret")
            dbClient = AmazonDynamoDBClient(auth)
            dbClient.signerRegionOverride = "us-east-1"
            dbClient.setEndpoint("http://localhost:$localDbPort")
        }

        @AfterClass @JvmStatic fun teardown() {
           localDb.stop()
        }
    }

    @Test fun testSomething() {
        dbClient.listTables().tableNames.forEach {
            println(it)
        }
    }
}

获得此代码后，您现在正在预期的端口上运行。

现在您可能遇到了第二个错误，例如 Sqlite4java 没有为您的平台找到正确的二进制文件。对于某些版本的 Mac OSX，它会生成一个实际上不存在的二进制文件名。并且 DynamoDBLocal 强制隐藏所有 Sqlite4java 日志记录（不可能覆盖它），所以你不会看到它。

可以通过downloading a distribution测试sqlite库，解压后运行：

java -jar sqlite4java-1.0.392.jar -d

它会报告它尝试加载的内容，以及它是如何失败的。您只需要从 Gradle、Maven 或您放置它的任何位置在您的系统上找到该 JAR。我的错误输出是：

sqlite4java 392
160212:002049.833 FINE [sqlite] Internal: loading library
160212:002049.834 FINE [sqlite] Internal: java.library.path=/Users/jminard/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
160212:002049.834 FINE [sqlite] Internal: sqlite4java.library.path=null
160212:002049.834 FINE [sqlite] Internal: cwd=/Users/jminard/DEV/Collokia/repos/collokia-web-back/.
160212:002049.834 FINE [sqlite] Internal: default path=/Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d
160212:002049.834 FINE [sqlite] Internal: forced path=null
160212:002049.834 FINE [sqlite] Internal: os.name=mac os x; os=osx
160212:002049.835 FINE [sqlite] Internal: os.arch=x86_64
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d.dylib
160212:002049.837 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-1.0.392
160212:002049.838 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
160212:002049.838 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-1.0.392
160212:002049.839 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-1.0.392 in java.library.path
160212:002049.839 FINE [sqlite] Internal: trying to load sqlite4java-osx-1.0.392
160212:002049.840 FINE [sqlite] Internal: cannot load sqlite4java-osx-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-1.0.392 in java.library.path
160212:002049.840 FINE [sqlite] Internal: trying to load sqlite4java-1.0.392
160212:002049.841 FINE [sqlite] Internal: cannot load sqlite4java-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-1.0.392 in java.library.path
160212:002049.841 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d-1.0.392 in java.library.path
160212:002049.842 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-osx-d-1.0.392
160212:002049.843 FINE [sqlite] Internal: cannot load sqlite4java-osx-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-d-1.0.392
160212:002049.844 FINE [sqlite] Internal: cannot load sqlite4java-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-d-1.0.392 in java.library.path
160212:002049.844 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx
160212:002049.846 FINE [sqlite] Internal: cannot load sqlite4java-osx: java.lang.UnsatisfiedLinkError: no sqlite4java-osx in java.library.path
160212:002049.846 FINE [sqlite] Internal: trying to load sqlite4java
160212:002049.847 FINE [sqlite] Internal: cannot load sqlite4java: java.lang.UnsatisfiedLinkError: no sqlite4java in java.library.path
160212:002049.847 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d
160212:002049.848 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d in java.library.path
160212:002049.848 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d in java.library.path
160212:002049.849 FINE [sqlite] Internal: trying to load sqlite4java-osx-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d in java.library.path
160212:002049.850 FINE [sqlite] Internal: trying to load sqlite4java-d
160212:002049.850 FINE [sqlite] Internal: cannot load sqlite4java-d: java.lang.UnsatisfiedLinkError: no sqlite4java-d in java.library.path
Error: cannot load SQLite
java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at com.almworks.sqlite4java.Internal.tryLoadFromSystemPath(Internal.java:352)
    at com.almworks.sqlite4java.Internal.loadLibraryX(Internal.java:124)
    at com.almworks.sqlite4java.SQLite.main(SQLite.java:368)

如果您在加载动态库时遇到错误，请阅读以下内容了解如何解决：

似乎可靠的唯一答案是：

其中我执行类似于第一个选项的操作，并将库添加到项目中，并确保构建添加了 -Djava.library.path=./lib/sqlite4java，其中动态库已解压缩。要获得更有弹性的测试，您可以使用此技巧在代码中以编程方式设置java.library.path（否则在代码中设置时会被忽略）：http://blog.cedarsoft.com/2010/11/setting-java-library-path-programmatically/

DynamoDBLocal 隐藏了所有 Sqlite 错误是邪恶的，所以你必须调试才能找到静默失败。该库充满了调整日志记录级别并使其难以调试的东西，因为它们会破坏您查看错误的能力。例如每次打开 SqlLite 文件时（类SQLiteDBAccess）：

LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java");
LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java.Internal");
java.util.logging.Logger.getLogger("com.almworks.sqlite4java").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("com.almworks.sqlite4java.Internal").setLevel(Level.OFF);

结语：我正在寻找替代方案，这不是地球上最好的构建物，查看 DynamoDbLocal 的代码和为它做出的决定给了我对它的信心很小。 Alternator 或 Jcabi - Dynamo Mock 在我的名单上。

【讨论】：

  • 我之前尝试过 Alternator ——我真的很喜欢在同一个 JVM 中运行 DynamoDB 模拟以加快我的测试的想法。但是我发现了大量的边缘案例，它们会被真正的 DynamoDB 支持，但在 Alternator 中却失败了。再说一次，很久以前，如果你有好的经验，请分享:)。现在，由于您正在寻找替代方案，我在不久前使用 Docker 为一个宠物项目运行 DynamoDB Local（即“官方”版本）获得了非常成功的经验。我发现一个可以在 Docker Hub 上使用的镜像，设置起来非常简单。
  • @BrunoReis 太好了，现在我正在使用上面嵌入的官方 DynamoDbLocal 并等到它失败后再尝试另一个。
  • 在 Mac 上，将 dylib 保存到 /Library/Java/Extensions 就可以了！
【解决方案2】：

2018 年 8 月 Amazon announced 新 Docker image 板载 Amazon DynamoDB Local。它不需要下载和运行任何 JAR 以及使用第三方操作系统特定的二进制文件添加（我说的是sqlite4java）。

就像在测试前启动一个 Docker 容器一样简单：

docker run -p 8000:8000 amazon/dynamodb-local

您可以手动执行此操作以进行本地开发，如上所述。 IDE 通常会提供在执行任务之前运行任意命令的方法，因此您可以make IDE 为您启动容器。

或者您可以在 CI 管道中使用它。许多 CI 服务提供了在管道期间启动其他容器的能力，这些容器可以为您的测试提供依赖项。以下是 Gitlab CI/CD 的示例：

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://dynamodb-local:8000 ./gradlew clean test

或 Bitbucket 管道：

definitions:
  services:
    dynamodb-local:
      image: amazon/dynamodb-local
…
step:
  name: test
  image:
    name: openjdk:8-alpine
  services:
    - dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://localhost:8000 ./gradlew clean test

【讨论】：

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2012-02-03
    • 2015-08-14
    • 2021-02-02
    • 2021-11-12
    • 1970-01-01
    • 2021-05-03
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode