Cassandra 另一个帧大小问题

Question

我知道 Apache Cassandra 的这种问题是不久前的，例如： Pentaho Frame size (17727647) larger than max length (16384000)! 和 thrift_max_message_length_in_mb not recognized Cassandra

但我的问题有点不同。我使用 Cassandra v2.0.7，我想将 Pentaho Kettle (5.1) 中的数据插入 Cassandra。我必须启用加密（SSL），所以我修改了 cassandra.yaml

client_encryption_options:
enabled: true
keystore: /usr/cassandra/cassandra/conf/.keystore
keystore_password: password
require_client_auth: true
# Set trustore and truststore_password if require_client_auth is true
truststore: /usr/cassandra/cassandra/conf/.truststore
truststore_password: password

现在，出现了一种奇怪的情况，因为当我禁用加密（启用：false）并通过 Pentaho 插入数据时一切正常。但是当我启用加密时，我得到了这个：

   2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : A problem occurred during initializ
ation of the step
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.apache.thrift.transport.TTransp
ortException: Frame size (352518400) larger than max length (16384000)!
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.recv_set_cql_version(Cassandra.java:1855)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.set_cql_version(Cassandra.java:1842)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.checkOpen(CassandraConnection.java:159)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.setKeyspace(CassandraConnection.java:174)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.LegacyKeyspace.setKeyspace(LegacyKeyspace.java:100)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.getKeyspace(CassandraConnection.java:277)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.initialize(CassandraOutput.java:218)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:353)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/05 17:42:38 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
        at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Unexpected error
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : java.lang.NullPointerException
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/05 17:42:38 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
child index = 1, logging object : org.pentaho.di.core.logging.LoggingObject@47406edf parent=277afd35-cec6-4972-a572-a68a58ff9ae7
2015/01/05 17:42:38 - t_product_rb - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
2015/01/05 17:42:38 - t_product_rb - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

我使用 CQL3 插入数据。 现在，我发现在 cassandra.yaml 中有两个属性： -thrift_framed_transport_size_in_mb 和 -thrift_max_message_length_in_mb

我用这个属性检查了一些配置。一旦我添加了这个 thrift_max_message_length_in_mb ，然后我删除了这个等等。但是每次我得到这个错误。

我注意到加密在我的框架中添加了一些重要的东西，但我不知道是什么以及如何。 有人知道如何解决这个问题吗？

有关此问题的其他一些信息： https://issues.apache.org/jira/browse/THRIFT-1324 https://issues.apache.org/jira/browse/THRIFT-1323

编辑 我再次注意到，当我将端口从 9160 更改为 9042（使用 CQL3）时，我得到了不同的错误。

2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : A problem occurred during initializ
ation of the step
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.apache.thrift.transport.TTransp
ortException: java.net.SocketTimeoutException: Read timed out
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.recv_set_cql_version(Cassandra.java:1855)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.set_cql_version(Cassandra.java:1842)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.checkOpen(CassandraConnection.java:159)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.setKeyspace(CassandraConnection.java:174)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.LegacyKeyspace.setKeyspace(LegacyKeyspace.java:100)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.getKeyspace(CassandraConnection.java:277)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.initialize(CassandraOutput.java:218)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:353)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
2015/01/09 12:53:21 - Cassandra Output.0 - Caused by: java.net.SocketTimeoutException: Read timed out
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.socketRead0(Native Method)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.read(SocketInputStream.java:152)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.read(SocketInputStream.java:122)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
2015/01/09 12:53:21 - Cassandra Output.0 -      ... 18 more
java.lang.NullPointerException
        at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Unexpected error
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : java.lang.NullPointerException
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
child index = 4, logging object : org.pentaho.di.core.logging.LoggingObject@6a4ea42b parent=c8d41836-602c-4a17-8977-62d691c419c5
2015/01/09 12:53:21 - Cassandra Output.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2015/01/09 12:53:21 - t_customer_cm - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
2015/01/09 12:53:21 - t_customer_cm - Transformation detected one or more steps with errors.
2015/01/09 12:53:21 - t_customer_cm - Transformation is killing the other steps!
2015/01/09 12:53:21 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=10002, W=10002, U=0, E=0)
2015/01/09 12:53:21 - Get System Info.0 - Finished processing (I=0, O=0, R=20003, W=20003, U=0, E=0)
2015/01/09 12:53:21 - t_customer_cm - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

在 cassandra 上我得到了这个：

ERROR 12:53:11,807 Unexpected exception during request
 org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00000028800100010000000f7365745f63716c5f76657273696f6e000000010b0001000000
5332e302e3100
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:871)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

有什么想法吗？

Answer 1

Frame size (352518400) larger than max length (16384000)!

我注意到加密在我的框架中添加了一些重要的东西，但我不知道是什么以及如何。有人知道如何解决这个问题吗？

关于帧大小，它是什么并不重要，重要的是它有多大。正如 Bryan 在 THRIFT-1324 中所写，成帧和“未成帧”之间大小的唯一区别是常量 4 字节（int 32），位于数据块之前并保持其大小。

理论上的解决方案对我来说似乎很明显：找出你最大的框架有多大，并相应地配置设置。每个请求和每个响应都必须符合这一点。

理论上，因为启用加密会使您的数据增加 21 倍的大小似乎很奇怪（您说它可以在没有的情况下工作，所以它必须是

Answer 2

检查以确保您是否使用不安全的 CQL 二进制协议（端口 9042）进行连接。堆栈跟踪看起来表明正在使用 Thrift，但 CQL3 使用完全不同的接口。是否有可能在使用 SSL 时使用节俭协议（端口 9160）而在使用不安全时使用二进制协议（端口 9042）？

编辑：这个答案可能不是一个好答案。你提到使用 CQL3 让我觉得 thrift 甚至不应该参与，仍然值得一看，看看不安全的版本是使用本机二进制协议还是 thrift。