从事务性 Kafka 侦听器发送消息：尝试从状态 IN_TRANSACTION 到状态 IN_TRANSACTION 的无效转换

Question

我正在尝试在同一事务中从 Kafka 侦听器发送消息以在同一事务中发送消息并提交偏移量，但收到异常：尝试从状态 IN_TRANSACTION 到状态 IN_TRANSACTION 的无效转换，并且事务回滚.

我使用以下方法为我的应用程序（Spring Boot 2、Spring Kafka 2.1.4）添加了事务支持。

application.properties

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.group-id=transaction-sample
spring.kafka.listener.ack-mode=RECORD

spring.kafka.producer.transaction-id-prefix=transaction-sample-${random.uuid}

我已将 kafkaTransactionManager 设置为 kafkaListenerContainerFactory containerProperties：

@SpringBootApplication
public class KafkaTransactionsSampleApplication {

  @Bean
  public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
      ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
      ConsumerFactory<Object, Object> kafkaConsumerFactory,
      KafkaTransactionManager<Object, Object> kafkaTransactionManager) {
    ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
    configurer.configure(factory, kafkaConsumerFactory);
    factory.getContainerProperties().setTransactionManager(kafkaTransactionManager);
    return factory;
  }

  public static void main(String[] args) {
    SpringApplication.run(KafkaTransactionsSampleApplication.class, args);
  }
}

测试监听器应该从输入主题（transaction-sample-topic-in）接收消息，记录它，并将其转发到输出主题（transaction-sample-topic-out）。我希望在同一个事务中执行消息发送和偏移提交。

@Component
public class TestKafkaListener {

  private static final Logger LOGGER = LoggerFactory.getLogger(TestKafkaListener.class);

  public static final String INPUT_TEST_TOPIC = "transaction-sample-topic-in";
  public static final String OUTPUT_TEST_TOPIC = "transaction-sample-topic-out";

  @Autowired
  private KafkaTemplate<String, String> kafkaTemplate;

  @KafkaListener(topics = INPUT_TEST_TOPIC)
  public void listen(ConsumerRecord<String, String> record) {
    LOGGER.info("Received Kafka record from {}: {}", INPUT_TEST_TOPIC, record);
    kafkaTemplate.send(OUTPUT_TEST_TOPIC, record.key(), record.value());
    LOGGER.info("Forwarded Kafka record to {}: {}", OUTPUT_TEST_TOPIC, record);
  }
}

在测试中，我定义了具有 borker 属性的嵌入式 Kafka，允许使用 1 个代理实例支持事务。

@Bean
public KafkaEmbedded kafkaEmbedded() {
  KafkaEmbedded kafkaEmbedded = new KafkaEmbedded(1, false, 1,
      INPUT_TEST_TOPIC, OUTPUT_TEST_TOPIC);
  kafkaEmbedded.setKafkaPorts(9092);
  kafkaEmbedded.brokerProperty(KafkaConfig.TransactionsTopicReplicationFactorProp(), "1");
  kafkaEmbedded.brokerProperty(KafkaConfig.TransactionsTopicMinISRProp(), "1");
  return kafkaEmbedded;
}

当我运行一个向输入主题发送消息的简单测试时，我收到一个异常：尝试从状态 IN_TRANSACTION 到状态 IN_TRANSACTION 的无效转换。然后事务回滚，消息不会转发到输出主题。

@TestPropertySource("classpath:test.properties")
@RunWith(SpringRunner.class)
@SpringBootTest
public class KafkaListenerTest {

  @Autowired
  private KafkaEmbedded kafkaEmbedded;

  @Autowired
  private KafkaTemplate<String, String> kafkaTemplate;

  @Test
  public void shouldProcessEvent() throws Exception {
    String testKey = "test_key";
    String testData = "test_data";

    kafkaTemplate.send(INPUT_TEST_TOPIC, testKey, testData);

    try (Consumer<String, String> consumer = createConsumer()) {
      kafkaEmbedded.consumeFromAnEmbeddedTopic(consumer, OUTPUT_TEST_TOPIC);
      ConsumerRecords<String, String> records = KafkaTestUtils.getRecords(consumer);
      Iterator<ConsumerRecord<String, String>> iterator = records.iterator();
      ConsumerRecord<String, String> record = iterator.next();

      assertEquals(testKey, record.key());
      assertEquals(testData, record.value());
      assertFalse(iterator.hasNext());
    }
  }

  private Consumer<String, String> createConsumer() {
    Map<String, Object> consumerProps =
        KafkaTestUtils.consumerProps("test-consumer", "true", kafkaEmbedded);
    consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    DefaultKafkaConsumerFactory<String, String> cf = new DefaultKafkaConsumerFactory<>(
        consumerProps, new StringDeserializer(), new StringDeserializer());
    return cf.createConsumer();
  }
}

例外是

2018-04-13 16:22:46.856 ERROR 141936 --- [ntainer#0-0-C-1] essageListenerContainer$ListenerConsumer : Transaction rolled back

org.springframework.transaction.CannotCreateTransactionException: Could not create Kafka transaction; nested exception is org.apache.kafka.common.KafkaException: TransactionalId transaction-sample-209b149f-7f97-42f6-82e7-257e1ac0d1950: Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION
    at org.springframework.kafka.transaction.KafkaTransactionManager.doBegin(KafkaTransactionManager.java:141) ~[spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378) ~[spring-tx-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137) ~[spring-tx-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.innvokeRecordListenerInTx(KafkaMessageListenerContainer.java:949) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:929) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:801) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:689) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_144]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.apache.kafka.common.KafkaException: TransactionalId transaction-sample-209b149f-7f97-42f6-82e7-257e1ac0d1950: Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION
    at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:755) ~[kafka-clients-1.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:749) ~[kafka-clients-1.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.TransactionManager.beginTransaction(TransactionManager.java:215) ~[kafka-clients-1.0.0.jar:na]
    at org.apache.kafka.clients.producer.KafkaProducer.beginTransaction(KafkaProducer.java:564) ~[kafka-clients-1.0.0.jar:na]
    at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.beginTransaction(DefaultKafkaProducerFactory.java:285) ~[spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.kafka.core.ProducerFactoryUtils.getTransactionalResourceHolder(ProducerFactoryUtils.java:60) ~[spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.kafka.transaction.KafkaTransactionManager.doBegin(KafkaTransactionManager.java:126) ~[spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    ... 9 common frames omitted

Answer 1

请不要在多个地方问同一个问题；它浪费了我们和你的时间。

我已在您的 GitHub 问题上对此作出回应：

您的测试send() 需要在自己的事务中运行：

kafkaTemplate.executeInTransaction(kt ->
    kt.send(INPUT_TEST_TOPIC, testKey, testData));

（或不使用事务模板）。

我还修正了你的断言：

assertEquals(testKey, record.key());
assertEquals(testData, record.value());

（他们都在比较值）。