【问题标题】:Data inconsistency in multithread program多线程程序中的数据不一致
【发布时间】:2015-05-08 14:52:15
【问题描述】:

我在使用多线程编程时遇到了数据一致性问题。

用例:我将在队列中获取消息(人员信息)。我有一个多线程代码,它从队列中获取数据并将其放入另一个数据库。这里我需要比较个人信息,如果有重复,我需要合并/更新并插入到另一个数据库中。

问题:如果两个相似的人对象同时在两个不同的线程中,则两者都将这个人视为第二个数据库中不存在并且都尝试插入它 - 所以这里我们会有重复的记录。

我该如何解决上述问题?

从概念上讲,如果我知道该怎么做,我可以用 Java 编写代码,或者使用 Apache Storm 并运行并行进程。

【问题讨论】:

  • 您需要同步线程。
  • messages 队列是否有重复项?
  • 我遇到了类似的问题。所以我使用了分发器。如果没有类似的任务正在处理 atm,它基本上会抓取一个任务,然后它会创建一个新线程并将任务交给它。但是在我的情况下,Distrobuter 最多可以创建 400 个线程
  • @Dagriel 我应该处理大量数据,所以如果我同步它像单线程这样的东西,那么我的性能就会下降。
  • @JAtkin 是的,队列中的消息会有重复

标签: java multithreading parallel-processing apache-storm


【解决方案1】:

可能的解决方案:

  1. 插入队列时检查重复项。在队列之外维护一个哈希表。每次插入队列时,检查数据是否已经在哈希表中。如果是这样,丢弃插入。插入的复杂度仍然是 O(1),但增加了内存成本。

  2. 不是插入单个队列,而是根据哈希值插入多个队列。一个消费者线程处理一个队列。这也是维护时间序列数据的常用方法。

【讨论】:

    【解决方案2】:

    很久以前我写了一个简单的锁定机制,它锁定对象的值而不是实例的值。这有点慢,但如果你有一些在两个线程上相等的键,它可能会起作用。

    /*
     * Copyright (c) 2012, Isaiah van der Elst (isaiah.v@comcast.net)
     * All rights reserved.
     * 
     * Redistribution and use in source and binary forms, with or without
     * modification, are permitted provided that the following conditions are met:
     * 
     * - Redistributions of source code must retain the above copyright notice,
     *   this list of conditions and the following disclaimer.
     *   
     * - Redistributions in binary form must reproduce the above copyright notice,
     *   this list of conditions and the following disclaimer in the documentation
     *   and/or other materials provided with the distribution.
     *   
     * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
     * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
     * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
     * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
     * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
     * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
     * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
     * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
     * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
     * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
     * POSSIBILITY OF SUCH DAMAGE.
     */
    
    package org.gearman.impl.util;
    import java.util.HashMap;
    import java.util.Map;
    
    
    /**
     * A simple lock based on the value of an object instead of the object's instance.
     * 
     * The synchronizing problem in the server is that sometimes it's required to
     * synchronize on a key value for a hash table. However, the key being used will
     * never be the same instance from one thread to another, and synchronizing on the
     * hash table itself will be too slow.  Currently synchronization is done by this
     * lock which locks based on Object value, not the Object's instance.
     * 
     * Synchronization could have been done using primitive Objects, like Integers,
     * but I decided not to because this program is designed to be embedded. That
     * kind of synchronization may interfere with the wrapping program, possibly
     * causing a deadlock that is impossible to find. 
     * 
     * @author isaiah.v
     */
    public class EqualsLock {
    
            /** The set of all keys and lock owners */
            private final Map<Object, Thread> keys = new HashMap<Object, Thread>();
    
            /**
             * Accrues a lock for the given key. If this thread acquires a lock with
             * key1, any subsequent threads trying to acquire the lock with key2 will
             * block if key1.equals(key2).  If key1.equals(key2) is not true, the
             * subsequent thread will acquire the lock for key2 and continue execution.
             * 
             * @param key
             *              The key 
             */
            public final void lock(final Object key) {
                    boolean isInterrupted = false;
    
                    try {
                            synchronized(keys){
    
                                    while(!acquireLock(key, Thread.currentThread())) {
                                            keys.wait();
                                    }
                            }
    
                    } catch (InterruptedException e) {
                            // Ignore the interruption until we've finished
                            isInterrupted = Thread.interrupted();
                    }
    
                    if(isInterrupted) {
                            // re-interrupt thread if an interruption occured
                            Thread.currentThread().interrupt();
                    }
            }
    
            /**
             * Acquires the lock only if it is free at the time of invocation.
             * 
             * Acquires the lock if it is available and returns immediately with the
             * value true. If the lock is not available then this method will return
             * immediately with the value false.
             * 
             * @param key
             *              The key to acquire the lock
             * @return      
             *              true if the lock was acquired, false if the lock was not acquired 
             */
            public final boolean tryLock(final Object key) {
                    synchronized(keys) {
                            return acquireLock(key, Thread.currentThread());
                    }
            }
    
            /**
             * Releases the lock of the given key.  The lock is only released if the
             * calling thread owns the lock for the given key
             * 
             * @param key The key
             */
            public final void unlock(final Object key) {
                    synchronized(keys){
                            if(keys.get(key)==Thread.currentThread()) {
                                    keys.remove(key);
                                    keys.notifyAll();
                            }
                    }
            }
    
            /**
             * Adds the (Object,Thread) pair if the key is not already in the key set.
             * 
             * @param key   The key to add
             * @param t             The Thread to be associated with the key
             * @return
             *              true if the Thread t and the Object key is successfully added, or
             *              Thread t is already associated with Object key. false if the Object
             *              key has already been added but Thread t is not associated with it.
             */
            private final boolean acquireLock (final Object key, final Thread t) {
                    final Thread value = keys.get(key);
    
                    if(value == t)
                            return true;
                    if(value != null)
                            return false;
    
                    keys.put(key, t);
                    return true;
            }
    }
    

    【讨论】:

    • 注意:如果线程在锁定时被中断,我想我会看到一个错误。
    【解决方案3】:

    如果您使用的数据库支持事务和事务隔离,您可以依赖它。您可能需要使用Serializable 隔离级别来避免Phantom Reads。每个验证 + 更新/插入操作都应在单个事务中执行。

    解释: 您描述的问题是一种并发效应。它被称为幻读。想象一下,首先您使用选择查询检查数据库表是否已经包含一个名字为“test”的人。该查询返回一个空的结果集。所以你决定把这个人插入数据库。与此同时,在您发出选择查询之后但在您发出插入查询之前,另一个线程正在尝试做同样的事情,即检查数据库是否包含名为“test”的人。第二个线程将人员插入数据库。如果第一个线程在执行插入后发出相同的选择查询并观察到有 2 行而不是预期的 1 行(它刚刚插入),那就是幻读。您可以在这篇 Wikipedia 文章 Isolation (database systems)

    中阅读有关隔离和并发效果的更多信息

    如果您的数据库不支持事务或可序列化隔离级别,您将需要自己同步。如果所有线程都在一个 JVM 中,您可以使用同步关键字或 ReentrantReadWriteLock。如果线程在不同的 JVM 中,您可以使用分布式锁服务(Terracotta 或 Hazelcast)。

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多