从大型未排序数组中删除重复项并保持顺序

Question

我有一个未排序的整数数组，其中值的范围从 Integer.MIN_VALUE 到 Integer.MAX_VALUE。数组中的任何整数都可以有多个重复项。 我需要返回一个删除了所有重复项并保持元素顺序的数组。

示例：

int[] input = {7,8,7,1,9,0,9,1,2,8}

输出应该是 {7,8,1,9,0,2}

我知道这个问题可以使用LinkedHashSet 解决，但我需要一个不涉及大量缓冲区空间的解决方案。

Answer 1

您可以使用 java 8 Arrays stream.distinct() 方法从数组中获取不同的值，它将仅保留输入顺序

public static void main(String[] args) {
    int[] input = {7,8,7,1,9,0,9,1,2,8};
    int[] output = Arrays.stream(input).distinct().toArray();
    System.out.println(Arrays.toString(output)); //[7, 8, 1, 9, 0, 2]
}

Answer 2

一个聪明的方法是使用LinkedHashSet 来表示输入数组。 LinkedHashSet 具有维护插入顺序（链表行为）的属性，但也会忽略再次插入的相同键（映射行为）。这意味着，例如，值7 只会在第一次出现时插入到列表/映射中一次。这是我们想要的行为。

LinkedHashSet<Integer> lhs = new LinkedHashSet<>();
int[] input = new int[] {7, 8, 7, 1, 9, 0, 9, 1, 2, 8};
for (int val : input) lhs.add(val);
int[] output = new int[lhs.size()];
int i = 0;
for (Integer val : lhs) {
    output[i++] = val;
}
System.out.println(Arrays.toString(output));

[7, 8, 1, 9, 0, 2]

Demo

Answer 3

就这样直截了当

    int[] input =  {7,8,7,1,9,0,9,1,2,8};
    ArrayList<Integer> list = new ArrayList<>();
    for(int i = 0;i<input.length;i++)
    {
        if(!list.contains(input[i]))
        {
            list.add(i);
        }
    }
    int[] output = new int[list.size()];
    for(int i = 0;i<list.size();i++)
    {
        output[i] = list.get(i);
    }

Answer 4

您可以使用HashSet 代替LinkedHashSet，但这仍然使用缓冲区：

 private static int[] noDups(int[] arr) {
    Set<Integer> set = new HashSet<>();
    int nextIndex = 0;
    for (int i = 0; i < arr.length; ++i) {
        if (set.add(arr[i])) {
            arr[nextIndex++] = arr[i];
        }

    }

    return Arrays.copyOfRange(arr, 0, nextIndex);
}

请注意，这会改变原始数组，如果您不希望需要另一个调用来创建副本：int[] copy = Arrays.copyOfRange(arr, 0, arr.length);

否则，如果您想在原地执行此操作，而没有 any 额外的缓冲区，则必须通过嵌套数组迭代数组，复杂性将变为：O(n*n)，这很漂亮不好。

Answer 5

我尝试了几种方法 - 使用 Map 和 Set 来获得不同的元素。根据要求，这些不要使用LinkedHashSet 或LinkedHashMap。此外，订单被保留。

我在这两种情况下都使用了这个示例输入数组并得到了预期的输出。
输入：int [] arr = new int [] {1, 99, 2, 82, 99, -20, 9, 2, 9, 45, -319, 1};
结果：[1, 99, 2, 82, -20, 9, 45, -319]

代码示例：

使用地图：

int [] arrayWithDistictElements = new int [arr.length];
int noOfDistinctElements = 0;
HashMap<Integer, Integer> map = new HashMap<>();

for (int i = 0, j = 0; i < arr.length; i++) {

    if (map.put(arr [i], 1) == null) {

        arrayWithDistictElements [j++] = arr [i];
        ++noOfDistinctElements;
    }
}

int [] result = Arrays.copyOf(arrayWithDistictElements, noOfDistinctElements);

使用集合：

Set<Integer> set = new HashSet<>();
int [] arrayWithDistictElements = new int [arr.length];
int noOfDistinctElements = 0;

for (int i = 0, j = 0; i < arr.length; i++) {

    if (set.add(arr [i])) {

        arrayWithDistictElements [j++] = arr [i];
        ++noOfDistinctElements;
    }
}

int [] result = Arrays.copyOf(arrayWithDistictElements, noOfDistinctElements);

守则：

这是我尝试过的测试的完整代码和一些结果：

import java.util.*;
import java.util.stream.*;
import java.time.*;
import java.text.*;
public class UniqueArrayTester {

    private final static int ARRAY_SIZE = 10_000_000;
    private static Random r = new Random();

    public static void main(String [] args) {

        DecimalFormat formatter = new DecimalFormat("###,###,###,###");
        System.out.println("Input array size: " + formatter.format(ARRAY_SIZE));

        for (int i = 0; i < 5; i++) {

            // For testing with a small input and print the result use this as input:
            //int [] arr = new int [] {1, 99, 2, 82, 99, -20, 9, 2, 9, 45, -319, 1};
            //System.out.println(Arrays.toString(arr));

            System.out.println("[Test " + Integer.toString(i+1) + "]");
            int [] arr = getArray();
            process1(arr);
            process2(arr);
            process3(arr);
        }
    }

    private static int [] getArray() {  
        return IntStream.generate(() -> r.nextInt())
                        .limit(ARRAY_SIZE)
                        .toArray();
    }

    /*
     * Process uses Stream API.
     */
    private static void process1(int [] arr) {
        LocalTime time1 = LocalTime.now();
        int [] result = IntStream.of(arr).distinct().toArray();
        LocalTime time2 = LocalTime.now();
        System.out.println("Process 1 (using streams) out array size: " + result.length);
        System.out.println("    Duration in millis: " + Duration.between(time1, time2).toMillis());
        //System.out.println(Arrays.toString(result));
    }

    /*
     * Process uses a Map to arrive at distinct elements.
     */ 
    private static void process2(int [] arr) {
        LocalTime time1 = LocalTime.now();
        int [] arrayWithDistictElements = new int [arr.length];
        int noOfDistinctElements = 0;
        HashMap<Integer, Integer> map = new HashMap<>();
        for (int i = 0, j = 0; i < arr.length; i++) {
            if (map.put(arr [i], 1) == null) {
                arrayWithDistictElements [j++] = arr [i];
                ++noOfDistinctElements;
            }
        }
        int [] result = Arrays.copyOf(arrayWithDistictElements, noOfDistinctElements);
        LocalTime time2 = LocalTime.now();
        System.out.println("Process 2 (using map) out array size: " + result.length);
        System.out.println("    Duration in millis: " + Duration.between(time1, time2).toMillis());
        //System.out.println(Arrays.toString(result));
    }

    /*
     * Process uses a Set to arrive at distinct elements.
     */ 
    private static void process3(int [] arr) {
        LocalTime time1 = LocalTime.now();
        Set<Integer> set = new HashSet<>();
        int [] arrayWithDistictElements = new int [arr.length];
        int noOfDistinctElements = 0;
        for (int i = 0, j = 0; i < arr.length; i++) {
            if (set.add(arr [i])) {
                arrayWithDistictElements [j++] = arr [i];
                ++noOfDistinctElements;
            }
        }
        int [] result = Arrays.copyOf(arrayWithDistictElements, noOfDistinctElements);
        LocalTime time2 = LocalTime.now();
        System.out.println("Process 3 (using set) out array size: " + result.length);
        System.out.println("    Duration in millis: " + Duration.between(time1, time2).toMillis());
        //System.out.println(Arrays.toString(result));
    }
}

测试结果：

使用的设备：Intel CORE i3 处理器，Windows 7 64 位，Java 8

Input array size: 10,000,000
[Test 1]
Process 1 (using streams) out array size: 9988498
    Duration in millis: 10649
Process 2 (using map) out array size: 9988498
    Duration in millis: 10294
Process 3 (using set) out array size: 9988498
    Duration in millis: 8982
[Test 2]
Process 1 (using streams) out array size: 9988331
    Duration in millis: 7839
Process 2 (using map) out array size: 9988331
    Duration in millis: 5567
Process 3 (using set) out array size: 9988331
    Duration in millis: 4155
[Test 3]
Process 1 (using streams) out array size: 9988286
    Duration in millis: 9138
Process 2 (using map) out array size: 9988286
    Duration in millis: 6799
Process 3 (using set) out array size: 9988286
    Duration in millis: 7155
[Test 4]
Process 1 (using streams) out array size: 9988431
    Duration in millis: 7908
Process 2 (using map) out array size: 9988431
    Duration in millis: 6909
Process 3 (using set) out array size: 9988431
    Duration in millis: 7205
[Test 5]
Process 1 (using streams) out array size: 9988334
    Duration in millis: 7971
Process 2 (using map) out array size: 9988334
    Duration in millis: 6910
Process 3 (using set) out array size: 9988334
    Duration in millis: 7196