【问题标题】:Group Dataset into different Sub Datasets based on the value根据值将数据集分组为不同的子数据集
【发布时间】:2017-08-13 09:12:27
【问题描述】:

我想在由如下几列组成的数据集上实现一个程序:

+-----------+---------------+-------------------+-----------------------+
|Item_ID    |Product_Name   |Manufacturer_Name  |Product_Description    |
+-----------+---------------+-------------------+-----------------------+
|12345      |Pen            |Cello              |Ball Pen Soft Nib...   |
|12346      |Pencil         |Nataraja           |Pencil HB Extra D...   |
|42345      |Ruler          |Nataraja           |Scale No.1103 15c...   |
|12677      |Sharpener      |Nataraja           |Pencil Shraperner...   |
|12987      |Pen            |Reynolds           |Dot Pen Extra Gr...    |
|44326      |Pen            |Reynolds           |Gel Pen German T...    |
|13456      |Pen            |Cello              |Dot Pen 0.5mm Nib...   |
|19876      |Eraser         |Cello              |Dust free Eraser ...   |
|43246      |Ink Pen        |Hero               |Ink Pen Smooth Ha...   |
+-----------+---------------+-------------------+-----------------------+

我想根据Manufacturer_Name 对数据集进行分组,如下所示

Manufacturer = Cello
+-----------+---------------+-------------------+-----------------------+
|Item_ID    |Product_Name   |Manufacturer_Name  |Product_Description    |
+-----------+---------------+-------------------+-----------------------+
|12345      |Pen            |Cello              |Ball Pen Soft Nib...   |
|13456      |Pen            |Cello              |Dot Pen 0.5mm Nib...   |
|19876      |Eraser         |Cello              |Dust free Eraser ...   |
+-----------+---------------+-------------------+-----------------------+

Manufacturer = Nataraja
+-----------+---------------+-------------------+-----------------------+
|Item_ID    |Product_Name   |Manufacturer_Name  |Product_Description    |
+-----------+---------------+-------------------+-----------------------+
|12346      |Pencil         |Nataraja           |Pencil HB Extra D...   |
|42345      |Ruler          |Nataraja           |Scale No.1103 15c...   |
|12677      |Sharpener      |Nataraja           |Pencil Shraperner...   |
+-----------+---------------+-------------------+-----------------------+

Manufacturer = Reynolds
+-----------+---------------+-------------------+-----------------------+
|Item_ID    |Product_Name   |Manufacturer_Name  |Product_Description    |
+-----------+---------------+-------------------+-----------------------+
|12987      |Pen            |Reynolds           |Dot Pen Extra Gr...    |
|44326      |Pen            |Reynolds           |Gel Pen German T...    |
+-----------+---------------+-------------------+-----------------------+

Manufacturer = Hero
+-----------+---------------+-------------------+-----------------------+
|Item_ID    |Product_Name   |Manufacturer_Name  |Product_Description    |
+-----------+---------------+-------------------+-----------------------+
|43246      |Ink Pen        |Hero               |Ink Pen Smooth Ha...   |
+-----------+---------------+-------------------+-----------------------+

我尝试使用以下代码,但效果不佳。帮助我改进这个程序。这是我使用的代码:

Dataset<Row> countsBy = src.select("Manufacturer_Name").distinct();
List<Row> lsts = countsBy.collectAsList();
for (Row lst : lsts) {
    String man = lst.toString();
    System.out.println("Records of " + man + " only");
    Dataset<Row> mandataset = src.filter("Manufacturer_Name='" + man + "'");
    mandataset.show();
}

【问题讨论】:

  • 你能更具体地描述一下不好的结果吗?是缓慢还是错误?
  • 我希望数据集的子集在迭代部分之外可用。由于它是在本地声明的并且每次迭代都会被覆盖,因此我不能使用除上次迭代期间生成的子集之外的所有子集。 @AugustinBocken

标签: java apache-spark dataset grouping


【解决方案1】:

也许您可以尝试制作数据集的映射,键为字符串(Manufacturer_Name),对于每次迭代,您检查Manufacturer_Name,然后检查它是否已经在映射中(如果需要,您可以创建它)和最后,将行添加到好的数据集中。

你会有类似的东西:

Map<string,ArrayList<ShopItem>> dic = new HashMap<string,ArrayList<ShopItem>>();
for(/*...*/)
{
  string Manufacturer_Name = //you get the name
  if(/*the Manufacturer_Name is not in dic*/)
  {
    dic.put(Manufacturer_Name,new ArrayList<ShopItem>());
  }
  dic.get(Manufacturer_Name).Add(/*what you want to add*/);
}

然后您需要第二个循环,但仅用于打印数据。

希望能解决你的问题!

编辑:用地图替换字典(对不起)并提供链接

How do you create a dictionary in Java?

编辑:更改代码以匹配新想法

【讨论】:

  • 只是在实施前澄清一些事情。可以实例化字典吗?方法 add 和 AddValue 在 java 中是否可用?
  • 也许只有在图书馆里...你是对的,我需要检查更多,或者你可以实现你自己的字典!这是一个条目列表,每个条目都有一个对象键和对象值,应该足够你使用了......但我会看看并编辑我的答案
  • 在这里,我更正了答案,所以它更正确:D
  • ` Map> dic = new HashMap>(); for(Row row : srcrows) { String Manufacturer_Name = row.getString(3); if(!dic.values().equals(Manufacturer_Name)) { dic.put(Manufacturer_Name,new Dataset()); } dic.get(Manufacturer_Name).Add(row); } ` 这是正确的吗?
  • 错误一:dic.put(Manufacturer_Name,new Dataset());它说“构造函数 Dataset() 未定义”错误二: dic.get(Manufacturer_Name).Add(row);它说“Dataset 类型的方法 Add(Row) 是未定义的” 我遇到了一些错误,请你帮我解决它吗?
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2020-08-08
  • 1970-01-01
  • 2021-07-24
  • 2022-10-20
  • 2020-12-19
相关资源
最近更新 更多