【发布时间】:2017-08-13 09:12:27
【问题描述】:
我想在由如下几列组成的数据集上实现一个程序:
+-----------+---------------+-------------------+-----------------------+
|Item_ID |Product_Name |Manufacturer_Name |Product_Description |
+-----------+---------------+-------------------+-----------------------+
|12345 |Pen |Cello |Ball Pen Soft Nib... |
|12346 |Pencil |Nataraja |Pencil HB Extra D... |
|42345 |Ruler |Nataraja |Scale No.1103 15c... |
|12677 |Sharpener |Nataraja |Pencil Shraperner... |
|12987 |Pen |Reynolds |Dot Pen Extra Gr... |
|44326 |Pen |Reynolds |Gel Pen German T... |
|13456 |Pen |Cello |Dot Pen 0.5mm Nib... |
|19876 |Eraser |Cello |Dust free Eraser ... |
|43246 |Ink Pen |Hero |Ink Pen Smooth Ha... |
+-----------+---------------+-------------------+-----------------------+
我想根据Manufacturer_Name 对数据集进行分组,如下所示
Manufacturer = Cello
+-----------+---------------+-------------------+-----------------------+
|Item_ID |Product_Name |Manufacturer_Name |Product_Description |
+-----------+---------------+-------------------+-----------------------+
|12345 |Pen |Cello |Ball Pen Soft Nib... |
|13456 |Pen |Cello |Dot Pen 0.5mm Nib... |
|19876 |Eraser |Cello |Dust free Eraser ... |
+-----------+---------------+-------------------+-----------------------+
Manufacturer = Nataraja
+-----------+---------------+-------------------+-----------------------+
|Item_ID |Product_Name |Manufacturer_Name |Product_Description |
+-----------+---------------+-------------------+-----------------------+
|12346 |Pencil |Nataraja |Pencil HB Extra D... |
|42345 |Ruler |Nataraja |Scale No.1103 15c... |
|12677 |Sharpener |Nataraja |Pencil Shraperner... |
+-----------+---------------+-------------------+-----------------------+
Manufacturer = Reynolds
+-----------+---------------+-------------------+-----------------------+
|Item_ID |Product_Name |Manufacturer_Name |Product_Description |
+-----------+---------------+-------------------+-----------------------+
|12987 |Pen |Reynolds |Dot Pen Extra Gr... |
|44326 |Pen |Reynolds |Gel Pen German T... |
+-----------+---------------+-------------------+-----------------------+
Manufacturer = Hero
+-----------+---------------+-------------------+-----------------------+
|Item_ID |Product_Name |Manufacturer_Name |Product_Description |
+-----------+---------------+-------------------+-----------------------+
|43246 |Ink Pen |Hero |Ink Pen Smooth Ha... |
+-----------+---------------+-------------------+-----------------------+
我尝试使用以下代码,但效果不佳。帮助我改进这个程序。这是我使用的代码:
Dataset<Row> countsBy = src.select("Manufacturer_Name").distinct();
List<Row> lsts = countsBy.collectAsList();
for (Row lst : lsts) {
String man = lst.toString();
System.out.println("Records of " + man + " only");
Dataset<Row> mandataset = src.filter("Manufacturer_Name='" + man + "'");
mandataset.show();
}
【问题讨论】:
-
你能更具体地描述一下不好的结果吗?是缓慢还是错误?
-
我希望数据集的子集在迭代部分之外可用。由于它是在本地声明的并且每次迭代都会被覆盖,因此我不能使用除上次迭代期间生成的子集之外的所有子集。 @AugustinBocken
标签: java apache-spark dataset grouping