错误：线程“main”中的异常 java.lang.ClassCastException：sun.nio.fs.UnixPath 无法转换为 org.apache.parquet.io.OutputFile

Question

我正在尝试在不使用大数据工具的情况下将 xml 文件转换为 avro，然后再转换为 parquet 文件格式。我可以将其转换为 avro，但之后出现错误：

线程“main”java.lang.ClassCastException 中的异常：sun.nio.fs.UnixPath 无法转换为 org.apache.parquet.io.OutputFile 在 Parquet.ConversionToParquet.main(ConversionToParquet.java:65)

下面是我的代码：

import java.io.ByteArrayOutputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.hadoop.conf.Configuration;
import org.apache.parquet.avro.AvroParquetWriter;
import org.apache.parquet.hadoop.ParquetWriter;
import org.apache.parquet.hadoop.metadata.CompressionCodecName;
import org.apache.parquet.io.OutputFile;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.xml.XMLConstants;
import javax.xml.bind.JAXBContext;
import javax.xml.validation.SchemaFactory;


public class ConversionToParquet {

    private static final Logger LOGGER = LoggerFactory.getLogger(ConversionToParquet.class);

    private static final Path inputPath = Paths.get("/home/lucky/output.avro");
    private static final Path outputPath = Paths.get("/home/lucky/Desktop/sample.parquet");

    public static void main(String[] args) throws Exception {

        JAXBContext jaxbContext = JAXBContext.newInstance(ObjectFactory.class);
        javax.xml.validation.Schema newSchema = SchemaFactory
                .newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI)
                .newSchema(ConversionToParquet.class.getResource("/question.xsd"));
        XmlSerializer<Question> xmlSerializer = new XmlSerializer<>(
                jaxbContext, newSchema);
        InputStream questionStream = ConversionToParquet.class.getResourceAsStream(
                "/question.xml");
        Question question = xmlSerializer.readFromXml(questionStream);

        AvroSchemaGenerator schemaGenerator = new AvroSchemaGenerator();
        Schema questionSchema = schemaGenerator.generateSchema(Question.class);

        AvroSerializer<Question> avroSerializer = schemaGenerator.createAvroSerializer(questionSchema);
        ByteArrayOutputStream avroByteStream = new ByteArrayOutputStream();

        avroSerializer.writeToAvro(avroByteStream, question);
        byte[] avroBytes = avroByteStream.toByteArray();

        avroSerializer.writeToAvro(new FileOutputStream("/home/lucky/output.avro"), question);

        System.out.println("File Converted to Avro");


        try (ParquetWriter writer = AvroParquetWriter
                .builder((OutputFile) outputPath)
                .withSchema(questionSchema)
                .withConf(new Configuration())
                .withCompressionCodec(CompressionCodecName.SNAPPY)
                .build()){

            for (Path record : inputPath) {
                writer.write(record);
            }
        }
        System.out.println("File Convereted Successfully");
    }
}

Answer 1

错误在这一行：

.builder((OutputFile) outputPath)

最简单的解决方案是使用带有org.apache.hadoop.fs.Path 参数的已弃用builder 方法，这与您当前导入的java.nio.file.Path 类不同。