使用 Jersey 客户端上传大文件时如何避免 OutOfMemoryError

Question

我正在使用 Jersey 客户端进行基于 http 的请求。如果文件很小但在我发布大小为 700M 的文件时出错：

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
    at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
    at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:90)
    at com.sun.jersey.core.util.ReaderWriter.writeTo(ReaderWriter.java:115)
    at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:76)
    at com.sun.jersey.core.impl.provider.entity.FileProvider.writeTo(FileProvider.java:103)
    at com.sun.jersey.core.impl.provider.entity.FileProvider.writeTo(FileProvider.java:64)
    at com.sun.jersey.multipart.impl.MultiPartWriter.writeTo(MultiPartWriter.java:224)
    at com.sun.jersey.multipart.impl.MultiPartWriter.writeTo(MultiPartWriter.java:71)
    at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568)
    at TestHttpRequest.main(TestHttpRequest.java:42)

这是我的代码：

ClientConfig cc = new DefaultClientConfig();
        Client client = Client.create(cc);
        WebResource resource = client.resource("http://localhost:8080/JerseyWithServletTest/helloworld");
        FormDataMultiPart form = new FormDataMultiPart();
        File file = new File("E:/CN_WXPPSP3_v312.ISO");
        form.field("username", "ljy");
        form.field("password", "password");
        form.field("filename", file.getName());
        form.bodyPart(new FileDataBodyPart("file", file, MediaType.MULTIPART_FORM_DATA_TYPE));
        ClientResponse response = resource.type(MediaType.MULTIPART_FORM_DATA).post(ClientResponse.class, form);

Answer 1

您可以使用流。在客户端上尝试这样的操作：

InputStream fileInStream = new FileInputStream(fileName);
String sContentDisposition = "attachment; filename=\"" + fileName.getName()+"\"";
WebResource fileResource = a_client.resource(a_sUrl);       
ClientResponse response = fileResource.type(MediaType.APPLICATION_OCTET_STREAM)
                        .header("Content-Disposition", sContentDisposition)
                        .post(ClientResponse.class, fileInStream);      

在服务器上有这样的资源：

@PUT
@Consumes("application/octet-stream")
public Response putFile(@Context HttpServletRequest a_request,
                         @PathParam("fileId") long a_fileId,
                         InputStream a_fileInputStream) throws Throwable
{
    // Do something with a_fileInputStream
    // etc

Answer 2

为了让您的代码不依赖于上传文件的大小，您需要：

1. 使用流
2. 定义球衣客户端的夹头尺寸。例如： client.setChunkedEncodingSize(1024);

服务器：

    @POST
    @Path("/upload/{attachmentName}")
    @Consumes(MediaType.APPLICATION_OCTET_STREAM)
    public void uploadAttachment(@PathParam("attachmentName") String attachmentName, InputStream attachmentInputStream) {
        // do something with the input stream
    }

客户：

    ...
    client.setChunkedEncodingSize(1024);
    WebResource rootResource = client.resource("your-server-base-url");
    File file = new File("your-file-path");
    InputStream fileInStream = new FileInputStream(file);
    String contentDisposition = "attachment; filename=\"" + file.getName() + "\"";
    ClientResponse response = rootResource.path("attachment").path("upload").path("your-file-name")
            .type(MediaType.APPLICATION_OCTET_STREAM).header("Content-Disposition", contentDisposition)
            .post(ClientResponse.class, fileInStream);

Answer 3

以下是使用 Jersey 2.11 上传具有分块传输编码（即流）的（可能很大）文件的代码。

马文：

<properties>
    <jersey.version>2.11</jersey.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-client</artifactId>
        <version>${jersey.version}</version>
    </dependency>
    <dependency>
    <groupId>org.glassfish.jersey.media</groupId>
        <artifactId>jersey-media-multipart</artifactId>
        <version>${jersey.version}</version>
    </dependency>
<dependencies>

Java：

Client client = ClientBuilder.newClient(clientConfig);
client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");

WebTarget target = client.target(SERVICE_URI); 
InputStream fileInStream = new FileInputStream(inFile);
String contentDisposition = "attachment; filename=\"" + inFile.getName() + "\"";
System.out.println("sending: " + inFile.length() + " bytes...");
Response response = target
            .request(MediaType.APPLICATION_OCTET_STREAM_TYPE)
            .header("Content-Disposition", contentDisposition)
            .header("Content-Length", (int) inFile.length())
            .put(Entity.entity(fileInStream, MediaType.APPLICATION_OCTET_STREAM_TYPE));
System.out.println("Response status: " + response.getStatus());

Answer 4

在我的情况下（泽西 2.23.2）rschmidt13's solution 给出了这个警告：

WARNING: Attempt to send restricted header(s) while the [sun.net.http.allowRestrictedHeaders] system property not set. Header(s) will possibly be ignored.

这可以通过添加以下行来解决：

System.setProperty("sun.net.http.allowRestrictedHeaders", "true");

但是我认为使用StreamingOutput 接口可以获得更清洁的解决方案。 我发布了一个完整的示例，希望它可能有用。

客户端（文件上传）

WebTarget target = ClientBuilder.newBuilder().build()
            .property(ClientProperties.CHUNKED_ENCODING_SIZE, 1024)
            .property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED")
            .target("<your-url>");

StreamingOutput out = new StreamingOutput() {

    @Override
    public void write(OutputStream output) throws IOException, 
            WebApplicationException {

        try (FileInputStream is = new FileInputStream(file)) {

            int available;
            while ((available = is.available()) > 0) {
                // or use a buffer
                output.write(is.read());
            }
        }
    }
};

Response response = target.request().post(Entity.text(out));

服务器

@Path("resourcename")
public class MyResource {

    @Context
    HttpServletRequest request;

    @POST
    @Path("thepath")
    public Response upload() throws IOException, ServletException {

        try (InputStream is = request.getInputStream()) {
            // ...
        }
    }
}

Answer 5

如果可能，您能否将发送的文件拆分成更小的部分？这样会减少内存占用，但是需要在上传/下载代码两边更改代码。

如果你不能，那么你的堆空间太小了，试着用这个 JVM 参数增加它。在您的应用程序服务器中添加/更改Xmx JVM 选项。例如

-Xmx1024m

将最大堆空间设置为 1Gb

Answer 6

@Consumes("multipart/form-data")
@Produces(MediaType.TEXT_PLAIN + ";charset=utf-8")
public String upload(MultipartFormDataInput input,  @QueryParam("videoId") String  videoId,
        @Context HttpServletRequest a_request) {

    String fileName = "";
    for (InputPart inputPart : input.getParts()) {
        try {

            MultivaluedMap<String, String> header = inputPart.getHeaders();
            fileName = getFileName(header);
            // convert the uploaded file to inputstream
            InputStream inputStream = inputPart.getBody(InputStream.class, null);               
            // write the inputStream to a FileOutputStream
            OutputStream outputStream = new FileOutputStream(new File("/home/mh/Téléchargements/videoUpload.avi"));
            int read = 0;
            byte[] bytes = new byte[1024];
            while ((read = inputStream.read(bytes)) != -1) {
                outputStream.write(bytes, 0, read);
            }
            System.out.println("Done!");
        } catch (IOException e) {
            e.printStackTrace();
            return "ko";
        }

    }