【问题标题】:How to read or parse MHTML (.mht) files in java如何在 java 中读取或解析 MHTML (.mht) 文件
【发布时间】:2011-03-14 21:42:43
【问题描述】:

我需要挖掘大多数已知文档文件的内容,例如:

  1. pdf
  2. html
  3. doc/docx 等

对于我计划使用的大多数文件格式:

http://tika.apache.org/

但截至目前Tika 不支持 MHTML (*.mht) 文件.. (http://en.wikipedia.org/wiki/MHTML) C# (http://www.codeproject.com/KB/files/MhtBuilder.aspx) 中的示例很少,但我在 Java 中没有找到。

我尝试在 7Zip 中打开 *.mht 文件,但失败了...尽管 WinZip 能够将文件解压缩为图像和文本(CSS、HTML、脚本)作为文本和二进制文件...

根据 MSDN 页面 (http://msdn.microsoft.com/en-us/library/aa767785%28VS.85%29.aspx#compress_content) 和我之前提到的 code project 页面...mht 文件使用 GZip 压缩...。

尝试在 java 中解压缩会导致以下异常: 与java.uti.zip.GZIPInputStream

java.io.IOException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(Unknown Source)
at java.util.zip.GZIPInputStream.<init>(Unknown Source)
at java.util.zip.GZIPInputStream.<init>(Unknown Source)
at GZipTest.main(GZipTest.java:16)

还有java.util.zip.ZipFile

 java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at GZipTest.main(GZipTest.java:21)

请高手指点解压方法....

谢谢....

【问题讨论】:

    标签: java parsing compression mhtml


    【解决方案1】:

    你可以试试http://www.chilkatsoft.com/mht-features.asp,它可以打包/解包,你可以像普通文件一样处理它。下载链接为:http://www.chilkatsoft.com/java.asp

    【讨论】:

    • 感谢 Roki.. 我已经看到 chilkatsoft 产品但无法使用它,因为: 1. 它作为本机实现 chilkat.dll .. 我需要用纯 Java 编写的东西。 2. 这是一个商业产品.. 在我的项目中我使用的是开源产品... :o 无论如何谢谢 :)
    【解决方案2】:

    我使用http://jtidy.sourceforge.net 来解析/读取/索引 mht 文件(但作为普通文件,而不是压缩文件)

    【讨论】:

    • 感谢 Wajdy Essam...大多数 *.mht 文件是使用 save as 选项从 IE 保存的...默认情况下它们是压缩的...但我仍然会尝试 jtidy ...
    【解决方案3】:

    坦率地说,我并没有期待在不久的将来有一个解决方案,并且正要放弃,但是我偶然发现了这个页面:

    http://en.wikipedia.org/wiki/MIME#Multipart_messages

    http://msdn.microsoft.com/en-us/library/ms527355%28EXCHG.10%29.aspx

    虽然,乍一看不是很吸引人。但如果你仔细看,你会得到线索。读完这篇文章后,我启动了我的 IE 并随机开始将页面保存为 *.mht 文件。让我一行一行的说吧……

    但让我事先解释一下,我的最终目标是分离/提取 html 内容并对其进行解析...解决方案本身并不完整,因为它取决于我选择的 character setencoding在保存的同时。但即使它会提取单个文件,但会遇到一些小问题......

    我希望这对任何试图解析/解压缩*.mht/MHTML文件的人有用:)

    ======= 解释======== ** 取自 mht 文件 **

    From: "Saved by Windows Internet Explorer 7"
    

    是用来保存文件的软件

    Subject: Google
    Date: Tue, 13 Jul 2010 21:23:03 +0530
    MIME-Version: 1.0
    

    主题、日期和 mime 版本……很像邮件格式

      Content-Type: multipart/related;
    type="text/html";
    

    这部分告诉我们它是一个multipart 文档。多部分文档将一组或多组不同的数据组合在一个正文中,multipart Content-Type 字段必须出现在实体的标题中。在这里,我们也可以看到类型为"text/html"

    boundary="----=_NextPart_000_0007_01CB22D1.93BBD1A0"
    

    所有这些是最重要的部分。这是分隔两个不同部分(html、图像、css、脚本等)的唯一分隔符。 一旦你掌握了这一点,一切都会变得简单......现在,我只需要遍历文档并找出不同的部分并按照他们的 Content-Transfer-Encoding 保存它们(base64,quoted-printable等等) ... . . .

    示例

     ------=_NextPart_000_0007_01CB22D1.93BBD1A0
     Content-Type: text/html;
     charset="utf-8"
     Content-Transfer-Encoding: quoted-printable
     Content-Location: http://www.google.com/webhp?sourceid=navclient&ie=UTF-8
    
     <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" =
    .
    .
    .
    

    ** JAVA代码**

    定义常量的接口。

    public interface IConstants 
    {
        public String BOUNDARY = "boundary";
        public String CHAR_SET = "charset";
        public String CONTENT_TYPE = "Content-Type";
        public String CONTENT_TRANSFER_ENCODING = "Content-Transfer-Encoding";
        public String CONTENT_LOCATION = "Content-Location";
    
        public String UTF8_BOM = "=EF=BB=BF";
    
        public String UTF16_BOM1 = "=FF=FE";
        public String UTF16_BOM2 = "=FE=FF";
    }
    

    主要的解析器类...

    /**
     * This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0
     * which accompanies this distribution, and is available at
     * http://www.eclipse.org/legal/epl-v10.html
     */
    package com.test.mht.core;
    
    import java.io.BufferedOutputStream;
    import java.io.BufferedReader;
    import java.io.BufferedWriter;
    import java.io.File;
    import java.io.FileOutputStream;
    import java.io.FileReader;
    import java.io.OutputStreamWriter;
    import java.util.regex.Matcher;
    import java.util.regex.Pattern;
    
    import sun.misc.BASE64Decoder;
    
    /**
     * File to parse and decompose *.mts file in its constituting parts.
     * @author Manish Shukla 
     */
    
    public class MHTParser implements IConstants
    {
        private File mhtFile;
        private File outputFolder;
    
        public MHTParser(File mhtFile, File outputFolder) {
            this.mhtFile = mhtFile;
            this.outputFolder = outputFolder;
        }
    
        /**
         * @throws Exception
         */
        public void decompress() throws Exception
        {
            BufferedReader reader = null;
    
            String type = "";
            String encoding = "";
            String location = "";
            String filename = "";
            String charset = "utf-8";
            StringBuilder buffer = null;
    
            try
            {
                reader = new BufferedReader(new FileReader(mhtFile));
    
                final String boundary = getBoundary(reader);
                if(boundary == null)
                    throw new Exception("Failed to find document 'boundary'... Aborting");
    
                String line = null;
                int i = 1;
                while((line = reader.readLine()) != null)
                {
                    String temp = line.trim();
                    if(temp.contains(boundary)) 
                    {
                        if(buffer != null) {
                            writeBufferContentToFile(buffer,encoding,filename,charset);
                            buffer = null;
                        }
    
                        buffer = new StringBuilder();
                    }else if(temp.startsWith(CONTENT_TYPE)) {
                        type = getType(temp);
                    }else if(temp.startsWith(CHAR_SET)) {
                        charset = getCharSet(temp);
                    }else if(temp.startsWith(CONTENT_TRANSFER_ENCODING)) {
                        encoding = getEncoding(temp);
                    }else if(temp.startsWith(CONTENT_LOCATION)) {
                        location = temp.substring(temp.indexOf(":")+1).trim();
                        i++;
                        filename = getFileName(location,type);
                    }else {
                        if(buffer != null) {
                            buffer.append(line + "\n");
                        }
                    }
                }
    
            }finally 
            {
                if(null != reader)
                    reader.close();
            }
    
        }
    
        private String getCharSet(String temp) 
        {
            String t = temp.split("=")[1].trim();
            return t.substring(1, t.length()-1);
        }
    
        /**
         * Save the file as per character set and encoding 
         */
        private void writeBufferContentToFile(StringBuilder buffer,String encoding, String filename, String charset) 
        throws Exception
        {
    
            if(!outputFolder.exists())
                outputFolder.mkdirs();
    
            byte[] content = null; 
    
            boolean text = true;
    
            if(encoding.equalsIgnoreCase("base64")){
                content = getBase64EncodedString(buffer);
                text = false;
            }else if(encoding.equalsIgnoreCase("quoted-printable")) {
                content = getQuotedPrintableString(buffer);         
            }
            else
                content = buffer.toString().getBytes();
    
            if(!text)
            {
                BufferedOutputStream bos = null;
                try
                {
                    bos = new BufferedOutputStream(new FileOutputStream(filename));
                    bos.write(content);
                    bos.flush();
                }finally {
                    bos.close();
                }
            }else 
            {
                BufferedWriter bw = null;
                try
                {
                    bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filename), charset));
                    bw.write(new String(content));
                    bw.flush();
                }finally {
                    bw.close();
                }
            }
        }
    
        /**
         * When the save the *.mts file with 'utf-8' encoding then it appends '=EF=BB=BF'</br>
         * @see http://en.wikipedia.org/wiki/Byte_order_mark
         */
        private byte[] getQuotedPrintableString(StringBuilder buffer) 
        {
            //Set<String> uniqueHex = new HashSet<String>();
            //final Pattern p = Pattern.compile("(=\\p{XDigit}{2})*");
    
            String temp = buffer.toString().replaceAll(UTF8_BOM, "").replaceAll("=\n", "");
    
            //Matcher m = p.matcher(temp);
            //while(m.find()) {
            //  uniqueHex.add(m.group());
            //}
    
            //System.out.println(uniqueHex);
    
            //for (String hex : uniqueHex) {
                //temp = temp.replaceAll(hex, getASCIIValue(hex.substring(1)));
            //}     
    
            return temp.getBytes();
        }
    
        /*private String getASCIIValue(String hex) {
            return ""+(char)Integer.parseInt(hex, 16);
        }*/
        /**
         * Although system dependent..it works well
         */
        private byte[] getBase64EncodedString(StringBuilder buffer) throws Exception {
            return new BASE64Decoder().decodeBuffer(buffer.toString());
        }
    
        /**
         * Tries to get a qualified file name. If the name is not apparent it tries to guess it from the URL.
         * Otherwise it returns 'unknown.<type>'
         */
        private String getFileName(String location, String type) 
        {
            final Pattern p = Pattern.compile("(\\w|_|-)+\\.\\w+");
            String ext = "";
            String name = "";
            if(type.toLowerCase().endsWith("jpeg"))
                ext = "jpg";
            else
                ext = type.split("/")[1];
    
            if(location.endsWith("/")) {
                name = "main";
            }else
            {
                name = location.substring(location.lastIndexOf("/") + 1);
    
                Matcher m = p.matcher(name);
                String fname = "";
                while(m.find()) {
                    fname = m.group();
                }
    
                if(fname.trim().length() == 0)
                    name = "unknown";
                else
                    return getUniqueName(fname.substring(0,fname.indexOf(".")), fname.substring(fname.indexOf(".") + 1, fname.length()));
            }
            return getUniqueName(name,ext);
        }
    
        /**
         * Returns a qualified unique output file path for the parsed path.</br>
         * In case the file already exist it appends a numarical value a continues
         */
        private String getUniqueName(String name,String ext)
        {
            int i = 1;
            File file = new File(outputFolder,name + "." + ext);
            if(file.exists())
            {
                while(true)
                {
                    file = new File(outputFolder, name + i + "." + ext);
                    if(!file.exists())
                        return file.getAbsolutePath();
                    i++;
                }
            }
    
            return file.getAbsolutePath();
        }
    
        private String getType(String line) {
            return splitUsingColonSpace(line);
        }
    
        private String getEncoding(String line){
            return splitUsingColonSpace(line);
        }
    
        private String splitUsingColonSpace(String line) {
            return line.split(":\\s*")[1].replaceAll(";", "");
        }
    
        /**
         * Gives you the boundary string
         */
        private String getBoundary(BufferedReader reader) throws Exception 
        {
            String line = null;
    
            while((line = reader.readLine()) != null)
            {
                line = line.trim();
                if(line.startsWith(BOUNDARY)) {
                    return line.substring(line.indexOf("\"") + 1, line.lastIndexOf("\""));
                }
            }
    
            return null;
        }
    }
    

    问候,

    【讨论】:

    • 感谢您分享此内容。不过我有一个问题:您如何确定部分 HEADER 的结尾?我的意思是,该部分从上面某处定义的边界开始,然后是类型/编码/位置等标头内容,在此之后,在我看来,每个标头都由 ASCII 0D0A0D0A(两倍 CR+LF)分隔。我对此是否正确,还是有任何其他有保证的分隔符?
    • @favonius:您的代码是否将 .mht 文件转换为 html?
    • @UmeshKumar :代码用于数据提取。但我认为将 mht 转换为 html 应该很简单。 Google 周围可能是一个已经存在的库。
    • 我尝试使用此解决方案,最初它有效,但很快我发现了大问题。请注意,BufferedReader.readLine() 方法采用 UTF-8 编码并将原始输入字节转换为 Java 16 位字符,按照 UTF-8 规范重新编码。因此,如果您有一个二进制文件,即使是一个带有传输编码“二进制”和 HTML 中不同字符编码的 HTML 文件(我有 Shift_JIS 作为测试样本),所有地狱都会丢失。最终用 C++ 从头开始​​实现我自己的 MHTML (MIME) 解码器,并使用 JNI 从 Java 访问它。
    【解决方案4】:

    你不必自己做。

    有依赖关系

    <dependency>
        <groupId>org.apache.james</groupId>
        <artifactId>apache-mime4j</artifactId>
        <version>0.7.2</version>
    </dependency>
    

    滚动你的 mht 文件

    public static void main(String[] args)
    {
        MessageTree.main(new String[]{"YOU MHT FILE PATH"});
    }
    

    MessageTree

    /**
     * Displays a parsed Message in a window. The window will be divided into
     * two panels. The left panel displays the Message tree. Clicking on a
     * node in the tree shows information on that node in the right panel.
     *
     * Some of this code have been copied from the Java tutorial's JTree section.
     */
    

    那你可以调查一下。

    ;-)

    【讨论】:

      【解决方案5】:

      派对迟到了,但扩展了@wener 的答案,以解决其他遇到此问题的人。

      Apache Mime4J 库似乎为EML or MHTML 处理提供了最容易访问的解决方案,比自己滚动要容易得多!

      下面我的原型“parseMhtToFile”函数从 Cognos 活动报告“mht”文件中提取 html 文件和其他工件,但可以针对其他目的进行定制。

      这是用 Groovy 编写的,需要 Apache Mime4J 'core' and 'dom' jars(当前为 0.7.2)。

      import org.apache.james.mime4j.dom.Message
      import org.apache.james.mime4j.dom.Multipart
      import org.apache.james.mime4j.dom.field.ContentTypeField
      import org.apache.james.mime4j.message.DefaultMessageBuilder
      import org.apache.james.mime4j.stream.MimeConfig
      
      /**
       * Use Mime4J MessageBuilder to parse an mhtml file (assumes multipart) into
       * separate html files.
       * Files will be written to outDir (or parent) as baseName + partIdx + ext.
       */
      void parseMhtToFile(File mhtFile, File outDir = null) {
          if (!outDir) {outDir = mhtFile.parentFile }
          // File baseName will be used in generating new filenames
          def mhtBaseName = mhtFile.name.replaceFirst(~/\.[^\.]+$/, '')
      
          // -- Set up Mime parser, using Default Message Builder
          MimeConfig parserConfig  = new MimeConfig();
          parserConfig.setMaxHeaderLen(-1); // The default is a mere 10k
          parserConfig.setMaxLineLen(-1); // The default is only 1000 characters.
          parserConfig.setMaxHeaderCount(-1); // Disable the check for header count.
          DefaultMessageBuilder builder = new DefaultMessageBuilder();
          builder.setMimeEntityConfig(parserConfig);
      
          // -- Parse the MHT stream data into a Message object
          println "Parsing ${mhtFile}...";
          InputStream mhtStream = mhtFile.newInputStream()
          Message message = builder.parseMessage(mhtStream);
      
          // -- Process the resulting body parts, writing to file
          assert message.getBody() instanceof Multipart
          Multipart multipart = (Multipart) message.getBody();
          def parts = multipart.getBodyParts();
          parts.eachWithIndex { p, i ->
              ContentTypeField cType = p.header.getField('content-type')
              println "${p.class.simpleName}\t${i}\t${cType.mimeType}"
      
              // Assume mime sub-type is a "good enough" file-name extension 
              // e.g. text/html = html, image/png = png, application/json = json
              String partFileName = "${mhtBaseName}_${i}.${cType.subType}"
              File partFile = new File(outDir, partFileName)
      
              // Write part body stream to file
              println "Writing ${partFile}...";
              if (partFile.exists()) partFile.delete();
              InputStream partStream = p.body.inputStream;
              partFile.append(partStream);
          }
      }
      

      用法很简单:

      File mhtFile = new File('<path>', 'Report-en-au.mht')
      parseMhtToFile(mhtFile)
      println 'Done.'
      

      输出是:

      Parsing <path>\Report-en-au.mht...
      BodyPart    0   text/html
      Writing <path>\Report-en-au_0.html...
      BodyPart    1   image/png
      Writing <path>\Report-en-au_1.png...
      Done.
      

      关于其他改进的想法:

      • 对于 'text' mime parts,您可以访问 Reader 而不是 Stream,这可能更适合 OP 要求的文本挖掘。

      • 对于生成的文件扩展名,我会使用另一个库来查找适当的扩展名,而不是假设 mime 子类型就足够了。

      • 处理单体(非多部分)和递归多部分 mhtml 文件和其他复杂性。这些可能需要带有自定义Content Handler 实现的MimeStreamParser

      【讨论】:

        【解决方案6】:

        使用 Java Mail API 的更紧凑的代码

        import java.io.File;
        import java.io.FileInputStream;
        import java.io.FileOutputStream;
        import java.net.URL;
        import java.util.Properties;
        
        import javax.mail.BodyPart;
        import javax.mail.Session;
        import javax.mail.internet.MimeMessage;
        import javax.mail.internet.MimeMultipart;
        
        import org.apache.commons.io.IOUtils;
        
        public class MhtParser {
        
            private File mhtFile;
            private File outputFolder;
        
            public MhtParser(File mhtFile, File outputFolder) {
                this.mhtFile = mhtFile;
                this.outputFolder = outputFolder;
            }
        
            public void decompress() throws Exception {
                MimeMessage message = 
                    new MimeMessage(
                            Session.getDefaultInstance(new Properties(), null),
                            new FileInputStream(mhtFile));
        
                if (message.getContent() instanceof MimeMultipart) {
                    outputFolder.mkdir();
                    MimeMultipart mimeMultipart = (MimeMultipart) message.getContent();
        
                    for (int i = 0; i < mimeMultipart.getCount(); i++) {
                        BodyPart bodyPart = mimeMultipart.getBodyPart(i);
                        String fileName = bodyPart.getFileName();
        
                        if (fileName == null) {
                            String[] locationHeader = bodyPart.getHeader("Content-Location");
                            if (locationHeader != null && locationHeader.length > 0) {
                                fileName = 
                                    new File(new URL(locationHeader[0]).getFile()).getName();
                            }
                        }
        
                        if (fileName != null) {
                            FileOutputStream out = 
                                new FileOutputStream(new File(outputFolder, fileName));
        
                            IOUtils.copy(bodyPart.getInputStream(), out);
                            out.flush();
                            out.close();
                        }
                    }
                }
            }
        }
        

        【讨论】:

          猜你喜欢
          • 1970-01-01
          • 1970-01-01
          • 2020-09-21
          • 1970-01-01
          • 2012-04-19
          • 2011-06-17
          • 1970-01-01
          • 1970-01-01
          • 2018-05-30
          相关资源
          最近更新 更多