【问题标题】:Java - Problem with multiple, concurrent runtime.exec() InputStreamsJava - 多个并发 runtime.exec() InputStreams 的问题
【发布时间】:2011-01-20 23:32:44
【问题描述】:

我别无选择,只能通过对 VBScript 的多次Runtime.exec() 调用来检索一些外部数据。我真的很讨厌这种实现,因为我失去了跨平台的灵活性,但我最终可能会开发类似的 *nix 脚本来至少缓解这个问题。在有人问之前,我无法解决调用外部脚本来收集数据的需要。我会忍受导致的问题。

exec() 进程在扩展 Runnable 的自定义类中运行。它使用BufferedReadergetInputStream() 读取数据。

编辑:根据要求添加了更多代码,但我看不出额外代码的相关性:) 我希望它有所帮助,因为格式化需要一段时间!哦,如果我的代码风格很丑,请放轻松,但鼓励建设性的批评......

public class X extends JFrame implements Runnable {

   ...
   static final int THREADS_MAX = 4;
   ExecutorService  exec;
   ...
   public static void main(String[] args) {
      ...
      SwingUtilities.invokeLater(new X("X"));
   } // End main(String[])

   public X (String title) {
      ...
      exec = Executors.newFixedThreadPool(THREADS_MAX);
      ...

      // Create all needed instances of Y
      for (int i = 0; i < objects.length; i++) {
         Y[i] = new Y(i);
      } // End for(i)

      // Initialization moved here for easy single-thread testing
      // Undesired, of course
      for (int i = 0; i < objects.length; i++) {
         Y[i].initialize(parent);
      } // End for(i)

   } // End X

   class Y implements Runnable {
      // Define variables/arrays used to capture data here
      String computerName = "";
      ...

      public Y(int rowIndex) {
         row          = rowIndex;
         ...
         computerName = (String)JTable.getValueAt(row, 0);
         ...
         exec.execute(this);
      } // End Y(int)

      public void run() {
         // Initialize variables/arrays used to capture data here
         ...

         // Initialization should be done here for proper threading
         //initialize(parent);
      } // End run()

      public void initialize(Z obj) {
         runTime = Runtime.getRuntime();
         ...

         try {
            process = runTime.exec("cscript.exe query.vbs " + computerName);
            stdErr  = process.getErrorStream();
            stdIn   = process.getInputStream();
            isrErr  = new InputStreamReader(stdErr);
            isrIn   = new InputStreamReader(stdIn);
            brErr   = new BufferedReader(isrErr);
            brIn    = new BufferedReader(isrIn);

            while ((line = brIn.readLine()) != null) {
               // Capture, parse, and store data here
               ...
            } // End while

         } catch (IOException e) {
            System.out.println("Unable to run script");
         } catch (Exception e) {
            e.printStackTrace();
         } finally {
            try {
               stdErr.close();
               stdIn. close();
               isrErr.close();
               isrIn. close();
               brErr. close();
               brIn.  close();
            } catch (IOException e) {
               System.out.println("Unable to close streams.");
            } // End try
         } // End try
      } // End initialize(Z)
      ...
   } // End class Y
} // End class X

如果我单独执行命令,我会按预期收集数据。但是,如果我在类的run() 块中执行命令(这意味着调用是并发的,正如我所希望的那样),看起来好像只生成了一个输入流,所有BufferedReaders 都同时使用。

为了调试问题,我在控制台上输出每个消耗的行,前缀是我的类的哪个实例正在读取输入流。我期望如下所示,理解它们可能因实例而异,但单个实例的行顺序将保持不变:

exec 0: Line1
exec 1: Line1
exec 2: Line1
exec 0: Line2
exec 1: Line2
exec 2: Line2
exec 0: Line3
exec 1: Line3
exec 2: Line3
...

奇怪的是我得到了输出的第一行(Microsoft (R) Windows Script Host Version 5.7)的预期实例数,但是在这一行之后,只有一个进程继续在输入流中生成数据,并且所有读者随机消费这一个流,比如这个例子:

exec 2: Microsoft (R) Windows Script Host Version 5.7
exec 0: Microsoft (R) Windows Script Host Version 5.7
exec 1: Microsoft (R) Windows Script Host Version 5.7
exec 0: line2
exec 1: line3
exec 2: line4
...

更糟糕的是,阅读器停滞不前,readLine() 永远不会返回 null。我读到这种行为可能与缓冲区大小有关,但是当我只运行两个并发线程时,即使输出很短,它仍然表现出相同的行为。在stdErr 中未捕获任何内容以表明存在问题。

为了查看这是否是脚本宿主的限制,我创建了一个批处理文件,STARTs 脚本的多个实例同时进行。 我应该声明这是在 Java 的 外部 cmd shell 中运行的,并启动了几个自己的 shell。 但是,每个并发实例都完全返回了预期的结果并且表现良好.

编辑:作为另一个故障排除思路,我决定重新启用并发,但通过将以下内容插入到我的 Y.run() 块中来错开我的初始化方法:

try {
   Thread.sleep((int)(Math.random() * 1200));
} catch (InterruptedException e) {
   System.out.println("Can't sleep!");
} // End try
initialize(monitor);

进入我的代码。我开始看到前几行的多个输出,但它很快恢复为多个消费者使用同一个生产者,并且一旦第一个完成的流关闭,其余消费者就会触发异常。下一个消费者触发IOException: Read error,其余的触发IOException: Stream closed

根据 maaartinus 的说法,可以同时运行多个 InputStreams,所以现在问题变成了导致不良行为的原因是什么?如何独立获取他们的输入流?如果可以避免的话,我不想写一个临时文件来处理数据。

【问题讨论】:

  • So I'm confused - is Java capable of processing multiple input streams concurrently or not? 当然,它是。 Windows 脚本宿主中可能存在问题,无论它是什么。我会尝试一个简单的脚本(或另一个程序)产生一个简单的输出。
  • 听起来很奇怪,您的某些字段可能是静态的。你能发布更多你的课程吗?
  • To make matters worse, the readers stall and readLine() never returns null. 尝试为每个流使用自己的线程,即每个进程两个线程。每个流上都有一个小缓冲区,当它填满时进程会被阻塞(你从另一个流中什么也得不到)。
  • 这听起来像是特定于脚本主机的问题,您将哪些命令行开关传递给cscript.exe
  • 我在 TechNet 文章中发现了这一点。我不知道它是否适用于您的情况,但如果适用,它可能是解决方案的关键。 “在 MOM 2000 中,一次只能执行特定脚本的单个实例,这意味着您不必担心脚本的多个实例相互冲突。在 MOM 2005 中删除了该限制后,您获得了灵活性以牺牲一些额外的责任为代价。”

标签: java concurrency exec inputstream


【解决方案1】:

我认为您需要注意 IO 变量的范围。这是一个运行良好的快速代码,具有来自 4 个子进程的并发输入流...

import java.io.*;

public class MultiExec {

        private final static String[] comLines = {
                        "date",
                        "ls /var/spool/postfix",
                        "ls -F /usr/local/bin",
                        "wc -l /etc/apache2/apache2.conf"};

        public void execute() {
                for (int i = 0 ; i < comLines.length ; i++) {
                        ExecutableChild ec = new ExecutableChild (i, comLines[i]);
                        new Thread (ec).start();
        }}

        public class ExecutableChild implements Runnable {

                private int prIndex;
                private String executable;

                public ExecutableChild (int k, String cmd) {
                        prIndex = k;
                        executable = cmd;
                }

                public void run () {
                        try {
                                Process child = Runtime.getRuntime().exec(executable);
                                BufferedReader br = new BufferedReader (new InputStreamReader (
                                                                child.getInputStream()));
                                for (String s = br.readLine() ; s != null ; s = br.readLine())
                                        System.out.println ("[" + prIndex + "] " + s);
                                br.close();
                        } catch (IOException ioex) {
                                System.err.println ("IOException for process #"+
                                                prIndex+ ": " + ioex.getMessage());
        }}}

        public static void main (String[] args) {
                new MultiExec().execute();
        }
}

上述代码的输出(% javac MultiExec.java; java MultiExec)

[2] tomcat*
[0] Thu Jan 20 18:38:31 CST 2011
[3] 368 /etc/apache2/apache2.conf
[1] active
[1] bounce
[1] corrupt
[1] defer
[1] deferred
[1] etc
[1] flush
[1] hold
[1] incoming
[1] lib
[1] maildrop
[1] pid
[1] private
[1] public
[1] saved
[1] trace
[1] usr
[1] var

如果您向我们提供了您尝试的源代码,我们可以讨论它。祝你好运,-M.S.

================================================ ===============================

编辑: DN:我理解您对 1 线输出的担忧。让我们有一个小脚本...

#!/usr/bin/perl -w
foreach (1..50) {
        print "$_\n";
}

以及上述 Java 代码的编辑版本... comLines 发生了变化,并且在每个 println() 之后添加了一个 Thread.sleep

公共类 MultiExec {

        private final static String[] comLines = {
                        "ls /var/spool/postfix",
                        "perl count50.pl",
                        "cat MultiExec.java",
                        "head -40 /etc/apache2/apache2.conf"};

        public void execute() {
                for (int i = 0 ; i < comLines.length ; i++) {
                        ExecutableChild ec = new ExecutableChild (i, comLines[i]);
                        new Thread (ec).start();
        }}

        public class ExecutableChild implements Runnable {

                private int prIndex;
                private String executable;

                public ExecutableChild (int k, String cmd) {
                        prIndex = k;
                        executable = cmd;
                }

                public void run () {
                        try {
                                Process child = Runtime.getRuntime().exec(executable);
                                BufferedReader br = new BufferedReader (new InputStreamReader (
                                                                child.getInputStream()));
                                for (String s = br.readLine() ; s != null ; s = br.readLine()) {
                                        System.out.println ("[" + prIndex + "] " + s);
                                        try {
                                                Thread.sleep (20);
                                        } catch (InterruptedException intex) {
                                }}
                                br.close();
                        } catch (IOException ioex) {
                                System.err.println ("IOException for process #"+
                                                                prIndex+ ": " + ioex.getMessage());
        }}}

        public static void main (String[] args) {
                new MultiExec().execute();
}}

这是现在的输出(编译/运行后)...

[0] active
[1] 1
[2] import java.io.*;
[3] #
[2]
[0] bounce
[1] 2
[3] # Based upon the NCSA server configuration files originally by Rob McCool.
[2] public class MultiExec {
[1] 3
[0] corrupt
[3] #
[1] 4
[2]
[0] defer
[3] # This is the main Apache server configuration file.  It contains the
[2]     private final static String[] comLines = {
[0] deferred
[1] 5
[3] # configuration directives that give the server its instructions.
[2]                     "ls /var/spool/postfix",
[0] etc
[1] 6
[3] # See http://httpd.apache.org/docs/2.2/ for detailed information about
[2]                     "perl count50.pl",
[0] flush
[1] 7
[3] # the directives.
[2]                     "cat MultiExec.java",
[1] 8
[0] hold
[3] #
[1] 9
[2]                     "head -40 /etc/apache2/apache2.conf"};
[0] incoming
[3] # Do NOT simply read the instructions in here without understanding
[2]
[0] lib
[1] 10
[3] # what they do.  They're here only as hints or reminders.  If you are unsure
[1] 11
[2]     public void execute() {
[0] maildrop
[3] # consult the online docs. You have been warned.
[2]             for (int i = 0 ; i < comLines.length ; i++) {
[0] pid
[1] 12
[3] #
[1] 13
[2]                     ExecutableChild ec = new ExecutableChild (i, comLines[i]);
[0] private
[3] # The configuration directives are grouped into three basic sections:
[1] 14
[2]                     new Thread (ec).start();
[0] public
[3] #  1. Directives that control the operation of the Apache server process as a
[2]     }}
[1] 15
[0] saved
[3] #     whole (the 'global environment').
[1] 16
[0] trace
[2]
[3] #  2. Directives that define the parameters of the 'main' or 'default' server,
[0] usr
[2]     public class ExecutableChild implements Runnable {
[1] 17
[3] #     which responds to requests that aren't handled by a virtual host.
[0] var
[2]
[1] 18
[3] #     These directives also provide default values for the settings
[1] 19
[2]             private int prIndex;
[3] #     of all virtual hosts.
[1] 20
[2]             private String executable;
[3] #  3. Settings for virtual hosts, which allow Web requests to be sent to
[2]
[1] 21
[3] #     different IP addresses or hostnames and have them handled by the
[1] 22
[2]             public ExecutableChild (int k, String cmd) {
[3] #     same Apache server process.
[1] 23
[2]                     prIndex = k;
[3] #
[1] 24
[2]                     executable = cmd;
[3] # Configuration and logfile names: If the filenames you specify for many
[2]             }
[1] 25
[3] # of the server's control files begin with "/" (or "drive:/" for Win32), the
[2]
[1] 26
[3] # server will use that explicit path.  If the filenames do *not* begin
[1] 27
[2]             public void run () {
[3] # with "/", the value of ServerRoot is prepended -- so "/var/log/apache2/foo.log"
[1] 28
[2]                     try {
[3] # with ServerRoot set to "" will be interpreted by the
[1] 29
[2]                             Process child = Runtime.getRuntime().exec(executable);
[3] # server as "//var/log/apache2/foo.log".
[1] 30
[2]                             BufferedReader br = new BufferedReader (new InputStreamReader (
[3] #
[1] 31
[2]                                                             child.getInputStream()));
[3]
[1] 32
[2]                             for (String s = br.readLine() ; s != null ; s = br.readLine()) {
[3] ### Section 1: Global Environment
[1] 33
[2]                                     System.out.println ("[" + prIndex + "] " + s);
[3] #
[1] 34
[2]                                     try {
[3] # The directives in this section affect the overall operation of Apache,
[1] 35
[2]                                             Thread.sleep (20);
[3] # such as the number of concurrent requests it can handle or where it

......

输入流工作得很好,不要认为我有问题。很抱歉回复这么久。祝您一切顺利,并等待查看您的代码,-M.S.

【讨论】:

  • 我不禁注意到,您使用的示例 a) 不如 cscript 命令复杂 b) 除了一个示例外,所有示例都返回 1 行。出于所有意图和目的,从技术上讲,您可能会遇到类似的问题 :) 无论如何,这几乎是所有代码。其余的类变量用于存储提取的数据,并且都在任何构造函数或方法之外定义(非静态)。我将发布更多结构(很多无关紧要)以帮助推动对话。非常感谢您迄今为止的努力!
  • 您好,DN,修改后的回复已发布在那里。很高兴能提供任何帮助,-M.S.
  • 显然我花了半个多小时(其中一部分时间很忙,呵呵)来写剩下的。慢慢来,今晚我不能再看它了。 :P
  • 你是对的! instanceofTom 让我意识到你的答案中最重要的部分,我在哪里声明了流!我在 X 而不是 Y 中声明它们,导致它们被覆盖。但是你先答对了,所以对胜利者... :)
【解决方案2】:

确保您在正确的范围内声明 stdErrstdIn。在这种情况下,您需要在 Y 中声明它们。

如果您在X 中声明它们,则每次运行以下代码时:

stdErr  = process.getErrorStream();
stdIn   = process.getInputStream();

变量将被重新分配,Y 的所有实例都将引用同一个流。

【讨论】:

  • 像这样的简单错误解释了为什么我不是专业的程序员。但正如我上面所说,他首先回答了这个问题,我只是没有意识到。谢谢!
猜你喜欢
  • 2011-07-25
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2012-07-01
  • 1970-01-01
  • 2011-04-28
  • 1970-01-01
相关资源
最近更新 更多