恐慌：运行时错误：当作为 goroutine 同时运行时，切片边界超出范围

Question

我将一个函数作为 goroutine 调用，并使用 WaitGroup 来防止在共享扫描器全部完成之前关闭它们。 myfunc() 函数迭代文件。我想内存映射这个文件并在所有 goroutine 之间共享它，而不是每次都从磁盘读取 I/O 阻塞点。有人告诉我这种方法可以工作in an answer to another question. 但是，虽然这个功能可以独立工作，但它不能同时工作。我收到错误消息：

panic: runtime error: slice bounds out of range

但错误是当我调用 Scan() 方法（不是在切片上）时，这令人困惑。

这是一个 MWE：

// ... package declaration; imports; yada yada

// the actual Sizes map is much more meaningful, this is just for the MWE
var Sizes = map[int]string {
    10: "Ten",
    20: "Twenty",
    30: "Thirty",
    40: "Forty",
}

type FileScanner struct {
    io.Closer
    *bufio.Scanner
}

func main() {
    // ... validate path to file stored in filePath variable
    filePath := "/path/to/file.txt"

    // get word list scanner to be shared between goroutines
    scanner := getScannerPtr(&filePath)

    // call myfunc() for each param passed
    var wg sync.WaitGroup
    ch := make(chan string)
    for _, param := range os.Args[1:] {
        wg.Add(1)
        go myfunc(&param, scanner, ch)
        wg.Done()
    }

    // print results received from channel
    for range os.Args[1:] {
        fmt.Println(<-ch)  // print data received from channel ch
    }

    // don't close scanner until all goroutines are finished
    wg.Wait()
    defer scanner.Close()
}

func getScannerPtr(filePath *string) *FileScanner {
    f, err := os.Open(*filePath)
    if err != nil {
        fmt.Fprint(os.Stderr, "Error opening file\n")
        panic(err)
    }
    scanner := bufio.NewScanner(f)
    return &FileScanner{f, scanner}
}

func myfunc(param *string, scanner *FileScanner, ch chan<-string) {
    for scanner.Scan() {
        line := strings.TrimSpace(scanner.Text())
        // ... do something with line (read only)
        // ... access shared Sizes map when doing it (read only)
        ch <- "some string result goes here"
    }
}

我最初认为问题在于对共享 Sizes 映射的并发访问，但将其移入 myfunc()（并且每次都重新声明/重新定义效率低下）仍然导致相同的错误，这与调用 Scan() 有关.我正在尝试遵循我收到的指导in this answer.

这是恐慌的完整堆栈跟踪：

panic: runtime error: slice bounds out of range

goroutine 6 [running]:
bufio.(*Scanner).Scan(0xc42008a000, 0x80)
        /usr/local/go/src/bufio/scan.go:139 +0xb3e
main.crack(0xc42004c280, 0xc42000a080, 0xc42001c0c0)
        /Users/dan/go/src/crypto_ctf_challenge/main.go:113 +0x288
created by main.main
        /Users/dan/go/src/crypto_ctf_challenge/main.go:81 +0x1d8
exit status 2

第 81 行是：

go myfunc(&param, scanner, ch)

第 113 行是：

for scanner.Scan() {

Answer 1

实际上在查看Scan 源代码后，它似乎不是线程安全的。您可以通过从扫描仪读取一个例程来解决此问题，并且任何数量的其他例程都会消耗行并处理它们：

func main() {
    // ... validate path to file stored in filePath variable
    filePath := "/path/to/file.txt"

    // get word list scanner to be shared between goroutines
    scanner := getScannerPtr(&filePath)
    defer scanner.Close()

    // call myfunc() for each param passed
    var wg sync.WaitGroup
    ch := make(chan string)
    lines := make(chan string)
    go func() {
        for scanner.Scan() {
            lines <- scanner.Text()
        }
        close(lines)
    }()
    for _, param := range os.Args[1:] {
        wg.Add(1)
        go myfunc(param, lines, ch)
        wg.Done()
    }

    // print results received from channel
    for range os.Args[1:] {
        fmt.Println(<-ch)  // print data received from channel ch
    }

    // don't close scanner until all goroutines are finished
    wg.Wait()
}

func myfunc(param string, lines chan []byte, ch chan<-string) {
    for line := range lines {
        line = strings.TrimSpace(line)
        // ... do something with line (read only)
        // ... access shared Sizes map when doing it (read only)
        ch <- "some string result goes here"
    }
}

还要注意defering 函数的最后一行是没有意义的； defer 的全部意义在于在函数体的某处调用它，并且知道它会在函数返回后被调用。由于您使用WaitGroup 来防止函数返回，直到您完成扫描仪，您可以安全地立即推迟关闭。