运行 mapreduce 函数时出错答案

【问题标题】：Error when running mapreduce function运行 mapreduce 函数时出错
【发布时间】：2016-06-09 13:13:26
【问题描述】：

我有以下数据框

userID  movieID rating  timestamp
1   1   9   12
1   2   10  13

我调用了这个数据框 mapper1.txt 并将它存储在与这个 python 文件相同的目录中：

from mrjob.job import MRJob

class MRRatingCounter(MRJob):
    def mapper(self, key, line):
        (userID, movieID, rating, timestamp) = line.split('\t')
        yield rating, 1

    def reducer(self, rating, occurences):
        yield rating, sum(occurences)

if __name__ == '__main__':
    MRRatingCounter.run()

不，我想通过运行以下代码来运行此功能

!python Rating-Counter.py mapreduce\mapper1.txt

然而，这会引发以下错误：

!python Rating-Counter.py mapreduce\mapper1.txt
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
Traceback (most recent call last):
  File "Rating-Counter.py", line 12, in <module>
    MRRatingCounter.run()
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\job.py", line 461, in run
    mr_job.execute()
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\job.py", line 479, in execute
    super(MRJob, self).execute()
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\launch.py", line 153, in execute
    self.run_job()
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\launch.py", line 216, in run_job
    runner.run()
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\runner.py", line 470, in run
    self._run()
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\sim.py", line 164, in _run
    _error_on_bad_paths(self.fs, self._input_paths)
  File "C:\Users\Marc\AppData\Local\Enthought\Canopy\User\lib\site-packages\mrjob\sim.py", line 549, in _error_on_bad_paths
    "None found in %s" % paths)
ValueError: At least one valid path is required. None found in ['mapreduce\\mapper1.txt']

!python Rating-Counter.py mapreduce\mapper1.txt

我不明白出了什么问题。谁能向我解释为什么这段代码不起作用？

【问题讨论】：

您是否使用/依赖于任何类似 mrjob.conf 的东西？如果是，请尝试附加 --conf-path mrjob.conf 并调用您的脚本。
您确定文件路径中的 \ 不应该是 / 斜线吗？错误是At least one valid path is required. None found in ['mapreduce\\mapper1.txt']。

标签： python mapreduce mrjob

【解决方案1】：

输入文件内容应该是制表符分隔..

| userID | movieID | rating | timestamp |
|--------|---------|--------|-----------|
| 1      | 1       | 9      | 12        |
| 1      | 2       | 10     | 13        |

而要运行的命令是

!python Rating-Counter.py mapper1.txt

确保py文件和txt文件在同一个文件夹/当前目录

我的输出：

C:\>python mrjobfirst.py
No configs found; falling
Creating temp directory c
0170424.081057.565000
Running step 1 of 1...
Streaming final output fr
mm.20170424.081057.565000
"10"    1
"9"     1
"rating"        1
Removing temp directory c
0170424.081057.565000...

【讨论】：