windows hadoop MapReduce 罕见错误:Cleaning up the staging area file:/tmp/hadoop/mapred/staging ... No such file or directory

描述错误

在 windows 中,执行 MapReduce 时,报错如下:

2021-11-05 19:21:43,340 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Cleaning up the staging area file:/tmp/hadoop/mapred/staging/mym1580568728/.staging/job_local1580568728_0001
Exception in thread "main" 0: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:236)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:773)
    at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:508)
    at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:489)
    at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:511)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:726)
    at org.apache.hadoop.mapreduce.JobResourceUploader.mkdirs(JobResourceUploader.java:658)
    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:172)
    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
    at top.malaoshi.Main.main(Main.java:41)

分析

跟踪 NativeIO.java 的 236 行代码可知,执行到 233 行代码

chmodImpl(path, mode);

报错,

好像是检查 D:\tmp\hadoop\mapred\staging\mym510547688\.staging\job_local510547688_0001 目录的权限

解决

在 windows 中,hadoop所在的盘下,有 tmp 目录,将其删除 或 改名

如:

我的 hadoop 安装在 D:\devtools\hadoop-3.0.3 目录下

那么就有 D:\tmp 目录,该目录可能是因为权限等一些原因,导致 MapReduce 无法在该目录下创建文件夹

所以将 D:\tmp 目录 删除 或 改名 即可


原文出处:/show_1IX2AmzPVVkz.html