hadoop3.x MapReduce-Partition分区的注意事项

本文代码在 hadoop3.x MapReduce-Partition分区的实现 基础上修改

注意1

如果 1 < job.setNumReduceTasks(n) 中的 n < getPartition() 返回值数量 ,则有一部分分区数据无处存放,会报错;

例子

数据如下,有3个日期:

1,100,2020-01-01 09:30:50,2020-01-01 09:40:50
2,200,2020-01-11 10:40:50,2020-01-11 11:05:50
3,150,2020-03-01 11:20:50,2020-03-01 11:32:50
1,50,2020-02-02 12:30:50,2020-02-02 12:35:50
1,300,2020-01-03 14:30:50,2020-01-03 15:20:10
2,500,2020-02-02 15:30:30,2020-02-02 17:20:30
3,400,2020-03-02 17:30:20,2020-03-02 18:20:20

设置 reduce task 数量是2:

job.setNumReduceTasks(2);

执行报错:

java.lang.Exception: java.io.IOException: Illegal partition for 3 (2)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Illegal partition for 3 (2)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1096)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:727)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
    at partition.MyMapper.map(MyMapper.java:50)
    at partition.MyMapper.map(MyMapper.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

注意2

如果 job.setNumReduceTasks(n) 中的 n > getPartition() 返回值数量 ,会产生几个空的输出文件

提示:一般不要这么设置,会浪费 CPU、内存资源,除非有此业务需求

注意3

如果 job.setNumReduceTasks(n) 中的 n = 1 ,最终只产生一个结果文件

而且,没有执行我们自定义的 Partitioner 类

注意4

分区号从零开始,每个分区加1,不能跳着加


原文出处:/show_1IX2CpVFXkKw.html