flink1.12.x 重分区算子-forward()

说明

当下游算子 与 上游算子 并行度 一样时,默认的数据传递方式是 forward

上下游并行度相同时,一对一发送,将元素转发到本地运行的下游操作的分区器

方法声明

filter、keyBy 后可以调用

forward()

案例

package chongfenqu;

import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;

import java.util.Properties;

public class F10forward {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // 定义kafka相关配置项
        Properties pro = new Properties();
        pro.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"hadoop1:9092,hadoop2:9092,hadoop3:9092");
        pro.setProperty(ConsumerConfig.GROUP_ID_CONFIG,"consumer-group");
        pro.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"latest");

        // 创建FlinkKafka消费者,需要指定主题、反序列化、kafka配置
        FlinkKafkaConsumer<String> fkc = new FlinkKafkaConsumer<>("kafka2flink", new SimpleStringSchema(), pro);

        DataStreamSource<String> dss = env.addSource(fkc);

        dss.print("dss:");

        DataStream<Tuple2<Integer,String>> mapDS = dss.map(new RichMapFunction<String, Tuple2<Integer,String>>() {
            @Override
            public Tuple2<Integer, String> map(String s) throws Exception {
                return Tuple2.of(getRuntimeContext().getIndexOfThisSubtask(),s);
            }
        });
        mapDS.print("mapDS::");

        System.out.println(env.getExecutionPlan());
        env.execute();
    }
}

生成计划可视图

访问:https://flink.apache.org/visualizer/

将上面打印的执行计划生成图,由于没有设置并行度,所以并行度等于CPU线程数,即:都是 8,所以都是 forward,如下:

参考

https://blog.51cto.com/u_9928699/5670559


原文出处:/show_1IX44ZOFaE7X.html