map()
将 RDD 中的 每个元素 传入 map()函数 执行,返回新元素,并将 新元素 组成一个 新RDD
可以是 类型 的转换,也可以是 值 的转换。
例子
将 list 中的元素乘以2
val sparConf = new SparkConf().setMaster("local[*]").setAppName("RDD")
val sc = new SparkContext(sparConf)
val rdd: RDD[Int] = sc.makeRDD(List(1,2,3,4))
val resRDD = rdd.map(_*2)
resRDD.collect().foreach(println)
sc.stop()
读取tomcat访问日志
读取tomcat日志文件,解析出 IP 地址
日志文件:
49.74.3.207 - - [14/Jun/2021:00:01:12 +0800] "GET /show_1EF4h7aLPSFB.html HTTP/1.1" 200 17300
121.226.148.220 - - [14/Jun/2021:00:01:14 +0800] "GET /show_1C7cGrpMhQ2.html HTTP/1.1" 200 21979
117.95.183.206 - - [14/Jun/2021:00:02:41 +0800] "GET /show_1EF1cWTFezUR.html HTTP/1.1" 200 15432
157.55.39.186 - - [14/Jun/2021:00:03:58 +0800] "GET /show_1EF5fipw0O0n.html HTTP/1.1" 200 20482
203.208.60.42 - - [14/Jun/2021:00:04:12 +0800] "GET /show_1IX51xrrA8z.html HTTP/1.1" 200 16986
203.208.60.29 - - [14/Jun/2021:00:04:14 +0800] "POST /ajaxApi/addViewNum HTTP/1.1" 200 40
203.208.60.114 - - [14/Jun/2021:00:04:48 +0800] "GET /show_1EF3uYu8CMSP.html HTTP/1.1" 200 20777
49.87.13.200 - - [14/Jun/2021:00:06:03 +0800] "GET /show_1EF2Pg2sje3N.html HTTP/1.1" 200 29320
49.87.13.200 - - [14/Jun/2021:00:06:04 +0800] "GET /ghost/css/bootstrap.min.css HTTP/1.1" 200 117305
49.87.13.200 - - [14/Jun/2021:00:06:04 +0800] "GET /ghost/css/font-awesome.min.css HTTP/1.1" 200 23739
49.87.13.200 - - [14/Jun/2021:00:06:04 +0800] "GET /ghost/css/magnific-popup.min.css HTTP/1.1" 200 5913
49.87.13.200 - - [14/Jun/2021:00:06:04 +0800] "GET /ghost/css/screen.css?v=1 HTTP/1.1" 200 17496
49.87.13.200 - - [14/Jun/2021:00:06:04 +0800] "GET /editor.md/css/editormd.preview.min.css HTTP/1.1" 200 44903
代码:
val sparConf = new SparkConf().setMaster("local[*]").setAppName("RDD")
val sc = new SparkContext(sparConf)
val rdd: RDD[String] = sc.textFile("C:\\localhost_access_log.2021-06-14.txt")
val resRDD = rdd.map(_.split(" ")(0))
resRDD.collect()
resRDD.foreach(println)
sc.stop()
