此文已由作者岳猛授权网易云社区发布。
欢迎访问网易云社区,了解更多网易技术产品运营经验。
code examples
import org.apache.spark.sql.streaming._ val df = spark.readStream.text("/home/testhdfs") val ps = df.writeStream.format("console").outputMode(OutputMode.Append).start
basedir = /home/testhdfs
支持:mv file to basedir(/home/testhdfs)
不支持:mv directory to basedir
如果往basedir里面添加文件夹会出现ERROR:
java.lang.AssertionError: assertion failed: Conflicting directory structures detected. Suspicious paths: hdfs://172.17.1.180:9000/home/testhdfs/data1 hdfs://172.17.1.180:9000/home/testhdfsIf provided paths are partition directories, please set "basePath" in the options of the data source to specify the root directory of the table. If there are multiple root directories, please load them separately and then union them.
测试textFile接口使用
import org.apache.spark.streaming._ val ssc = StreamingContext.getActiveOrCreate(() => new StreamingContext(sc, Seconds(120))) val ds1 = ssc.textFileStream("/home/testhdfs2") ds1.print ssc.start
支持:mv file to basedir(/home/testhdfs2)
支持:mv directory to basedir
链接:https://www.jianshu.com/p/9eb8ff8f0660
更多网易技术、产品、运营经验分享请点击。
相关文章:
【推荐】 基于Redis+Kafka的首页曝光过滤方案
【推荐】 SpringBoot入门(三)——入口类解析