springboot集成spark并使用spark-sql的方法

这篇文章主要介绍"springboot集成spark并使用spark-sql的方法"的相关知识，小编通过实际案例向大家展示操作过程，操作方法简单快捷，实用性强，希望这篇"springboot集成spark并使用spark-sql的方法"文章能帮助大家解决问题。

首先添加相关依赖：

  4.0.0      org.springframework.boot    spring-boot-starter-parent    1.5.6.RELEASE        com.cord  spark-example  1.0-SNAPSHOT  spark-example    http://www.example.com      UTF-8    UTF-8    1.8    2.10.3    1.8    1.8              org.springframework.boot      spring-boot-starter      1.5.6.RELEASE                                    org.springframework.boot                spring-boot-starter-logging                                  org.apache.spark      spark-core_2.10      1.6.1      provided                                    org.slf4j                slf4j-log4j12                                        log4j                log4j                                  org.apache.spark      spark-sql_2.10      1.6.1      provided              org.apache.spark      spark-hive_2.10      1.6.1      provided              org.scala-lang      scala-library      ${scala.version}      provided                  mysql      mysql-connector-java      5.1.22                                              org.apache.maven.plugins                maven-shade-plugin                                                            org.springframework.boot                        spring-boot-maven-plugin                        1.5.6.RELEASE                                                                        false                    false                                                                        *:*                                                            META-INF/*.SF                                META-INF/*.DSA                                META-INF/*.RSA                                                                                                                                                META-INF/spring.handlers                                                                            META-INF/spring.factories                                                                            META-INF/spring.schemas                                                                                                    com.cord.StartApplication                                                                                                                        package                                                    shade                                                                                    

需要注意的是依赖中排除掉的日志模块，以及特殊的打包方式

定义配置类:

SparkContextBean.class

@Configurationpublic class SparkContextBean {    private String appName = "sparkExp";    private String master = "local";    @Bean    @ConditionalOnMissingBean(SparkConf.class)    public SparkConf sparkConf() throws Exception {        SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);        return conf;    }    @Bean    @ConditionalOnMissingBean    public JavaSparkContext javaSparkContext() throws Exception {        return new JavaSparkContext(sparkConf());    }    @Bean    @ConditionalOnMissingBean    public HiveContext hiveContext() throws Exception {        return new HiveContext(javaSparkContext());    }    ......}

启动类：

StartApplication.class

@SpringBootApplicationpublic class StartApplication implements CommandLineRunner {    @Autowired    private HiveContext hc;    public static void main(String[] args) {        SpringApplication.run(StartApplication.class, args);    }    @Override    public void run(String... args) throws Exception {        DataFrame df = hc.sql("select count(1) from LCS_DB.STAFF_INFO");        List result = df.javaRDD().map((Function) row -> {            return row.getLong(0);        }).collect();        result.stream().forEach(System.out::println);}

执行方式：

spark-submit \    --class com.cord.StartApplication  \    --executor-memory 4G \    --num-executors 8 \    --master yarn-client \/data/cord/spark-example-1.0-SNAPSHOT.jar

关于"springboot集成spark并使用spark-sql的方法"的内容就介绍到这里了，感谢大家的阅读。如果想了解更多行业相关的知识，可以关注行业资讯频道，小编每天都会为大家更新不同的知识点。