데이터전처리
반응형
-
[Spark] 스파크 Dataframe 데이터프레임 가공하기DataProcessing/Spark 2021. 1. 31. 23:57
모듈 import import findspark findspark.init() from pyspark import SparkContext from pyspark.sql import SQLContext ## Cassandra import os os.environ['PYSPARK_SUBMIT_ARGS'] = \ '--packages com.datastax.spark:spark-cassandra-connector_2.11:2.4.1' \ ' --conf spark.cassandra.connection.host=localhost:port pyspark-shell' sc = SparkContext(appName="app name") sqlContext = SQLContext(sc) CassandraDB 데이터 조..