Python 3.6.8 (default, May 2 2019, 20:40:44)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyspark.context import SparkContext
>>> from pyspark.sql.session import SparkSession
>>> sc = SparkContext('local')
19/07/23 05:35:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
>>> spark = SparkSession(sc)
>>> flightPerFilePath='hdfs://10.18.3.151/export/departuredelays.csv'
>>> airportFilePath='hdfs://10.18.3.151/export/airport-codes-na.txt'
>>> aireports=spark.read.csv(airportFilePath,header='true',inferSchema='true',sep='\t')
>>> aireports.createOrReplaceTempView("airports")
>>> flightPerf=spark.read.csv(flightPerFilePath,header='true')
>>> flightPerf.createOrReplaceTempView("FlightPerformance")
>>> flightPerf.cache()
DataFrame[date: string, delay: string, distance: string, origin: string, destination: string]
>>> spark.sql("""select a.City,f.origin,sum(f.delay) as Delays from FlightPerformance f join airports a on a.IATA=f.origin
... where a.State='WA' group by a.City,f.origin order by sum(f.delay) desc""").show()
+-------+------+--------+
| City|origin| Delays|
+-------+------+--------+
|Seattle| SEA|159086.0|
|Spokane| GEG| 12404.0|
| Pasco| PSC| 949.0|
+-------+------+--------+
>>>
分类