Problem
All presented frameworks are great, but ...
they are limited by performance of our machines.
We need to handle ~TBs datasets.
Apache Spark
“Apache Spark™ is a fast and general engine for large-scale data processing.”
“Fast”
- Holds world record in 100TB dataset sorting.
- Able to cache datasets into memory.
Apache Spark
“Apache Spark™ is a fast and general engine for large-scale data processing.”
“General”
- Contains a lot of high-level data-processing functions.
- Built-in libs: SQL, Streaming, MLlib, GraphX
- API in Scala , Java, Python and R.
- Works with HDFS, HBase, Hive, Postgres, etc.
- Runs locally, over Hadoop(YARN), Mesos or in standalone mode.
PySpark
PySpark is Python wrapper around Spark.
Architecture
Performance?
“Spark’s core developers have worked extensively to bridge the performance gap between JVM languages and Python. In particular, PySpark can now run on PyPy to leverage the just-in-time compiler. (Up to 50x speedup) The way Python processes communicate with the main Spark JVM programs have also been redesigned to enable worker reuse.”
Comparable with native Spark...
Enough said...
let's give it a try.