Project Zen: Making Data Science Easier in PySpark

Project Zen Goals:

• make spark more pythonic
• better ops with python libraries

Changes:

• Python Hints
• better documentation
• support conda, pip, pipex to deploy package

Shipping pacakge in conda

• pandas Api on Spark

During my Metis bootcamp, I did an intro presentation to my class about spark.

The presentation explored the content and length of a sample of the commit messages on github.

Notes on setting up pyspark and jupyter notebook.