Project Zen: Making Data Science Easier in PySpark
Project Zen Goals:
- make spark more pythonic
- better ops with python libraries
Changes:
- Python Hints
- better documentation
- support conda, pip, pipex to deploy package
Shipping pacakge in conda
-
pandas Api on Spark
-
visualization
Links: Project Zen Databricks Blog