Thus, by default, python points to /databricks/python3/bin/python and pip points to /databricks/python3/bin/pip for init scripts, SSH sessions, and notebooks. By default, PYSPARK_PYTHON is set to /databricks/python3/bin/python. You can use python and pip commands that are defined in the activated environment directly without having to specify the absolute locations of these commands.

If PYSPARK_PYTHON points to a Python executable that is in an environment managed by Virtualenv, this environment will be activated for init scripts, SSH sessions, and notebooks.

However, Python notebooks and Python commands will not work, that is, Python command cells will fail with a “Canceled” error and a Python shell failed to start error will appear in the driver logs. A cluster with such setting can still start.

Setting PYSPARK_PYTHON to /databricks/python2/bin/python is not supported because Databricks Runtime 6.0 does not support Python 2.

The display function on Python ggplot objects is no longer supported because the ggplot package is not compatible with newer version of pandas.

Compared with Databricks Runtime 5.5 LTS, the following Python packages are not installed: ansi2html, brewer2mpl, colorama, configobj, enum34, et-xmlfile, freetype-py, funcsigs, fusepy, ggplot, html5lib, ipaddress, jdcal, Jinja2, llvmlite, lxml, MarkupSafe, mpld3, msgpack-python, ndg-httpsclient, numba, openpyxl, pathlib2, Pillow, ply, pyasn1, pypng, python-geohash, scour, simplejson, and singledispatch.

Compared with Databricks Runtime 5.5 LTS, the following Python packages are newly included: asn1crypto, backcall, jedi, kiwisolver, parso, and PySocks.Install Python 3.7 compatible versions of Python packages instead. Installing such a package can trigger errors similar to ‘PyThreadState’ has no member named ‘exc_type’ (see GitHub issue 1978 for details). Some old versions of Python packages may not be forward compatible with Python 3.7 because they depend on old versions of Cython that are not compatible with Python 3.7. In addition, as was previously announced, Databricks Runtime 6.0 does not support Python 2. For details, see Installed Python libraries.

Databricks Runtime 6.0 includes major changes to Python and the way Python environments are configured, including upgrading Python to 3.7.3, refining the list of installed Python packages, and upgrading those installed packages to newer versions.

Install openpyxl python 3.5