Creaing Multiple Kernels in Jupyter Notebook

Different Kernel Has Different Versions of a Package

Recently, I have been heavily working with Python and Jupyer Notebook. When I tried to run Tensorflow 1.0 code in Tensorflow 2.0 environment, so many errors occured.

That’s hard to solve at the begining, after I changed the tf.abc() to tf.compact.v1.abc(), it works a little bit normal. But I just want run the old code in its original way, this way also does not change the code to the pattern and paradigm of Tensorflow 2.0. It just let your code can run(no crash) in Tensorflow 2.0 environment.

The best way and far way is the understand 1.0 and 2.0, and totally refactor 1.0 code to 2.0.

Then I created two Python environments, one has tf 1.0 and one has 2.0. I thought if I change the env to tf 1.0, the Notebook will use tf 1.0. But, it’s not worked.

Then I began to run between pip install tensorflow==1.13.1 and pip install tensorflow==2.2.0. It’s hard.

I recalled in a platform, we are allowed to change the kernel of a Notebook. I guess that should be the right way. Indeed, that’s it, it works.

Suppose env tf1 has Tensorflow 1.0 and env tf2 has Tensorflow 2.0. But, there is another problem, if you want to use pandas, pandas not changed much, you need also install the version of pandas in both tf1 and tf2, the pandas’ code will duplicate in your system.

So, you want use the system’s packages at the same time, you need to add --system-site-packages parameter when you are creating your env.

virtualenv venv --system-site-packages

Now, to switch different versions of package, the only thing you need to do is to change kernel in Notebook, the following will show you how to achieve this setup.

Install Anaconda.

https://www.anaconda.com/

Create Virtual Environment with Virtualenv/venv

You can also model venv to create the virtual env.

Install virtualenv package.

pip install virtualenv

Create virtual env of tf1

virtualenv tf1 --system-site-packages

Install Tensorflow 1 in tf1

source ~/tf1/bin/activate
pip install tensorflow==1.13.1
deactivate

Create virtual env of tf2

virtualenv tf2 --system-site-packages

Install Tensorflow 2 in tf2

source ~/tf2/bin/activate
pip install tensorflow==2.2.0
deactivate

Add Virtual Environment to Jupyter Notebook

source ~/tf1/bin/activate
pip install ipykernel

python3 -m ipykernel install --name=tf1
deactivate
source ~/tf2/bin/activate
pip install ipykernel

python3 -m ipykernel install --name=tf2

Then you should see the tf1 and tf2 kernel in Notebook.

Other Commands

Activate a virtual env

source env/bin/activate

Deactivate a env

deactivate

Specify a Python version when creating env.

virtualenv -p /usr/python/python3.7 py37_env

Save all packages in an env into a txt file

pip freeze --local > requirements.txt

List all package you have installed

pip list 

Upgrade a package

pip install tensorflow --upgrade

Create a env using venv module

python3 -m venv project_env

# see the env
which python

# see packages installed
pip list

delete the env, delete the directory

rm -rf project_env

create a venv in project dir

python3 -m venv my_project/venv

pip install -r requirements.txt

create a venv and using the system packages

python3 -m venv venv --system-site-packages

# see local packages(only in env, system packages is not included)
pip list --local

Using tf-nightly

virtualenv -p /usr/local/bin/python3.7m  tf-nightly --system-site-packages
source tf-nightly/bin/activate
python3 -m ipykernel install --name=tf-nightly

Why I use /usr/local/bin/python3.7m? because tf-nightly has been installed in python3.7m environment, I don’t know install it under another python version.

Remove Virtual Environment from Jupyter Notebook

list kernels in Jupyter notebook

➜  ~ jupyter kernelspec list
Available kernels:
  python3       /Users/yq/Library/Python/3.7/lib/python/site-packages/ipykernel/resources
  tf1           /Users/yq/Library/Jupyter/kernels/tf1
  tf2           /Users/yq/Library/Jupyter/kernels/tf2
  tf-nightly    /usr/local/share/jupyter/kernels/tf-nightly

delete a kernel

jupyter kernelspec uninstall myenv

Checking a package belongs to which version of Python

A package is tied to a version of Python. Suppose you installed a package Tenserflow=2.3.0-dev20200615 on python3.7, if you enter a wrong version of Python, say Python2.7, you will not be able use it.

check how may Pythons are in you system.

➜  ~ which python
/usr/bin/python
➜  ~ which python3
/usr/local/bin/python3
➜  ~ ls /usr/local/bin/python*
/usr/local/bin/python3           /usr/local/bin/python3.7-config  /usr/local/bin/python3.8
/usr/local/bin/python3-config    /usr/local/bin/python3.7m        /usr/local/bin/python3.8-config
/usr/local/bin/python3.7         /usr/local/bin/python3.7m-config
➜  ~ ls /usr/bin/python*
/usr/bin/python           /usr/bin/python2          /usr/bin/python2.7-config /usr/bin/pythonw
/usr/bin/python-config    /usr/bin/python2.7        /usr/bin/python3          /usr/bin/pythonw2.7

Where the pythons are pointing to? We can see that python, python2 and python2.7 are all the same python.

➜  ~ ls -l /usr/bin/python*
lrwxr-xr-x  1 root  wheel     75 Feb 17 21:44 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel     82 Feb 17 21:44 /usr/bin/python-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
lrwxr-xr-x  1 root  wheel     75 Feb 17 21:44 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel     75 Feb 17 21:44 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel     82 Feb 17 21:44 /usr/bin/python2.7-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
-rwxr-xr-x  1 root  wheel  31488 Apr  7 04:38 /usr/bin/python3
lrwxr-xr-x  1 root  wheel     76 Feb 17 21:44 /usr/bin/pythonw -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/pythonw2.7
lrwxr-xr-x  1 root  wheel     76 Feb 17 21:44 /usr/bin/pythonw2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/pythonw2.7

Python2.7 is the system version, it is in /System/Library/Frameworks/Python.framework/Versions/ directory.

➜  Versions cd /System/Library/Frameworks/Python.framework/Versions/
➜  Versions ls
2.3     2.5     2.6     2.7     Current

Where the python3s are pointing to?

➜  ~ ls -l /usr/local/bin/python*
lrwxr-xr-x  1 root  wheel  69 Jun 22 11:41 /usr/local/bin/python3 -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3
lrwxr-xr-x  1 root  wheel  76 Jun 22 11:41 /usr/local/bin/python3-config -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3-config
lrwxr-xr-x  1 yq    wheel  38 May 26 10:37 /usr/local/bin/python3.7 -> ../Cellar/python/3.7.6_1/bin/python3.7
lrwxr-xr-x  1 yq    wheel  45 May 26 10:37 /usr/local/bin/python3.7-config -> ../Cellar/python/3.7.6_1/bin/python3.7-config
lrwxr-xr-x  1 yq    wheel  39 May 26 10:37 /usr/local/bin/python3.7m -> ../Cellar/python/3.7.6_1/bin/python3.7m
lrwxr-xr-x  1 yq    wheel  46 May 26 10:37 /usr/local/bin/python3.7m-config -> ../Cellar/python/3.7.6_1/bin/python3.7m-config
lrwxr-xr-x  1 root  wheel  71 Jun 22 11:41 /usr/local/bin/python3.8 -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8
lrwxr-xr-x  1 root  wheel  78 Jun 22 11:41 /usr/local/bin/python3.8-config -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8-config

Python3.8 is installed via pkg file from python.org.

➜  Versions cd /Library/Frameworks/Python.framework/Versions
➜  Versions ls
3.8

To explore 3.8 directory.

3.8 ll
total 6848
lrwxr-xr-x   1 root  admin    17B Jun 22 11:41 Headers -> include/python3.8
-rwxrwxr-x   1 root  admin   3.3M May 14 04:33 Python
drwxrwxr-x   5 root  admin   160B May 14 04:30 Resources
drwxrwxr-x   3 root  admin    96B May 14 04:33 _CodeSignature
drwxrwxr-x  28 root  admin   896B Jun 26 11:51 bin
drwxrwxr-x   3 root  admin    96B May 14 04:30 etc
drwxrwxr-x   3 root  admin    96B May 14 04:30 include
drwxrwxr-x  36 root  admin   1.1K Jun 22 11:41 lib
drwxrwxr-x   4 root  admin   128B May 14 04:30 share
➜  3.8 cd bin
➜  bin ls
2to3                 google-oauthlib-tool pydoc3.8             python3
2to3-3.8             idle3                pyrsa-decrypt        python3-config
chardetect           idle3.8              pyrsa-encrypt        python3.8
easy_install-3.8     markdown_py          pyrsa-keygen         python3.8-config
f2py                 pip3                 pyrsa-priv2pub       wheel
f2py3                pip3.8               pyrsa-sign
f2py3.8              pydoc3               pyrsa-verify
➜  bin ./python3
Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
➜  bin ./python3-config 
Usage: /usr/local/bin/python3-config [--prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir|--embed]

If we want to use Tenserflow=2.3.0-dev20200615, we must enter python3.7 environment.

➜  ~ python3.7
Python 3.7.6 (default, Dec 30 2019, 19:38:26)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'2.3.0-dev20200615'

If you type python3, you may enter python3.8, Tenserflow=2.3.0-dev20200615 can not be used.

➜  ~ python3
Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'tensorflow' has no attribute '__version__'
>>>

If you type python, you will enter python2.7

➜  ~ python

WARNING: Python 2.7 is not recommended.
This version is included in macOS for compatibility with legacy software.
Future versions of macOS will not include Python 2.7.
Instead, it is recommended that you transition to using 'python3' from within Terminal.

Python 2.7.16 (default, Feb 29 2020, 01:55:37)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.

References

相关文章: