Recently, I have been heavily working with Python and Jupyer Notebook. When I tried to run Tensorflow 1.0 code in Tensorflow 2.0 environment, so many errors occured.
That’s hard to solve at the begining, after I changed the tf.abc() to tf.compact.v1.abc(), it works a little bit normal. But I just want run the old code in its original way, this way also does not change the code to the pattern and paradigm of Tensorflow 2.0. It just let your code can run(no crash) in Tensorflow 2.0 environment.
The best way and far way is the understand 1.0 and 2.0, and totally refactor 1.0 code to 2.0.
Then I created two Python environments, one has tf 1.0 and one has 2.0. I thought if I change the env to tf 1.0, the Notebook will use tf 1.0. But, it’s not worked.
Then I began to run between pip install tensorflow==1.13.1 and pip install tensorflow==2.2.0. It’s hard.
I recalled in a platform, we are allowed to change the kernel of a Notebook. I guess that should be the right way. Indeed, that’s it, it works.
Suppose env tf1 has Tensorflow 1.0 and env tf2 has Tensorflow 2.0. But, there is another problem, if you want to use pandas, pandas not changed much, you need also install the version of pandas in both tf1 and tf2, the pandas’ code will duplicate in your system.
So, you want use the system’s packages at the same time, you need to add --system-site-packages parameter when you are creating your env.
virtualenv venv --system-site-packages
Now, to switch different versions of package, the only thing you need to do is to change kernel in Notebook, the following will show you how to achieve this setup.
Install Anaconda.
Create Virtual Environment with Virtualenv/venv
You can also model venv to create the virtual env.
Install virtualenv package.
pip install virtualenv
Create virtual env of tf1
virtualenv tf1 --system-site-packages
Install Tensorflow 1 in tf1
source ~/tf1/bin/activate
pip install tensorflow==1.13.1
deactivate
Create virtual env of tf2
virtualenv tf2 --system-site-packages
Install Tensorflow 2 in tf2
source ~/tf2/bin/activate
pip install tensorflow==2.2.0
deactivate
Add Virtual Environment to Jupyter Notebook
source ~/tf1/bin/activate
pip install ipykernel
python3 -m ipykernel install --name=tf1
deactivate
source ~/tf2/bin/activate
pip install ipykernel
python3 -m ipykernel install --name=tf2
Then you should see the tf1 and tf2 kernel in Notebook.
Other Commands
Activate a virtual env
source env/bin/activate
Deactivate a env
deactivate
Specify a Python version when creating env.
virtualenv -p /usr/python/python3.7 py37_env
Save all packages in an env into a txt file
pip freeze --local > requirements.txt
List all package you have installed
pip list
Upgrade a package
pip install tensorflow --upgrade
Create a env using venv module
python3 -m venv project_env
# see the env
which python
# see packages installed
pip list
delete the env, delete the directory
rm -rf project_env
create a venv in project dir
python3 -m venv my_project/venv
pip install -r requirements.txt
create a venv and using the system packages
python3 -m venv venv --system-site-packages
# see local packages(only in env, system packages is not included)
pip list --local
Using tf-nightly
virtualenv -p /usr/local/bin/python3.7m tf-nightly --system-site-packages
source tf-nightly/bin/activate
python3 -m ipykernel install --name=tf-nightly
Why I use /usr/local/bin/python3.7m? because tf-nightly has been installed in python3.7m environment, I don’t know install it under another python version.
Remove Virtual Environment from Jupyter Notebook
list kernels in Jupyter notebook
➜ ~ jupyter kernelspec list
Available kernels:
python3 /Users/yq/Library/Python/3.7/lib/python/site-packages/ipykernel/resources
tf1 /Users/yq/Library/Jupyter/kernels/tf1
tf2 /Users/yq/Library/Jupyter/kernels/tf2
tf-nightly /usr/local/share/jupyter/kernels/tf-nightly
delete a kernel
jupyter kernelspec uninstall myenv
Checking a package belongs to which version of Python
A package is tied to a version of Python. Suppose you installed a package Tenserflow=2.3.0-dev20200615 on python3.7, if you enter a wrong version of Python, say Python2.7, you will not be able use it.
check how may Pythons are in you system.
➜ ~ which python
/usr/bin/python
➜ ~ which python3
/usr/local/bin/python3
➜ ~ ls /usr/local/bin/python*
/usr/local/bin/python3 /usr/local/bin/python3.7-config /usr/local/bin/python3.8
/usr/local/bin/python3-config /usr/local/bin/python3.7m /usr/local/bin/python3.8-config
/usr/local/bin/python3.7 /usr/local/bin/python3.7m-config
➜ ~ ls /usr/bin/python*
/usr/bin/python /usr/bin/python2 /usr/bin/python2.7-config /usr/bin/pythonw
/usr/bin/python-config /usr/bin/python2.7 /usr/bin/python3 /usr/bin/pythonw2.7
Where the pythons are pointing to? We can see that python, python2 and python2.7 are all the same python.
➜ ~ ls -l /usr/bin/python*
lrwxr-xr-x 1 root wheel 75 Feb 17 21:44 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x 1 root wheel 82 Feb 17 21:44 /usr/bin/python-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
lrwxr-xr-x 1 root wheel 75 Feb 17 21:44 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x 1 root wheel 75 Feb 17 21:44 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x 1 root wheel 82 Feb 17 21:44 /usr/bin/python2.7-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
-rwxr-xr-x 1 root wheel 31488 Apr 7 04:38 /usr/bin/python3
lrwxr-xr-x 1 root wheel 76 Feb 17 21:44 /usr/bin/pythonw -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/pythonw2.7
lrwxr-xr-x 1 root wheel 76 Feb 17 21:44 /usr/bin/pythonw2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/pythonw2.7
Python2.7 is the system version, it is in /System/Library/Frameworks/Python.framework/Versions/ directory.
➜ Versions cd /System/Library/Frameworks/Python.framework/Versions/
➜ Versions ls
2.3 2.5 2.6 2.7 Current
Where the python3s are pointing to?
➜ ~ ls -l /usr/local/bin/python*
lrwxr-xr-x 1 root wheel 69 Jun 22 11:41 /usr/local/bin/python3 -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3
lrwxr-xr-x 1 root wheel 76 Jun 22 11:41 /usr/local/bin/python3-config -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3-config
lrwxr-xr-x 1 yq wheel 38 May 26 10:37 /usr/local/bin/python3.7 -> ../Cellar/python/3.7.6_1/bin/python3.7
lrwxr-xr-x 1 yq wheel 45 May 26 10:37 /usr/local/bin/python3.7-config -> ../Cellar/python/3.7.6_1/bin/python3.7-config
lrwxr-xr-x 1 yq wheel 39 May 26 10:37 /usr/local/bin/python3.7m -> ../Cellar/python/3.7.6_1/bin/python3.7m
lrwxr-xr-x 1 yq wheel 46 May 26 10:37 /usr/local/bin/python3.7m-config -> ../Cellar/python/3.7.6_1/bin/python3.7m-config
lrwxr-xr-x 1 root wheel 71 Jun 22 11:41 /usr/local/bin/python3.8 -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8
lrwxr-xr-x 1 root wheel 78 Jun 22 11:41 /usr/local/bin/python3.8-config -> ../../../Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8-config
Python3.8 is installed via pkg file from python.org.
➜ Versions cd /Library/Frameworks/Python.framework/Versions
➜ Versions ls
3.8
To explore 3.8 directory.
➜ 3.8 ll
total 6848
lrwxr-xr-x 1 root admin 17B Jun 22 11:41 Headers -> include/python3.8
-rwxrwxr-x 1 root admin 3.3M May 14 04:33 Python
drwxrwxr-x 5 root admin 160B May 14 04:30 Resources
drwxrwxr-x 3 root admin 96B May 14 04:33 _CodeSignature
drwxrwxr-x 28 root admin 896B Jun 26 11:51 bin
drwxrwxr-x 3 root admin 96B May 14 04:30 etc
drwxrwxr-x 3 root admin 96B May 14 04:30 include
drwxrwxr-x 36 root admin 1.1K Jun 22 11:41 lib
drwxrwxr-x 4 root admin 128B May 14 04:30 share
➜ 3.8 cd bin
➜ bin ls
2to3 google-oauthlib-tool pydoc3.8 python3
2to3-3.8 idle3 pyrsa-decrypt python3-config
chardetect idle3.8 pyrsa-encrypt python3.8
easy_install-3.8 markdown_py pyrsa-keygen python3.8-config
f2py pip3 pyrsa-priv2pub wheel
f2py3 pip3.8 pyrsa-sign
f2py3.8 pydoc3 pyrsa-verify
➜ bin ./python3
Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
➜ bin ./python3-config
Usage: /usr/local/bin/python3-config [--prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir|--embed]
If we want to use Tenserflow=2.3.0-dev20200615, we must enter python3.7 environment.
➜ ~ python3.7
Python 3.7.6 (default, Dec 30 2019, 19:38:26)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'2.3.0-dev20200615'
If you type python3, you may enter python3.8, Tenserflow=2.3.0-dev20200615 can not be used.
➜ ~ python3
Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'tensorflow' has no attribute '__version__'
>>>
If you type python, you will enter python2.7
➜ ~ python
WARNING: Python 2.7 is not recommended.
This version is included in macOS for compatibility with legacy software.
Future versions of macOS will not include Python 2.7.
Instead, it is recommended that you transition to using 'python3' from within Terminal.
Python 2.7.16 (default, Feb 29 2020, 01:55:37)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.