ENV

Posted on Edited on

Python Data Science

Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
############
# Packages #
############
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import plotly.express as px
import plotly.graph_objects as go
import plotly.io as pio
import math as mt
import scipy
from scipy import stats
import sklearn
from sklearn.model_selection import train_test_split
# import tensorflow as tf
# import tensorflow_probability as tfp
# import datascientists as ds

%matplotlib inline
try:
    %load_ext autotime
except:
    !pip install autotime
    %load_ext autotime

############
# Settings #
############
# Pandas Settings
pd.set_option('max_rows', 20)
pd.set_option('display.colheader_justify', 'left')
# plotly.express Settings
#   px.defaults.template = 'plotly_white' # "plotly", "plotly_white", "plotly_dark", "ggplot2", "seaborn", "simple_white", "none"
px.defaults.width = 1200
px.defaults.height = 800
# plotly.io Settings for both plotly.graph_objects and plotly.express
pio.templates.default = "plotly_white" # "plotly", "plotly_white", "plotly_dark", "ggplot2", "seaborn", "simple_white", "none"
pio.kaleido.scope.default_format = 'svg'
pio.kaleido.scope.default_scale = 1
# Config for fig
#   https://plotly.com/python/configuration-options/
#   https://github.com/plotly/plotly.js/blob/master/src/plot_api/plot_config.js
config = {
    'editable': True, # Axes editable
    'toImageButtonOptions': {
        'format': 'svg', # one of png, svg, jpeg, webp
        'filename': 'fig',
        'height': None, # Set to download at the currently-rendered size by setting height and width to None
        'width': None, # # Set to download at the currently-rendered size by setting height and width to None
        'scale': 1 # Multiply title/legend/axis/canvas sizes by this factor
      }, 
    'scrollZoom': True, 
    'doubleClick': 'reset+autosize', 
    'showSources': False
}

#############
# Documents #
#############
# IPython
#    https://ipython.org/ipython-doc/stable/interactive/reference.html#dynamic-object-information
# Pandas
#    https://pandas.pydata.org/pandas-docs/stable/reference/index.html#api
# Numpy
#    https://numpy.org/doc/stable/reference/index.html
# Matplotlib
#    https://matplotlib.org/stable/api/
# Plotly
#    https://plotly.com/python-api-reference/index.html
# Plotly - Image Export Settings (Kaleido)
#    https://plotly.com/python/static-image-export/
# math
#    https://docs.python.org/3/library/math.html
# Statistical functions (scipy.stats)
#    https://docs.scipy.org/doc/scipy/reference/stats.html
# scikit-learn
#    https://scikit-learn.org/stable/
# TensorFlow
#    https://www.tensorflow.org/api_docs/python/tf
# TensorFlow Probability
#    https://www.tensorflow.org/probability
# datascientists
#    https://github.com/ZacksAmber/datascientists

Jupyter Lab

1
2
!pip3 list | grep ipywidgets #>=7.6.3
!jupyter nbextension enable --py widgetsnbextension

Kaggle

Markdown
1
2
3
4
5
6
7
# Project Name

- Author: [Zacks Shen](https://www.linkedin.com/in/zacks-shen/)<br>
- Blog: [Zacks.One](https://zacks.one)</br>
- Project GitHub: []()<br>
- Project Kaggle: []()<br>
- Source: []()
Markdown
1
2
3
---

## Dependencies
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
kaggle_project = 
data_dir = ('./data' if os.path.exists('data') else f'/kaggle/input/{kaggle_project}')

for dirname, _, filenames in os.walk(data_dir):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session

Windows

Anaconda

Add 3 paths

MacOS

VMware Fusion Pro 12

Windows Server 2019
Drag .iso to VMware Fusion Pro 12 window.

Configure the settings later

.NET Core

Setting-up Visual Studio Codespaces for .NET Core
Official Document: Configure Codespace Environments

Folder Structure
.devcontainer
├── Dockerfile - a bash script that the Docker container executes
├── devcontainer.json - that defines extensions.
└── setup.sh

Define Dockerfile

The 3.1-bionic tag is for Ubuntu 18.04 LTS (line #1). If you want to use a different Linux distro, choose a different tag.
Dockerfile
1
2
3
4
5
6
7
8
# Ubuntu 18.04 LTS
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-bionic

WORKDIR /home/

COPY . .

RUN bash ./setup.sh

Configure setup.sh

In the setup.sh, we’re going to install several applications:

  1. Update the existing packages through the apt-get command. If there’s a new package, the script will install the new ones. (line #1-9).
  2. Install nvm for ASP.NET Core application development, which uses node.js (line #12).
  3. The Docker image as on today includes PowerShell 7.0.2. If you want to install the latest version of PowerShell, run this part (line #15).
  4. If you want to use zsh instead of bash, oh my zsh enhances developer experiences (line #18-22).
setup.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
## Update and install some things we should probably have

apt-get update
apt-get install -y \
  curl \
  git \
  gnupg2 \
  jq \
  sudo \
  zsh \
  python3

## Instsall nvm

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash

## Install PowerShell Core

curl -sSL https://raw.githubusercontent.com/PowerShell/PowerShell/master/tools/install-powershell.sh | bash

## Setup and install oh-my-zsh

sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
cp -R /root/.oh-my-zsh /home/$USERNAME
cp /root/.zshrc /home/$USERNAME
sed -i -e "s/\/root\/.oh-my-zsh/\/home\/$USERNAME\/.oh-my-zsh/g" /home/$USERNAME/.zshrc
chown -R $USER_UID:$USER_GID /home/$USERNAME/.oh-my-zsh /home/$USERNAME/.zshrc

List of Extensions

devcontainer.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

{
  ...
  "extensions": [
    "coenraads.bracket-pair-colorizer-2",
    "docsmsft.docs-authoring-pack",
    "donjayamanne.githistory",
    "eamodio.gitlens",
    "editorconfig.editorconfig",
    "github.vscode-pull-request-github",
    "jongrant.csharpsortusings",
    "k--kato.docomment",
    "kreativ-software.csharpextensions",
    "mhutchie.git-graph",
    "ms-dotnettools.csharp",
    "ms-vscode.powershell",
    "ms-vscode.vscode-node-azure-pack",
    "ms-vsliveshare.vsliveshare",
    "vscode-icons-team.vscode-icons",
    "visualstudioexptteam.vscodeintellicode",
    "yzhang.markdown-all-in-one",
    "ms-python.python",
    "vscodevim.vim"
  ],
  "dockerFile": "Dockerfile",
  "forwardPorts": [5000, 5001, 7071],
  "settings": {
    "terminal.integrated.shell.linux": "/usr/bin/zsh",
    "files.exclude": {
      "**/CODE_OF_CONDUCT.md": true,
      "**/LICENSE": true
    }
  }
}

What Else Does devcontainer.json Do?

  • dockerFile: Set the value as Dockerfile that we defined above.
  • forwardPorts: When VS CS takes some ports, they need to be forwarded so that we can debug that on our web browser. For example, ASP.NET Core application needs both 5000 and 5001 ports, and Azure Functions takes 7071. Put these ports into an array and assign it to this attribute.
  • settings: It’s to configure the VS CS editor settings.