EzPyPI

Definitions

PEP 440 – Version Identification and Dependency Specification


The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

  • Projects“ are software components that are made available for integration. Projects include Python libraries, frameworks, scripts, plugins, applications, collections of data or other resources, and various combinations thereof. Public Python projects are typically registered on the Python Package Index.
  • Releases“ are uniquely identified snapshots of a project.
  • Distributions“ are the packaged files which are used to publish and distribute a release.
  • Build tools“ are automated tools intended to run on development systems, producing source and binary distribution archives. Build tools may also be invoked by integration tools in order to build software distributed as sdists rather than prebuilt binary archives.
  • Index servers“ are active distribution registries which publish version and dependency metadata and place constraints on the permitted metadata.
  • Publication tools“ are automated tools intended to run on development systems and upload source and binary distribution archives to index servers.
  • Installation tools“ are integration tools specifically intended to run on deployment targets, consuming source and binary distribution archives from an index server or other designated location and deploying them to the target system.
  • Automated tools“ is a collective term covering build tools, index servers, publication tools, integration tools and any other software that produces or consumes distribution version and dependency metadata.

Setup Environments

Prerequisites

Git and Github

Git
An GitHub account.

Install git on your operating system and setup the github account.


Python

Python


TestPyPI and PyPI

An TestPyPI account.
An PyPI account.


Name your project and distribution.

Python Naming Convention
How to Write Beautiful Python Code With PEP 8
PEP 423 – Naming conventions and recipes related to packaging

We need to name the project and the distribution.

For this project, the project name is EzPyPI.
For this distribution, the distribution name of the package is ezpypi. To simplify, I will use distribution name.

  • A distribution is the integration of all of your packages.
    For the packages inside your distribution, they can have their own names.

Why don’t we use the same name for the project and distribution?

In order to comply with the naming convention of PEP 423, the distribution name should be like lower_with_under.

For making our project more attractive and clear, we would better to give the project capital words, such as:

You can also give your project and distribution the same name, such as:

But remember, for this project,

  • the project name is EzPyPI
  • the distribution name is ezpypi

Check if your distribution name available

Next, we need to check if the distribution name available in TestPyPI and PyPI.

Let’s search for the distribution name on the two websites.

TestPyPI

PyPI

Good news! Our distribution name ezpypi is available.


Create a repository for the project

If you want to build a large project, you probably need a GitHub account name as the same as your project name. Here I assume you are a individual developer who is willing to put the project under the person GitHub repository.

Modern projects need version control. So we are going to make a brand new repository for this project EzPyPI.

Login with your account on GitHub.

  • Repository name: your project name
  • Description (optional): description for your project

Initialize this repository with:
Add a README file
Add .gitignore: Python
Choose a license: MIT License

If you’d like to choose a different license, click on Licensing a repository for more information.

Click on Create repository


Click on Code, then copy the command for cloning your GitHub repository to your computer.


Project Structure

Best Practices for Python Project Structure

Structuring Your Project
Python Application Layouts: A Reference
samplemod
Packaging Python Projects

In short, your project structure should follow the structure like these:

Installable Single Package

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
project_name/

├── package_name/
│ ├── __init__.py
│ ├── core.py
│ └── helpers.py

├── tests/
│ ├── core.py
│ └── helpers_tests.py

├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.cfg

Project EzPyPI will has the project structure as the following:

1

This is the final structure

Git clone your project

Let’s find a directory to put on our project. I will choose ~/Desktop

Change directory to ~/Desktop. Then clone your GitHub repository.

CLI
1
2
3
cd ~/Desktop

gh repo clone ZacksAmber/EzPyPI

If you don’t have a GitHub CLI, you can clone your repository through git clone:

CLI
1
git clone git@github.com:ZacksAmber/EzPyPI.git

Make a directory for distribution

Change directory to your project directory. Then make a directory for your distribution with the distribution name.

CLI
1
2
3
cd EzPyPI

mkdir ezpypi

You can check your project structure through command tree.

CLI
1
tree

The latest project structure:

1
2
3
4
.
├── LICENSE
├── README.md
└── ezpypi

Virtual Environment Configuration

Installing packages using pip and virtual environments


Make an Python virtual environment for your project

Make an virtual environment through python3 -m venv/[project name].
This command will first make a directory called venv under the project directory EzPyPI.
Then make an Python virtual environment under directory venv with the directory name EzPyPI

CLI
1
python3 -m venv venv/EzPyPI

The latest project structure:

CLI
1
tree -L 4 # -L 4 to restrict the max directory levels as 4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
.
├── LICENSE
├── README.md
├── ezpypi
└── venv
└── EzPyPI
├── bin
│   ├── Activate.ps1
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── pip
│   ├── pip3
│   ├── pip3.9
│   ├── python -> python3.9
│   ├── python3 -> python3.9
│   └── python3.9 -> /usr/local/opt/python@3.9/bin/python3.9
├── include
├── lib
│   └── python3.9
└── pyvenv.cfg

Remember the path of activate, it is located in [project directory]/venv/[venv directory]/bin/activate


Activate your venv

Activate your venv through source [project directory]/venv/[venv directory]/bin/activate

CLI
1
source venv/EzPyPI/bin/activate

As you can see, once you activate your venv, the user prefix will change to your venv name.


Deactivate your venv

If you don’t wan to use this venv again, type the following command:

CLI
1
deactivate


Upgrade and Install required Python tools

Distributing Python Modules

Key terms

  • the Python Packaging Index is a public repository of open source licensed packages made available for use by other Python users
  • distutils is the original build and distribution system first added to the Python standard library in 1998. While direct use of distutils is being phased out, it still laid the foundation for the current packaging and distribution infrastructure, and it not only remains part of the standard library, but its name lives on in other ways (such as the name of the mailing list used to coordinate Python packaging standards development).
  • setuptools is a (largely) drop-in replacement for distutils first published in 2004. Its most notable addition over the unmodified distutils tools was the ability to declare dependencies on other packages. It is currently recommended as a more regularly updated alternative to distutils that offers consistent support for more recent packaging standards across a wide range of Python versions.
  • wheel (in this context) is a project that adds the bdist_wheel command to distutils/setuptools. This produces a cross platform binary packaging format (called “wheels” or “wheel files” and defined in PEP 427) that allows Python libraries, even those including binary extensions, to be installed on a system without needing to be built locally.

Our Practice

Activate your venv if you deactivate it.

Using pip to install or upgrade the three packages: setuptools, wheel, and twine.

CLI
1
pip install setuptools wheel twine --upgrade

List all of your current packages under the venv.

CLI
1
pip list


Building and packaging the project

Building and packaging the project


Creating a test directory

tests/ is a placeholder for test files. Leave it empty for now.

CLI
1
mkdir test/

The latest project structure:

CLI
1
tree -L 2 # -L 2 to restrict the max directory levels as 2
1
2
3
4
5
6
7
.
├── LICENSE
├── README.md
├── ezpypi
├── tests
└── venv
└── EzPyPI

Creating pyproject.toml

Official Guides

pyproject.toml tells build tools (like pip and build) what is required to build your project. This tutorial uses setuptools, so open pyproject.toml and enter the following content:

1
2
3
4
5
6
[build-system]
requires = [
"setuptools>=42",
"wheel"
]
build-backend = "setuptools.build_meta"
  • build-system.requires gives a list of packages that are needed to build your package. Listing something here will only make it available during the build, not after it is installed.
  • build-system.build-backend is the name of Python object that will be used to perform the build. If you were to use a different build system, such as flit or poetry, those would go here, and the configuration details would be completely different than the setuptools configuration described below.
  • See PEP 517 and PEP 518 for background and details.

Our Practice

Create pyproject.toml

CLI
1
touch pyproject.toml

Add the above content to pyproject.toml then save the file.
The pyproject.toml for this project is:

CLI
1
2
3
4
5
6
[build-system]
requires = [
"setuptools>=42",
"wheel"
]
build-backend = "setuptools.build_meta"

The latest project structure:

CLI
1
tree -L 2 # -L 2 to restrict the max directory levels as 2
1
2
3
4
5
6
7
8
.
├── LICENSE
├── README.md
├── ezpypi
├── pyproject.toml
├── tests
└── venv
└── EzPyPI

Configuring metadata

Official Guides

Building and packaging the project
Configuring setup() using setup.cfg files
Packaging and distributing projects

There are two types of metadata: static and dynamic.

  • Static metadata (setup.cfg): guaranteed to be the same every time. This is simpler, easier to read, and avoids many common errors, like encoding errors.
  • Dynamic metadata (setup.py): possibly non-deterministic. Any items that are dynamic or determined at install-time, as well as extension modules or extensions to setuptools, need to go into setup.py.

Static metadata (setup.cfg) should be preferred. Dynamic metadata (setup.py) should be used only as an escape hatch when absolutely necessary. setup.py used to be required, but can be omitted with newer versions of setuptools and pip.


setup.cfg is the configuration file for setuptools. It tells setuptools about your package (such as the name and version) as well as which code files to include. Eventually much of this configuration may be able to move to pyproject.toml.

Open setup.cfg and enter the following content. Change the name to include your username; this ensures that you have a unique distribution name and that your package doesn’t conflict with packages uploaded by other people following this tutorial.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[metadata]
name = example-pkg-YOUR-USERNAME-HERE
version = 0.0.1
author = Example Author
author_email = author@example.com
description = A small example package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/pypa/sampleproject
project_urls =
Bug Tracker = https://github.com/pypa/sampleproject/issues
classifiers =
Programming Language :: Python :: 3
License :: OSI Approved :: MIT License
Operating System :: OS Independent

[options]
package_dir =
= src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

There are a variety of metadata and options supported here. This is in configparser format; do not place quotes around values. This example package uses a relatively minimal set of metadata:

  • name is the distribution name of your package. This can be any name as long as it only contains letters, numbers, _ , and -. It also must not already be taken on pypi.org. Be sure to update this with your username, as this ensures you won’t try to upload a package with the same name as one which already exists.
  • version is the package version. See PEP 440 for more details on versions. You can use file: or attr: directives to read from a file or package attribute.
  • author and author_email are used to identify the author of the package.
  • description is a short, one-sentence summary of the package.
  • long_description is a detailed description of the package. This is shown on the package detail page on the Python Package Index. In this case, the long description is loaded from README.md (which is a common pattern) using the file: directive.
  • long_description_content_type tells the index what type of markup is used for the long description. In this case, it’s Markdown.
  • url is the URL for the homepage of the project. For many projects, this will just be a link to GitHub, GitLab, Bitbucket, or similar code hosting service.
  • project_urls lets you list any number of extra links to show on PyPI. Generally this could be to documentation, issue trackers, etc.
  • classifiers gives the index and pip some additional metadata about your package. In this case, the package is only compatible with Python 3, is licensed under the MIT license, and is OS-independent. You should always include at least which version(s) of Python your package works on, which license your package is available under, and which operating systems your package will work on. For a complete list of classifiers, see https://pypi.org/classifiers/.

In the options category, we have controls for setuptools itself:

  • package_dir is a mapping of distribution names and directories. An empty distribution name represents the “root package” — the directory in the project that contains all Python source files for the package — so in this case the src directory is designated the root package.
  • packages is a list of all Python import packages that should be included in the distribution package. Instead of listing each package manually, we can use the find: directive to automatically discover all packages and subpackages and options.packages.find to specify the package_dir to use. In this case, the list of packages will be example_package as that’s the only package present.
  • python_requires gives the versions of Python supported by your project. Installers like pip will look back through older versions of packages until it finds one that has a matching Python version.

setup.py is the build script for setuptools. It tells setuptools about your package (such as the name and version) as well as which code files to include.

Open setup.py and enter the following content. Change the name to include your username; this ensures that you have a unique distribution name and that your package doesn’t conflict with packages uploaded by other people following this tutorial.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()

setuptools.setup(
name="example-pkg-YOUR-USERNAME-HERE",
version="0.0.1",
author="Example Author",
author_email="author@example.com",
description="A small example package",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/pypa/sampleproject",
project_urls={
"Bug Tracker": "https://github.com/pypa/sampleproject/issues",
},
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
package_dir={"": "src"},
packages=setuptools.find_packages(where="src"),
python_requires=">=3.6",
)

setup() takes several arguments. This example package uses a relatively minimal set:

  • name is the distribution name of your package. This can be any name as long as it only contains letters, numbers, _ , and -. It also must not already be taken on pypi.org. Be sure to update this with your username, as this ensures you won’t try to upload a package with the same name as one which already exists.
  • version is the package version. See PEP 440 for more details on versions.
  • author and author_email are used to identify the author of the package.
  • description is a short, one-sentence summary of the package.
  • long_description is a detailed description of the package. This is shown on the package detail page on the Python Package Index. In this case, the long description is loaded from README.md, which is a common pattern.
  • long_description_content_type tells the index what type of markup is used for the long description. In this case, it’s Markdown.
  • url is the URL for the homepage of the project. For many projects, this will just be a link to GitHub, GitLab, Bitbucket, or similar code hosting service.
  • project_urls lets you list any number of extra links to show on PyPI. Generally this could be to documentation, issue trackers, etc.
  • classifiers gives the index and pip some additional metadata about your package. In this case, the package is only compatible with Python 3, is licensed under the MIT license, and is OS-independent. You should always include at least which version(s) of Python your package works on, which license your package is available under, and which operating systems your package will work on. For a complete list of classifiers, see https://pypi.org/classifiers/.
  • package_dir is a dictionary with distribution names for keys and directories for values. An empty distribution name represents the “root package” — the directory in the project that contains all Python source files for the package — so in this case the src directory is designated the root package.
  • packages is a list of all Python import packages that should be included in the distribution package. Instead of listing each package manually, we can use find_packages() to automatically discover all packages and subpackages under package_dir. In this case, the list of packages will be example_package as that’s the only package present.
  • python_requires gives the versions of Python supported by your project. Installers like pip will look back though older versions of packages until it finds one that has a matching Python version.

There are many more than the ones mentioned here. See Packaging and distributing projects for more details.


Our Practice

Specifications
PEP 440 – Version Identification and Dependency Specification

We are going to use Sequence-based scheme for versioning.

Finial releases:

1
MAJOR.MINOR[.MICRO]

Development releases:

1
MAJOR.MINOR[.MICRO].devN

This project does need setup.py. So we will only create a setup.cfg for setuptools.

Make setup.cfg then copy and edit the above template.

CLI
1
touch setup.cfg

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
.
├── LICENSE
├── README.md
├── ezpypi
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

The configuration meta data for project EzPyPI are:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[metadata]
name = ezpypi
# Reference: https://www.python.org/dev/peps/pep-0440/#version-scheme
version = 0.0.1.dev0
author = Zacks Shen
author_email = zacks.shen@gmail.com
description = A Python publication tool for painlessly uploading your package to GitHub, TestPyPI, and PyPI.
long_description = file: README.md, CHANGELOG.md, LICENSE
long_description_content_type = text/markdown
url = https://github.com/ZacksAmber/EzPyPI
license = MIT
project_urls =
Bug Tracker = https://github.com/ZacksAmber/EzPyPI/issues
Changelog = https://github.com/ZacksAmber/EzPyPI/blob/main/CHANGELOG.md
# Reference: https://pypi.org/classifiers/
classifiers =
Development Status :: 1 - Planning
# Development Status :: 2 - Pre-Alpha
# Development Status :: 3 - Alpha
# Development Status :: 4 - Beta
# Development Status :: 5 - Production/Stable
# Development Status :: 6 - Mature
# Development Status :: 7 - Inactive
Environment :: Console
Intended Audience :: Developers
License :: OSI Approved :: MIT License
Natural Language :: English
Operating System :: OS Independent
Programming Language :: Python
Programming Language :: Python :: 3
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Topic :: Software Development :: Libraries :: Python Modules

[options]
zip_safe = False
package_dir=
= .
packages = find:
python_requires = >=3.6
include_package_data = True
setup_requires =
setuptools
install_requires =
requests

[options.packages.find]
where = .

[flake8]
max-line-length = 80
doctests = True
exclude = .git, .eggs, __pycache__, tests/, docs/, build/, dist/

Create CHANGELOG.md

Changelog: Changelog (or release notes) helps developers to stay up to date with project changes and helps migrating to new versions.
Semantic Versioning 2.0.0

As you can see, we need three files README.md, CHANGELOG.md, and LICENSE.
We already have README.md and LICENSE. So let’s create CHANGELOG.md.

CLI
1
touch CHANGELOG.md

The content inside CHANGELOG.md is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

---

## [Unreleased]

### Added

- Initialize project [EzPyPI](https://github.com/ZacksAmber/EzPyPI/)

### Fixed

- None

---

## [0.0.1] - 2021-08-01

### Added

- None

### Changed

- None

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
10
.
├── CHANGELOG.md
├── LICENSE
├── README.md
├── ezpypi
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

Including other files

Official Guides

Including other files
Including files in source distributions with MANIFEST.in

The files listed above will be included automatically in your source distribution. If you want to control what goes in this explicitly, see Including files in source distributions with MANIFEST.in.

The final built distribution will have the Python files in the discovered or listed Python packages. If you want to control what goes here, such as to add data files, see Including Data Files from the setuptools docs.


Our Practice

SciPy MANIFEST.in

CLI
1
touch MANIFEST.in

Edit the MANIFEST.in.
The content inside MANIFEST.in is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
include MANIFEST.in
include LICENSE

# Top-level build script
include setup.cfg
include pyproject.toml

# All source files
recursive-include ezpypi *

# All documentation
recursive-include doc *
include README.md

# Add build and testing tools

# Exclude what we don't want to include
prune */venv
prune */__pycache__
global-exclude *.pyc *~ *.bak *.swp *.pyo *.json

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
10
11
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── ezpypi
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

Create directory doc

Let’s create directory doc for future use.

CLI
1
mkdir doc

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
10
11
12
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── doc
├── ezpypi
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

Add the basic Python scripts

Create __init__.py

Create __init__.py into your package directory.

CLI
1
touch ezpypi/__init__.py

Add the following code:

Python
1
from ezpypi.core import EzPyPI

Create core.py

Create core.py into your package directory.

CLI
1
touch ezpypi/core.py

Add the following code:

Python
1
2
3
4
5
6
class EzPyPI:
def __init__(self):
pass

def main(self):
return 'Hello World!'

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── doc
├── ezpypi
│   ├── __init__.py
│   └── core.py
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

Local Installation

CLI
1
pip install .


Validation

List the Python packages under your venv.

CLI
1
pip list

Now we can try to import ezpypi in different directories.
Change directory to venv.

CLI
1
2
3
cd venv

python3
Python
1
2
3
4
5
6
import ezpypi

ezpypi.EzPyPI().main()

# OR
ezpypi.core.EzPyPI().main()


Exit Python console.

Python
1
exit()

Change directory to the last one.

CLI
1
2
3
4
5
6
7
cd ..
````

The latest project structure:

```sh CLI
tree -L 2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── doc
├── ezpypi
│   ├── __init__.py
│   ├── __pycache__
│   └── core.py
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

since we successfully validate the package, we remove the .dev0 in the 3rd line of setup.cfg.

The latest setup.cfg should like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[metadata]
name = ezpypi
# Reference: https://www.python.org/dev/peps/pep-0440/#version-scheme
version = 0.0.1
author = Zacks Shen
author_email = zacks.shen@gmail.com
description = A Python publication tool for painlessly uploading your package to GitHub, TestPyPI, and PyPI.
long_description = file: README.md, CHANGELOG.md, LICENSE
long_description_content_type = text/markdown
url = https://github.com/ZacksAmber/EzPyPI
license = MIT
project_urls =
Bug Tracker = https://github.com/ZacksAmber/EzPyPI/issues
Changelog = https://github.com/ZacksAmber/EzPyPI/blob/main/CHANGELOG.md
# Reference: https://pypi.org/classifiers/
classifiers =
Development Status :: 1 - Planning
# Development Status :: 2 - Pre-Alpha
# Development Status :: 3 - Alpha
# Development Status :: 4 - Beta
# Development Status :: 5 - Production/Stable
# Development Status :: 6 - Mature
# Development Status :: 7 - Inactive
Environment :: Console
Intended Audience :: Developers
License :: OSI Approved :: MIT License
Natural Language :: English
Operating System :: OS Independent
Programming Language :: Python
Programming Language :: Python :: 3
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Topic :: Software Development :: Libraries :: Python Modules

[options]
zip_safe = False
package_dir=
= .
packages = find:
python_requires = >=3.6
include_package_data = True
setup_requires =
setuptools
install_requires =
requests

[options.packages.find]
where = .

[flake8]
max-line-length = 80
doctests = True
exclude = .git, .eggs, __pycache__, tests/, docs/, build/, dist/

Working with GitHub

Our package works well!
So let’s push the package to GitHub with main branch and maintenance/0.0.x


branch main

View current branch

List the available branches and your current branch.

CLI
1
git branch

As you can see, I only have one branch – main.


View status

Press q to quit.

CLI
1
git status

Git reminds us that we need to commit the changes before uploading.


Add and Commit

Add all objects then commit them.

CLI
1
2
git add .
git commit -m 'Initialize Project EzPyPI. Ver: 0.0.1.dev0'


Push

Push the local repository to GitHub with branch main

CLI
1
git push

Check our GitHub repository EzPyPI with the branch main.

Check the git status

CLI
1
git status

Everything is up to date!


branch maintenance

Make an maintenance branch

17.1. Branches
3.2 Git Branching - Basic Branching and Merging

Let’s make a maintenance branch for our project with the name maintenance/0.0.x.

CLI
1
git checkout -b maintenance/0.0.x

CLI
1
git status

Now we are on branch maintenance/0.0.x and everything is the same as them on branch main.


Push

Git Push

Since nothing to commit, we are ready to push branch maintenance/0.0.x to GitHub.

  • git push -u origin [branch]: Useful when pushing a new branch, this creates an upstream tracking branch with a lasting relationship to your local branch
CLI
1
git push -u origin maintenance/0.0.x

Check branch maintenance/0.0.x on GitHub.

Click on main.
On the list, click on maintenance/0.0.x .

Everything is up to date!


Switch branch

Git Branches: List, Create, Switch to, Merge, Push, & Delete

To list all of the local branches:

CLI
1
git branch

To switch to another branch:

  • git checkout [branch name]
CLI
1
git checkout main


Update your project

In the future, you should yield the following steps to update your project:

  1. Assume your latest distribution version is 0.0.1
  2. Switch to branch maintenance/0.0.x through command git checkout maintenance/0.0.x
  3. Modify your project version from 0.0.1 to 0.0.2.devN in setup.cfg
  4. Every time when the project works well with your latest changes:
    1. git add [changes]
    2. git commit -m 'Update/Fix xx features. ver=x.x.x.devN'
  5. When everything pass the unit test and integration test, remove .devN in setup.cfg. Then push the project to GitHub with branch maintenance/0.0.x.
  6. Release the new project to TestPyPI. Then test your project by your users.
  7. If no bug, switch branch to main through command git checkout main.
  8. Merge your changes from maintenance/0.0.x to main through command git merge maintenance/0.0.x. Then push the project to GitHub with branch main.
  9. Release the new project to PyPI

Generating distribution archives

Generating distribution archives

Since we successfully tested the package, it’s time to generate distribution archives for uploading the distribution to TestPyPI PyPI.

Check the git status and checkout to maintenance/0.0.x

CLI
1
2
3
git status

git checkout maintenance/0.0.x


The next step is to generate distribution packages for the package. These are archives that are uploaded to the Python Package Index and can be installed by pip.

Make sure you have the latest version of PyPA’s build installed:

CLI
1
pip install --upgrade build

Tip: If you have trouble installing these, see the Installing Packages tutorial.

Now run this command from the same directory where pyproject.toml is located:

CLI
1
python3 -m build

This command should output a lot of text and once completed should generate two files in the dist directory:

CLI
1
ls -l dist 

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── dist
│   ├── ezpypi-0.0.1-py3-none-any.whl
│   └── ezpypi-0.0.1.tar.gz
├── doc
├── ezpypi
│   ├── __init__.py
│   ├── __pycache__
│   └── core.py
├── ezpypi.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── not-zip-safe
│   ├── requires.txt
│   └── top_level.txt
├── pyproject.toml
├── setup.cfg
├── tests
└── venv
└── EzPyPI

The tar.gz file is a source archive whereas the .whl file is a built distribution. Newer pip versions preferentially install built distributions, but will fall back to source archives if needed. You should always upload a source archive and provide built archives for the platforms your project is compatible with. In this case, our example package is compatible with Python on any platform so only one built distribution is needed.


Uploading the distribution archives to TestPyPI

Finally, it’s time to upload your package to the TestPyPI!

To securely upload your project, you’ll need a PyPI API token. Create one at https://test.pypi.org/manage/account/#api-tokens, setting the “Scope” to “Entire account”. Don’t close the page until you have copied and saved the token — you won’t see that token again.

Now that you are registered, you can use twine to upload the distribution packages. You’ll need to install Twine:

CLI
1
python3 -m pip install --upgrade twine

Once installed, run Twine to upload all of the archives under dist:

CLI
1
python3 -m twine upload --repository testpypi dist/*

You will be prompted for a username and password. For the username, use __token__. For the password, use the token value, including the pypi- prefix.

After the command completes, you should see output similar to this:

Once uploaded your package should be viewable on TestPyPI, for example, https://test.pypi.org/project/example-pkg-YOUR-USERNAME-HERE

To view our package on TestPyPI

Now search for ezpypi on TestPyPI


Installing your newly uploaded package

Check your local installed packages and grep ezpypi:

CLI
1
pip list | grep ezpypi

Uninstall ezpypi

CLI
1
pip uninstall ezpypi

You can use pip to install your package and verify that it works. Create a virtual environment and install your package from TestPyPI:

CLI
1
pip install -i https://test.pypi.org/simple/ ezpypi==0.0.1

Validation:

Change directory to venv

CLI
1
cd venv

You can test that it was installed correctly by importing the package. Make sure you’re still in your virtual environment, then run Python:

CLI
1
python3
Python
1
2
3
4
5
6
import ezpypi

ezpypi.EzPyPI().main()

# OR
ezpypi.core.EzPyPI().main()


Python
1
exit()

Change directory to your project directory

CLI
1
cd ..

Uploading the distribution archives to PyPI

Since we successfully tested the package from Test PyPI, it’s time to upload the package to PyPI.

Check the git status and checkout to maintenance/0.0.x

CLI
1
2
3
git status

git checkout main

Run Twine to upload all of the archives under dist:

CLI
1
python3 -m twine upload --repository pypi dist/*

View our project on PyPI

Search for ezpypi on PyPI


Installing your newly uploaded package

Check your local installed packages and grep ezpypi:

CLI
1
pip list | grep ezpypi

Uninstall ezpypi

CLI
1
pip uninstall ezpypi

You can use pip to install your package and verify that it works. Create a virtual environment and install your package from PyPI:

CLI
1
pip install ezpypi==0.0.1

Validation:

Change directory to venv

CLI
1
cd venv

You can test that it was installed correctly by importing the package. Make sure you’re still in your virtual environment, then run Python:

CLI
1
python3
Python
1
2
3
4
5
6
import ezpypi

ezpypi.EzPyPI().main()

# OR
ezpypi.core.EzPyPI().main()


Python
1
exit()

Change directory to your project directory

CLI
1
cd ..

One more Example

We made it! And now we want to improve our project:

  • update docs
  • add more dependencies
  • update the project
    • add git flow chart
    • make more packages to make the project powerful
    • make the packages organized

BUG WARNING: all of the following version should be 0.1.0 instead of 0.2.0.

PyPI and TestPyPI do not support downgrade your version. So make sure your version be continuous.


Git

But we need checkout to maintenance/0.0.x since we’re updating the stable distribution.
Remember, any update may causing unpredictable bugs. That’s why we need version control!

CLI
1
2
3
git status

git checkout maintenance/0.0.x

Since we are updating the distribution, rather than fixing bugs, we will update the version in setup.cfg from 0.0.1 to 0.2.0.dev0.

Commit this change.

CLI
1
2
3
git status
git add setup.cfg
git commit -m 'Update setup.cfg. ver=0.2.0.dev0'


Update Docs

Since we have more information now, it’s time to update the docs.


README.md

The README.md is displayed on the your project home page of TestPyPI and PyPI, since long_description = file: README.md is in setup.cfg


CHANGELOG.md

The CHANGELOG.md should be updated every time when you make or want to make an update for your project.


setup.cfg

Change it as your needs. For example, we can switch the classifiers from Development Status :: 1 - Planning to Development Status :: 2 - Pre-Alpha.


Add files to doc

Add extra files to doc, such as an README.md

1
2
3
4
# EzPyPI Documentation

The `doc` directory contains:
- `EzPyPI_Git_flow.svg`

Notice: If you don’t add any file to doc/, you cannot push an empty directory to GitHub.


Add workflows

draw.io

Before we get start, let’s push the current commit to GitHub.

CLI
1
2
3
4
git status

git add .
git commit -m 'Update Project Structure. ver=0.2.0.dev0'

CLI
1
git push


Flowcharts or mind map can help us better understand users’ requirements and design your program.

Visit https://draw.io and click on GitHub.

Click on Create New Diagram

For example, under Software, select Git flow 3

Under extension, select .svg

Rename it then click on Create

Login with GitHub account.

Choose doc/ directory. Then click on OK

Leave the commit message as default.

a simple git branching model (written in 2013)
OneFlow – a Git branching model and workflow

Once you finished, save it and add a commit message.

Check the Git flow on our GitHub.


Check GitHub status.

CLI
1
git remote update

Check git local status

CLI
1
git status

Since we save the git flow chart on GitHub, we need to synchronize the local git objects.

CLI
1
git pull

List the directory doc

As we can see, we have the Git flow chart on the local machine.


Add dependencies

Edit your setup.cfg.
Under the install_requires, add more packages.

For example:

1
2
3
4
5
6
7
install_requires = 
requests
numpy
pandas
scipy
plotly
kaleido

Change your Project Structure

Since our project has the most basic project structure.
We’d like to update it and make it more organized and extendable.

Make sure your are on the root directory of your project directory.

CLI
1
2
3
pwd 

tree -L 1

Add a package directory called src as the distribution directory. Therefore, our three important directories are:

  • EzPyPi: the Project Directory
  • EzPyPi/src: the Distribution Directory, which is the root directory for your source code.
    • Python recommends to use src as the Distribution Directory
  • EzPyPi/src/ezpypi: the Main Package Directory

Make a directory called src.

CLI
1
mkdir src

Move the Main Package Directory to the Distribution Directory:

CLI
1
mv ezpypi src

The latest project structure:

CLI
1
tree -L 2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── dist
│   ├── ezpypi-0.0.1-py3-none-any.whl
│   └── ezpypi-0.0.1.tar.gz
├── doc
│   ├── EzPyPI_Git_flow.svg
│   └── README.md
├── ezpypi.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── not-zip-safe
│   ├── requires.txt
│   └── top_level.txt
├── pyproject.toml
├── setup.cfg
├── src
│   └── ezpypi
├── tests
└── venv
└── EzPyPI

Then we need to update setup.cfg: replace . with src

  • package_dir: line 39
  • where: 54

The latest setup.cfg for project EzPyPI is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
[metadata]
name = ezpypi
# Reference: https://www.python.org/dev/peps/pep-0440/#version-scheme
version = 0.2.0.dev0
author = Zacks Shen
author_email = zacks.shen@gmail.com
description = A Python publication tool for painlessly uploading your package to GitHub, Test PyPI, and PyPI.
long_description = file: README.md, CHANGELOG.md, LICENSE
long_description_content_type = text/markdown
url = https://github.com/ZacksAmber/EzPyPI
license = MIT
project_urls =
Bug Tracker = https://github.com/ZacksAmber/EzPyPI/issues
Changelog = https://github.com/ZacksAmber/EzPyPI/blob/main/CHANGELOG.md
# Reference: https://pypi.org/classifiers/
classifiers =
# Development Status :: 1 - Planning
Development Status :: 2 - Pre-Alpha
# Development Status :: 3 - Alpha
# Development Status :: 4 - Beta
# Development Status :: 5 - Production/Stable
# Development Status :: 6 - Mature
# Development Status :: 7 - Inactive
Environment :: Console
Intended Audience :: Developers
License :: OSI Approved :: MIT License
Natural Language :: English
Operating System :: OS Independent
Programming Language :: Python
Programming Language :: Python :: 3
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Topic :: Software Development :: Libraries :: Python Modules

[options]
zip_safe = False
package_dir=
= src
packages = find:
python_requires = >=3.6
include_package_data = True
setup_requires =
setuptools
install_requires =
requests
setuptools
wheel
twine
build

[options.packages.find]
where = src

[flake8]
max-line-length = 80
doctests = True
exclude = .git, .eggs, __pycache__, tests/, docs/, build/, dist/

Add core directory

You can use directory core to store your core code. core will be a package of your project.

  • mkdir src/[your main package directory]/core
CLI
1
mkdir src/ezpypi/core 

Move core.py to directory core

CLI
1
mv src/ezpypi/core.py src/ezpypi/core

rename core.py to any name since we already have directory core, such as main

CLI
1
mv src/ezpypi/core/core.py src/ezpypi/core/main.py 

Edit main.py

Python
1
2
def main():
return 'module: src/ezpypi/core/main.py works'

Add __init__.py for package core.
Then add from .main import main to __init__.py

CLI
1
echo 'from .main import main' > src/ezpypi/core/__init__.py

Update __init__.py

Then we need to update __init__.py in directory src/ezpypi/__init__.py

Python
1
from .core import *

Test

Update the version in setup.cfg from version = 0.2.0.dev0 to version = 0.2.0.dev1 since we make a big change.

Uninstall the package then install it locally.

Go to directory venv and test the package.

CLI
1
2
3
cd venv

python3
Python
1
2
3
import ezpypi

help(ezpypi)

Notice our distribution ezpypi now has a package core.

Python
1
ezpypi.core.main()

Change directory to your project directory

CLI
1
cd ..

TestPyPI

Make sure your are under maintenance/0.0.x

Git

Update the version in setup.cfg from version = 0.2.0.dev1 to version = 0.2.0 since the package is stable now.

CLI
1
git status

CLI
1
2
3
git add .
git commit -m 'Update distribution to 0.2.0. ver=0.2.0'
git push

Build distribution

Now run this command from the same directory where pyproject.toml is located:

CLI
1
python3 -m build

List the files inside directory dist.

CLI
1
ls -l dist


Upload distribution to TestPyPI

Run Twine to upload all of the archives end with 0.2.0 under dist:

CLI
1
python3 -m twine upload --repository testpypi dist/*0.2.0*


PyPI

Git

Merge your git branch.

CLI
1
2
3
4
5
git status

git checkout main

git merge maintenance/0.0.x

If you meets git merge CONFLICT, check here

We can fix it manually.

Since we have fixed the merge CONFLICT, we add the file and commit it.

CLI
1
2
3
4
5
git add setup.cfg

git commit -m 'Fix git merge setup.cfg. ver=0.2.0'

git status

Merge again.

CLI
1
git merge maintenance/0.0.x

CLI
1
git push


Upload distribution to TestPyPI

Run Twine to upload all of the archives end with 0.2.0 under dist:

CLI
1
python3 -m twine upload --repository pypi dist/*0.2.0*


Validation

Uninstall the local package.

Install the latest version from PyPI.

CLI
1
pip install ezpypi --upgrade

Since we already move ezpypi to src, we don’t have to worry about importing package from a local directory rather than Python site packages.

CLI
1
python3
Python
1
2
3
import ezpypi

ezpypi.main()


Make a new branch

Since our current version is 0.2.x, we should make a new branch maintenance/0.2.x for this branch.

CLI
1
2
3
git checkout -b maintenance/0.2.x

git push -u origin maintenance/0.2.x