Initial commit

.hgignore

@@ -0,0 +1,3 @@
+build/
+dist/
+venv/

LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2018 to present, Gregory Szorc
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors
+may be used to endorse or promote products derived from this software without
+specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.rst

@@ -0,0 +1,123 @@
+========================
+Python Standalone Builds
+========================
+
+This project contains code for building Python distributions that are
+self-contained and highly portable (the binaries can be executed
+on most target machines).
+
+The intended audience of this project are people wanting to produce
+applications that embed Python in a larger executable. The artifacts
+that this project produces make it easier to build highly-portable
+applications containing Python.
+
+Most consumers of this project can bypass the building of artifacts
+and consume the pre-built binaries produced from it.
+
+Project Status
+==============
+
+The project can be considered alpha quality. It is still in a heavy state
+of flux.
+
+Currently, it produces a nearly full-featured CPython distribution for
+Linux that is fully statically linked with the exception of some very
+common system libraries.
+
+Planned features include:
+
+* Support for Windows
+* Support for macOS
+* Static/dynamic linking toggles for dependencies
+
+Instructions
+============
+
+To build a Python distribution for Linux x64::
+
+    $ ./build-linux.py
+
+Requirements
+============
+
+Linux
+-----
+
+The host system must be 64-bit. A Python 3.5+ interpreter must be
+available. The execution environment must have access to a Docker
+daemon (all build operations are performed in Docker containers for
+isolation from the host system).
+
+How It Works
+============
+
+The first thing the ``build-*`` scripts do is bootstrap an environment
+for building Python. On Linux, a base Docker image based on a deterministic
+snapshot of Debian Wheezy is created. A modern binutils and GCC are built
+in this environment. That modern GCC is then used to build a modern Clang.
+Clang is then used to build all of Python's dependencies (openssl, ncurses,
+readline, sqlite, etc). Finally, Python itself is built.
+
+Python is built in such a way that extensions are statically linked
+against their dependencies. e.g. instead of the ``sqlite3`` Python
+extension having a run-time dependency against ``libsqlite3.so``, the
+SQLite symbols are statically inlined into the Python extension object
+file.
+
+From the built Python, we produce an archive containing the raw Python
+distribution (as if you had run ``make install``) as well as other files
+useful for downstream consumers.
+
+Setup.local Hackery
+-------------------
+
+Python's build system reads the ``Modules/Setup`` and ``Modules/Setup.local``
+files to influence how C extensions are built. By default, many extensions
+have no entry in these files and the ``setup.py`` script performs work
+to compile these extensions. (``setup.py`` looks for headers, libraries,
+etc, and sets up the proper compiler flags.)
+
+``setup.py`` doesn't provide a lot of flexibility and relies on a lot
+of default behavior in ``distutils`` as well as other inline code in
+``setup.py``. This default behavior is often undesirable for our
+desired outcome of producing a standalone Python distribution.
+
+Since the build environment is mostly deterministic and since we have
+special requirements, we generate a custom ``Setup.local`` file that
+builds C extensions in a specific manner. The undesirable behavior of
+``setup.py`` is bypassed and the Python C extensions are compiled just
+the way we want.
+
+Linux Runtime Requirements
+==========================
+
+The produced Linux binaries have minimal references to shared
+libraries and thus can be executed on most Linux systems.
+
+The following shared libraries are referenced:
+
+* linux-vdso.so.1
+* libpthread.so.0
+* libdl.so.2 (required by ctypes extension)
+* libutil.so.1
+* librt.so.1
+* libnsl.so.1 (required by nis extension)
+* libcrypt.so.1 (required by crypt extension)
+* libm.so.6
+* libc.so.6
+* ld-linux-x86-64.so.2
+
+Licensing
+=========
+
+Python and its various dependencies are governed by varied software use
+licenses. This impacts the rights and requirements of downstream consumers.
+
+The ``python-licenses.rst`` file contained in this repository and produced
+artifacts summarizes the licenses of various components.
+
+Most licenses are fairly permissive. Notable exceptions to this are GDBM and
+readline, which are both licensed under GPL Version 3.
+
+**It is important to understand the licensing requirements when integrating
+the output of this project into derived works.**

build-linux.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+import datetime
+import os
+import pathlib
+import subprocess
+import sys
+import venv
+
+
+ROOT = pathlib.Path(os.path.abspath(__file__)).parent
+BUILD = ROOT / 'build'
+DIST = ROOT / 'dist'
+VENV = BUILD / 'venv'
+PIP = VENV / 'bin' / 'pip'
+PYTHON = VENV / 'bin' / 'python'
+REQUIREMENTS = ROOT / 'requirements.txt'
+MAKE_DIR = ROOT / 'cpython-linux'
+
+
+def bootstrap():
+    BUILD.mkdir(exist_ok=True)
+    DIST.mkdir(exist_ok=True)
+
+    venv.create(VENV, with_pip=True)
+
+    subprocess.run([str(PIP), 'install', '-r', str(REQUIREMENTS)],
+                   check=True)
+
+    os.environ['PYBUILD_BOOTSTRAPPED'] = '1'
+    os.environ['PATH'] = '%s:%s' % (str(VENV / 'bin'), os.environ['PATH'])
+    os.environ['PYTHONPATH'] = str(ROOT)
+    subprocess.run([str(PYTHON), __file__], check=True)
+
+
+def run():
+    import zstandard
+    from pythonbuild.downloads import DOWNLOADS
+
+    now = datetime.datetime.utcnow()
+
+    subprocess.run(['make'],
+                   cwd=str(MAKE_DIR), check=True)
+
+    source_path = BUILD / 'cpython-linux64.tar'
+    dest_path = DIST / ('cpython-%s-linux64-%s.tar.zst' % (
+        DOWNLOADS['cpython-3.7']['version'], now.strftime('%Y%m%dT%H%M')))
+
+    print('compressing Python archive to %s' % dest_path)
+    with source_path.open('rb') as ifh, dest_path.open('wb') as ofh:
+        cctx = zstandard.ZstdCompressor(level=15)
+        cctx.copy_stream(ifh, ofh, source_path.stat().st_size)
+
+
+if __name__ == '__main__':
+    try:
+        if 'PYBUILD_BOOTSTRAPPED' not in os.environ:
+            bootstrap()
+        else:
+            run()
+    except subprocess.CalledProcessError as e:
+        sys.exit(e.returncode)

cpython-linux/Makefile

@@ -0,0 +1,82 @@
+ROOT := $(abspath $(CURDIR)/..)
+HERE := $(ROOT)/cpython-linux
+OUTDIR := $(ROOT)/build
+
+BUILD := $(HERE)/build.py
+NULL :=
+
+COMMON_DEPENDS := \
+#    $(BUILD) \
+    $(NULL)
+
+PLATFORM := linux64
+
+TOOLCHAIN_DEPENDS := \
+    $(OUTDIR)/binutils-linux64.tar \
+    $(OUTDIR)/gcc-linux64.tar \
+    $(OUTDIR)/clang-linux64.tar \
+    $(NULL)
+
+default: $(OUTDIR)/cpython-linux64.tar
+
+$(OUTDIR)/image-%.tar: $(HERE)/%.Dockerfile $(COMMON_DEPENDS)
+	$(BUILD) image-$*
+
+$(OUTDIR)/binutils-linux64.tar: $(OUTDIR)/image-gcc.tar $(HERE)/build-binutils.sh
+	$(BUILD) binutils
+
+$(OUTDIR)/gcc-linux64.tar: $(OUTDIR)/binutils-linux64.tar $(HERE)/build-gcc.sh
+	$(BUILD) gcc
+
+$(OUTDIR)/clang-linux64.tar: $(OUTDIR)/binutils-linux64.tar $(OUTDIR)/gcc-linux64.tar $(OUTDIR)/image-clang.tar $(HERE)/build-clang.sh
+	$(BUILD) clang
+
+$(OUTDIR)/bzip2-%.tar: $(OUTDIR)/image-build.tar $(TOOLCHAIN_DEPENDS) $(HERE)/build-bzip2.sh
+	$(BUILD) --platform $* bzip2
+
+$(OUTDIR)/gdbm-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-gdbm.sh
+	$(BUILD) --platform $* gdbm
+
+$(OUTDIR)/libffi-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-libffi.sh
+	$(BUILD) --platform $* libffi
+
+$(OUTDIR)/ncurses-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-ncurses.sh
+	$(BUILD) --platform $* ncurses
+
+$(OUTDIR)/openssl-%.tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-openssl.sh
+	$(BUILD) --platform $* openssl
+
+$(OUTDIR)/readline-%.tar: $(TOOLCHAIN_DEPENDS) $(OUTDIR)/ncurses-$(PLATFORM).tar $(HERE)/build-readline.sh
+	$(BUILD) --platform $* readline
+
+$(OUTDIR)/sqlite-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-sqlite.sh
+	$(BUILD) --platform $(PLATFORM) sqlite
+
+$(OUTDIR)/tcltk-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-tcltk.sh
+	$(BUILD) --platform $(PLATFORM) tcltk
+
+$(OUTDIR)/uuid-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-uuid.sh
+	$(BUILD) --platform $(PLATFORM) uuid
+
+$(OUTDIR)/xz-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-xz.sh
+	$(BUILD) --platform $(PLATFORM) xz
+
+$(OUTDIR)/zlib-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-zlib.sh
+	$(BUILD) --platform $(PLATFORM) zlib
+
+PYTHON_DEPENDS := \
+    $(OUTDIR)/bzip2-$(PLATFORM).tar \
+    $(OUTDIR)/gdbm-$(PLATFORM).tar \
+    $(OUTDIR)/libffi-$(PLATFORM).tar \
+    $(OUTDIR)/ncurses-$(PLATFORM).tar \
+    $(OUTDIR)/openssl-$(PLATFORM).tar \
+    $(OUTDIR)/readline-$(PLATFORM).tar \
+    $(OUTDIR)/sqlite-$(PLATFORM).tar \
+    $(OUTDIR)/uuid-$(PLATFORM).tar \
+    $(OUTDIR)/xz-$(PLATFORM).tar \
+    $(OUTDIR)/zlib-$(PLATFORM).tar \
+    $(HERE)/static-modules \
+    $(NULL)
+
+$(OUTDIR)/cpython-$(PLATFORM).tar: $(TOOLCHAIN_DEPENDS) $(HERE)/build-cpython.sh $(PYTHON_DEPENDS)
+	$(BUILD) --platform $(PLATFORM) cpython

cpython-linux/README.rst

@@ -0,0 +1,19 @@
+This project builds CPython for Linux in a mostly deterministic and
+reproducible manner. The resulting Python build is mostly self-contained
+and the binaries are capable of running on many Linux distributions.
+
+The produced binaries perform minimal loading of shared libraries.
+The required shared libraries are:
+
+* linux-vdso.so.1
+* libpthread.so.0
+* libdl.so.2 (required by ctypes extension)
+* libutil.so.1
+* librt.so.1
+* libnsl.so.1 (required by nis extension)
+* libcrypt.so.1 (required by crypt extension)
+* libm.so.6
+* libc.so.6
+* ld-linux-x86-64.so.2
+
+These shared libraries should be present on most modern Linux distros.

cpython-linux/base.Dockerfile

@@ -0,0 +1,30 @@
+# Debian Wheezy.
+FROM debian@sha256:37103c15605251b2e35b70a3214af626a55cff39abbaadccd01ff828ee7005e0
+MAINTAINER Gregory Szorc <[email protected]>
+
+RUN groupadd -g 1000 build && \
+    useradd -u 1000 -g 1000 -d /build -s /bin/bash -m build && \
+    mkdir /tools && \
+    chown -R build:build /build /tools
+
+ENV HOME=/build \
+    SHELL=/bin/bash \
+    USER=build \
+    LOGNAME=build \
+    HOSTNAME=builder \
+    DEBIAN_FRONTEND=noninteractive
+
+CMD ["/bin/bash", "--login"]
+WORKDIR '/build'
+
+RUN for s in debian_wheezy debian_wheezy-updates debian_wheezy-backports debian-security_wheezy/updates; do \
+      echo "deb http://snapshot.debian.org/archive/${s%_*}/20181129T234109Z/ ${s#*_} main"; \
+    done > /etc/apt/sources.list && \
+    ( echo 'quiet "true";'; \
+      echo 'APT::Get::Assume-Yes "true";'; \
+      echo 'APT::Install-Recommends "false";'; \
+      echo 'Acquire::Check-Valid-Until "false";'; \
+      echo 'Acquire::Retries "5";'; \
+    ) > /etc/apt/apt.conf.d/99cpython-portable
+
+RUN apt-get update

cpython-linux/build-binutils.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+set -e
+
+cd /build
+
+tar -xf binutils-${BINUTILS_VERSION}.tar.xz
+mkdir binutils-objdir
+pushd binutils-objdir
+
+../binutils-${BINUTILS_VERSION}/configure \
+    --build=x86_64-unknown-linux-gnu \
+    --prefix=/tools/host \
+    --enable-plugins \
+    --disable-nls \
+    --with-sysroot=/
+
+make -j `nproc`
+make install -j `nproc` DESTDIR=/build/out
+
+popd