blob: febdf16d453cdd792879b78751a3fed230d5241f [file] [log] [blame] [view]
Joshua Habermaneaa8f242023-03-03 08:57:13 -08001# Protocol Buffers Python
jessecd04e9b2015-03-16 15:15:59 -07002
Joshua Habermaneaa8f242023-03-03 08:57:13 -08003This directory contains the Protobuf library for Python.
temporal40ee5512008-07-10 02:12:20 +00004
Joshua Haberman8135fca2024-01-31 14:37:42 -08005For user documentation about how to use Protobuf Python, see
6https://protobuf.dev/getting-started/pythontutorial/
7
8# Installation
9
Joshua Habermaneaa8f242023-03-03 08:57:13 -080010In most cases you should install the library using `pip` or another package
11manager:
temporal40ee5512008-07-10 02:12:20 +000012
Joshua Habermaneaa8f242023-03-03 08:57:13 -080013```
14$ pip install protobuf
15```
temporalcc930432008-07-21 20:28:30 +000016
Joshua Habermaneaa8f242023-03-03 08:57:13 -080017The packages released on https://pypi.org/project/protobuf/#files include both a
18source distribution and binary wheels.
temporalcc930432008-07-21 20:28:30 +000019
Joshua Haberman8135fca2024-01-31 14:37:42 -080020## Building packages from this repo
temporal742e4092008-08-27 19:25:48 +000021
Joshua Habermaneaa8f242023-03-03 08:57:13 -080022If for some reason you wish to build the packages directly from this repo, you
23can use the following Bazel commands:
temporal742e4092008-08-27 19:25:48 +000024
Joshua Habermaneaa8f242023-03-03 08:57:13 -080025```
Adam Cozzette501ecec2023-09-26 14:36:20 -070026$ bazel build //python/dist:source_wheel
27$ bazel build //python/dist:binary_wheel
Joshua Habermaneaa8f242023-03-03 08:57:13 -080028```
temporal40ee5512008-07-10 02:12:20 +000029
Joshua Habermaneaa8f242023-03-03 08:57:13 -080030The binary wheel will build against whatever version of Python is installed on
31your system. The source package is always the same and does not depend on a
32local version of Python.
temporal40ee5512008-07-10 02:12:20 +000033
Joshua Haberman8135fca2024-01-31 14:37:42 -080034## Building from `setup.py`
35
36We support building from `setup.py`, but only from a Python source package.
37You cannot build from `setup.py` using the GitHub repo or the GitHub source
38tarball.
39
40To build a source package from this repo, see the instructions in the previous
41section.
42
Joshua Habermaneaa8f242023-03-03 08:57:13 -080043# Implementation backends
temporal40ee5512008-07-10 02:12:20 +000044
Joshua Habermaneaa8f242023-03-03 08:57:13 -080045There are three separate implementations of Python Protobuf. All of them offer
46the same API and are thus functionally the same, though they have very different
47performance characteristics.
Mike Kruskaled5c57a2022-08-10 22:51:29 -070048
Joshua Habermaneaa8f242023-03-03 08:57:13 -080049The runtime library contains a switching layer that can choose between these
50backends at runtime. Normally it will choose between them automatically, using
51priority-ordered list, and skipping any backends that are not available. However
52you can request a specific backend by setting the
53`PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` environment variable to one of the
54following values:
temporal40ee5512008-07-10 02:12:20 +000055
Joshua Habermaneaa8f242023-03-03 08:57:13 -0800561. **upb**: Built on the
Adam Cozzette501ecec2023-09-26 14:36:20 -070057 [upb C library](https://github.com/protocolbuffers/protobuf/tree/main/upb),
58 this is a new extension module
Joshua Habermaneaa8f242023-03-03 08:57:13 -080059 [released in 4.21.0](https://protobuf.dev/news/2022-05-06/). It offers
60 better performance than any of the previous backends, and it is now the
61 default. It is distributed in our PyPI packages, and requires no special
Adam Cozzette501ecec2023-09-26 14:36:20 -070062 installation. The code for this module lives in this directory.
Joshua Habermaneaa8f242023-03-03 08:57:13 -0800631. **cpp**: This extension module wraps the C++ protobuf library. It is
64 deprecated and is no longer released in our PyPI packages, however it is
65 still used in some legacy cases where apps want to perform zero-copy message
66 sharing between Python and C++. It must be installed separately before it
67 can be used. The code for this module lives in
68 [google/protobuf/pyext](https://github.com/protocolbuffers/protobuf/tree/main/python/google/protobuf/pyext).
691. **python**: The pure-Python backend, this does not require any extension
70 module to be present on the system. The code for the pure-Python backend
71 lives in [google/protobuf/internal](google/protobuf/internal)
temporal40ee5512008-07-10 02:12:20 +000072
Joshua Habermaneaa8f242023-03-03 08:57:13 -080073The order given above is the general priority order, with `upb` being preferred
74the most and the `python` backend preferred the least. However this ordering can
75be overridden by the presence of a
76`google.protobuf.internal._api_implementation` module. See the logic in
77[api_implementation.py](https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/internal/api_implementation.py)
78for details.
temporal40ee5512008-07-10 02:12:20 +000079
Joshua Habermaneaa8f242023-03-03 08:57:13 -080080You can check which backend you are using with the following snippet:
temporal40ee5512008-07-10 02:12:20 +000081
Joshua Habermaneaa8f242023-03-03 08:57:13 -080082```
83$ python
84Python 3.10.9 (main, Dec 7 2022, 13:47:07) [GCC 12.2.0] on linux
85Type "help", "copyright", "credits" or "license" for more information.
86>>> from google.protobuf.internal import api_implementation
87>>> print(api_implementation.Type())
88upb
89```
jieluo@google.com1eba9d92014-08-25 20:17:53 +000090
Joshua Habermaneaa8f242023-03-03 08:57:13 -080091This is not an officially-supported or stable API, but it is useful for ad hoc
92diagnostics.
Tamir Dubersteind632bc72015-04-10 15:26:58 -040093
Joshua Habermaneaa8f242023-03-03 08:57:13 -080094More information about sharing messages between Python and C++ is available
95here: https://protobuf.dev/reference/python/python-generated/#sharing-messages
Tamir Dubersteind632bc72015-04-10 15:26:58 -040096
Joshua Habermaneaa8f242023-03-03 08:57:13 -080097# Code generator
Tamir Dubersteind632bc72015-04-10 15:26:58 -040098
Joshua Habermaneaa8f242023-03-03 08:57:13 -080099The code for the Protobuf Python code generator lives in
100[//src/google/protobuf/compiler/python](https://github.com/protocolbuffers/protobuf/tree/main/src/google/protobuf/compiler/python).
101The code generator can output two different files for each proto `foo.proto`:
Tamir Dubersteind632bc72015-04-10 15:26:58 -0400102
Joshua Habermaneaa8f242023-03-03 08:57:13 -0800103* **foo_pb2.py**: The module you import to actually use the protos.
104* **foo_pb2.pyi**: A stub file that describes the interface of the protos.
Tamir Dubersteind632bc72015-04-10 15:26:58 -0400105
Joshua Habermaneaa8f242023-03-03 08:57:13 -0800106The `foo_pb2.pyi` file is useful for IDEs or for users who want to read the
107output file. The `foo_pb2.py` file is optimized for fast loading and is not
108readable at all.
Tamir Dubersteind632bc72015-04-10 15:26:58 -0400109
Joshua Habermaneaa8f242023-03-03 08:57:13 -0800110Note that the pyi file is only generated if you pass the `pyi_out` option to
111`protoc`:
Tamir Dubersteind632bc72015-04-10 15:26:58 -0400112
Joshua Habermaneaa8f242023-03-03 08:57:13 -0800113```
114$ protoc --python_out=pyi_out:output_dir
115```