Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 1 | # Protocol Buffers Python |
jesse | cd04e9b | 2015-03-16 15:15:59 -0700 | [diff] [blame] | 2 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 3 | This directory contains the Protobuf library for Python. |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 4 | |
Joshua Haberman | 8135fca | 2024-01-31 14:37:42 -0800 | [diff] [blame] | 5 | For user documentation about how to use Protobuf Python, see |
| 6 | https://protobuf.dev/getting-started/pythontutorial/ |
| 7 | |
| 8 | # Installation |
| 9 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 10 | In most cases you should install the library using `pip` or another package |
| 11 | manager: |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 12 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 13 | ``` |
| 14 | $ pip install protobuf |
| 15 | ``` |
temporal | cc93043 | 2008-07-21 20:28:30 +0000 | [diff] [blame] | 16 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 17 | The packages released on https://pypi.org/project/protobuf/#files include both a |
| 18 | source distribution and binary wheels. |
temporal | cc93043 | 2008-07-21 20:28:30 +0000 | [diff] [blame] | 19 | |
Joshua Haberman | 8135fca | 2024-01-31 14:37:42 -0800 | [diff] [blame] | 20 | ## Building packages from this repo |
temporal | 742e409 | 2008-08-27 19:25:48 +0000 | [diff] [blame] | 21 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 22 | If for some reason you wish to build the packages directly from this repo, you |
| 23 | can use the following Bazel commands: |
temporal | 742e409 | 2008-08-27 19:25:48 +0000 | [diff] [blame] | 24 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 25 | ``` |
Adam Cozzette | 501ecec | 2023-09-26 14:36:20 -0700 | [diff] [blame] | 26 | $ bazel build //python/dist:source_wheel |
| 27 | $ bazel build //python/dist:binary_wheel |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 28 | ``` |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 29 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 30 | The binary wheel will build against whatever version of Python is installed on |
| 31 | your system. The source package is always the same and does not depend on a |
| 32 | local version of Python. |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 33 | |
Joshua Haberman | 8135fca | 2024-01-31 14:37:42 -0800 | [diff] [blame] | 34 | ## Building from `setup.py` |
| 35 | |
| 36 | We support building from `setup.py`, but only from a Python source package. |
| 37 | You cannot build from `setup.py` using the GitHub repo or the GitHub source |
| 38 | tarball. |
| 39 | |
| 40 | To build a source package from this repo, see the instructions in the previous |
| 41 | section. |
| 42 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 43 | # Implementation backends |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 44 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 45 | There are three separate implementations of Python Protobuf. All of them offer |
| 46 | the same API and are thus functionally the same, though they have very different |
| 47 | performance characteristics. |
Mike Kruskal | ed5c57a | 2022-08-10 22:51:29 -0700 | [diff] [blame] | 48 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 49 | The runtime library contains a switching layer that can choose between these |
| 50 | backends at runtime. Normally it will choose between them automatically, using |
| 51 | priority-ordered list, and skipping any backends that are not available. However |
| 52 | you can request a specific backend by setting the |
| 53 | `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` environment variable to one of the |
| 54 | following values: |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 55 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 56 | 1. **upb**: Built on the |
Adam Cozzette | 501ecec | 2023-09-26 14:36:20 -0700 | [diff] [blame] | 57 | [upb C library](https://github.com/protocolbuffers/protobuf/tree/main/upb), |
| 58 | this is a new extension module |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 59 | [released in 4.21.0](https://protobuf.dev/news/2022-05-06/). It offers |
| 60 | better performance than any of the previous backends, and it is now the |
| 61 | default. It is distributed in our PyPI packages, and requires no special |
Adam Cozzette | 501ecec | 2023-09-26 14:36:20 -0700 | [diff] [blame] | 62 | installation. The code for this module lives in this directory. |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 63 | 1. **cpp**: This extension module wraps the C++ protobuf library. It is |
| 64 | deprecated and is no longer released in our PyPI packages, however it is |
| 65 | still used in some legacy cases where apps want to perform zero-copy message |
| 66 | sharing between Python and C++. It must be installed separately before it |
| 67 | can be used. The code for this module lives in |
| 68 | [google/protobuf/pyext](https://github.com/protocolbuffers/protobuf/tree/main/python/google/protobuf/pyext). |
| 69 | 1. **python**: The pure-Python backend, this does not require any extension |
| 70 | module to be present on the system. The code for the pure-Python backend |
| 71 | lives in [google/protobuf/internal](google/protobuf/internal) |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 72 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 73 | The order given above is the general priority order, with `upb` being preferred |
| 74 | the most and the `python` backend preferred the least. However this ordering can |
| 75 | be overridden by the presence of a |
| 76 | `google.protobuf.internal._api_implementation` module. See the logic in |
| 77 | [api_implementation.py](https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/internal/api_implementation.py) |
| 78 | for details. |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 79 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 80 | You can check which backend you are using with the following snippet: |
temporal | 40ee551 | 2008-07-10 02:12:20 +0000 | [diff] [blame] | 81 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 82 | ``` |
| 83 | $ python |
| 84 | Python 3.10.9 (main, Dec 7 2022, 13:47:07) [GCC 12.2.0] on linux |
| 85 | Type "help", "copyright", "credits" or "license" for more information. |
| 86 | >>> from google.protobuf.internal import api_implementation |
| 87 | >>> print(api_implementation.Type()) |
| 88 | upb |
| 89 | ``` |
jieluo@google.com | 1eba9d9 | 2014-08-25 20:17:53 +0000 | [diff] [blame] | 90 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 91 | This is not an officially-supported or stable API, but it is useful for ad hoc |
| 92 | diagnostics. |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 93 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 94 | More information about sharing messages between Python and C++ is available |
| 95 | here: https://protobuf.dev/reference/python/python-generated/#sharing-messages |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 96 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 97 | # Code generator |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 98 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 99 | The code for the Protobuf Python code generator lives in |
| 100 | [//src/google/protobuf/compiler/python](https://github.com/protocolbuffers/protobuf/tree/main/src/google/protobuf/compiler/python). |
| 101 | The code generator can output two different files for each proto `foo.proto`: |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 102 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 103 | * **foo_pb2.py**: The module you import to actually use the protos. |
| 104 | * **foo_pb2.pyi**: A stub file that describes the interface of the protos. |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 105 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 106 | The `foo_pb2.pyi` file is useful for IDEs or for users who want to read the |
| 107 | output file. The `foo_pb2.py` file is optimized for fast loading and is not |
| 108 | readable at all. |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 109 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 110 | Note that the pyi file is only generated if you pass the `pyi_out` option to |
| 111 | `protoc`: |
Tamir Duberstein | d632bc7 | 2015-04-10 15:26:58 -0400 | [diff] [blame] | 112 | |
Joshua Haberman | eaa8f24 | 2023-03-03 08:57:13 -0800 | [diff] [blame] | 113 | ``` |
| 114 | $ protoc --python_out=pyi_out:output_dir |
| 115 | ``` |