Application note: Field presence

This application note explains the various presence tracking disciplines for protobuf fields. It also explains how to enable experimental support for explicit presence tracking for singular proto3 fields with basic types.

Background

Field presence is the notion of whether a protobuf field has a value. There are two different manifestations of presence for protobufs: no presence, where the generated message API stores field values (only), and explicit presence, where the API also stores whether or not a field has been set.

Historically, proto2 has mostly followed explicit presence, while proto3 exposes only no presence semantics. Singular proto3 fields of basic types (numeric, string, bytes, and enums) which are defined with the optional label have explicit presence, like proto2 (this is an experimental feature added as of release 3.12, and must be enabled by passing a flag to protoc).

Presence disciplines

Presence disciplines define the semantics for translating between the API representation and the serialized representation. The no presence discipline relies upon the field value itself to make decisions at (de)serialization time, while the explicit presence discipline relies upon the explicit tracking state instead.

Presence in tag-value stream (wire format) serialization

The wire format is a stream of tagged, self-delimiting values. By definition, the wire format represents a sequence of present values. In other words, every value found within a serialization represents a present field; furthermore, the serialization contains no information about not-present values.

The generated API for a proto message includes (de)serialization definitions which translate between API types and a stream of definitionally present (tag, value) pairs. This translation is designed to be forward- and backward-compatibile across changes to the message definition; however, this compatibility introduces some (perhaps surprising) considerations when deserializing wire-formatted messages:

When serializing, fields with no presence are not serialized if they contain their default value.
- For numeric types, the default is 0.
- For enums, the default is the zero-valued enumerator.
- For strings, bytes, and repeated fields, the default is the zero-length value.
- For messages, the default is the language-specific null value.
“Empty” length-delimited values (such as empty strings) can be validly represented in serialized values: the field is “present,” in the sense that it appears in the wire format. However, if the generated API does not track presence, then these values may not be re-serialized; i.e., the empty field may be “not present” after a serialization round-trip.
When deserializing, duplicate field values may be handled in different ways depending on the field definition.
- Duplicate repeated fields are typically appended to the field's API representation. (Note that serializing a packed repeated field produces only one, length-delimited value in the tag stream.)
- Duplicate optional field values follow the rule that “the last one wins.”
oneof fields expose the API-level invariant that only one field is set at a time. However, the wire format may include multiple (tag, value) pairs which notionally belong to the oneof. Similar to optional fields, the generated API follows the “last one wins” rule.
Out-of-range values are not returned for enum fields in generated proto2 APIs. However, out-of-range values may be stored as unknown fields in the API, even though the wire-format tag was recognized.

Presence in named-field mapping formats

Protobufs can be represented in human-readable, textual forms. Two notable formats are TextFormat (the output format produced by generated message DebugString methods) and JSON.

These formats have correctness requirements of their own, and are generally stricter than tagged-value stream formats. However, TextFormat more closely mimics the semantics of the wire format, and does, in certain cases, provide similar semantics (for example, appending repeated name-value mappings to a repeated field). In particular, similar to the wire format, TextFormat only includes fields which are present.

JSON is a much stricter format, however, and cannot validly represent some semantics of the wire format or TextFormat.

Notably, JSON elements are semantically unordered, and each member must have a unique name. This is different from TextFormat rules for repeated fields.
JSON may include fields that are “not present,” unlike the no presence discipline for other formats:
- JSON defines a null value, which may be used to represent a defined but not-present field.
- Repeated field values may be included in the formatted output, even if they are equal to the default (an empty list).
Because JSON elements are unordered, there is no way to unambiguously interpret the “last one wins” rule.
- In most cases, this is fine: JSON elements must have unique names: repeated field values are not valid JSON, so they do not need to be resolved as they are for TextFormat.
- However, this means that it may not be possible to interpret oneof fields unambiguously: if multiple cases are present, they are unordered.

In theory, JSON can represent presence in a semantic-preserving fashion. In practice, however, presence correctness can vary depending upon implementation choices, especially if JSON was chosen as a means to interoperate with clients not using protobufs.

Presence in proto2 APIs

This table outlines whether presence is tracked for fields in proto2 APIs (both for generated APIs and using dynamic reflection):

Field type	Explicit Presence
Singular numeric (integer or floating point)	✔️
Singular enum	✔️
Singular string or bytes	✔️
Singular message	✔️
Repeated
Oneofs	✔️
Maps

Singular fields (of all types) track presence explicitly in the generated API. The generated message interface includes methods to query presence of fields. For example, the field foo has a corresponding has_foo method. (The specific name follows the same language-specific naming convention as the field accessors.) These methods are sometimes referred to as “hazzers” within the protobuf implementation.

Similar to singular fields, oneof fields explicitly track which one of the members, if any, contains a value. For example, consider this example oneof:

oneof foo {
  int32 a = 1;
  float b = 2;
}

Depending on the target language, the generated API would generally include several methods:

A hazzer for the oneof: has_foo
A oneof case method: foo
Hazzers for the members: has_a, has_b
Getters for the members: a, b

Repeated fields and maps do not track presence: there is no distinction between an empty and a not-present repeated field.

Presence in proto3 APIs

This table outlines whether presence is tracked for fields in proto3 APIs (both for generated APIs and using dynamic reflection):

Field type	`optional`	Explicit Presence
Singular numeric (integer or floating point)	No
Singular enum	No
Singular string or bytes	No
Singular numeric (integer or floating point)	Yes	✔️
Singular enum	Yes	✔️
Singular string or bytes	Yes	✔️
Singular message	Yes	✔️
Singular message	No	✔️
Repeated	N/A
Oneofs	N/A	✔️
Maps	N/A

Similar to proto2 APIs, proto3 does not track presence explicitly for repeated fields. Without the optional label, proto3 APIs do not track presence for basic types (numeric, string, bytes, and enums), either. (Note that optional for proto3 fields is only experimentally available as of release 3.12.) Oneof fields affirmatively expose presence, although the same set of hazzer methods may not generated as in proto2 APIs.

Under the no presence discipline, the default value is synonymous with “not present” for purposes of serialization. To notionally “clear” a field (so it won't be serialized), an API user would set it to the default value.

The default value for enum-typed fields under no presence is the corresponding 0-valued enumerator. Under proto3 syntax rules, all enum types are required to have an enumerator value which maps to 0. By convention, this is an UNKNOWN or similarly-named enumerator. If the zero value is notionally outside the domain of valid values for the application, this behavior can be thought of as tantamount to explicit presence.

Semantic differences

The no presence serialization discipline results in visible differences from the explicit presence tracking discipline, when the default value is set. For a singular field with numeric, enum, or string type:

No presence discipline:
- Default values are not serialized.
- Default values are not merged-from.
- To “clear” a field, it is set to its default value.
- The default value may mean:
  - the field was explicitly set to its default value, which is valid in the application-specific domain of values;
  - the field was notionally “cleared” by setting its default; or
  - the field was never set.
Explicit presence discipline:
- Explicitly set values are always serialized, including default values.
- Un-set fields are never merged-from.
- Explicitly set fields -- including default values -- are merged-from.
- A generated has_foo method indicates whether or not the field foo has been set (and not cleared).
- A generated clear_foo method must be used to clear (i.e., un-set) the value.

Considerations for merging

Under the no presence rules, it is effectively impossible for a target field to merge-from its default value (using the protobuf's API merging functions). This is because default values are skipped, similar to the no presence serialization discipline. Merging only updates the target (merged-to) message using the non-skipped values from the update (merged-from) message.

The difference in merging behavior has further implications for protocols which rely on partial “patch” updates. If field presence is not tracked, then an update patch alone cannot represent an update to the default value, because only non-default values are merged-from.

Updating to set a default value in this case requires some external mechanism, such as FieldMask. However, if presence is tracked, then all explicitly-set values -- even default values -- will be merged into the target.

Considerations for change-compatibility

Changing a field between explicit presence and no presence is a binary-compatible change for serialized values in wire format. However, the serialized representation of the message may differ, depending on which version of the message definition was used for serialization. Specifically, when a “sender” explicitly sets a field to its default value:

The serialized value following no presence discipline does not contain the default value, even though it was explicitly set.
The serialized value following explicit presence discipline contains every “present” field, even if it contains the default value.

This change may or may not be safe, depending on the application's semantics. For example, consider two clients with different versions of a message definition.

Client A uses this definition of the message, which follows the explicit presence serialization discipline for field foo:

syntax = "proto3";
message Msg {
  optional int32 foo = 1;
}

Client B uses a definition of the same message, except that it follows the no presence discipline:

syntax = "proto3";
message Msg {
  int32 foo = 1;
}

Now, consider a scenario where client A observes foo's presence as the clients repeatedly exchange the “same” message by deserializing and reserializing:

// Client A:
Msg m_a;
m_a.set_foo(1);                  // non-default value
assert(m_a.has_foo());           // OK
Send(m_a.SerializeAsString());   // to client B

// Client B:
Msg m_b;
m_b.ParseFromString(Receive());  // from client A
assert(m_b.foo() == 1);          // OK
Send(m_b.SerializeAsString());   // to client A

// Client A:
m_a.ParseFromString(Receive());  // from client B
assert(m_a.foo() == 1);          // OK
assert(m_a.has_foo());           // OK
m_a.set_foo(0);                  // default value
Send(m_a.SerializeAsString());   // to client B

// Client B:
Msg m_b;
m_b.ParseFromString(Receive());  // from client A
assert(m_b.foo() == 0);          // OK
Send(m_b.SerializeAsString());   // to client A

// Client A:
m_a.ParseFromString(Receive());  // from client B
assert(m_a.foo() == 0);          // OK
assert(m_a.has_foo());           // FAIL

If client A depends on explicit presence for foo, then a “round trip” through client B will be lossy from the perspective of client A. In the example, this is not a safe change: client A requires (by assert) that the field is present; even without any modifications through the API, that requirement fails in a value- and peer-dependent case.

How to enable explicit presence in proto3

These are the general steps to use the experimental field tracking support for proto3:

Add an optional field to a .proto file.
Run protoc (from release 3.12 or later) with an extra flag to recognize optional (i.e,. explicit presence) in proto3 files.
Use the generated “hazzer” methods and “clear” methods in application code, instead of comparing or setting default values.

`.proto` file changes

This is an example of a proto3 message with fields which follow both no presence and explicit presence semantics:

syntax = "proto3";
package example;

message MyMessage {
  // No presence:
  int32 not_tracked = 1;

  // Explicit presence:
  optional int32 tracked = 2;
}

`protoc` invocation

To enable presence tracking for proto3 messages, pass the --experimental_allow_proto3_optional flag to protoc. Without this flag, the optional label is an error in files using proto3 syntax. This flag is available in protobuf release 3.12 or later (or at HEAD, if you are reading this application note from Git).

Using the generated code

The generated code for proto3 fields with explicit presence (the optional label) will be the same as it would be in a proto2 file.

This is the definition used in the “no presence” examples below:

syntax = "proto3";
package example;
message Msg {  
  int32 foo = 1;
}

This is the definition used in the “explicit presence” examples below:

syntax = "proto3";
package example;
message Msg {
  optional int32 foo = 1;
}

In the examples, a function GetProto constructs and returns a message of type Msg with unspecified contents.

C++ example

No presence:

Msg m = GetProto();
if (m.foo() != 0) {
  // "Clear" the field:
  m.set_foo(0);
} else {
  // Default value: field may not have been present.
  m.set_foo(1);
}

Explicit presence:

Msg m = GetProto();
if (m.has_foo()) {
  // Clear the field:
  m.clear_foo();
} else {
  // Field is not present, so set it.
  m.set_foo(1);
}

C# example

No presence:

var m = GetProto();
if (m.Foo != 0) {
  // "Clear" the field:
  m.Foo = 0;
} else {
  // Default value: field may not have been present.
  m.Foo = 1;
}

Explicit presence:

var m = GetProto();
if (m.HasFoo) {
  // Clear the field:
  m.ClearFoo();
} else {
  // Field is not present, so set it.
  m.Foo = 1;
}

Go example

No presence:

m := GetProto()
if m.Foo != 0 {
  // "Clear" the field:
  m.Foo = 0
} else {
  // Default value: field may not have been present.
  m.Foo = 1
}

Explicit presence:

m := GetProto()
if m.Foo != nil {
  // Clear the field:
  m.Foo = nil
} else {
  // Field is not present, so set it.
  m.Foo = proto.Int32(1)
}

Java example

These examples use a Builder to demonstrate clearing. Simply checking presence and getting values from a Builder follows the same API as the message type.

No presence:

Msg.Builder m = GetProto().toBuilder();
if (m.getFoo() != 0) {
  // "Clear" the field:
  m.setFoo(0);
} else {
  // Default value: field may not have been present.
  m.setFoo(1);
}

Explicit presence:

Msg.Builder m = GetProto().toBuilder();
if (m.hasFoo()) {
  // Clear the field:
  m.clearFoo()
} else {
  // Field is not present, so set it.
  m.setFoo(1);
}

Python example

No presence:

m = example.Msg()
if m.foo != 0:
  # "Clear" the field:
  m.foo = 0
else:
  # Default value: field may not have been present.
  m.foo = 1

Explicit presence:

m = example.Msg()
if m.HasField('foo'):
  # Clear the field:
  m.ClearField('foo')
else:
  # Field is not present, so set it.
  m.foo = 1

Ruby example

No presence:

m = Msg.new
if m.foo != 0
  # "Clear" the field:
  m.foo = 0
else
  # Default value: field may not have been present.
  m.foo = 1
end

Explicit presence:

m = Msg.new
if m.has_foo?
  # Clear the field:
  m.clear_foo
else
  # Field is not present, so set it.
  m.foo = 1
end

Javascript example

No presence:

var m = new Msg();
if (m.getFoo() != 0) {
  // "Clear" the field:
  m.setFoo(0);
} else {
  // Default value: field may not have been present.
  m.setFoo(1);
}

Explicit presence:

var m = new Msg();
if (m.hasFoo()) {
  // Clear the field:
  m.clearFoo()
} else {
  // Field is not present, so set it.
  m.setFoo(1);
}

Objective C example

No presence:

Msg *m = [[Msg alloc] init];
if (m.foo != 0) {
  // "Clear" the field:
  m.foo = 0;
} else {
  // Default value: field may not have been present.
  m.foo = 1;
}

Explicit presence:

Msg *m = [[Msg alloc] init];
if (m.hasFoo()) {
  // Clear the field:
  [m clearFoo];
} else {
  // Field is not present, so set it.
  [m setFoo:1];
}

Application note: Field presence

Background

Presence disciplines

Presence in tag-value stream (wire format) serialization

Presence in named-field mapping formats

Presence in proto2 APIs

Presence in proto3 APIs

Semantic differences

Considerations for merging

Considerations for change-compatibility

How to enable explicit presence in proto3

.proto file changes

protoc invocation

Using the generated code

C++ example

C# example

Go example

Java example

Python example

Ruby example

Javascript example

Objective C example

`.proto` file changes

`protoc` invocation