Unit Grijjy.ProtocolBuffers

Description

Google Protocol Buffers for Delphi.

This unit contains declarations for serializing attributed Delphi records in Protocol Buffer format.

It does not use the Google Protocol Buffers framework. Instead, it uses a custom framework, but serializes in a Protocol Buffers compatible bitstream.

About Protocol Buffers

Protocol Buffers is both a framework and a bitstream specification:

https://developers.google.com/protocol-buffers/

It is used to serialize messages in an efficient binary format. A message is a collection of fields, where each field is uniquely identified with an integer Tag. Fields can be of various simple data types, including integers, enums, strings, booleans and strings, as well as compound data types such as nested messages, repeated messages (arrays) and binary data.

Delphi Implementation

This Delphi implementation uses attributed records to define messages. These are regular Delphi records, but fields that are decorated with the [Serialize] attribute can be serialized. For example:

  type
    TPhoneType = (Mobile, Home, Work);

  type
    TPhoneNumber = record
    public
      [Serialize(1)] Number: String;
      [Serialize(2)] PhoneType: TPhoneType;
    public
      procedure Initialize;
    end;

  type
    TPerson = record
    public
      [Serialize(1)] Name: String;
      [Serialize(2)] Id: Integer;
      [Serialize(3)] Email: String;
      [Serialize(4)] MainPhone: TPhoneNumber;
      [Serialize(5)] OtherPhones: TArray<TPhoneNumber>;
    public
      procedure Initialize;
    end;

Each serializable field must be decorated with a [Serialize] attribute with a single parameter containing the Tag for that field. Tags must be unique within the record, but you can use the same tag in different records or in nested records. When a record contains duplicate tags, an exception will be raised when the record is (de)serialized.

Tags start at 1 and must be positive. You should reserve tags 1-15 for the most common fields, since these tags are stored most efficiently (using 1 byte). Tags 16-2047 are stored in 2 bytes, and other tags take more bytes.

Records are serialized in an extensible way. You can add, delete and reorder fields without breaking compatibility with older bitstreams. However, you should never change the tag or data type of a field.

Supported Data Types

You can use a wide variety of Delphi data types for your serializable fields:

• UInt8 (Byte), UInt16 (Word), UInt32 (Longword/Cardinal), UInt64

• Int8 (Shortint), Int16 (Smallint), Int32 (Longint/Integer), Int64

• Single, Double

• Boolean

• Enumerated types, as long as the type does not contain any explicitly assigned values (Delphi does not provide RTTI for these)

• Records (that is, your field can be of another record type)

• Strings (of type UnicodeString)

• TBytes (for raw binary data)

• 1-dimensional dynamic arrays (TArray<>) of the types described above,

Tech note: all arrays of primitive numeric types (integers, floats and enums) are stored in "packed" format, which is supported since Protocol Buffers version 2.1.0. This is a more efficient format that doesn't repeat the tag for each element. All other array types are stored unpacked (where the tag is repeated for each element).

The integer data types are stored in an efficient VarInt format. This means that smaller values are stored in less bytes than larger values. 32-bit integer types are stored in 1-5 bytes, and 64-bit integer types are stored in 1-10 bytes. Sometimes, you can have integer data that contains random values across the entire 32-bit or 64-bit range. In those cases, it is more efficient to store these integers as fixed 32-bit or 64-bit values. You can do this by declaring the field as one of 4 fixed integer types:

• FixedInt32, FixedUInt32, FixedInt64, FixedUInt64

All other data types can not be used for serializable fields. An exception will be raised when an unsupported data type is encountered. However, you can still use these types for regular (non-serializable) fields. In particular, the following types are not supported:

• Extended, Comp, Currency

• Class, Object, Interface

• Enumerated types with explicitly assigned values

• AnsiString, RawByteString, UCS4String etc.

• Static arrays

• Multi-dimensional dynamic arrays

Using the (de)serializer

Serializing is very easy. You just fill your record with the values you want to serialize and call:

  TgoProtocolBuffer.Serialize<TPerson>(MyPerson, 'Person.dat');

This is a generic method with a type parameter that must match the type of record you are serializing.

Since Delphi is able to infer the generic type from the first parameter, you can also write this a little bit shorter:

  TgoProtocolBuffer.Serialize(MyPerson, 'Person.dat');

You can serialize to a file, stream or TBytes array.

Deserializing is equally simple:

  TgoProtocolBuffer.Deserialize(MyPerson, 'Person.dat');

Because all fields in a record are optional, some fields may not be in the stream. To prevent the record from having unitialized fields after deserialization, the record is cleared before it is deserialized (that is, all fields are set to 0 or nil).

You can also provide your own means of initializing the record with default values. To do that, you have to add a parameterless Initialize procedure to your record. Then, the deserialization process will call that routine after clearing the record (so it will still clear any fields you don't initialize yourself).

Overview

Classes, Interfaces, Objects and Records

Types

Description

Types