Unit Grijjy.Bson.Serialization

Description

Serializing Delphi records and objects to JSON and BSON format (or to TgoBsonDocument values).

Quick Start

  type
    TOrderDetail = record
    public
      Product: String;
      Quantity: Integer;
    end;

    TOrder = record
    public
      Customer: String;
      OrderDetails: TArray<<TOrderDetail>;
    end;

  procedure TestSerialization;
  var
    Order, Rehydrated: TOrder;
    Json: String;
  begin
    Order.Customer := 'John';

    SetLength(Order.OrderDetails, 2);
    Order.OrderDetails[0].Product := 'Pen';
    Order.OrderDetails[0].Quantity := 1;
    Order.OrderDetails[1].Product := 'Ruler';
    Order.OrderDetails[1].Quantity := 2;

    { Serialize Order record to JSON: }
    TgoBsonSerializer.Serialize(Order, Json);
    WriteLn(Json); // Outputs:
    // { "Customer" : "John",
    //   "OrderDetails" : [
    //     { "Product" : "Pen", "Quantity" : 1 },
    //     { "Product" : "Ruler", "Quantity" : 2 }
    //   ]
    // }

    { Deserialize JSON to Order record: }
    TgoBsonSerializer.Deserialize(Json, Rehydrated);
    { Rehydrated will have the same values as Order }
  end;

Features

The serialization engine supports the following features:

• You can serialize records and classes to JSON, BSON and TgoBsonDocument.

• You can also serialize dynamic arrays to JSON (but not to BSON and TgoBsonDocument).

• By default, all public fields and public and published read/write properties are serialized. Private and protected fields and properties are never serialized.

• Fields can be of type boolean, integer (all sizes and flavors), floating-point (Single and Double), WideChar, UnicodeString, TDateTime, TGUID, TgoObjectId and TBytes (for binary data).

• Fields can also be of an enumerated type, as long as that type does not have any explicitly declared values (since Delphi provides no RTTI for those).

• Likewise, a set of an enumerated type is also supported.

• Furthermore, a field can also be of a serializable record or class type, or a dynamic array of a serializable type.

• You can customize some behavior and output using attributes.

Representations

By default, the serialization engine serializes fields and properties (collecively called "members" from here onward) as their native types. That is, integers are serialized as integers and strings are serialized as strings. However, you can use the BsonRepresentation attribute to change the way a member is serialized to JSON:

  type
    TColor = (Red, Green, Blue);

  type
    TOrderDetail = record
    public
      Color: TColor;

      [BsonRepresentation(TgoBsonRepresentation.String)]
      ColorAsString: TColor;
    end;

This serializes the Color member as a integer (which is the default serialization type for enums), but serializes the ColorAsString member as a string (using the name of the enum, eg "Red", "Green" or "Blue"). Not all types can be serialized as other types. Below is a list of the types that can be serialized as another type, and the conversion that will take place.

Boolean, can be serialized as:

• Boolean (default)

• Int32, Int64, Double (False=0, True=1)

• String (False="false", True="true")

Integer types:

• Int32, Int64 (default)

• Double

• String (IntToStr-conversion)

Enumerated types:

• Int32 (default, ordinal value)

• Int64 (ordinal value)

• String (name of the enum value)

Set types:

• Int32, Int64 (default, stored as a bitmask)

• String (comma-separated list of elements, without any (square) brackets)

Floating point types:

• Double (default)

• Int32, Int64 (truncated version)

• String (FloatToStr-conversion, in US format)

TDateTime:

• DateTime (default)

• Int64 (number of UTC ticks since midnight 1/1/0001, using 10,000 ticks per millisecond)

• String (DateToISO8601-conversion)

• Document (a document with two elements: TimeStamp serialized as a DateTime value, and Ticks serialized as the number of ticks since midnight 1/1/0001). For example: { "DateTime" : ISODate("2016-05-01T15:28:57.784Z"), "Ticks" : NumberLong("635977133377840000") }

String:

• String (default)

• Symbol

• ObjectId (if the string is a valid TgoObjectId)

WideChar:

• Int32 (default, ordinal value)

• Int64 (ordinal value)

• String (single-character string)

TGUID:

• Binary (default)

• String (without curly braces)

TgoObjectId:

• TgoObjectId (default)

• String (string value of ObjectId)

TBytes:

• Binary (default)

• String (hex string, using 2 hex digits per byte)

Note that for array members, the BsonRepresentation attribute applies to the element types, not to the array itself:

  type
    TColor = (Red, Green, Blue);

  type
    TMyColors = record
    public
      [BsonRepresentation(TgoBsonRepresentation.String)]
      Colors: TArray<<TColor>;
    end;

This will serialize each color as a string (not the entire array as a string).

Handling Extra Elements

When a JSON/BSON stream is deserialized, the name of each element is used to look up a matching member in the record or class. Normally, if no matching member is found, the element is ignored. This also means that when the record or class is rendered back to JSON/BSON, those extra exlements will not exist and may be lost forever.

You can also treat extra members in the JSON/BSON stream as an error condition. In that case, an exception will be raised when extra elements are found. To enable this error, use the BsonErrorOnExtraElements attribute at the record or class level:

    [BsonErrorOnExtraElements]
    TOrderDetail = record
    public
      ...
    end;

Member Customization

Normally, read-only properties are not serialized (unless the property is of a class type, and the object property has already been created). If you want to serialize read-only properties, you can mark them with a BsonElement attribute:

    TOrder = class
    public
      [BsonElement]
      property TotalAmount: Double read GetTotalAmount;
    end;

Of course, read-only properties are never deserialized.

Also, you may wish to serialize a member using a different name than the member name. A common use for this is if you want to serialize using a C-style name (lower case with underscores) but you would like the member to have a Pascal-style name (with camel caps). Another situation where you may want to use this is if the serialization name includes a character that is invalid in a Delphi identifier. You can use the BsonElement attribute to provide the serialization name:

    TOrder = record
    public
      [BsonElement('customer_name')]
      CustomerName: String;

      [BsonElement('$id')]
      Id: TgoObjectId;
    end;

You may also choose to ignore a public member when serializing using the BsonIgnore attribute:

    TOrder = record
    public
      CustomerName: String;

      [BsonIgnore]
      CustomerAge: Integer;
    end;

This will only serialize the CustomerName member. This would be the same as making the CustomerAge field private or protected, with the difference that the CustomAge field is still accessible in code outside of the TOrder class.

You can also ignore a field only when it has a default value, using the BsonIgnoreIfDefault attribute:

    TOrder = record
    public
      [BsonIgnoreIfDefault]
      CustomerName: String;
    end;

This will only serialize the customer name if it is not an empty string. For other types the default value will be 0, False, [] etc. For Boolean, integral and String types, you can specify the default value using the BsonDefaultValue attribute:

    TOrder = record
    public
      [BsonIgnoreIfDefault]
      [BsonDefaultValue('John Smith')]
      CustomerName: String;
    end;

This will only serialize the customer name if it isn't 'John Smith'.

Note: an exception will be raised if you apply the BsonDefaultValue attribute to a member that is not of a Boolean, integral or String type.

Note: the BsonIgnoreIfDefault attribute can be used on all types except record types.

Using Records

The easiest way to serialize to/from JSON/BSON is by declaring record types as shown above.

When a record is deserialized, all its values will be cleared first. This assures that no values will be left uninitialized if certain members are not deserialized.

If you want to customize the initialization behavior, then you can add a method called Initialize without parameters. If such a method exists, then it is called instead of clearing all fields:

    TOrder = record
    public
      // This method gets called before deserializing a TOrder
      procedure Initialize;
    end;

Using Classes

Serialization is easiest and most efficient when used with record types. You can also serialize objects (class instances), but need to be aware of a different behavior.

When you deserialize the object, and the object you pass has a value of nil, then a new instance will be created. You are responsible for releasing the instance at some later point:

  type
    TOrder = class
    public
      Customer: String;
    end;

  procedure TestDeserialization;
  var
    Order: TOrder;
  begin
    Order := nil;
    TgoBsonSerializer.Deserialize('{ "Customer" : "John" }', Order);
  end;

This will create a new TOrder instance and return it in the Order parameter. The TOrder instance is created by calling a parameterless constructor. If the TOrder class has constructor without parameters, then that constructor will be called. Otherwise, a parameterless constructor of the ancestor class will be used. If the ancestor class also doesn't have a parameterless constructor, then we keep going up one ancestor in the chain, until TObject is reached, which always has a parameterless constructor.

If you pass a non-nil value to Deserialize, then the existing object will be updated and no new instance will be created.

When deserializing a field or property of a class type, the behavior depends on whether the member is already assigned.

Deserializing Assigned object-properties

Usually, it is best to make sure that the member is always assigned, by creating it in the constructor and destroying it in the destructor:

  type
    TOrderDetail = class
    ...
    end;

    TOrder = class
    private
      FCustomer: String;
      FDetail: TOrderDetail;
    public
      constructor Create;
      destructor Destroy; override;

      property Customer: String read FCustomer write FCustomer;
      property Detail: TOrderDetail read FDetail; // Read-only
    end;

    constructor TOrder.Create;
    begin
      inherited;
      FDetail := TOrderDetail.Create;
    end;

    destructor TOrder.Destroy;
    begin
      FDetail.Free;
      inherited;
    end;

This is a very common design pattern when using composition. Properties that are of a class type (like Detail in this example) are usually read-only.

When deserializing the TOrder.Detail property in this example, its members are deserialized as usual. Even though the Detail property is read-only, it will still be deserialized (other read-only properties are usually ignored).

Deserializing Non-Assigned object-properties

If the member is not assigned, it is only created and assigned if it is a read/write property (or field):

  type
    TOrderDetail = class
    ...
    end;

    TOrder = class
    private
      FCustomer: String;
      FDetail: TOrderDetail;
    public
      property Customer: String read FCustomer write FCustomer;
      property Detail: TOrderDetail read FDetail write FDetail; // Read/write
    end;

In this case, when deserializing the Detail property, it will be created (using a parameterless constructor) and assigned to Detail. You need to make sure though that the Detail property will be destroyed at some point. You could make the Order class the owner and have it destroy the property in the destructor.

This design pattern is less common and not recommended. The recommended approach is to always make sure the Detail property is assigned (and read-only), as mentioned previously.

Polymorphism

A complication that arises when serializing classes (instead of records) it that they may be part of a class hierarchy:

  type
    TAnimal = class
    public
      Weight: Double;
    end;

    TDog = class(TAnimal)
    public
      FurColor: String;
    end;

All animals have a weight, but only dogs have fur. When serializing a TDog, the output is as expected:

  var
    Dog: TDog;
    Json: String;
  begin
    Dog.Weight := 30;
    Dog.FurColor := 'Blond';
    TgoBsonSerializer.Serialize(Dog, Json); // Result:
    // { "Weight" : 30.0, "FurColor" : "Blond" }
  end;

However, output is different when a TDog is serialized as a TAnimal:

  var
    Dog: TDog;
    Animal, Rehydrated: TAnimal;
    Json: String;
  begin
    Dog.Weight := 30;
    Dog.FurColor := 'Blond';
    Animal := Dog;
    TgoBsonSerializer.Serialize(Animal, Json); // Result:
    // { "_t" : "TDog", "Weight" : 30.0, "FurColor" : "Blond" }

    TgoBsonSerializer.Deserialize(Json, Rehydrated);
    // This will actually create a TDog instance (instead of TAnimal)
  end;

In this case, an extra "_t" element is added (called a Discriminator) that specifies the actual type that is serialized. This way, when you deserialize a TAnimal, and the JSON/BSON contains a discriminator, it knows what actual type of class to instantiate.

However, this only works if the serialization engine "knows" about the TDog type. You have to let the engine know what kind of sub classes can be expected when deserializing. You do this by calling RegisterSubClass(es):

  TgoBsonSerializer.RegisterSubClass(TDog);

Note that this is only needed if you plan to deserialize dogs using type TAnimal. If you always serialize and deserialize dogs as TDog, then you don't need to do this.

You can choose to always serialize a descriminator, even if not strictly necessary, by adding a BsonDiscriminator attribute to the class:

  type
    [BsonDiscriminator(True)]
    TAnimal = class
    public
      Weight: Double;
    end;

The True argument indicates that the descriminator is required. You can also specify a custom discriminator name using the same attribute:

  type
    [BsonDiscriminator('animal', True)]
    TAnimal = class
    public
      Weight: Double;
    end;

This will serialize the descriminator as { "_t" : "animal" } instead of using the Delphi type name { "_t" : "TAnimal" }. In this case, the second parameter (True) is optional. If not specified, the descriminator is not required.

Custom Serialization

Is some situations, you may want to customize the way a certain type is (de)serialized entirely. For example, the TgoAlias type in Grijjy.Protocol.Types is a record consisting of a prefix and a value. But when serializing records of this type, you always want to serialize them as a single string containing both the prefix and value.

To do this, you have to create and register a custom serializer for this type. You can find an example of a custom serializer for TgoAlias in the Grijjy.Protocol.Types unit.

First, you create a class derived from TgoBsonSerializer.TCustomSerializer. You only need to override its Serialize and Deserialize methods. In those methods you perform the type-specific (de)serialization. Both methods have an untyped AValue parameter that you must cast to the actual type (TgoAlias in this example):

type
  TgoAliasSerializer = class(TgoBsonSerializer.TCustomSerializer)
  public
    procedure Serialize(const AValue; const AWriter: IgoBsonBaseWriter); override;
    procedure Deserialize(const AReader: IgoBsonBaseReader; out AValue); override;
  end;

procedure TgoAliasSerializer.Deserialize(const AReader: IgoBsonBaseReader;
  out AValue);
var
  Value: TgoAlias absolute AValue;
begin
  // TgoAlias has in implicit conversion operator to convert from a String
  Value := AReader.ReadString;
end;

procedure TgoAliasSerializer.Serialize(const AValue;
  const AWriter: IgoBsonBaseWriter);
var
  Value: TgoAlias absolute AValue;
begin
  // TgoAlias has in implicit conversion operator to convert to a String
  AWriter.WriteString(Value);
end;

Next, you need to register the custom serializer for the type. For our example:

TgoBsonSerializer.RegisterCustomSerializer<TgoAlias>(TgoAliasSerializer);

Note that custom serializers currently only work for record types.

Notes

• The Serialize and Deserialize methods will raise an exception if the type is not serializable, or if the JSON/BSON to deserialize is invalid. To prevent exceptions, you can use the TrySerialize and TryDeserialize methods instead. These return False if (de)serialization failed.

• Members of type TDateTime are expected to be un UTC format. No attempt is made to convert from local time to UTC and vice versa.

Overview

Classes, Interfaces, Objects and Records

Types

Description

Types