Construct (python library)

Construct is a python library for the construction and deconstruction of data structures in a declarative fashion. In this context, construction, or building, refers to the process of converting (serializing) a programmatic object into a binary representation. Deconstruction, or parsing, refers to the opposite process of converting (deserializing) binary data into a programmatic object. Being declarative means user code defines the only the data structure, instead of the convention of writing procedural code to accomplish the goal. Construct can work seamlessly with bit- and byte-level data granularity and various byte-ordering.

Using declarative code has many benefits. For example, the same code that can parse can also build (symmetrical), debugging and testing are much simpler (provable to some extent), creating new constructs is easy (wrapping components), and many more. If you are familiar with C (programming language), it would be easiest to think of constructs is casting of char * to struct foo * and vice versa, rather than writing code which unpacks the data.

Example

The following example will show how one might define a TCP/IP protocol stack using Construct. Note that some code is omitted for brevity and complexity. Also note that the following code is just python code that creates objects.

First, the ethernet header (layer 2):

ethernet = Struct("ethernet_header",
   Bytes("destination", 6),
   Bytes("source", 6),
   Enum(UBInt16("type"),
       IPv4 = 0x0800,
       ARP = 0x0806,
       RARP = 0x8035,
       X25 = 0x0805,
       IPX = 0x8137,
       IPv6 = 0x86DD,
   ),
)

Next, the IP header (layer 3):

ip = Struct("ip_header",
   EmbeddedBitStruct(
       Const(Nibble("version"), 4),
       Nibble("header_length"),
   ),
   BitStruct("tos",
       Bits("precedence", 3),
       Flag("minimize_delay"),
       Flag("high_throuput"),
       Flag("high_reliability"),
       Flag("minimize_cost"),
       Padding(1),
   ),
   UBInt16("total_length"),
   # ...
)

And finally, the TCP header (layer 4):

tcp = Struct("tcp_header",
   UBInt16("source"),
   UBInt16("destination"),
   UBInt32("seq"),
   UBInt32("ack"),
   # ...
)

Now define the hierarchy of the protocol stack. The following code "binds" each pair of adjacent protocols into a separate unit. Each such unit will "select" the proper next layer based on its contained protocol.

layer4tcp = Struct("layer4",
   Embed(tcp),
   # ... payload
)

layer3ip = Struct("layer3",
   Embed(ip),
   Switch("next", lambda ctx: ctx["protocol"],
       { 
           "TCP" : layer4tcp,
       }
   ),
)

layer2ethernet = Struct("layer2",
   Embed(ethernet),
   Switch("next", lambda ctx: ctx["type"], 
       { 
           "IP" : layer3ip,
       }
   ),
)

At this point, the code can parse captured TCP/IP frames into "packet" objects and build these packet objects back into binary representation.

tcpip_stack = layer2ethernet
pkt = tcpip_stack.parse("...raw captured packet...")
raw_data = tcpip_stack.build(pkt)

Ports and spin-offs

Perl

Data::ParseBinary is a CPAN module, which originated as a port of Construct to the Perl programming language. (see its main POD document for its inspiration). Since the initial version, some parts of the original API have been deprecated.

Java

A port to Java is available on GitHub. Examples in Java, the ethernet header (layer 2):

Construct ethernet_header = Struct("ethernet_header",
     MacAddress("destination"),
     MacAddress("source"),
     Enum(UBInt16("type"), 
         "IPv4",  0x0800,
         "ARP",   0x0806,
         "RARP",  0x8035,
         "X25",   0x0805,
         "IPX",   0x8137,
         "IPv6",  0x86DD,
         "_default_",  Pass
  ));

External links

1. Construct's homepage