colon3 file format specs
Return to colon3 Home page
Author: EternalAbby
License: CC BY-SA 4.0
Abstract
colon3 is a binary file format for configuration files.
It is designed to compact as much data as possible without using sophisticated compression algorithms,
except for the ones described within this document.
By design, colon3 uses a character map to define characters,
control characters to change parser states, and prioritizes implicit over explicit to compact the data.
Status
This document is not associated with any entities that governs the RFC making process.
It only uses the style guide for its content. All further associations are null and void with the exception of references.
This version of the document is work-in-progress and a proof of concept. It may change in the future.
Copyright Notice
This document is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).
You are allowed to Share and Adapt the content of this document as long as you give Attribution and distribute it with a compatible ShareAlike license. Refer to the license for more details.
Copyrighted EternalAbby
Table of Contents
- Introduction
-
Problems
- JavaScript Object Notation (JSON)
- Tom's Obvious, Minimal Language (TOML)
- YAML Ain't Markup Language (YAML)
- Lua
- Python
- Other Problems
- Name
-
Design
- Details
- Character
- Character Map
- Control Character
- Escape Character
- Comment
- String
- Number
- Boolean
- Array
- Dictionary
null- Custom Data Type
- File Extension
-
Format Structure
- File Header
- Character Map Identifier
- File Content
- Example
- Default Character Map
- Networking
- Security Considerations
-
References
- Informative References
Many configuration files are in a human-readable format, such as JSON and TOML.
They're designed without enough restrictions on how the same data structure can be represented,
which leads to declaring them in many different ways.
This is the result of allowing too many keywords, or multiple complex and verbose idioms to accompany readability for humans.
As a side effect, it leads to many unnecessary arguments and debates over which is better,
and also too many ambiguous edge cases to consider for processors.
See Problems for more information.
To avoid encountering these problems, colon3 restricts its syntax into a small set of control characters,
limits its grammar structure, and forces the optimal result so that there can only be one correct way to represent the entire structure.
This document mostly follows the RFC Style Guide [RFC7322]
and omits parts of it whenever sections aren't necessary for the scale of this document.
In other words, it doesn't strictly follow the guideline.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as
described in BCP 14 [RFC2119] [RFC8174] when, and only when, they
appear in all capitals, as shown here.
For digital information consistency, a byte is 8-bit of information, also known as an octet.
In this document, a byte MUST be able to represent any values from 0 to 7 or 7 to 0 depending on bit endianness.
This section describes problems from human-readable configuration files.
Specifically, it will go through problems encountered in JSON, TOML, YAML, Lua, and Python when they're used as a configuration file.
At the top-level scope, it is an object, or a dictionary of key-value pairs.
It uses parentheses to enclose the data structures, and commas to separate the elements.
However, these parentheses must be explicitly written rather than implicitly known.
Furthermore, it requires explicit corresponding closing characters to end a value, which adds extra characters to close the value.
It's unnecessary to use many keywords to represent meaningful data when it can be omitted with defined implicit behaviors.
At its compressed structure, which removes unnecessary whitespaces, it is affected by the closing characters.
Overall, JSON allows human readability with easy to read structures as a configuration file.
However, it reduces filesize efficiency without adding any meaningful data.
It is possible to specify the same data structures using different syntax and grammar structure.
This causes major confusion and ambiguity on the implementations as it creates many edge cases and undesirable behaviors.
Overall, TOML is a simple configuration file format when its scale is contained and the needs for it grows slowly.
However, it loses its roots of being obvious to represent data for humans and less minimal for processors because of its excessive syntax and grammar structure.
Indentations defines the scope of the data structure.
This can cause accidental mistakes from humans when trying to use it as a configuration file format,
because a piece of data may be defined in a different scope than where it should be.
Non-standardized whitespaces also creates more complexity for processors, and its excessive syntax makes it harder to implement the parser correctly.
Overall, it is a mixture between the excessive syntax of JSON and ambiguity of TOML.
Traditionally, Lua files may be used as configuration files for programs as long as it returns a Lua table.
This also benefits from the language itself as it can process data before the information is created,
allowing dynamic changes to the data without user interactions.
Unfortunately, the immutability of a Lua table allows it to be modified after the information is exposed into the rest of the codebase.
This is a security vulnerability and can cause malicious changes into the data to possibly perform remote code execution.
Furthermore, leveraging the language, or any programming languages, for dynamic changes to the data can cause a zero-day security vulnerability to the users.
Embedding a Lua Virtual Machine (VM) for this specific task may not be a wise choice as there are many functionalities from Lua that needs to be dropped in order to safely use it.
It is possible to restrict it inside of a sandboxed environment so that no internal infrastructures may be exposed to any vulnerabilities,
but this restrictions only make other solutions more desirable for the users.
An alternative to embedding the language is using one of its dialect. A dialect of a programming language is a programming language that is based on the original programming language.
For example, Luau is a Lua 5.1 dialect that offers sandboxing capabilities, a faster VM, immutability of Lua tables.
However, it also suffers from arbitrary code execution if the file itself is maliciously crafted to break the systems.
Overall, Lua is an embedded scripting language that has better use cases other than being simply a configuration file.
It allows too much control over the configurations and dynamic changes can cause major security risks, even in a sandboxed environment.
Python files are used similarly to Lua files and also suffers from similar security vulnerabilities.
Python also has much larger binaries compared to Lua, so it is less of a desirable choice over Lua.
Overall, Python should not be used as a configuration file. It has more use cases elsewhere than this task.
Configuration files are human-readable. However, it does not mean that it should be verbose.
Examples used in this section only highlights popular configuration file formats and ignoring dialects that attempt to make it more human-readable.
Files should be easily parsed by machines and only read by humans through applications that can convert these information into a human-readable format.
colon3 accepts losing human accessibility so that machines can benefit from being able to parse the file easier.
colon3 originated from the emoticon :3 that depicts a cat face.
This emoticon suggests playfulness and carelessness intentions, similar to a cat.
This animal is also a mysterious, quick, intelligent, flexible, and adaptable animal.
Although modern usages of this emoticon had partially lost its actual origins, it is still widely used in the modern Internet.
colon3 had chosen its name as it comprises all aspects of a cat.
It is a binary file format meant for representing data as a configuration file in environments with memory storage constraints.
colon3 is designed around minimizing its syntax and grammar structure while being able to represent a variety of primitive data types.
To achieve this goal, it uses control characters to indicate primitive data types and implicit rather than explicit.
In general, if a common schema is supplied for the sender and receiver, data types SHOULD be implicitly interpreted using the schema.
Otherwise, all data types MUST use control characters to explicitly identify their data type, or implicitly considered as an unsealed string.
When designing a custom character map, all control characters are OPTIONAL,
except `<VALUE_SEPARATOR>` and `<NEWLINE_SEPARATOR>`.
This section explains some terminology details used in this document.
A processor MUST be a program or an application or any tool that can parse a colon3 file format.
The parser refers to the processor step that reads the data from a colon3 file to be interpreted differently.
The parser state refers to how the values are being read by the processor.
It MAY translate the format into any representation.
For example, an intermediate representation before translating into another programming language's equivalent data structure,
another configuration file format, or a human-readable representation.
The processor MAY define and use a custom character map. See Character Map for more information.
The programming language implements the processor so that it can satisfy these requirements.
The sender and receiver is the origin and destination for a colon3 file.
For example, between server and client over the network or a video game application and the player's filesystem.
This document includes examples in a human-readable representation. However, this representation is for illustrative purposes only.
A character is a value such that it is defined and can be found in the character map using a byte.
A valid character MUST exist in the character map, and MUST be known by the sender and receiver.
Otherwise, it is an invalid character.
For example, a byte takes a value of `1A` in hexadecimal,
which is mapped to a `B` character in a character map.
A character map is a 8-bit array (0-indexed, 256 total slots), which uses a byte to index a character.
Processors MAY define a custom character map. However, slots SHOULD NOT be undefined.
All character maps MUST have a defined identifier known by the sender and receiver. This identifier MUST be 4-bytes long.
The default character map uses `00 00 00 00` in hexadecimal. Invalid identifiers MUST NOT be parsed.
The default character map in this document MUST be defined by processors.
A processor MAY change the default character map if and only if the default character map in this document is not used.
In this case, it MUST be documented as the processor-defined default character map.
A control character is a character that changes the parser state for the next bytes until the processor reaches
another control character, newline character, or end-of-file.
All processors MUST implement `<VALUE_SEPARATOR>` and `<NEWLINE_SEPARATOR>`.
A `<VALUE_SEPARATOR>` separates the previous value from the next value.
In general, it ends the previous value and changes the parser state back to default.
Additional behaviors depends on the current primitive data type, explained in detail below.
This control character MUST be `0000 0000` of the character map.
A `<NEWLINE_SEPARATOR>` begins a new key at the top-level dictionary,
and by default, implicitly ends the previous key-value pair.
Another control character MAY override this behavior.
This control character MUST be `1111 1111` of the character map.
Other control characters are defined in sections below and can be omitted depending on the current character map or a common schema.
A `<ESCAPE_CHARACTER>` is a control character that explicitly changes the meaning of the next byte into a normal character.
Any character after this control character is parsed as their value instead of changing the parser state. This is an escape sequence.
After that, the parser state can be changed.
A comment is undefined in a colon3 file format.
Processors MAY implement comments during a human-readable representation,
such as from an output file for humans to read.
There MUST NOT be any comments in the final result of the file.
Use a `<STRING_IDENTIFIER>` to explicitly begin a value as a string.
All control characters, except `<VALUE_SEPARATOR>` and `<NEWLINE_SEPARATOR>` no longer change the parser state.
Processors MUST implicitly treat values as a string, as long as the value contains only bytes for non-control characters defined in the character map.
If a schema is supplied for the sender and receiver, the processor MAY parse the values as the corresponding data type in the schema instead of a string.
Explicitly beginning a value as a string is useful to prevent triggering control characters and allow escape characters for escape sequences.
Explicitly ending a value as a string is useful for arrays and dictionaries, especially nested ones.
Implicitly beginning a value as a string reduces filesize and compacts the value further, while constraining to only bytes for non-control characters.
Implicitly ending a value as a string reduces filesize.
Use a `<NUMBER_IDENTIFIER>` to explicitly begin a value as an unsigned number.
When used, the value is interpreted as itself, as long as it fits within the unsigned integer format.
For example, if the processor chooses unsigned 8-bit for the unsigned number format,
`<SIGNED_IDENTIFIER> 200` is interpreted as `200`
or equivalent to `<UNSIGNED_IDENTIFIER> 200`.
Use a `<SIGNED_IDENTIFIER>` to explicitly begin a value as a signed number.
When used, the value is interpreted as shifting the value to fit within the signed integer format range.
For example, if the processor chooses signed 8-bit for the signed number format,
`<SIGNED_IDENTIFIER> 200` is interpreted as `72`,
or equivalent to `<UNSIGNED_IDENTIFIER> 72`.
This value MAY NOT be limited to any standards. However, it may be limited by the programming language.
The programming language SHOULD support IEEE 754 double-precision floating-point number format and/or 64-bit signed integer and/or 64-bit unsigned integer formats.
If applicable, the programming language MAY use higher precision to represent this value.
In practice, processors SHOULD interpret all number values as a 64-bit signed integer by default.
When the value is out of range, the processor SHOULD trigger the behavior for underflowing or overflowing the number format.
A boolean is undefined in a colon3 file format. Alternatively, use numbers to represent booleans.
Booleans SHOULD use `0` to represent `FALSE` and `1` to represent `TRUE`.
This design encourages bit-packing data into a number or an array so that bits are not wasted.
In practice, the sender and receiver SHOULD know the schema for this situation.
When designing a custom character map, a <BOOLEAN_IDENTIFIER> MAY be defined as a control character,
or two control characters for TRUE and FALSE.
Use a `<OPEN_ARRAY_IDENTIFIER>` to begin an array.
The value before the control character is the identifier for the array.
When defined within an array, the index is the identifier.
This control character also creates a scope for a collection of values.
An array's index MUST be a non-negative integer.
If an unsigned number and a non-negative signed number represents to the same value, they point to the same index.
An array can be a heterogeneous array, also known as a mixed array. Each element may be a different data type.
However, the programming language may only allow homogeneous arrays, or disallow mixed arrays. Processors should account for this.
It is RECOMMENDED to treat arrays as homogeneous.
When followed by a `<NUMBER_IDENTIFIER>` control character, the value of the index,
and then a `<VALUE_SEPARATOR>` control character, this explicitly begins the operation to set the value of the index.
For example, `<NUMBER_IDENTIFIER> 4 <VALUE_SEPARATOR> abc` sets index `4` of the array to the value `abc`.
Otherwise, every value after `<OPEN_ARRAY_IDENTIFIER>` is stored within the array, sequentially appended into the array in chronological order from left to right.
When followed by a `<ESCAPE_CHARACTER>` control character, it changes the parser state to this behavior.
This escape sequence allows representing heterogeneous arrays without triggering the indexing behavior.
Use a `<CLOSE_ARRAY_IDENTIFIER>` to explicitly end an array and return to the previous scope.
A `<NEWLINE_SEPARATOR>` can also implicitly end an array, but it returns to the top-level scope.
However, it can also return to any scope absolute to or relative to the array's scope.
Using run-length encoding, a byte represents the amount of runs plus one, and then a `<SCOPE_TRAVERSE_ABSOLUTE>` to change the parser state to the scope from the top-level to the array's scope.
This byte MUST NOT exceed the depth of the scope. Every subsequent `<SCOPE_TRAVERSE_ABSOLUTE>` adds `256` more runs.
For example. `3 <SCOPE_TRAVERSE_ABSOLUTE> <SCOPE_TRAVERSE_ABSOLUTE>` translate to traversing from the top-level scope 260 depth towards the array's scope.
Similarly, `<SCOPE_TRAVERSE_RELATIVE>` changes the parser context to the scope from the array's scope to the top-level.
A `<OPEN_DICTIONARY_IDENTIFIER>` begins a dictionary. A `<CLOSE_DICTIONARY_IDENTIFIER>` ends a dictionary.
Other behaviors for beginning and ending a dictionary is similar to the array data type,
including creating a scope, setting the identifier for the value and its behavior in an array,
and absolute and relative scopes, are explained above.
An array and a dictionary MUST NOT be a key of a dictionary.
After `<OPEN_DICTIONARY_IDENTIFIER>`, each pair of values has a key-value relationship, a key-value pair.
A key is associated to a value separated by a `<VALUE_SEPARATOR>`,
and then the next key-value pair is separated by another `<VALUE_SEPARATOR>`.
`null` represents the existence of the absence of a value.
`undefined` represents the value of an undefined key and doesn't exist.
It is RECOMMENDED to include `null` if array or dictionary data type is included in the character map.
If a `<VALUE_SEPARATOR>` is followed by another `<VALUE_SEPARATOR>`, the value SHALL be null.
For example, `EMPTY_KEY <VALUE_SEPARATOR> <VALUE_SEPARATOR>` is equivalent to assigning `EMPTY_KEY` to null.
If an index of an array is defined, but its value is not. It is implicitly null.
This happens when an index is explicitly defined, but it is followed by a `<NEWLINE_SEPARATOR>` or reached end-of-file.
If a key of a dictionary is defined, but its value is not. It is implicitly null.
This happens when a key is explicitly defined, but it is followed by a `<NEWLINE_SEPARATOR>` or reached end-of-file.
`null` and `undefined` MUST NOT be an identifier for an array or a dictionary.
`null` and `undefined` MUST NOT be a key of a dictionary.
If the programming language doesn't distinguish between intentional absence of values and undefined values,
the processor MAY omit indices or keys with `null` or `undefined` as values from the file,
or consider both values as the same values.
When designing a custom character map, `null` and `undefined` MAY be defined as a character.
In this case, the processor MAY no longer parse `null` in implicit situations explained above.
However, the restrictions SHOULD NOT be changed.
When designing a custom character map, processors MAY include data types that aren't present in this document. These are called custom data types.
To prevent undefined behaviors, processors SHOULD NOT define implicit parsing state changes.
All colon3 files SHOULD include a file extension.
This extension identifier MUST be .colon3 or .c3.
For example, `foo.c3` is a colon3 file.
This section defines the file structure of a colon3 file format. Every sections are in order of their appearance.
There is also an example of a valid colon3 file.
A colon3 file format MUST begin with c3 in ASCII (hexadecimal: 63 33),
followed by <VALUE_SEPARATOR> (hexadecimal: FE)
and then <NEWLINE_SEPARATOR> (hexadecimal: FF).
In other words, the start of a colon3 file is `63 33 FE FF` in hexadecimal.
The character map identifier is 4-byte long. Each byte of the identifier is written from left to right.
If the identifier is shorter than 4 bytes, any unused byte MUST be set to 00 in hexadecimal.
For example, MEOW is encoded as `4D 45 4F 57`, while CAT is encoded as `43 41 54 00`.
For the default character map, it uses `00 00 00 00` in hexadecimal.
Any bytes after the character map identifier is the file content.
The file content SHOULD be optimized into the most optimal result, which has the smallest filesize.
When the parser reaches end-of-file (EOF), it MUST be treated as an implicit <NEWLINE_SEPARATOR> before stopping the parser.
The top-level scope is a dictionary that is implicitly defined.
When a processor converts this format into the language's representation, it SHOULD use a hashmap or an equivalent.
In this example, assume a character map is defined with the identifier `EX` (hexadecimal: 45 58).
In this character map, <STRING_IDENTIFIER> is defined at index 42 (hexadecimal: 2A)
and all alphabetical characters are in their ASCII positions.
Both the control character usage and human-readable views are for illustrative purposes only.
In a human-readable view, representing control characters as their ASCII symbol:
c3þÿ
EX��
FOOþ"BAR
In a human-readable view, without representing control characters as their ASCII symbol:
c3
EX
FOO "BAR
In a hex editor, reading the file:
63 33 FE FF 45 58 00 00
46 4F 4F FE 2A 42 41 52
In JavaScript Object Notation (JSON) format equivalent:
{"FOO": "BAR"}
All colon3 implementations MUST be able to use the default character map defined in this document.
As of this version, a default character map hasn't been designed.
colon3 is not designed for network transmission. It does not have a checksum field to ensure integrity.
Processors SHOULD generate and include a checksum file to ensure integrity if this file is sent over the network.
By design, it has a smaller filesize than human-readable configuration file formats.
Because of this, less packets are required to be sent over the network.
colon3 file format may be formatted to cause arbitrary remote code execution
due to its minimal design and using unconventional data to represent its file content.
colon3 SHOULD NOT contain sensitive information or be sent over the network.
It does not have security measures to protect against attacks that modify the file content.
Processors MUST NOT parse any colon3 file format with an invalid character map identifier.
As of this version, this section isn't formatted like in the style guide.
- RFC7322
- BCP 14
- RFC2119
- RFC8174
- JSON
- TOML
- YAML
- Lua
- Python
You've reached the end of the page!
Back to top
Return to colon3 Home page