Differences

This shows you the differences between two versions of the page.

--- zxt:format [2021/07/02 22:00] – questions 1-4 answered + BRD tweaks asie
+++ zxt:format [2021/08/10 13:00] (current) – asie
@@ Line 1: / Line 1: @@
-====== ZXT Extended World Format ======
+====== ZXT Extension Format Specification (Draft) ======
 <WRAP center round important 50%>
-This is a draft! Things are still subject to change.
+This page is for draft work! For the latest official version of the ZXT specification, please check [[zxt:format_specification|this page]].
-To discuss the draft, please log in and see the bottom of the documentation.
 </WRAP>
+Version ?.?.?
+By Adrian "asie" Siekierka
+Special thanks: endgame, GreaseMonkey, Lancer-X, Noser, The Mysterious KM, WiL, zzo38
+The specification follows a MAJOR.MINOR.PATCH numbering scheme, where:
+  * Changes to the MAJOR number are expected to introduce breaking changes, resulting in a change of the extension header magic number;
+  * Changes to the MINOR number are expected to introduce changes which don't break backwards compatibility, such as giving meaning to reserved values or defining undefined behaviour;
+  * Changes to the PATCH number are expected to introduce changes which don't impact the meaning of the specification, such as fixing typos or clarifying used language.
+For example, a ZXT 1.1.x world which doesn't utilize functionality specific to ZXT 1.1.x would be expected to work as-is in a ZXT 1.0.x engine implementation; there is no such requirement placed on a ZXT 2.0.x world.
 ===== Introduction =====
-Over the past thirty years, many ideas have been theorized and approaches tried with regards to extending and tweaking the behaviour and capabilities of the ZZT and Super ZZT game engines. With the release of the Reconstruction of ZZT greatly lowering the bar for creating such modified engine versions, the complexity and amount of forks, as well as interest in them, has begun to grow significantly. As such, in order to preserve to the extent possible the interoperability ZZT's ecosystem is known for, a standardized way to declare worlds which use or depend on extensions to the original ZZT and Super ZZT game engines is necessary.
+Extending and tweaking the functionality of the ZZT game engines ((ZZT and Super ZZT)) has always been an undercurrent in their world development community. While many games relying on edited executables or TSRs ((Terminate-Stay Resident programs)) have been released in the past, they ran into key problems, keeping their count small and the overall idea unpopular:
+  * The lack of source code greatly increased the difficulty of performing non-trivial modifications.
+  * The requirement for special batch scripts or modified executables, typically made specifically for a given game, created additional hassle for the end user when trying to play such a game.
+  * No consistent means of signaling required extensions was devised. For most worlds, this made them indistinguishable from ZZT 3.2-compatible ones for the end user or reimplementations, hurting compatibility.
+However, the release of the Reconstruction of ZZT in 2020 greatly lowered the bar for creating such modified engine versions, solving the first problem. With it, interest in creating game-specific forks and providing enhanced functionality has reappeared. Nonetheless, the remaining issues persisted and, as such, a standardized way to declare engine extensions utilized by a given world was deemed necessary.
-A common format for declaring extensions will allow:
+The following use cases were considered:
-  * Creating forks and source ports of ZZT which support a more complete set of the diversified (non-ZZT 3.2-strict) game world catalog,
+  * Creating forks and source ports of ZZT which can support any set of extensions at will,
-  * Mixing and matching extensions between forks at will,
+  * Mixing and matching extensions from various creators within worlds,
-  * Broader editor support for handling extended worlds.
+  * Usage of extensions within typical ZZT game operations (playing, saving, editing).
-The game world types that the ZXT format considers are as follows:
+The following categories of worlds were considered:
-  * Creating completely custom experiences derived from ZZT or Super ZZT. Example: "The King in Yellow Borders", WiL.
+  * Completely custom experiences derived from ZZT game engines. Example: "The King in Yellow Borders", WiL.
-  * Creating games which can benefit from enhanced functionality, but are playable in unmodified ZZT or Super ZZT engines. Example: "Variety", WiL.
+  * Creating games which can benefit from enhanced functionality, but are playable in unmodified ZZT game engines. Example: "Variety", WiL.
   * Standardizing pre-fork games which modify ZZT or Super ZZT in an incompatible way. Example: "ZZT Enhancer", Craig Boston; "Banana Quest", WiL; 64K intros by bitbot.
-  * Standardizing pre-fork games which modify ZZT or Super ZZT in partially or fully compatible ways. Example: "Angelis Finale: Episode 1", Commodore (custom character set); "Daedalus' Obelisk", Darren Hewer (optional ZZT modification to remove certain sounds and messages).
+  * Standardizing pre-fork games which modify ZZT or Super ZZT in ways which are technically compatible. Example: "Angelis Finale: Episode 1", Commodore (custom character set); "Daedalus' Obelisk", Darren Hewer (optional ZZT modification to remove certain sounds and messages).
+===== Coverage =====
+The specification covers:
+  * The format of extension header data and resulting world files,
+  * Rules for processing extension data and handling identified edge cases,
+  * Expectations placed upon game engines implementing the specification.
+Notably, the specification currently does **NOT** cover:
+  * The format of save (.SAV) files. We have decided not to mandate this at this time.
+  * The format of board (.BRD) files. This is due to the niche nature of the subject, and is likely to be expanded upon in a future version of the specification.
 ===== Definitions =====
@@ Line 43: / Line 74: @@
     * **plstring** - a Pascal-style long string; a length u16 (N, of range between 0 and 65535) followed by N characters.
-Things marked "//COMMENT://" below are "non-normative" and are not considered a part of the specification.
+Where the specification mentions the terms **ZZT game engine** or **.ZZT**, one may substitute **Super ZZT game engine** and **.SZT** respectively.
-===== File Types =====
+===== File Format =====
+==== Attachment ====
 The extension data can be provided in one of two ways:
@@ Line 52: / Line 85: @@
   * Providing a .ZXT file, being formed as a concatenation of an Extension Header and a world file (''cat TOWN.ZAX TOWN.ZZT > TOWN.ZXT'' is a valid operation) or board file.
-This is done because neither solution is strictly superior:
+The two approaches are made with specific intents in mind, but may be used in any way the creator sees fit:
-  * .ZAX files can be used to extend games with additional functionality (keybind shortcuts, metadata, etc.) without the need to distribute a modified .ZZT file.
+  * The intent of .ZAX files is to extend games with additional functionality (keybind shortcuts, metadata, etc.), while preserving compatibility with the ZZT game engine. (They can also serve as patches on top of an otherwise unmodified .ZZT file.)
-  * A .ZXT file cannot be understood by unmodified versions of ZZT or Super ZZT, which would require ZZT-compatible games making use of extensions to distribute two variants - a .ZZT and a .ZXT.
+  * The intent of .ZXT files is to contain a modified ZZT game within one world file. This is especially useful for games which cannot be supported by unmodified versions of the ZZT game engine.
-  * Distributing the world as a single .ZXT allows for greater user-friendliness than having to carry around two files, especially for non-ZZT-compatible worlds.
-Notably, this does not cover save files. Saves are NOT guaranteed by the specification to be interchangeable between different implementations of the engine. (However, the authors expect a lot of implementations to simply follow ZZT's lead in terms of SAV files being valid ZXT files.)
+If a .ZXT file is being loaded, a .ZAX file should be ignored - .ZAX files apply for .ZZT files, not .ZXT files.
-===== Extension Header =====
+The behaviour of multiple concatenated extension headers contained within one .ZXT or .ZAX file is undefined; distributed .ZXT or .ZAX files MUST NOT rely on it.
+==== Extension Header ====
 The extension header's format is as follows:
@@ Line 72: / Line 106: @@
 ^ Offset ^ Type ^ Name ^ Description ^
 | 0 | u16 | flags | Extension flags; defined below. |
-| 2 | u32 | owner_id | Extension owner ID |
+| 2 | u32 | owner_id | Extension owner ID. |
-| 6 | u16 | selector_id | Extension selector ID |
+| 6 | u16 | selector_id | Extension selector ID. |
-| 8 | u8 | reserved_0 | Reserved. |
+| 8 | u8 | reserved_0 | Reserved; must be set to 0. |
-| 9 | u16 | field_length | Field length. Can be between 0 and 65534; if set to 65535, an u32 containing the 32-bit field length follows. |
+| 9 | u16 | field_length | Field length. Values between 0 and 65534 refer to the length in bytes; if set to 65535, an u32 containing the 32-bit field length in bytes follows. |
 | 11 | u8[field_length] | field_data | Field data. Extension-defined. |
@@ Line 90: / Line 124: @@
 | 5 | editing_should | Recommended for editing. | If set, an implementation which does not understand this extension SHOULD signal this to the end user if an attempt at editing the world is made. An engine does not have to make any effort to support such a world further. | Non-format-breaking (format-breaking go into writing_must, reading_must or both) editor-side changes. Unknown element IDs count, for example. |
 | 6 | preserve_should | Recommended to preserve on resave. | If set, an implementation SHOULD preserve the extension unchanged upon resave; if cleared, an implementation MUST NOT do so, and MUST discard the extension. | Generally, you want this set to 1 - unless the field data relies on other board, world or extension data outside itself. Meant in particular for metadata. |
-| 7 .. 15 | reserved | Reserved. | If set, an implementation MUST NOT continue parsing of the extension block. | |
+| 7 | vanilla_behavior | Is this extension something ZZT does anyway? | If set, an unmodified implementation of ZZT 3.2 or Super ZZT 2.0 may be assumed to support the ''playing_must'' and ''playing_should'' conditions without further additional support. | This is meant for extensions which either (a) impose additional restrictions on the engine implementation not present in regular ZZT, or (b) depend on ZZT 3.2/Super ZZT 2.0 implementation details which are not required for an otherwise ZXT-compliant implementation. The intent is for external tooling to be able to declare a world ZZT 3.2-compatible. |
+| 8 .. 15 | reserved | Reserved. | If set, an implementation MUST NOT continue parsing of the extension block. | |
-It is important to note that the flags can be distinct from the ID pair; for instance, the same ZZT-OOP extension can be defined as "recommended for playing" if optional for gameplay, but "mandatory for playing" if required for gameplay. However, an extension standard may require you to set or clear certain bits.
+It is important to note that the flags can be distinct from the ID pair; for instance, the same ZZT-OOP extension can be defined as "recommended for playing" if optional for gameplay, but "mandatory for playing" if required for gameplay. However, an extension standard may require you to set or clear certain bits for compliance.
-===== Interactions between multiple extensions =====
+===== Extension IDs =====
+Extension IDs are allocated on a per-owner ID basis. Within one owner ID, selector IDs SHOULD be used in a "one per extension" manner.
+==== Owner ID Ranges ====
+  * The range ''00000000'' - ''000000FF'' is reserved for **standard extensions** - types of changes everyone agrees upon, as well as potential expansions to this standard.
+  * The range ''00000100'' - ''FFFFFEFF'' is available for **public extensions** - you are free to claim them via signaling intent on the [[zxt:owner_ids|ZXT Owner IDs]] page.
+  * The range ''FFFFFF00'' - ''FFFFFFFF'' is reserved for **private use extensions** - prototypes and/or experiments within internal/development versions of worlds and engine implementations. Publicly released worlds and engine implementations SHOULD NOT use this range.
+===== Interactions =====
+==== Cross-Extension Interactions ====
   * **Extension Order:** Extensions MAY depend on their order of definition in the file. Implementations MUST, as such, preserve the order of extension definition when writing a ZXT file.
-  * **Repeated Extensions:** Extensions SHOULD allow being defined multiple times within a single list of extension blocks. This can be done for a variety of purposes:
+  * **Repeated Extensions:** Extensions MAY allow being defined multiple times within a single list of extension blocks. Implementations MUST apply the extensions in order; that is to say, a later extension MAY override the settings of an earlier extension.
-    * defining multiple variants of an extension - custom, per-extension lists would lead to increased parsing complexity. For example, a custom character set extension could allow repetitions for different text cell width/height pairs.
-    * defining multiple flag variants for the extension. For example, a stat limit change extension could set a different mandatory (playing_must) and optional (playing_should) limit.
-Any interactions not listed above are undefined behaviour and MUST NOT be relied upon. Here's an example scenario:
+Any interactions not explicitly defined and not part of any enabled extension's specification are undefined behaviour and MUST NOT be relied upon. For example:
   * One extension is defined as "increasing the number of Keys from 7 to 16",
@@ Line 107: / Line 152: @@
   * It is not certain what should happen if both extensions are in place, unless additional detail is provided. One could interpret it as only allowing the seven vanilla ZZT Keys to be held multiple at a time, or as allowing all sixteen to do so.
-===== Impact on world/board parsing =====
+==== World Format ====
   * **Header/Data Expansion:** If an extension extends the board header, the world header, the tile data, the stat data, or any other existing in-file structure outside of the extension block, the order of parsing MUST be in ascending order of the 32-bit owner ID, then the 16-bit selector ID.
@@ Line 114: / Line 159: @@
     * If a ZXT or ZAX file is present, the implementation MUST ignore the world ID present in the file itself, and trust the ZXT/ZAX file instead.
-===== Reserved extension owner IDs ====
+==== Engine Extension Scope ====
+  * ZXT standard compliance concerns only the game engine's operation. The user interface and other external factors not impacting gameplay or game presentation are not in scope.
+  * An implementation MAY choose to be compatible only with ZXT worlds containing certain extensions. Compatibility with un-extended ZZT worlds for a ZXT implementation is OPTIONAL.
+==== Engine Accuracy ====
+Engines are expected to be accurate to the ZZT game engine's intended behaviour. However, preserving ZZT's memory and stack layouts - and the respective bugs - are OPTIONAL, with the following exceptions:
+  * Implementations MUST emulate the Black Key. The behaviour, including the gem value change and the message duration, MUST be preserved, although the specific text message emitted MAY be changed.
+  * Implementations MUST support reading the state of stat -1. While the state of stat -1 in ZZT may change in rare circumstances, it is recommended to assume the following values:
+^ Field ^ Value ^
+| X | 0 |
+| Y | 1 |
+| Step X | 256 |
+| Step Y | 256 |
+| Cycle | 256 |
+| P1 | 0 |
+| P2 | 1 |
+| P3 | 0 |
+| Follower | 1 |
+| Leader | 1 |
+| Under | Element 1, Color 0x00 |
+| Data Pos | 1 |
+| Data Len | 1 |
+Note that any feature which crashes regular ZZT (or causes double-frees, particularly double #BIND) remains undefined behaviour. ZXT-compliant implementations may choose to fix them.
+Of course, any of the assumptions in this section can be overridden by an extension.
+===== Best Practices =====
+This is not a formal part of the specification - it's more of an "advice" section.
+==== World Developers ====
+  * Distributing the world as a single .ZXT allows for greater user-friendliness than having to carry around two files, especially for non-ZZT-compatible worlds.
+  * Once you opt into the .ZXT format, ZZT tricks relying on memory/stack layout are not guaranteed to work. If you're relying on such behaviours, consider making an extension standard for them instead! This means in particular, but not limited to:
+    * Most out-of-bounds memory reads and writes,
+    * The argTile2 trick pioneered in “Wake Up and Unlock The Door".
-  * The range ''00000000'' - ''000000FF'' is reserved for **standard extensions** - expansions to this standard, as well as potential "obvious" changes everyone agrees upon.
+==== Extension Specification Authors ====
-  * The range ''FFFFF000'' - ''FFFFFFFF'' is reserved for **private extensions** - usage when a standard for a given extension is not yet defined, or for internal/private experiments. Publicly released worlds SHOULD NOT use this range.
-===== Considerations for .ZXT worlds and engines =====
+  * As soon as your extension is used by publicly-released worlds or implemented in publicly-released implementations, the expectation is for it not to change in a breaking manner! While undefined or explicitly reserved behaviour may be clarified further, breaking changes should be introduced via new extensions. There are two ways to avoid that issue during development:
+    * Use an ID inside the area of private use extensions while the extension's shape is not yet fully formed,
+    * Mark your public specification as "Draft" while details are not fully decided upon.
+  * If your extension is intended to work only with ZZT or Super ZZT, but not the other, it's a good idea to mention that explicitly in the specification.
-  * An implementation consuming a .ZXT world, or a world with a .ZAX file, MUST NOT be expected to preserve ZZT's memory layout exactly when running on a ZXT-compliant implementation.
+==== Engine Developers ====
-    * An exception is that ZXT-compliant implementations MUST emulate the Black Key, unless the extensions decide otherwise. The behaviour, including the gem value change and the message duration, MUST be preserved, although the specific text message emitted MAY be changed.
-    * Most out-of-bounds memory reads and writes, as such, are not guaranteed to work correctly.
-    * If you're relying on out-of-bounds memory writes, consider making an extension standard for it instead!
-  * A .ZXT world, or a world with a .ZAX file, MUST treat any feature which crashes regular ZZT (or causes double-frees, particularly double #BIND) as undefined behaviour, unless an extension states otherwise.
-    * That is to say, ZXT-compliant implementations are free to fix them or not.
-===== Unfinalized questions =====
+  * Not every ZXT-compliant engine implementation must provide every extension! Don't fall neck-deep into code bloat, especially on your first try. Focus on the ones you care about, or which games you enjoy utilize.
+  * Remember that your save files need to be aware of which extensions ought to be enabled in order to correctly restore a gameplay session. An easy way for this is to do what ZZT itself does, and share the code for creating world files and save files.
-These questions/concerns must be decided upon before the specification is finalized.
+===== Side Notes =====
-  - ~~Should extension definitions allow depending on their order of placement within the ZXT extension data list? This will complicate implementation, but allow additional flexibility.~~ YES
+This is not a formal part of the specification - rather, it provides insight into some contested decisions.
-  - ~~Should extension definitions allow for the same extension to be present multiple times in the same file? This will complicate implementation, but allow additional flexibility. ~~ YES
-  - ~~Should there be any considerations on preserving extension information as part of save files?~~ THE SPEC DOES NOT REGULATE SAVE FILES
-    * Defining save files as implementation-specific is the easy way out and handy for some use-cases, but hinders save cross-compatibility and loading saves as game worlds in the editor.
-    * Low-memory platforms may be hindered by requiring to load a lot of unnecessary extension data and keep it in memory. (For example, character sets.)
-      * However, so long as the complete data is unnecessary to play the world, does it really hurt a //save file//?
-      * Maybe we could allow a save file to be augmented with a .ZXT or .ZAX to restore blocks which are missing?
-        * This would be best done with a "playing_mutates" bit - if it is 0, the extension could plausibly be restored from the original .ZXT file.
-    * The compromise route, with no clear drawbacks, is to define save file rules as a SHOULD but not a MUST, allowing implementation-specific variants but encouraging a route for typical implementations.
-  - ~~The use-case of ZAXes was defined as being strictly for worlds whose .ZZT file is compatible with ZZT 3.2. However, zzo38 pointed out that he'd prefer to use ZAXes even for worlds incompatible with ZZT 3.2. Should the ZXT format reserve magic world IDs for this scenario?~~ PROPOSAL A
-    * Proposal A:
-      * Define a world ID (f.e. 0xE227) which means "this world file requires a ZXT header or ZAX file to be correctly parsed",
-      * Implementations must ignore the world ID if ZXT/ZAX information is present.
-      * The drawback is that using a valid, but incorrect ZAX will break loading... but a valid, but incorrect ZAX is possible anyway (OOP changes, element changes, etc).
-    * Proposal B:
-      * Allow extensions to define acceptable world IDs.
-      * Drawback: doesn't handle multiple world format changes
-  - How should extensions changing the world format (board header, world header, etc.) be handled?
-    * Proposal A: In order of definition (see "allowing depending on order of placement")
-    * Proposal B: In ascending order of extension IDs
-  - Should there be more or fewer restrictions with regards to ZZT 3.2 behaviours allowed for a ZXT/ZAX-utilizing world?
-    * The current ones are solely to not force implementations to emulate the entire ZZT 3.2 data segment; while Roton (commendably) tries to do it, it is a very large amount of effort that is also difficult to accomplish efficiently on some lower-end platforms.
-===== Implementations =====
+  * I am aware that the "owner/selector ID" system is not ideal. However, other alternatives considered had flaws deemed more important:
+    * A linear ID allocation system would most likely cause conflicts;
+    * A string-based ID system would increase implementation difficult, particularly on older compilers;
+    * The owner/selector divide was introduced after the decision to faciliate 48-bit IDs on older compilers, particularly Turbo Pascal. While 32-bit IDs were considered, they were rejected as potentially being too constraining.
+  * Save files are not mandated by the specification for two key reasons:
+    * Some extensions may require data specific to save files, and implementations may wish to have the freedom to decide how to store them in this case;
+    * Some implementations may store ZZT data in a custom format internally, and removing this requirement can make their code simpler.
+  * Multiple concatenated extension headers may be used as a feature in the future - for example, for faciliating multi-patch application for ZZT worlds.
-==== Final specification ====
+===== Future Suggestions =====
-As the specification has not been finalized, no such implementations exist yet.
+==== ZXT 2.0 ====
-==== Draft specification ====
+==== ZXT 1.1 ====
-  * [[https://github.com/OpenZoo/ClassicZoo/blob/feature/zxt-support/SRC/ZXTMGR.PAS|ZXTMGR]] (Pascal, Zero Clause BSD)
+  * Unified way of handling external file dependencies (for example, large music files).
-  * [[http://zzo38computer.org/fossil/freezzt.ui/dir?ci=tip&name=zxt|zxtsplit]] (C, public domain)
+    * Approach A: Add a bit which signifies that the data of an extension block is actually the filename of the file containing the real data.
+    * Approach B: Add a single extension with a list of external filenames used by the world.
+    * Approach C: Add a set of extensions to allocate "asset IDs" or "asset names", which may point to in-file data or out-of-file assets.
+    * Approach D: Do nothing.
+    * Only approach A would need to be discussed as part of a spec update.
+  * Add a bit which signifies "if this extension is not supported, the immediately following extension may be used as its substitute."
+    * Rationale: For example, multiple charset sizes (8x14, 6x8, 4x6, etc.) for different platforms.
 ===== Comments =====
 ~~DISCUSSION~~