├── ACT.MD ├── Assets ├── Images │ ├── GND_HeightVectors.png │ ├── GND_LightmapTextureSlices.png │ ├── GND_LightmapVisualization.png │ ├── GND_ShadowmapTextureSlices.png │ ├── GND_ShadowmapVisualization.png │ ├── GND_SurfacesAndCubes.png │ ├── RSW_QuadTreeVisualization.png │ ├── RSW_WaterPlaneAlpha.png │ ├── RSW_WaveOffsets_alberta.png │ ├── RSW_WaveOffsets_aldebaran.png │ ├── RSW_WaveOffsets_comodo.png │ ├── RSW_WaveTroughsAndCrest.png │ └── RSW_WaveTroughsAndCrestWithGrid.png └── WaveCycleVisualizationSheet.xlsx ├── CommunityProjects.md ├── EBM.MD ├── EZV.MD ├── GAT.MD ├── GND.MD ├── GR2.MD ├── Images ├── ACT_EmptyLayersExample.png ├── GAT_WaterTileFlags.png ├── GND_AntiBleedingBufferArea.bmp ├── GND_ColorMapOriginal.png ├── GND_ColorMapPosterized.png ├── GND_DynamicLightsEstimation.png ├── GND_DynamicLightsOriginalPosition.png ├── GND_SurfacesVisualization.png ├── GND_UnwrappedColorMap.png ├── GND_UnwrappedShadowMap.png ├── GND_VertexConnections_ORIGINAL.png ├── GND_VertexConnections_RIGHT.png ├── GND_VertexConnections_WRONG.png ├── GR2_BaseSkeleton.png ├── GR2_OodleString.png ├── WindowsSystemPalette.png ├── coordinatesCorrectionEN.png └── interpretingHeightVectors.png ├── PAL.MD ├── README.MD ├── RSM.MD ├── RSW.MD ├── SPR.MD ├── STR.MD └── Videos └── ACT_AnchorPointsVisualization.mkv /ACT.MD: -------------------------------------------------------------------------------- 1 | # ACT Format Specification 2 | 3 | ## Overview 4 | 5 | ACT files store the animation data for all actions of an ingame entity, compiled as a single (binary) file. They're also used for static objects that are rendered using a spritesheet, i.e. anything that's rendered from an SPR file and not a regular image/BMP file. 6 | 7 | They encode the following information: 8 | 9 | * Available actions, such as "idle", "attack", "pickup" (looting an item), or "flinch" (getting hit) 10 | * Animation frames, consisting of individual layers 11 | * Sound effects (referenced), to be played during certain phases of the animation 12 | 13 | Most notably, there are ACT files for items and even skill icons, which don't usually have an animation but still are assigned both an SPR and an ACT file regardless. Since there also exist regular bitmaps for them in the client files, I expect that these SPR/ACT files are only really needed for rendering equipment on characters, as well as lootable items in the game world (in short, anything part of the 3D environment). 14 | 15 | Static skill or item icons displayed in the UI overlays could then simply use the actual images, which would make processing somewhat easier, though wasting disk space doesn't seem like something the developers would do if the other file formats are anything to go by. 16 | 17 | While information from the SPR file is used to generate the actual bitmaps (images), the animation data allows the client to render objects in various phases of each animation, using the bitmaps contained in the SPR spritesheet. 18 | 19 | Therefore the easiest way to think about ACT files is to simply consider them animation metadata, defining what images to display when and for how long for any given action. For static objects the only action then would be "idle", which means they aren't animated at all. 20 | 21 | ## Status 22 | 23 | Mostly complete 24 | 25 | ## Open Questions 26 | 27 | * Incomplete information on anchor points (they have an unknown field containing seemingly random data, not sure if it's actually used though) 28 | * Are there more anchor points than one (players) or zero (creatures)? I've only seen these two, where the one existing anchor point refers to the body and is used to align the head, weapon, shield, headgear sprites. *Update: The file ``kagerou_male.act`` defines two anchor points, but they're both storing the exact same values. What is the point of this? Are there others? Can we safely ignore them?* 29 | * Are the inconsistent frame timings ignored by the client? If they aren't overridden, visible glitches occur, so presumably they just messed up the data for some frames (e.g., compare the [timings for rogue_female.act](ACT_InconsistentFrameTimings.png ) ). *Update: I was able to render animations without these glitches by simply forcing all anchored sprites (i.e., head, weapon etc.) to always synchronize themselves with the current frame of the anchor (body sprite) before they're rendered in the scene. So the additional frames are simply skipped and unequal frame timings are ignored?* 30 | 31 | I've been able to render sprite animations correctly, so I figured it'll do for now. 32 | 33 | ## Prerequisites 34 | 35 | In order to understand the ACT file format, some familiarity with the following concepts is required: 36 | 37 | * The [SPR](SPR.MD) file format, which is tightly coupled to the interpretation of ACT files 38 | * Basic [animation techniques](https://en.wikipedia.org/wiki/Animation#Techniques) and how they can be used to animate sprites 39 | 40 | ## Layout 41 | 42 | The file structure differs slightly with each file version, as newer versions added more features. Please consult the tables below for a detailed specification. 43 | 44 | It appears that (similar to SPR files) the header encodes the version in reverse, i.e. ``41 43 05 02`` reads as ``AC 5.2`` but it's actually version 2.5 and not 5.2. 45 | 46 | The following versions are said to exist: 47 | 48 | * 0x101 (to be confirmed) 49 | * 0x200 (to be confirmed) 50 | * 0x201 (to be confirmed) 51 | * 0x203 (to be confirmed) 52 | * 0x204 (to be confirmed) 53 | * 0x205 (confirmed) 54 | 55 | ### Version 0x205 56 | 57 | | Field | Bytes | Type | Description | 58 | | ---- | ---- | ---- | ---- | 59 | | Header | 2 | char | 'AC' | 60 | | Version | 2 | uint | 0x205, 0x204, 0x203, 0x202, 0x201, 0x200 | 61 | | nActions | 2 | uint | The number of [Actions](#action-layout) in this ACT file. | 62 | | Reserved | 10 | 10*byte | Reserved, unused bytes. | 63 | | | variable | [Action](#action-layout) | times * nActions read [Action](#action-layout) part below. | 64 | | nEvents | 4 | uint | The number of [Events](#event-layout) (e.g. sounds) this ACT file has. | 65 | | | variable | [Event](#event-layout) | times * nEvents read [Event](#event-layout) part below. | 66 | | Intervals | 4 * nActions | float | Interval for each [Action](#action-layout). First float corresponds to the first action, second float to the second action, etc. | 67 | 68 | ### Action layout 69 | | Field | Bytes | Type | Description | 70 | | --- | --- | --- | --- | 71 | | nFrames | 4 | uint | The number of [Frames](#frame-layout) in this Action. | 72 | | | variable | [Frame](#frame-layout) | times * nFrames read the [Frame](#frame-layout) part below. | 73 | 74 | ### Frame layout 75 | | Field | Bytes | Type | Description | 76 | | --- | --- | --- | --- | 77 | | Range1 | 16 | 4*uint | Left, top, right, bottom. In that order. Seems to be unused. Commented with "Attack range". | 78 | | Range2 | 16 | 4*uint | Left, top, right, bottom. In that order. Seems to be unused. Commented with "Fit range". | 79 | | nLayers | 4 | uint | The number of [Layers](#layer-layout) used in this Frame. | 80 | | | variable | [Layer](#layer-layout) | times * nLayers read [Layer](#layer-layout) part below. | 81 | | EventId | 4 | int | Id of the [Event](#event-layout) that this frame fires. | 82 | | nAnchorPoints | 4 | uint | The number of [AnchorPoints](#anchorpoint-layout) this Sprite uses. | 83 | | | variable | [AnchorPoint](#anchorpoint-layout) | times * nAnchorPoints read [AnchorPoint](#anchorpoint-layout) part below. | 84 | 85 | ### Layer layout 86 | | Field | Bytes | Type | Description | 87 | | --- | --- | --- | --- | 88 | | X | 4 | int | The x coordinate of the layer. | 89 | | Y | 4 | int | The y coordinate of the layer. | 90 | | SprNo | 4 | int | The sprite number used (this references the sprite inside a SPR file). | 91 | | Flags | 4 | uint | Bitflags. So far only first bit is used to indicate whether the layer should be mirrored on the y-Axis. | 92 | | Color | 4 | 4*ubyte | In the order: red, green, blue, alpha. Each color is one byte. This is the tint of this layer. | 93 | | XScale | 4 | float | The scale factor for the width of the layer. Below version 0x204 this is also the scale factor for the height (yScale). | 94 | | YScale | 4 | float | The scale factor for the height of the layer. | 95 | | Rotation | 4 | float | The rotation of the layer around its center in angle degrees. | 96 | | SprType | 4 | int | The type of the sprite used from SPR. 0 == Palette sprite, 1 == RGBA sprite. | 97 | | Width | 4 | int | The width of the layer. Assumingly always the same as the referenced sprite width in the SPR file. | 98 | | Height | 4 | int | The height of the layer. Assumingly always the same as the referenced sprite height in the SPR file. | 99 | 100 | ### AnchorPoint layout 101 | | Field | Bytes | Type | Description | 102 | | --- | --- | --- | --- | 103 | | Unknown | 4 | 4*byte | Unknown bytes. When parsing the ACT file 16 bytes will be read from the buffer directly into a *class* (not struct) that holds the three members x, y and attr. Funny that this actually works. | 104 | | X | 4 | int | Offset x coordinate of this anchor point. | 105 | | Y | 4 | int | Offset y coordinate of this anchor point. | 106 | | Attr | 4 | int | Apparently this value is used to compare against another anchor point when stitching together the separate sprites (e.g. body and head). However, only the first anchor point of each sprite is used for the comparison. For that to work it implies that this value is always the same (in this case 0). | 107 | 108 | ### Event layout 109 | | Field | Bytes | Type | Description | 110 | | --- | --- | --- | --- | 111 | | Name | 40 | 40*char | The null-terminated string name of the event (e.g. "atk" or some sound file). | 112 | 113 | **TODO: Add file layouts for the different versions.** 114 | 115 | I have yet to find any version other than just 2.5 to research them. 116 | 117 | ## Actions 118 | 119 | Each action represents an activity the ingame entity (monster, player, homunculus, NPC) can perform. To be precise, it represents an action that can be *animated*; it's perfectly possible to have entities perform actions that *can't* be animated, as is evident by immobile monsters, such as plants, mushrooms or Pupa, attacking players and even moving without having a proper animation. 120 | 121 | This is usually the result of a glitch or explicit serverside scripting forcing this behaviour and quite clearly unintended. However, in general the different entity types have animations for all of the relevant ingame actions. 122 | 123 | Some ACT files contained in the client have glitched animations, e.g. they are too slow, have missing frames, or contain incomplete and unused actions. (**TODO: Examples/screenshots?**) 124 | 125 | For example, there exists a "crying" animation for Rocker and an animation where Fabre is morphing into a Pupa. These can never seen ingame without modifying the client and possibly the server, too. 126 | 127 | ### Player Characters 128 | 129 | The following animated actions have been observed: 130 | 131 | * Idle 132 | * Walking 133 | * Sitting 134 | * Picking up an item 135 | * Standby (used after combat) 136 | * Casting 137 | * Attacking, in three different flavours (unarmed/armed with different weapon types) 138 | * Two "frozen" animations (in combat, standing) 139 | * A flinch (getting hit/taking damage) animation 140 | * One featuring the player dead on the floor 141 | 142 | ### Monsters 143 | 144 | The following animated actions have been observed: 145 | 146 | * Idle 147 | * Walking 148 | * Attacking, in three different flavours (unarmed/armed with different weapon types) 149 | * A flinch (getting hit/taking damage) animation 150 | * One featuring the monster dying 151 | 152 | Additionally, some monsters feature other animations, most of which I've never really seen ingame. 153 | 154 | ### Homunculi 155 | 156 | They use the same animations as monsters, but (usually?) come with three attack animations that are probably used for the different skills? (**TODO: Confirm**) 157 | 158 | ### NPCs 159 | 160 | They are usually static, though there's at least one (unused?) Kafra NPC that also has a walking animation. 161 | 162 | ### Item and Spell Icons 163 | 164 | They are completely static, and their "standing" animation is an image that never changes. Some appear to have copy/pasted their standing animation into other actions, but clearly they weren't intended to actually be animated. 165 | 166 | ### Shadow 167 | 168 | As item and spell icons the shadow underneath the players/npcs/monsters and whatnot consists of just one action and frame. 169 | 170 | ## Frames 171 | 172 | Each animatable action consists of frames. A frame defines the actual animation, and consists of a number of *layers* as well as the animation delay and optionally a sound effect. 173 | 174 | The animation for a unit's current action will proceed to play these frames with the given animationDelay from first to last, playing any sound effect for the given frame. only one can be played and subsequent sound effects would override those already playing. 175 | 176 | The delay is given in units of 25ms each, so an animationDelay of 1 means 25ms in between each frame. A sound effect index of -1 means that there is no sound. Otherwise, the index refers to a sound file listed in the ACT, which is loaded separately and not part of the file itself. 177 | 178 | Depending on the type of action, frames can be repeated. Standing, standby or walking animations are always looping, while attack, cast and dying (for monsters) are not. Frozen/dead animations could be implemented either way, as it doesn't make any difference. 179 | 180 | ## Layers 181 | 182 | A layer is essentially the metadata defining each individual sprite instance, and multiple can be stacked on top of each other to generate complex effects. Most animations feature only a small number, or even just one, but casting effects often can assemble many smaller effects also contained in the SPR spritesheet to form advanced visual effects. 183 | 184 | The information contained tells the client what part of the spritesheet to display, how to scale or position it, tint or mirror it, and may even rotate the image. It's probably easiest to see for yourself how they work using Tokei's GRF Editor or ActEditor. 185 | 186 | ### Invalid (unused) layers 187 | 188 | It can be observed that some sprites seem to contain "empty" layers, which might be leftovers from an earlier version of the game, or were left in there for some other reason. These layers reference sprites with index ``-1``. These are presumably skipped by the client and not rendered, nor animated, in any way. 189 | 190 | Example: [ACT_EmptyLayersExample.png](Images/ACT_EmptyLayersExample.png) 191 | 192 | In the head sprite for male characters, the first layer appears to be an invisible copy of the second. I've not found any information as to the purpose of this, but it doesn't appear to be relevant to rendering the sprites correctly. 193 | 194 | ## Animation Events 195 | 196 | ACT files also contain information about special events that can take place whenever the respective animation frame is rendered. There are at least the following types of animation events: 197 | 198 | * Audio playback (indicated by the name of a sound effect file) 199 | * Display damage numbers (indicated by a virtual ``atk`` event) 200 | 201 | If the damage numbers aren't explicitly triggered during any animation frame, they're displayed at the very end of the animation sequence. This seems like an error state/fallback, and can be seen in the attack animation of Dark Illusion, which is clearly somewhat broken. 202 | 203 | ### Sound Effects 204 | 205 | As mentioned above, playing multiple sound effects in a row will cut off those already playing. This means that idle frames are needed to play sounds without animations if a longer sound is desired and there aren't enough frames otherwise. 206 | 207 | Apparently, headgear sprites cannot contain sound effects and the client will just ignore them if added manually? 208 | 209 | ## Scaling and you - 3D vs. 2D rendering layers 210 | 211 | The following quotes summarize the issue with scaling sprites in the client: 212 | 213 | > The original client does not seem to support the operations (scaling and rotation) inside the act format when the sprite is being projected onto a window. So if he scales the sprite images up and reduces the size inside the act format, all sprite images in the login interface (which might not be important) and others (equip window, skill window, etc.) will have huge sizes. 214 | 215 | > This is why changing the ACT file to use upscaled sprites in Gravity's client inevitably causes the equipment/talent tree UI to display a vastly oversized sprite. Unfortunately, this cannot be easily "fixed" given the technological limitations. 216 | 217 | Essentially, there are two different layers with different technical limitations: 218 | 219 | * The 2D UI layer, which does not implement sprite scaling 220 | * The 3D layer, which supports scaling (based on each layer's scaleU/V properties) 221 | 222 | The 2D layer contains all interface frames and therefore also item icons displayed in the player's inventory, as well as the character preview in the equipment and talent tree windows. 223 | 224 | The 3D layer contains all sprites seen in the game world, as well as items dropped to the floor (which also live as entities in the game world). I believe it also includes all effects, even those that are basically sprites and purely 2D. 225 | 226 | ## ACT vs IMF (char select window) 227 | 228 | The character selection screen appears to be even weirder; it's not really on the UI layer and therefore seems to use special files in the IMF format to display sprites. 229 | 230 | This is probably an artifact of some weird design remnant of the early alpha/Arcturus times, so I wouldn't worry about replicating or even fixing this. 231 | 232 | 233 | ## SpriteTypes 234 | 235 | So far, there are two sprite types being used in an ACT which correspond to the type of the spritesheet image (SPR file): Indexed palette/BMP (type 0) and RGBA/TGA (type 1). 236 | 237 | Each spriteType uses their own sprite numbering. Means if you wish to use the third palette sprite of a SPR you would set the spriteType to 0 and the sprite number to 2. Respectively if you wish to use the third RGBA sprite you would set the spriteType to 1 and the sprite number to 2. 238 | 239 | Technically the spriteType is not a true/false flag indicating whether the image is of indexed palette or RGBA nature. But instead is used as an index to refer to the underlying images of a SPR. It would be possible to simply add another type and give it the value 3 (the official client would obviously either crash or ignore it). 240 | 241 | ## Attachments 242 | 243 | This is where the fun begins! As you may be aware, character sprites consist of different parts, like head, headgear, body, weapon, shield, cart, and they're all different sprites and therefore stored in different ACT/SPR files. 244 | 245 | How the client stitches these together is very much akin to Frankenstein's monstrosity, and there are layering issues to be considered, too. 246 | 247 | ### Draw Layers 248 | 249 | The weapon sprites, for example, must be drawn behind or before the body based on the character's direction. 250 | 251 | Depending on the type of sprite (players, monsters, npc, etc.) different draw layers will be used. The most complex one being the players sprite. It uses the following layers: 252 | 253 | `Shadow`, `Body`, `Head`, `Lower Headgear`, `Middle Headgear`, `Top Headgear`, `Weapon`, `Weapon Slash`, `Shield` and `Garment`. 254 | 255 | The draw order mostly depends on the current direction of the player which can be divided into two parts. Either the player is facing "top left" depicted as an 'x' or "bottom right" depicted as an 'o': 256 | ``` 257 | x x x 258 | \ | / 259 | x -¤- o 260 | / | \ 261 | o o o 262 | ``` 263 | The draw order might look like this then: 264 | 265 | | Order | Top Left | Bottom Right | 266 | |-------|-----------------|-----------------| 267 | | 0 | Shadow | Shadow | 268 | | *x* | *Garment* | *Garment* | 269 | | 1 | Shield | Body | 270 | | 2 | Body | Head | 271 | | 3 | Head | Lower Headgear | 272 | | 4 | Lower Headgear | Middle Headgear | 273 | | 5 | Middle Headgear | Upper Headgear | 274 | | 6 | Upper Headgear | Weapon Slash | 275 | | 7 | Weapon Slash | Weapon | 276 | | 8 | Weapon | Shield | 277 | | *x* | *Garment* | *Garment* | 278 | 279 | In this example, higher draw layers will be drawn over lower ones. 280 | The body and head sprites might be reversed depending on the actual direction (not just the two divided parts). That value is read from the associated IMF file. The Garment layer takes a special place because its order might change on a per frame basis. It is either drawn as the first sprite directly after the shadow (beneath everything) or as the last sprite (on top of everything). The order for the garment is read from the lua file `_New_2DLayerDir_F.lub` or `2dlayerdir_f.lub`. 281 | 282 | **TODO: What about 2D and 3D effects, the combat log (text plane)?** 283 | 284 | These layers also interact with the game world, and avoiding clipping issues is in fact very difficult (or even impossible), depending on the camera's position. I assume this is why they fixated the camera angles, as you can clearly see the sprites being warped when modifiying it beyond what they originally intended. 285 | 286 | **TODO: Screenshot and more details, approaches to circumventing it (override shader, forced draw layer priorization, possibly others?)** 287 | 288 | ### Anchor Points 289 | 290 | ACT files for player characters, as well as those for equipment worn by players, contain special information used to anchor sprites of arbitrary sizes together so that they fit perfectly. This is the needle and thread of the stitched abomination they created and it truly was a pain to get right, but it's of crucial importance. 291 | 292 | Monster sprites and items don't appear to be containing anchor points, though I've not done an exhaustive search to be sure. 293 | 294 | The exact calculation is clearly convoluted, and I only found one source giving more details on this. I have translated the picture they gave here from the Chinese original, with a little help from a friendly Redditor: 295 | 296 | ![](Images/coordinatesCorrectionEN.png) 297 | 298 | #### Example: Visualization of Anchor Points 299 | 300 | In order to really understand this concept, it's probably easiest to just see it in action. As an example, I made a [(very short) visualization video](Videos/ACT_AnchorPointsVisualization.mkv). 301 | 302 | In it you will see three anchor points, represented by the three squares positioned with their center directly on the anchor point that is implied by the offsets stored in the ACT file. 303 | 304 | The blue one is the sprite's origin, meaning the center of the tile where the unit itself is positioned in the game world. It's the same for both body and head because both are part of the same unit. If you ignore anchor points completely, this is also where the center of the body and the head itself would be rendered, but then the head, equipment etc. will of course look very wrong. 305 | 306 | The interesting parts are magenta and yellow: The yellow square is the body's anchor point, as defined in its ACT file, while the magenta one is taken from the head's ACT file. The head sprite is placed exactly in between those two and relocated at every frame, which gives the illusion of both parts being one sprite as they're moving in unison. 307 | 308 | You can hopefully see that the center of the head (which is its position in 2D and 3D coordinates) is the point in between the two squares, calculated by simply subtracting them and obtaining the mathematical displacement. Coincidentally (or not), this is exactly what the image above explained, albeit in a more realistic environment. 309 | 310 | You can additionally see an arrow depicting the direction the unit is facing in, and the red cube is simply a bounding box placed on the tile the unit is located at, though these aren't immediately relevant to the understanding of anchor points. 311 | 312 | #### Calculations 313 | 314 | Calculating the transformation from 2D to 3D space for any two given anchor points is a little tricky. For example, taking the frame's anchor point (for the head sprite ``1_male.act``) defined by 315 | 316 | offsetU = 2 317 | offsetV = -23 318 | 319 | we have to do the following: 320 | 321 | 1. Invert the ``Y`` axis (here labelled ``V`` axis, since we're working in 2D space) 322 | 2. Translate the pixel coordinates of all the added offsets (body minus head plus displacement, as picture above) into world coordinates, by mapping it to whatever scaling factor is used. This factor is given in pixels per world unit, which I *think* might be 32 (purely based on the size of the tile selector image, ``grid.tga``) 323 | 3. Find the origin of the unit represented by the sprite, in 3D world coordinates (XYZ) 324 | 4. Calculate the translation of those two dimensions (UV) in 3D space (XYZ), according to the formulae given in the picture above. This usually involves projecting the two UV unit bases ``(1, 0, 0)`` and ``(0, 1, 0)`` to 3D via the camera's view matrix and then scaling them by the amount calculated from all the offsets, in step 2, resulting in the sprite plane's actual 3D transformation that will be perceived as "moving on the UV axis" by the viewer 325 | 5. Translate the origin to the so-calculated anchor point, by simply adding the two 326 | 327 | Since this is somewhat complicated, I'll omit further details, but you can contact me or look at the code I wrote if you need to understand the algorithm better. 328 | 329 | #### /doridori and "look directions" 330 | 331 | There's one more stumbling block, and it's the fact that player characters in RO can not only have a "face direction"(as visualized by the arrow in the above video), but also a "look direction". This is the direction that the character's *head** is "looking" in, and it doesn't affect the actual face direction, which DOES have implications in the game world, like Backstab not being able to be used on targets that are facing towards you. 332 | 333 | Most notably, you can type /doridori to change your character's look direction, and SHIFT-clicking in the correct pose will alter it without affecting the face direction. Only the STANDING and SITTING poses support look directions, and if you SHIFT-CLICK in any other pose your character will simply change its face direction. 334 | 335 | Generally, a player's look direction is equal to its face direction unless manually changed by the above. If interpreted in angles, the look direction consists of three points on a semi-circle (180 degree field of view) representing LEFT, FORWARD, and RIGHT, but face directions are defined by eight points in a full circle (360 degree), with increments of 45 degree and usually sprites having only 2 poses per side which are mirrored and change in increments of 90 degree. 336 | 337 | The solution is to fixate the frame index to 0 (looking forward) used by the head sprite if a unit is currently STANDING or SITTING, and to set it to the look direction (1 or 2 for left and right, respectively) otherwise. 338 | 339 | *Update: This only works if the animation pose consists of 3 frames. I've now found some headgears that are animated with 27 frames, where 9 frames are used per look direction rather than just one. This means that the calculation will be more complex and needs to take into account the maximum frame index, dividing it by three (the number of look directions), or so I believe.* 340 | -------------------------------------------------------------------------------- /Assets/Images/GND_HeightVectors.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/GND_HeightVectors.png -------------------------------------------------------------------------------- /Assets/Images/GND_LightmapTextureSlices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/GND_LightmapTextureSlices.png -------------------------------------------------------------------------------- /Assets/Images/GND_LightmapVisualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/GND_LightmapVisualization.png -------------------------------------------------------------------------------- /Assets/Images/GND_ShadowmapTextureSlices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/GND_ShadowmapTextureSlices.png -------------------------------------------------------------------------------- /Assets/Images/GND_ShadowmapVisualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/GND_ShadowmapVisualization.png -------------------------------------------------------------------------------- /Assets/Images/GND_SurfacesAndCubes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/GND_SurfacesAndCubes.png -------------------------------------------------------------------------------- /Assets/Images/RSW_QuadTreeVisualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_QuadTreeVisualization.png -------------------------------------------------------------------------------- /Assets/Images/RSW_WaterPlaneAlpha.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_WaterPlaneAlpha.png -------------------------------------------------------------------------------- /Assets/Images/RSW_WaveOffsets_alberta.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_WaveOffsets_alberta.png -------------------------------------------------------------------------------- /Assets/Images/RSW_WaveOffsets_aldebaran.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_WaveOffsets_aldebaran.png -------------------------------------------------------------------------------- /Assets/Images/RSW_WaveOffsets_comodo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_WaveOffsets_comodo.png -------------------------------------------------------------------------------- /Assets/Images/RSW_WaveTroughsAndCrest.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_WaveTroughsAndCrest.png -------------------------------------------------------------------------------- /Assets/Images/RSW_WaveTroughsAndCrestWithGrid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/Images/RSW_WaveTroughsAndCrestWithGrid.png -------------------------------------------------------------------------------- /Assets/WaveCycleVisualizationSheet.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Assets/WaveCycleVisualizationSheet.xlsx -------------------------------------------------------------------------------- /CommunityProjects.md: -------------------------------------------------------------------------------- 1 | # Community Projects 2 | 3 | The RO community has created numerous projects over the years. While many are of questionable practical use, some could still be of interest today. 4 | 5 | ## Third-Party Client Implementations 6 | 7 | There have been countless attempts at creating a custom client to pair up with the existing server emulators. The following list is not exhaustive, but should contain all the most recent ones, particularly those that I've been following more closely: 8 | 9 | | Name | Language | Open Source | Actively Maintained | Description and Notes 10 | | :---: | :---: | :---: | :---: | :---: | 11 | | [UnityRO](https://github.com/guilhermelhr/unityro) | C# (Unity) | √ | √ | 12 | | [Ragna.roBrowser](https://github.com/MrAntares/Ragna.roBrowser) | JavaScript (Browser) | √ | √ | Updated fork of RO Browser; announced [here](https://rathena.org/board/topic/130469-robrowser-continued-join-the-effort/) 13 | | [Midgarts Client](https://github.com/drgomesp/midgarts) | Go | √ | ? | Announcements [here](https://old.reddit.com/r/RagnarokOnline/comments/m9yf9p/im_working_on_an_alternative_opensource_ro_client/) and [here](https://old.reddit.com/r/RagnarokOnline/comments/my9dml/midgarts_an_alternative_ro_client_current_project/) 14 | | [Aesir](https://github.com/Temtaime/aesir) | D | √ | ? | Announcement [here](https://rathena.org/board/topic/111484-make-ro-great-again/) 15 | | [RagnarokRebuild](https://github.com/Doddler/RagnarokRebuild) | C# (Unity) | √ | ? | More details on [Doddler's Twitter](https://twitter.com/RoDoddler/); Continued [here](https://github.com/Doddler/RagnarokRebuildTcp)? 16 | | [roBrowser](https://github.com/vthibault/roBrowser) | JavaScript (Browser) | √ | ✗ | Announcement [here](https://rathena.org/board/topic/53323-robrowser-ragnar%C3%B6k-online-in-browser/) 17 | | [RagnarokJS](https://github.com/GodLesZ/rangarok-js) | JavaScript (Browser) | √ | ✗ | Announcement [here](https://rathena.org/board/topic/74394-also-ragnarok-in-browser) 18 | | [Fimbulwinter Client](https://github.com/greenboxal/fimbulclient)| C++ | √ | ✗ | Announcement [here](https://rathena.org/board/topic/74415-fimbulwinter-client/) 19 | | [OpenRagnarok](https://github.com/open-ragnarok/roint) | C | √ | ✗ | 20 | | [Rustarok](https://github.com/bbodi/rustarok) | Rust | √ | ✗ | Not actually a RO client, but related 21 | | curiosity's "Sakexe" | C++ | ✗ | ✗ | Announcement [here](https://rathena.org/board/topic/104827-wip-native-ragnarok-client/) 22 | | Shinryo's Custom Client | C++ (Ogre3D) | ✗ | ✗ | Announcement [here](https://rathena.org/board/topic/57955-custom-ragnarok-online-client/) 23 | 24 | ## Editing Tools 25 | 26 | There are a large number of obsoleted and abandoned tools. This list contains only those that I found particularly useful: 27 | 28 | | Name | Language | Open Source | Actively Maintained | Description and Notes 29 | | :---: | :---: | :---: | :---: | :---: | 30 | | [GRF Editor](https://rathena.org/board/topic/77080-grf-grf-editor) | C# | ✗ | √ | 31 | | [ACT Editor](https://rathena.org/board/files/file/3304-act-editor/) | C# | ✗ | √ | 32 | | [STR Editor](https://rathena.org/board/topic/130296-a-more-friendly-str-editor/) | C# | ✗ | √ | 33 | | [BrowEdit](https://github.com/Borf/browedit) | C++ | √ | √ | Continued [here](https://github.com/Borf/BrowEdit3) 34 | | [GndEdit](https://dotalux.com/ro/GNDedit/) | ? | ✗ | ✗ | Might help understanding the terrain layout better 35 | -------------------------------------------------------------------------------- /EBM.MD: -------------------------------------------------------------------------------- 1 | # EBM Format Specification 2 | 3 | EBM files contain compressed bitmaps and are used exclusively to represent guild emblems. 4 | 5 | ## Prerequisites 6 | 7 | There really isn't much to read up on here: 8 | 9 | * [Bitmap (BMP)](https://en.wikipedia.org/wiki/Bitmap_file_format) is one of the standard image formats used in the RO client 10 | * [DEFLATE](https://en.wikipedia.org/wiki/Deflate) is a standard compression algorithm and serves to reduce the file size on disk 11 | 12 | ## Overview 13 | 14 | EBM files are what the game creates when players upload regular image files as their guild emblem. They are always stored in the ``_tmpEmblem`` folder, which I'd call "emblem cache". 15 | 16 | The emblems are decompressed and can then be displayed in only a few places: 17 | 18 | * Next to the character and guild name when hovering over a player character's sprite 19 | * In the Guild management UI that also allows uploading new emblems 20 | * On the guild flag model, when a guild has conquered the corresponding castle 21 | 22 | ## Compression 23 | 24 | The client simply applies the ``INFLATE`` (for decompressing) and ``DEFLATE``(for compressing) standard algorithms, provided by [zlib](https://en.wikipedia.org/wiki/Zlib), to the raw bitmaps to convert from/to EBM. 25 | 26 | This makes obtaining the original image file trivial as long as zlib can be used directly: 27 | 28 | ```javascript 29 | const fs = require('fs'); 30 | const zlib = require("zlib"); 31 | 32 | const SOME_EBM_PATH = "myEmblem.ebm"; 33 | 34 | const buffer = fs.readFileSync(SOME_EBM_PATH); 35 | const bitmap = zlib.inflateSync(buffer); 36 | 37 | fs.writeFileSync(SOME_EBM_PATH + ".bmp", bitmap); 38 | ``` 39 | 40 | *Example: Decompressing EBM files using [Node.js' builtin zlib module](https://nodejs.org/api/zlib.html)* 41 | 42 | ## See also 43 | 44 | * The [GR2 specification](GR2.MD) includes information about displaying emblems on guild flags 45 | -------------------------------------------------------------------------------- /EZV.MD: -------------------------------------------------------------------------------- 1 | 2 | ### EZV 3 | 4 | Status: Ignored (for now) 5 | 6 | Not sure if they're really useful, but they contain somewhat-readable text so hopefully understanding them won't be as difficult. -------------------------------------------------------------------------------- /GAT.MD: -------------------------------------------------------------------------------- 1 | # GAT Format Specification 2 | 3 | ## Overview 4 | 5 | GAT files contain two pieces of information: The altitude data (as height vectors) for each tile of a map, and the terrain type that represents the terrain on a given tile. 6 | 7 | The contents can be interpreted as a navigation map that is used by both client and server in pathfinding algorithms, to determine the legality of movements, and line of sight checks. 8 | 9 | ## Prerequisites 10 | 11 | GAT files are inherently simple and do not require much in the way of previous knowledge, or at least none that would be specific to them. 12 | 13 | ## Tiles 14 | 15 | Tiles are the elementary building block of RO terrain. In the GAT files, each tile for the map is defined by the following properties: 16 | 17 | * Four height vectors defining the height at each of the four corners 18 | * A flag indicating the terrain type, used to determine terrain attributes 19 | 20 | Tiles are arranged from bottom-left to top-right. 21 | 22 | ### Height Vectors 23 | 24 | Thei corners of each tiles are adjusted via the altitude parameters (height vectors) to realise many different slope variations. 25 | 26 | #### Interpreting Height Vectors 27 | 28 | It should be noted that the height vectors in both GND and GAT files are given in the client's (very much unintuitive) coordinate system; this means the values are actually negative and scaled. They should be interpreted as "value on the negative Y axis multiplied by the global GAT zoom level" (see next section). 29 | 30 | The distance between each corner defined by the height vector is defined by the geometry's scale factor. The heights themselves also have to be interpreted to account for this fact, lest you get very odd results indeed: 31 | 32 | ![](Images/interpretingHeightVectors.png) 33 | 34 | In this screenshot, we can see that the creator of GNDEdit correctly calculated the positions of tiles and cubes, but somehow forgot to normalize the height vectors, resulting in terrain that appears visibly "stretched". 35 | 36 | You'd have to either do what Gravity did and use a consistent scale factor for everything to keep things in proportion, or do what any sane person would do and normalize all coordinates to a basis of one world unit (sigh). 37 | 38 | #### Texture Mapping and Geometry Scaling 39 | 40 | Similar to the geometry scale factor stored in the map's GND file, interpreting GAT files also requires a "zoom" level (scale factor), which can be used to convert from map to world coordinates (and vice versa). See [GND specification](GND.MD) for more details on this. 41 | 42 | However, this value is hardcoded in the client rather than included in the GAT file itself, and set to be ``5`` for all maps, meaning that each tile is five world units large. Since GND cubes are usually 10 world units large, this merely means that the ratio of tiles to cubes is fixed, i.e. "each cube contains exactly two tiles". 43 | 44 | As far as I know, the textures used are always 256 x 256 pixels wide, from which a 64x64px slice is mapped to each cube and consequently, a 32x32px slice is mapped to each tile. Comparing this to the tile selector image seems to confirm this fact, and we find it is exactly 32x32 pixels large. 45 | 46 | ### Terrain Types 47 | 48 | Each tile comes with a "tile flag" that determines the type of terrain that the tile is intended to represent. These flags are used to create a simplistic simulation of the different terrains that can be found in the game world, and should be interpreted as follows: 49 | 50 | | Value | Walkable | Interpretation | 51 | | :---: | :------: | :------------------------------------: | 52 | | 0 | YES | Walkable terrain | 53 | | 1 | NO | Obstructed (impassable) terrain | 54 | | 5 | NO | Impassable snipeable terrain ("cliff") | 55 | 56 | There are other flags, as detailed in numerous online sources, but these don't seem to have any relevance to the client's interpretation. They are most certainly used by the zone server to determine illegal game states and therefore must be supplied as a "map cache" file in the Hercules/athena emulators, but I haven't looked into them more. 57 | 58 | From what I've seen, the client seemingly ignores the terrain type itself and instead only cares about whether or not the tile is "walkable" or "obstructed" (impassable or cliff). Consequently, the only types that the server can inform the client about are ``0``, ``1`` and ``5`` (or walkable, obstructed and snipeable/cliff, respectively). 59 | 60 | This is also reflected in the client's pathfinding routines, where the latter two types are considered impassable, though no heed is paid to whether or not a tile counts as "water" or not. More research is needed as to what is the exact behaviour on the serverside, presumably. 61 | 62 | ## Layout 63 | 64 | ### Version 1.2 65 | 66 | Just one version (1.2) is used in the modern RO client and there appears to be no support for any other version either. 67 | 68 | | Field | Offset | Size | Type | Description and notes | 69 | | :-----: | :----: | :------: | :----: | :-----------------------------------------------------------------: | 70 | | Header | 0 | 4 Bytes | int | "GRAT" as an ASCII-encoded string | 71 | | Version | 4 | 2 Bytes | binary | Versioning information (Major.Minor) | 72 | | Width | 6 | 4 Bytes | int | The horizontal dimensions of the map (maximum X coordinate) | 73 | | Height | 10 | 4 Bytes | int | The vertical dimensions of the map (maximum Z coordinate) | 74 | | Tiles | 14+ | 20 Bytes | struct | The given size is per tile (see below for details about the layout) | 75 | 76 | Each entry in the ``Tiles`` section represents a block (tile) of the given map, and is structured as follows: 77 | 78 | | Field | Offset | Size | Type | Description and notes | 79 | | :--------: | :----: | :---: | :----: | :-----------------------------------------------------------------------: | 80 | | Altitude | 0 | 4 | float | Altitude at the bottom left corner, i.e., at (0, 0) relative coordinates | 81 | | Altitude | 4 | 4 | float | Altitude at the bottom right corner, i.e., at (X, 0) relative coordinates | 82 | | Altitude | 8 | 4 | float | Altitude at the top left corner, i.e., at (0, Z) relative coordinates | 83 | | Altitude | 12 | 4 | float | Altitude at the top right corner, i.e., at (X, Z) relative coordinates | 84 | | Terrain Type | 16 | 4 | uint32 | Used to determine walkable/snipable properties | 85 | 86 | Only the low word of the terrain type field is actually used, so the rest appears to be irrelevant garbage data. The "32-bit unsigned integer" is therefore misleading, because there's really only a few terrain types (tile flags) that the client actually supports. 87 | 88 | ### Version 1.3 89 | 90 | This version is almost identical to the previous one, but it changes the terrain type field of each tile: Whenever the high byte of the value is ``0x0080`` (``128`` if interpreted as an unsigned 8-byte integer), the tile is marked as a "water tile". The low byte is still the same as before and defines obstructed tiles. 91 | 92 | What exactly the effect of this is in the client isn't quite clear, but it looks like it simply marks the intersection area of the ground mesh and the water plane(s)? At least this is what appears to be the result, here illustrated by visualizing the GAT properties of ``clock_01``: 93 | 94 | ![](Images/GAT_WaterTileFlags.png) 95 | 96 | It's unclear whether this special "water tile" flag (here rendered blue) has any gameplay implications or UI interactions on the client-side. It looks like every water surface tile in GAT 1.3 files is marked by the above terrain type, while those maps using 1.2 don't differentiate between land and water tiles with the same terrain type. 97 | 98 | More research is needed since this format was recently introduced in kRO and isn't available to me (for proper testing) at this time, as the emulators don't appear to support the latest client versions yet. 99 | -------------------------------------------------------------------------------- /GND.MD: -------------------------------------------------------------------------------- 1 | # GND Format Specification 2 | 3 | ## Overview 4 | 5 | GND files contain the entirety of a map's static geometry (i.e., the ground/terrain). In the latest version they also include the configuration of the water surfaces, which were previously stored in the RSW file. 6 | 7 | ## Prerequisites 8 | 9 | In order to understand the GND file format, some familiarity with the following concepts is required: 10 | 11 | * The [GAT](GAT.MD) specification, as it closely ties into how the GND file is interpreted 12 | * [Polygon meshes](https://en.wikipedia.org/wiki/Polygon_mesh), used to represent objects in 3D space 13 | * [Lightmaps](https://en.wikipedia.org/wiki/Lightmap), a standard way to use precomputed lighting and improve rendering performance 14 | * [Posterization](https://en.wikipedia.org/wiki/Posterization), an effect of reducing the number of colors which is applied to the lightmap texture 15 | 16 | ## Ground Mesh 17 | 18 | The ground mesh is a static object that represents the ground (or terrain) of each map. It is the defining part of the GND file and consists chiefly of geometry to which diffuse textures, color highlights and lightmap textures are applied. 19 | 20 | ## Relationship between GAT and GND 21 | 22 | Both files are defining different parts of a map's geometry. See also the following quote: 23 | 24 | > What you can take away from the RO method, is that they actually use two heightmaps: one invisible heightmap used for collision detection, and a visible one used to draw the map. That way, they can extend the "invisible wall or platforms" around various objects (eg those boats or docks placed in the scene that are not part of the heightmap 25 | 26 | *Source: [Game Development @ StackExchange](https://gamedev.stackexchange.com/questions/25823/how-to-create-a-3d-world-with-2d-sprites-similar-to-ragnorak-online)* 27 | 28 | The "collision map" is basically the contents of the GAT file (terrain types and "walkable" flags), while the "height map" is chiefly equivalent to the ground mesh as defined by its height vectors. The server needs to have the GAT information but doesn't care about GND, while the client uses the GAT to place units at their correct height and the GND to render the terrain geometry itself. 29 | 30 | ### Reconstruction 31 | 32 | Recreating the terrain involves a complex set of operations; It is stored in a memory-efficient format and therefore not directly compatible with how modern rendering hardware expects 3D objects to be described (see [Face-Vertex Mesh](https://en.wikipedia.org/wiki/Polygon_mesh#Face-vertex_meshes)). 33 | 34 | Instead of storing the vertex data directly, individual chunks of terrain ("cubes") are implicitly defined by the "height values" that are effectively defining the upwards-facing surface geometry by giving the relative position of its corners, offset by the surface cube's position in an imaginary "grid" of cubes. 35 | 36 | To better explain how the geometry can be reconstructed, it might be helpful to view it as a combination of different (simpler) primitive shapes that are "stitched together" to form the ground mesh. 37 | 38 | #### Tiles, Cubes, and Surfaces (Oh my!) 39 | 40 | RO uses tiles as the basic unit of measurement for game logic, but its world actually consists of surfaces larger than that, which are part of what you could call "cubes" (boxes). These boxes are combined like a jigsaw puzzle to form the actual terrain. 41 | 42 | Cubes are defined by only three surfaces (TOP, NORTH, EAST), which can be one of the two types mentioned below. In order to realize the BOTTOM, SOUTH and WEST sides, the adjacent cube's TOP, NORTH and EAST surface are used, respectively. The only two types of surfaces are GROUND and WALL, where GROUND is essentially the square defined by connecting the four height vectors stored in the GND file, and WALLS are the implied surfaces formed by connecting the GROUND surface of one cube to the GROUND surface of the neighboring cube, forming a wall where they differ in height. 43 | 44 | Each surface of a cube is 2x2 GAT tiles large and may be textured or not. Surfaces without textures (i.e., texture ID is ``-1``) aren't rendered, nor can they be used to attach "wall" surfaces to from their adjacent cube. This is relevant, because in at least two instances, namely ``c_tower4`` and ``juperos_01``, ground surfaces on the map boundaries are erroneously assigned EAST and NORTH surfaces, but you can't render walls on map boundaries since there's no vertices that you could connect with. 45 | 46 | There's no direct reference to the tiles as defined in the world coordinate system, but rather surfaces/cubes are combined in a predefined order (bottom/left to top/right, with new rows starting whenever the "map width" has been exceeded - very much like pixels in a bitmap image). 47 | 48 | Since this process is somewhat convoluted, more details are given below. 49 | 50 | Also, here's an amazing elementary-school-level illustration of the above concepts: 51 | 52 | ![Assets/Images/GND_HeightVectors.png](Assets/Images/GND_HeightVectors.png) 53 | 54 | *Pictured: The height values from the GND file's cube grid define a set of height vectors for each cube.* 55 | 56 | These then have to be turned into an equivalent representation of the actual geometry: 57 | 58 | ![Assets/Images/GND_SurfacesAndCubes.png](Assets/Images/GND_SurfacesAndCubes.png) 59 | 60 | *Any surfaces and cubes that are implicitly defined by the height vectors can easily be reconstructed.* 61 | 62 | The green tile selector cursor here serves as a comparison for the dimensions; as mentioned above, each surface takes up the space of two GAT tiles. Wall surfaces may often appear bigger, but from what I can tell they're still mapped just the same and appear stretched if they happen to be larger. 63 | 64 | ##### Tiles 65 | 66 | As previously mentioned, [tiles](https://en.wikipedia.org/wiki/Tile) are the basic building block for the terrain, so in a way RO could be described as a [tile-based game](https://en.wikipedia.org/wiki/Tile-based_game). It's not technically accurate, but that's definitely how the end result is perceived. Characters are positioned (at the center) on a tile, map coordinates refer to the tile and everything that requires thinking about positions usually works with the tile as the smallest unit... except when it comes to rendering. 67 | 68 | In practical terms, the actual size of each tile, as rendered on the screen, depends on the scale of the world, camera zoom level and screen resolution. However, it's the baseline for measuring distance in the world coordinate system, where one unit of distance can be expressed in "number of tiles". 69 | 70 | As far as textures are concerned, a texture appears to fit on a single tile if its dimensions are 32 pixels, as can easily be seen by checking the size of the ["grid selector"](https://i.imgur.com/GBuVjXe.png) texture rendered on top of the currently selected tile. 71 | 72 | ##### Surfaces 73 | 74 | Surfaces are the visible areas containing tiles. The entirety of the map's geometry is made out of surfaces, which can be textured or "invisible" and as far as I can tell, each surface contains exactly four tiles (two in each dimension). 75 | 76 | This is mostly of relevance for texturing, since texture slices usually are 64 pixels wide, so larger surfaces that appear seamless in the game are frequently pieced together by individual surfaces to give the appearance of a continuous textured plane (and they're indeed rendered similarly). Unfortunately this can cause problems if texture coordinates aren't applied correctly (and sometimes even if they are), which is why there are visible "gaps" in the terrain that can be seen at the right camera angle and distance. 77 | 78 | The vertices forming the terrain's surface geometry must be connected from bottom-right to top-left for each textured surface. If done the other way around some slopes will look very wrong. 79 | 80 | This beautiful part of early 2000s wall geometry can be found at approximately ``pay_dun00 (40, 35)``: 81 | 82 | ![Images/GND_VertexConnections_ORIGINAL.png](Images/GND_VertexConnections_ORIGINAL.png) 83 | 84 | Connecting the vertices the wrong way (bottom-left to top-right) results in a visible glitch: 85 | 86 | ![Images/GND_VertexConnections_WRONG.png](Images/GND_VertexConnections_WRONG.png) 87 | 88 | Clearly, the makers of this cave did not intend for things to go so horribly wrong! 89 | 90 | Luckily, connecting them the other way around eliminates the problem: 91 | 92 | ![Images/GND_VertexConnections_RIGHT.png](Images/GND_VertexConnections_RIGHT.png) 93 | 94 | For comparison, here's the original rendition once more: 95 | 96 | ![Images/GND_VertexConnections_ORIGINAL.png](Images/GND_VertexConnections_ORIGINAL.png) 97 | 98 | So remember kids, always draw your triangles the right way... or someone will be very sad when they see the result. 99 | 100 | ##### Cubes 101 | 102 | Generally speaking, the terrain is defined by a heightmap indicating the positions of all four corners for any given tile. Each chunk of the heightmap (2x2 tiles) is sometimes called cube, which implies the presence of up to 6 surfaces: Two for the top and bottom of the cube, two to either side, and two to the front and back, where of course any number of them could be missing. 103 | 104 | Now, the reason the cube metaphor doesn't work perfectly is that each chunk is only defined by *three* surfaces at most: Top, north, and east. If interpreted as surfaces it doesn't really change anything; Since you can express a cube's left (or south)-facing side as the adjacent cube's right (or north)-facing side you get the same effect with a much smaller data structure. 105 | 106 | [This visualization](Images/GND_SurfacesVisualization.png) might help understand the implications. In the picture, surfaces are colored as follows: 107 | 108 | * Green if they're the TOP surface for a given cube 109 | * Blue if they're NORTH (wall surfaces) 110 | * Red if they're EAST (wall surfaces) 111 | 112 | What this means is that, counter to intuition, the blue parts to the south of each chunk aren't really the south surface of the respective ground cube, but rather the north one of the neighbouring cube to the south. The "walls" are implicitly defined by the difference in both cubes' height vector and the client renders them according to which of the top surface is higher, or not at all if the surface's aren't textured. 113 | 114 | A side effect is that the northernmost "cubes" of each map can't have a "northern" wall, nor can the westernmost "cubes" have a wall to the west. This is why every single map has a specially-crafted "border" beyond the actual visible "walls" (if there are any), which are black impassable surface that are just sitting there and can't usually be seen in the game, as they blend in with the background. 115 | 116 | In the GND files, you will merely find each corner's height and the overall scale factor, which is enough to render the geometry. You could do this by translating the positions to world coordinates and normalize the heights so they refer to a cube of width and height 1 (as in, "one unit in world coordinates"), or just make the cubes bigger to account for the scale factor and then "zoom out" farther to get the same perspective. 117 | 118 | The geometry is then rendered by simply glueing all the surfaces together, and walls are implicitly formed by two tiles that aren't on the same height: If there's a height difference there'll be a plane in between the two tiles; this essentially becomes a "wall" if assigning it a texture. However, this is only done if the neighbouring "cube" is higher or lower and has a textured surface of its own. 119 | 120 | ### The Normalizing Scale Factor 121 | 122 | Each entity in the game world is assigned a "zoom" factor that is used to calculate its rendered size on the screen. For objects of type "Ground", this is always initialized with the value ``10``. Starting at version 1.6, it is overwritten with whatever value is stored in the map's GND file. 123 | 124 | Since I haven't seen any map that used a different "zoom level" for its terrain, I have to assume they just kept the default value. 125 | 126 | The actual calculation appears to be as follows: 127 | 128 | normalizingScaleFactor = GND.mapDimensions / GAT.mapDimensions * GND.geometryScaleFactor 129 | 130 | Since each GND surface normally measures two GAT tiles, and the scale factor is seemingly always set to ten, this nets: 131 | 132 | normalizingScaleFactor = 1/2 * 10 = 1/5 133 | 134 | Therefore a constant scale factor of ``1/5`` should be correct for every map. Applying this factor scales the heights given by the height vectors of the GND file back to the standard coordinate system, with one tile being one world unit large. 135 | ## Lightmaps 136 | 137 | A lightmap is used to add static light and shadow detail to each map, without requiring the client to calculate lighting in real time. 138 | 139 | Similarly to the terrain's surface itself, the lightmap texture is sliced into unique parts, and each part must be mapped to the right surface on the assembled ground mesh. Here's an example: 140 | 141 | ![Assets/Images/GND_LightmapTextureSlices.png](Assets/Images/GND_LightmapTextureSlices.png) 142 | 143 | *One possible rendition of the lightmap for ``c_tower2.gnd``* 144 | 145 | Here, you can clearly see the individual slices that are mapped to each surface. The visible borders constitute the buffer areas intended to minimize artifacts from texture bleeding. 146 | 147 | The ambient occlusion values are stored in the same format, with "white" pixels representing areas where no ambient occlusion (i.e., no "shadows") should be rendered. In this example, the buffer areas are even more starkly visible: 148 | 149 | ![Assets/Images/GND_ShadowmapTextureSlices.png](Assets/Images/GND_ShadowmapTextureSlices.png) 150 | 151 | *One possible rendition of the ambient occlusion map for ``c_tower2.gnd``* 152 | 153 | The process for mapping slices to surfaces is otherwise similar to how the actual terrain is formed: Each surface references a slice of the lightmap texture, as well as a part of the regular (diffuse) texture, while each cube references up to three surfaces. 154 | 155 | With [GNDedit](https://dotalux.com/ro/GNDedit/), it's trivial to export a human-readable form of the same textures: 156 | 157 | ![Assets/Images/GND_LightmapVisualization.png](Assets/Images/GND_LightmapVisualization.png) 158 | 159 | *Surface-mapped lightmap texture for ``c_tower2.gnd``* 160 | 161 | The ambient occlusion map is derived by separating the alpha channel: 162 | 163 | ![Assets/Images/GND_ShadowmapVisualization.png](Assets/Images/GND_ShadowmapVisualization.png) 164 | 165 | *Surface-mapped ambient occlusion texture for ``c_tower2.gnd``* 166 | 167 | These images only include the GROUND surfaces (walkable area). The process for mapping lightmap slices to WALL surfaces is identical, although it's slightly more difficult to visualize in this form. 168 | 169 | ### Buffer Areas and Texture Bleeding 170 | 171 | The visible white borders in that screenshot bring us to another phenomenon: Texture bleeding. If all the tiny slices were to be combined seamlessly, there could be visible artifacts from texture bleeding as soon as linear filtering is enabled. 172 | 173 | To avoid this problem, there's a one-pixel area at the outside of each texture slice that isn't used directly. It serves to provide a buffer between the adjacent slices and removes or at least reduces the effect when texture filtering is applied. 174 | 175 | In this image I've manually merged two adjacent slices: 176 | 177 | ![Images/GND_AntiBleedingBufferArea.bmp](Images/GND_AntiBleedingBufferArea.bmp) 178 | 179 | The green area marks the visible portion of each slice that is contained within the area defined by the texture coordinates, with the area outside being the "buffer" that is blended with its adjacent texels when texture bleeding occurs. 180 | 181 | ### Light and Shadow 182 | 183 | Lastly, there isn't a single lightmap texture, but rather two: A "color map" that contains static lighting, and a "shadow map" that contains only greyscale data and is used to render shadows. The latter effectively stores the [ambient occlusion](https://en.wikipedia.org/wiki/Ambient_occlusion) percentage for each pixel. 184 | 185 | Example: [Color map](Images/GND_UnwrappedColorMap.png) vs [Shadow map](Images/GND_UnwrappedShadowMap.png) 186 | 187 | The "shadow map" must be applied after the lighting calculation to avoid having it blended in with the other light sources, which would cause shadows to appear "washed out" or even invisible, depending on when in the lighting process the colors are multiplied with the shadow/alpha values. 188 | 189 | I believe the client might simply use the alpha channel directly and store both in just one texture, since there's just one texture stored in the GND file and the ``/lightmap`` command seems to disable both parts, but this is merely an educated guess. 190 | 191 | ### Posterization 192 | 193 | The client reduces the number of colors when processing the lightmap texture, causing visible artifacts. These are not part of the lightmap itself, but rather introduced deliberately when loading the texture. 194 | 195 | I can only speculate as to why such a "feature" exists, but my guess would be technical limitations or a stylistic choice. Either way, it is possible to render lighting in the way it was originally computed using just the information that's part of the GND file. This smoothes the transition between colors, as can be seen in the following comparison: 196 | 197 | * [Here](Images/GND_ColorMapOriginal.png) are the lights of Comodo in their original glory 198 | * And [here](Images/GND_ColorMapPosterized.png) they are again after applying the posterization effect with a "posterization level" (see below) of ``16`` 199 | 200 | Note the jagged appearance that plays a huge part in giving RO maps their distinctive look. 201 | 202 | The effect is achieved by applying the following operation to the RGB values: 203 | 204 | r = (int)(r / LEVEL_COUNT) * LEVEL_COUNT 205 | g = (int)(g / LEVEL_COUNT) * LEVEL_COUNT 206 | b = (int)(b / LEVEL_COUNT) * LEVEL_COUNT 207 | In RO, LEVEL_COUNT is 16. 208 | 209 | For javascript it would be c = floor(c / LEVEL_COUNT) * LEVEL_COUNT; 210 | where c is each color component 211 | 212 | *Source: greenboxal* 213 | 214 | This effectively casts the color value from float to int, reducing the amount of visible colors and thereby introducing the "jagged" transitions in the color gradient. 215 | 216 | ## Diffuse Colors (AKA Vertex Colors) 217 | 218 | In order to give highlights to the terrain without requiring vast amounts of slightly different textures, the client can render specific (solid) colors at the corners of each tile to give this corner (and all adjacent ones) a different hue. 219 | 220 | This concept is known as [vertex colors](https://gamedev.stackexchange.com/questions/139059/what-is-a-vertex-color) and it works by submitting additional information for a given point in 3D space that is then used by the hardware/GPU software to color it differently. It was widely used in older games where dynamic lighting wasn't feasible due to technical limitations, and it can have a significant visual impact for barely any effort at all: 221 | 222 | Again they massively saved on file size by providing only one color value per tile, which is applied to the bottom left corner (vertex) of each tile and the vertices of all adjacent tiles that happen to be in the same position. With this, one can define colored spots on the corners of the tile grid with very little effort or overhead. 223 | 224 | In [this basic rendition of pay_dun00](https://i.imgur.com/HWrzkEE.png) you can see the textured terrain with vertex colors applied (lightmaps, models, etc. are not displayed). They're the "blotches" on the ground, which blend in with the rest of the scene after all other effects are applied and give the terrain a lot of variety while still using a very limited set of textures. 225 | 226 | > Each tile can have a diffuse color. This color, however, is not being applied to all four corners of a tile, but only to the bottom left vertex and all vertices that share the same coordinate. 227 | > 228 | ## Layout 229 | 230 | ### Alpha / Arcturus 231 | 232 | At least one version exists that doesn't feature the typical GND header. 233 | 234 | It seems to otherwise be largely identical to the later version 1.7, but I think it may use a different (simpler) lightmap format. 235 | 236 | This is only an educated guess based on comparing the converted legacy GND files (author unknown) published on the athena forums. 237 | 238 | ### Version 1.7 239 | 240 | This is the original version used in the modern RO client. 241 | 242 | It contains all the basic information about the map's surface geometry, as well as information about the diffuse textures and lightmap/ambient occlusion textures. 243 | 244 | | Field | Offset | Size | Type | Description and notes | 245 | | :---: | :---: | :---: | :---: | :---: | 246 | | Header | 0 | 4 Bytes | int | "GRGN" as an ASCII-encoded string 247 | | Version | 4 | 2 Bytes | binary | Versioning information (Major.Minor) 248 | | Width | 6 | 4 Bytes | int | Width of the cube grid 249 | | Height | 8 | 4 Bytes | int | Height of the cube grid 250 | | Scale | 12 | 4 Bytes | float | Geometry scale factor 251 | | Texture Count | 16 | 4 Bytes | int | Number of diffuse texture paths 252 | | Texture Path Length | 20 | 4 Bytes | int | Byte-length of each texture path string (always 80) 253 | | Texture Paths | 24+ | 80 Bytes | string | Null-terminated (discard garbage bytes at the end) 254 | | Lightmap Slices | variable | 268 Bytes | struct | Ambient occlusion and lightmap texture bitmaps (alternating) 255 | | Surface Count | variable | 4 Bytes | int | Number of textured surface blueprints 256 | | Surface Definitions | variable | 56 Bytes | struct | Shared texturing information (copy to each cube vertex) 257 | | Ground Mesh Cubes | variable | 28 Bytes | struct | Width * Height entries 258 | 259 | #### Lightmap Slices 260 | 261 | There are two textures encoded in the files, one is a prebaked lightmap and the other a "shadowmap" (ambient occlusion map). They are read in that same order, based on the lightmap pixel format. 262 | 263 | | Field | Offset | Size | Type | Description and notes | 264 | | :---: | :---: | :---: | :---: | :---: | 265 | | Lightmap Slice Count | 0 | 4 Bytes | int | Number of lightmap (and ambient occlusion) texture slices 266 | | Pixel Format | 4 | 4 Bytes | struct | Encoding of the lightmap and ambient occlusion texture bitmaps (always 1 = 8-bit RGBA, stored as ARGB) 267 | | Width | 8 | 4 Bytes | int | Width of each texture bitmap (always 8) 268 | | Height | 12 | 4 Bytes | int | Height of teach texture bitmap (always 8) 269 | | Shadowmap Pixels | 16 | 64 Bytes | bytes (array) | Width * Height ambient occlusion texture pixels (intensity values) 270 | | Lightmap Pixels | 16 | 192 Bytes | bytes (array) | Width * Height lightmap texture pixels (specularity values) 271 | 272 | #### Textured Surfaces 273 | 274 | These are just the blueprints for creating the terrain geometry, and have to be applied (by copying over the texturing information) to the actual vertices implied by the ground mesh cubes. 275 | 276 | | Field | Offset | Size | Type | Description and notes | 277 | | :---: | :---: | :---: | :---: | :---: | 278 | | Texture Coordinates | 0 | 32 Bytes | float | Diffuse texture UVs 279 | | Texture ID | 32 | 4 Bytes | int | Index of the diffuse texture (``-1`` means "none") 280 | | Lightmap Slice ID | 36 | 4 Bytes | int | Which lightmap (and ambient occlusion) texture bitmap to apply to the surface 281 | | Vertex Color | 40 | 16 Bytes | byte | ARGB vertex color (stored as BGRA) for the surface's bottom left vertex 282 | 283 | The texture coordinates are stored in order 284 | 285 | 1. ``bottomLeftU`` 286 | 2. ``bottomRightU`` 287 | 3. ``topLeftU`` 288 | 4. ``topRightU`` 289 | 5. ``bottomLeftV`` 290 | 6. ``bottomRightV`` 291 | 7. ``topLeftV`` 292 | 8. ``topRightV`` 293 | 294 | where the direction is always relative to the cube that the surface belongs to. 295 | 296 | #### Cube Grid 297 | 298 | Please note that the dimensions are implied by the "width" and "height" given at the start of the file, and that they are different (half as big) as the actual map width and height given in GAT tiles. This is because each GND cube surface is 2 GAT tiles large, as described above. 299 | 300 | | Field | Offset | Size | Type | Description and notes | 301 | | :---: | :---: | :---: | :---: | :---: | 302 | | Bottom left height | 0 | 4 Bytes | float | Altitude of the bottom left corner 303 | | Bottom right height | 4 | 4 Bytes | float | Altitude of the bottom right corner 304 | | Top left height | 8 | 4 Bytes | float | Altitude of the top left corner 305 | | Top right height | 12 | 4 Bytes | float | Altitude of the top right corner 306 | | Surface ID (TOP) | 16 | 4 Bytes | int | Upwards-facing surface ID 307 | | Surface ID (NORTH) | 20 | 4 Bytes | int | Surface ID of the northwards-facing wall 308 | | Surface ID (EAST) | 24 | 4 Bytes | int |Surface ID of the eastwards-facing wall 309 | 310 | ### Version 1.8 311 | 312 | The water plane configuration that was previously part of the [RSW](RSW.MD) file has been removed from the RSW files and is now stored in the map's GND file instead. 313 | 314 | All the information stored in version 1.7 is still present, plus the water plane configuration (appended). 315 | 316 | #### Water Plane Configuration 317 | 318 | In theory, the new GND format should support multiple water planes. In reality, all the maps that use it only feature one and are therefore indistinguishable from those using the old RSW format. 319 | 320 | The reason is that a new version was introduced quickly afterwards, replacing 1.8 with a more elaborate version 1.9 that also supports this feature. 321 | 322 | | Field | Offset | Size | Type | Description and notes | 323 | | :---: | :---: | :---: | :---: | :---: | 324 | | Water Level | 0 | 4 Bytes | float | Height of the default water plane (there's always at least one) 325 | | Water Type | 4 | 4 Bytes | int | Texture ID applied to the water plane (tiled) 326 | | Wave Height | 8 | 4 Bytes | float | Amplitude of the water plane's animation curve 327 | | Wave Speed | 12 | 4 Bytes | float | Phase of the water plane's animation curve 328 | | Wave Pitch | 16 | 4 Bytes | float | Surface curvature level (origin of the curve?) 329 | | Texture Cycling Interval | 20 | 4 Bytes | int | After how many frames (or ms?) the next texture should be swapped in? 330 | 331 | This is the basic water configuration, as present in both RSW (pre-2.6) and the next GND version. The configuration for the other planes follows after the above, and is (in this version) always of the following form: 332 | 333 | 1. ``numWaterPlanesU`` (int32): How many water planes there are in the horizontal dimension (always 1) 334 | 2. ``numWaterPlanesV`` (int32): How many water planes there are in the vertical dimension (always 1) 335 | 3. A float32 defining the ``waterLevel`` (altitude) of this plane. It's always identical to the water level above, since there is only one. 336 | 337 | If there were multiple values here, it would likely be the water level of each plane (assuming the presence of ``u * v`` planes), but since that doesn't happen and there is already a new version it seems unlikely that this would ever be used. 338 | 339 | ### Version 1.9 340 | 341 | All the information stored in version 1.7, plus the water plane configuration (appended). The latter consists of the same information as is present in 1.8, but the format for defining multiple water planes has changed (and they are in fact used in some of the episode 19 maps I've seen, such as ``icecastle.gnd``): 342 | 343 | 1. ``numWaterPlanesU`` (int32): How many water planes there are in the horizontal dimension (*not* always 1) 344 | 2. ``numWaterPlanesV`` (int32): How many water planes there are in the vertical dimension (*not* always 1) 345 | 3. The same structure as defined for the original water plane (see table above), repeated ``u * v`` times 346 | 347 | I'm not quite sure what the original configuration is still used for, but I'm guessing it would serve as a fallback in case the numbers (u, v) are both zero, i.e. there aren't any water planes defined in this section. 348 | 349 | The individual water planes are placed in a grid of ``u * v`` cells, ordered just like the ground mesh cubes from bottom-left to top-right (in map coordinates). Each plane is exactly ``mapWidth / u`` wide and ``mapHeight / v`` high (all given in GAT coordinates), in combination covering the entire map. This is different from the situation where there's just a single plane, which would instead cover the entire map on its own. 350 | 351 | ## See also 352 | 353 | * [This online demo](https://playground.babylonjs.com/#J48BT5#1) showing GND-style cube geometry defined in a standardized vertex data format (*Note: The triangles are formed bottom-right to top-left in the actual GND ground mesh and not bottom-left to top-right, as displayed here*) 354 | * [An intuitive explanation of the terrain in RO (archived; all screenshots are missing)](http://web.archive.org/web/20071015031008/http://www.neatocool.com/Projects/Ragnarok/Map_Editors_Text/) -------------------------------------------------------------------------------- /GR2.MD: -------------------------------------------------------------------------------- 1 | # GR2 Format Specification 2 | 3 | GR2 files contain 3D models and skeletons stored in the proprietary [Granny3D](http://www.radgametools.com/granny.html) format. They consist of a physical (data storage) and logical (animated 3D model resources) layer. 4 | 5 | This article only covers the files used in Gravity's RO client and does not aim to be a comprehensive resource for all possible variants and versions of Granny3D files. 6 | 7 | ## Prerequisites 8 | 9 | Before learning about Granny3D files (as used in RO), you'll probably want to read up on a few things first. 10 | 11 | For the data storage layer, a basic idea of the following is recommended: 12 | 13 | * [Endianness](https://en.wikipedia.org/wiki/Endianness), a term that describes the architecture-dependant byte order of data 14 | * [Marshalling](https://en.wikipedia.org/wiki/Marshalling_(computer_science)) can be employed to store data in an architecture-independent fashion 15 | * [The Wikipedia article on Data Compression](https://en.wikipedia.org/wiki/Data_compression), for an intro to compression algorithms 16 | 17 | If your interest lies with the 3D models they contain, give these a read first: 18 | 19 | * The simpler [RSM](RSM.MD) format description, which uses keyframed animations directly 20 | * [Skeletal animation](https://en.wikipedia.org/wiki/Skeletal_animation), a technique for animating 3D models via "bones" and "weights" 21 | * [B-Splines](https://en.wikipedia.org/wiki/B-spline#Computer-aided_design_and_computer_graphics), a way of compactly storing animation data as mathematical curves 22 | 23 | ## Overview 24 | 25 | GR2 files are used very sparsely in RO, and appear to be a (failed) experiment that took place late in the game's development (2003): They are only used to represent WOE Guardians, the Emperium as seen during WOE, castle flags, and treasure boxes. They also contain an unfinished version of what looks to be a 3D player model (skeleton and textures only). 26 | 27 | There are two "types" of Granny3D files used in RO: The model and idle pose of each 3D actor and can be found in files located in the ``data/model/3dmob`` directory, while the others each store a skeleton for another pose and are found in ``data/model/3dmob_bone``. 28 | 29 | Unlike the custom formats used in RO, Granny3D files cannot as easily be deserialized. This is because they use a proprietary compression algorithm called [Oodle](http://www.radgametools.com/oodle.htm), which comes in many different versions and is (unfortunately) not widely supported by existing open-source tools. 30 | 31 | Each Granny file encodes a hierarchy of objects and some metadata, with all data being stored in containers called sections. The compression is applied to (some of) the data sections, while the data stored inside, like textures and animations, can also be compressed (potentially using a different algorithm). This makes processing them somewhat difficult. 32 | 33 | There are ways to get at the data, of course, but they generally rely on specific format conversion (such as a GR2-to-DAE converter that I've seen) or may not be fully compatible with all possible GR2 versions (various open source projects for other games come to mind). 34 | 35 | Due to this logical separation of data and representation, which makes the format highly flexible, a number of pre-processing steps are needed before the data itself can be accessed. 36 | 37 | ## Pre-Processing Phases 38 | 39 | It appears there are multiple phases involved in restoring the structure of a Granny file: 40 | 41 | 1. Parsing of the headers and section metadata that describes the physical layout 42 | 2. Decompressing any compressed section buffer areas based on the section metadata 43 | 3. Resolving virtual pointers present in the data (educated guess; this needs confirmation) 44 | 4. Restoring hardware-dependant pointers based on the machine's architecture 45 | 5. Loading of the restored sections to create a "tree" of nodes as in-memory structure 46 | 6. Navigating this tree, starting from the root node, to access the actual data 47 | 48 | ## Features 49 | 50 | Granny files can store arbitrary data, but in the case of RO they alway seem to follow a relatively primitive "standard" format. In particular, they don't appear to extend the basic Granny file format with custom data types. This means only some of the vast Granny3D feature set is used, the relevant portions of which should be described here. 51 | 52 | ### Mixed Marshalling 53 | 54 | Going by what's been implemented in various open-source decompressor tools, the format allows storing data in an architecture-dependent form. This may necessitate a preprocessing step where it's updated for the current computer architecture (to restore the endianness). 55 | 56 | It seems that this feature is indeed used by some of the files (e.g., ``aguardian90_8.gr2`` features 4 mixed marshalling blocks). I haven't looked into the specifics as OSS tools exist that will handle the de-marshalling process and I therefore consider it a "solved problem". 57 | 58 | ### Compression 59 | 60 | The compression algorithm used for the compressed data sections is called Oodle0 and (from what I can tell) is an ancient, and almost certainly inferior, version of the same suite of compression algorithms still in use today. 61 | 62 | In the compiled ``granny2.dll`` we find this string message that indicates it's now deprecated: 63 | 64 | ![A message that seems to indicate Oodle0 is no longer supported](Images/GR2_OodleString.png) 65 | 66 | As far as the algorithm itself goes, the people at RAD have published many articles about their work on compression algorithms. Since I'm not an expert in data compression that, alongside some independent research, is the only source providing details about the design: 67 | 68 | From what I can gather Oodle(0) uses [arithmetic coding]( https://en.wikipedia.org/wiki/Arithmetic_coding) with an [adaptive probabilistic model](https://en.wikipedia.org/wiki/Adaptive_coding) to predict the input stream while encoding, with the symbols being stored in a [Huffman-encoded](https://en.wikipedia.org/wiki/Huffman_coding) table. This means it's a variation on the standard [Lempel-Ziv](https://en.wikipedia.org/wiki/LZ77_and_LZ78) class of algorithms. 69 | 70 | Some of the section data seems to consist of compressed data using different schemes, which (I think) are stored in separate sections without section compression, which does make sense. Examples for these are textures using the Bink texture format, and animation curves. 71 | 72 | Since there are converter tools that will transparently deal with all of this, I haven't spent more time on researching the implementation details of the algorithm (beyond the basics). 73 | 74 | ### Sections 75 | 76 | Granny files consist of multiple buffer areas that the decompression needs to be applied to individually before a proper and more useful Granny File structure can be obtained. 77 | 78 | Not all of these are using compression; it seems like the assets are categorized, so some classes of assets that are already compressed (e.g., textures) don't require it and are stored in an uncompressed section that can be read directly using the offsets from the header. 79 | 80 | I'm guessing that they are also grouped by their intended use case, since this design allows for loading only certain sections and reduce memory usage if the others aren't needed. 81 | 82 | Once all sections have been decompressed, the result is a representation of the actual 3D model, which can be accessed to retrieve the stored geometry, textures, and animations. 83 | 84 | ## Layout 85 | 86 | I have verified that all 21 GR2 files in the RO client use version 6 of the Granny file format. 87 | 88 | | Field | Offset | Size | Type | Description and notes | 89 | | :-----: | :------: | :----: | :-----: | :---------------------------: | 90 | | Header | 0 | 352 bytes | struct | The Granny File header structure (see below) | 91 | | Sections | 352 | variable | binary | Compressed Granny File data, split into multiple parts | 92 | 93 | The decompressed sections need to be relocated; it looks as though they are simply appended at the end of the header section: The ``relocationOffset`` is always ``352``, which is the length of the header structure. 94 | 95 | ### Granny Header Structure 96 | 97 | While the header clearly allows for flexible structures, in Ragnarok's GR2 files the size is fixed: 98 | 99 | | Field | Offset | Size | Type | Description and notes | 100 | | :-----: | :------: | :----: | :-----: | :---------------------------: | 101 | | Signature | 0 | 16 Bytes | binary | Always ``B8 67 B0 CA F8 6D B1 0F 84 72 8C 7E 5E 19 00 1E`` | 102 | | Header Size | 16 | 4 Bytes | uint32 | Always ``352`` (decimal) | 103 | | Compression Flag | 20 | 4 Bytes | uint32 | Always ``0`` (no compression) | 104 | | Unknown | 24 | 4 Bytes | ? | Unused (probably) since it's always zero | 105 | | Unknown | 28 | 4 Bytes | ? | Unused (probably) since it's always zero | 106 | | Version | 32 | 4 Bytes | uint32 | The Granny File version (always ``6``) | 107 | | File Size | 36 | 4 Bytes | uint32 | The total file size in bytes | 108 | | Checksum | 40 | 4 Bytes | uint32 | A CRC checksum over the file contents | 109 | | Section Offset | 44 | 4 Bytes | uint32 | Where the sections begin (always ``56``) | 110 | | Section Count | 48 | 4 Bytes | uint32 | Number of sections (always ``6``) | 111 | | Root Node Type | 56 | 8 Bytes | struct | Points to a definition of the top-level node object's type | 112 | | Root Node Object | 64 | 8 Bytes | struct | Points to the top-level node object in the data tree | 113 | | User Tag | 68 | 4 Bytes | uint32 | Always ``0F 00 00 80`` | 114 | | User Data | 72 | 16 Bytes | ? | Unused (probably) since it's always zero | 115 | | Section Headers | 88 | 264 Bytes | struct | ``44`` Bytes per section (here: ``6 * 28 = 264``)| 116 | 117 | It should be noted that the "header size" includes the section metadata, which I've listed separately here to make it easier to survey at a glance. That is to say, ``352`` bytes is the length from the beginning of the file to the actual section data and *not* just to the section headers. 118 | 119 | ### Section Reference Structure 120 | 121 | Objects reference the decompressed section buffer area using this simple format: 122 | 123 | | Field | Offset | Size | Type | Description and notes | 124 | | :-----: | :------: | :----: | :-----: | :---------------------------: | 125 | | Index | 0 | 4 Bytes | uint32 | The index of the section containing this object | 126 | | Offset | 4 | 4 Bytes | uint32 | Where the object is located inside the section | 127 | 128 | This serves as a "pointer" to the buffer area and can be used to read data without having to re-arrange the entire file in memory (after the decompression and demarshalling steps). 129 | 130 | ### Section Headers Structure 131 | 132 | | Field | Offset | Size | Type | Description and notes | 133 | | :-----: | :------: | :----: | :-----: | :---------------------------: | 134 | | Compression Mode | 0 | 4 Bytes | uint32 | ``0`` if uncompressed, ``1`` for Oodle0 | 135 | | Section Offset | 4 | 4 Bytes | uint32 | After how many bytes in the file the section starts | 136 | | Compressed Size | 8 | 4 Bytes | uint32 | After how many bytes the section ends (when compressed) | 137 | | Decompressed Size | 12 | 4 Bytes | uint32 | How large the section is after decompressing it | 138 | | Alignment Size | 16 | 4 Bytes | uint32 | Used for packing the data on hardware-aligned boundaries (?) | 139 | | Stop 0 | 20 | 4 Bytes | uint32 | "Compressor stop 0" - Oodle0 parameter (?) | 140 | | Stop 1 | 24 | 4 Bytes | uint32 | "Compressor stop 1" - Oodle0 parameter (?) | 141 | | Relocation Header | 28 | 8 Bytes | struct | Used to restore virtual pointers to an intermediate, hardware-dependant form (?) | 142 | | Marshalling Header | 32 | 8 Bytes | struct | Used to restore the fixed pointers to their final, hardware-dependant form (?) | 143 | 144 | It looks like the section metadata covers the layout and size of the buffer area, as well as the configuration data for all preprocessing steps needed to restore the data in its original form. 145 | 146 | ### Relocation Headers 147 | 148 | | Field | Offset | Size | Type | Description and notes | 149 | | :-----: | :------: | :----: | :-----: | :---------------------------: | 150 | | Relocation Offset | 0 | 4 Bytes | uint32 | Where the relocation data is stored in the section | 151 | | Relocation Count | 4 | 4 Bytes | uint32 | How many relocation data sets (?) are stored at this location | 152 | 153 | If I understand this correctly, it's used to "relocate" references (pointers) to the stored data. This would mean "virtual" pointers are being stored, possibly to save space or to abstract away hardware architecture details. They would then need to be "resolved" in a separate step (the "demarshalling step") before the data can actually be used? 154 | 155 | More research is needed on this topic. 156 | 157 | ### Marshalling Headers Structure 158 | 159 | | Field | Offset | Size | Type | Description and notes | 160 | | :-----: | :------: | :----: | :-----: | :---------------------------: | 161 | | Marshalling Data Offset | 36 | 4 Bytes | uint32 | Where the marshalling data is stored in the section | 162 | | Marshalling Block Count | 40 | 4 Bytes | uint32 | How many marshalling data sets (?) are stored at this location | 163 | 164 | After the virtual references have been resolved, the hardware-dependant pointer layout would have to be restored? After this step, the final representation should be available. 165 | 166 | More research is needed to confirm that this is what the preprocessing steps actually do. 167 | 168 | ### Compressed Section Buffer Structure 169 | 170 | The size is given by the ``compressedSize`` value of the respective section header. Sections are concatenated one after the other and can be accessed using the Section Reference directly. 171 | 172 | ### Decompressed Section Buffer Structure 173 | 174 | Once all sections are decompressed, their data can be read by following the Root Node Object offset stored in the Granny file header. What you'll find inside depends on the file and format used, which could be entirely arbitrary data. What's present in RO files is listed below. 175 | 176 | ## Contents 177 | 178 | While the above dealt with the Granny file format, the *actually* interesting parts are the contents of the individual data sections, i.e., the 3D models used for WOE Guardians, etc. 179 | 180 | The following deals with their structure and is assuming you've used one of the available tools to get at them in the first place, as you won't be able to see what's inside the compressed buffer areas otherwise. 181 | 182 | ## Decompressed File Structure 183 | 184 | ### Contents of the Data Tree 185 | 186 | The processed buffer area contains a hierarchy of objects used to represent the 3D model actor. In RO, each of the Granny files contains exactly **one** of the following: 187 | 188 | * An abstract 3D model object that acts as a container (root node) 189 | * A single skeleton, which is linked to the model, and consists of multiple bones 190 | * A single animation container that describes the motion of the bones for this skeleton 191 | 192 | The *base model* (as defined above) **additionally** contains shared assets for the model: 193 | 194 | * Several meshes that contain the geometry of the 3D model's "skin" 195 | * Compressed textures that are mapped to those meshes 196 | 197 | The secondary models don't contain all the data required to render the model. As such, I suspect they are loaded separately and combined to form a cohesive whole with all of the relevant poses for the given model actor, which can then be animated by setting the "active" skeleton to the one assigned to its current pose. Geometry and textures are shared. 198 | 199 | All files also contain *material* data, which describes the material (shader) settings used by the exporter (3DS Max) that the artist used to create the model. Some of the materials refer to textures (if present), but otherwise they're just shader properties that probably won't be useful... unless you want to import them in a modelling tool and replicate the exact rendition. 200 | 201 | The following table gives an overview of all the base and auxiliary pose models: 202 | 203 | | Model ID | Creature | Base Model (Idle Pose) | Attack Pose | Flinch Pose | Death Pose | Walk Pose | 204 | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | 205 | | 0 | Emperium | ``empelium90_0.gr2`` | --- | --- | --- | --- | 206 | | 1 | Guild Standard | ``guildflag90_1.gr2`` | ``1_attack.gr2`` (see below) | --- | --- | --- | 207 | | 2 | Treasure Box | ``treasurebox_2`` | --- | ``2_damage.gr2`` | ``2_dead.gr2`` | --- | 208 | | 7 | Knight Guardian | ``kguardian90_7.gr2`` | ``7_attack.gr2`` | ``7_damage.gr2`` | ``7_dead.gr2`` | ``7_move.gr2`` | 209 | | 8 | Bow Guardian |``aguardian90_8.gr2`` | ``8_attack.gr2`` | ``8_damage.gr2`` | ``8_dead.gr2`` | ``8_move.gr2`` | 210 | | 9 | Sword Guardian | ``sguardian90_9.gr2`` | ``9_attack.gr2`` | ``9_damage.gr2`` | ``9_dead.gr2`` | ``9_move.gr2`` | 211 | 212 | The astute observer will notice that the guild flag appears to have an attack pose. While that sounds hilarious in theory, the skeleton contained in it is clearly not compatible with the guild flag's model: It uses different bones and even includes textures, but no geometry. 213 | 214 | Going by the animation and texture names, it rather looks like a prototype of a player character (novice) remade as 3D model, though why it was left in the game files isn't clear. 215 | 216 | ### Models 217 | 218 | All models are assigned exactly one skeleton, and a transformation matrix that (I think) may describe its offset position in the object's local coordinate system. 219 | 220 | #### Guild Flags 221 | 222 | The guild flag model features a special "placeholder guild emblem" texture that is never displayed in the game. From the looks of it, this texture is simply replaced by the guild emblem assigned to it. If no emblem needs to be rendered, the placeholder remains hidden. 223 | 224 | Interestingly, the placeholder texture is only 16 pixels in size (both width and height), which makes it even smaller than the regular guild emblems (24 pixels in size). 225 | 226 | Compatible guild emblem textures can, for example, be created by decompressing [EBM](EBM.MD) files. 227 | 228 | ### Skeletons 229 | 230 | Each skeleton contains (usually) 32 bones and a unique identifier. That's it. 231 | 232 | Due to the extremely low bone count, they look pretty... *bare-bones* (ha...ha..): 233 | 234 | ![Images/GR2_BaseSkeleton.png](Images/GR2_BaseSkeleton.png) 235 | 236 | *Pictured: The actual skeleton of a [Sword Guardian](https://file5s.ratemyserver.net/mobs/1829.gif) unit* 237 | 238 | Skeletons aren't really a physical object that needs rendering; they mostly serve as a container for the hierarchy of transformation matrices represented by the bones. 239 | 240 | ### Bones 241 | 242 | Bones are another "virtual" object, serving solely as an abstraction for the transformations used to animate the visible geometry. The "skeleton" represented by them, defines a set of points ("bones") that move independently when animated, and by "linking" (binding) a mesh to these points it follows these movements, by applying the same transformations. 243 | 244 | The standard approach to skeletal animation is to "bind" the meshes on a per-vertex basis to any number of "bones", so that it's matrix describes how all connected vertices need to be transformed (for a given animation keyframe). There are usually multiple "influences" for each vertex, but in the case of RO each vertex is only influenced by exactly one bone. 245 | 246 | All bones have a similarly unique (for the skeleton, not across models) identifier. They're parented to another bone (or none if they're the root bone), and offset with a set of transformation matrices that describe the bone's position in object-local space. These are decomposed and stored as a separate matrix for translation, rotation and scaling. 247 | 248 | The scale matrix seems to only support mirroring the Y component (mirror alongside X axis) and no other scaling seems to be used, i.e. it's mostly an identity matrix except for a few cases where bones are mirrored and the ``1`` in the 2nd row becomes a ``-1`` instead (?). 249 | 250 | From what I can tell, this reflection alongside the Y axis has no visible impact, but since I'm unsure how the bones are supposed to be scaled that might just be my mistaken interpretation. Either way, it can't have much of an effect even if I did get it wrong. 251 | 252 | One last thing about the root bone... There's actually two of them: 253 | 254 | * A "synthetic" root bone is used to steer the model actor's movement and orientation 255 | * A "natural" root bone is parented to this and controls the actual hierarchy of bones 256 | 257 | In practice, this only means that one can move the entire model by changing the synthetic root bone, and separate the animation of the character's pose from this world-space movement. Since the existing animations are already designed in this manner, both bones can be treated just like any other part of the skeleton to compose the final transform. 258 | 259 | ### Spline Animation 260 | 261 | Animations are stored as curve data, represented by base splines (b-splines). 262 | 263 | This means they are given as a set of points (usually called *knots* or *knot values*) which represent time as points on the temporal axis, and multi-dimensional vectors called *control points* or *controls* that approximately describe the curve at a given point. 264 | 265 | In order to compute the transformation of each individual bone at a given point in time, the curve must be evaluated ("sampled"), which means setting the knot value to the desired time offset and computing the resulting control point. This is all handled by the Granny3D library in the RO client (or so I would assume), but it can be replicated via a keyframed animation that simply stores a sufficiently fine-grained sample as its keyframe and target value. 266 | 267 | Curves might also be multidimensional, with the degree referring to the dimension of the vectors used to represent its data. The animation vectors are again separated into the three base types of translation, rotation, and scale expressed as 3D vector to be added, 4D rotation quaternion, and a 3x3 scale/shear matrix, respectively. 268 | 269 | Any animated value overrides the bone's rest pose, i.e., the animation is not additive. 270 | 271 | ### Textures 272 | 273 | Each base model contains one diffuse texture per mesh. There's some other material data, added by 3DS Max (modeling tool that the authors used), but I don't think it's actually used in the game. Trying to make use of it didn't seem to have any visible effect in my testing. 274 | 275 | Auxiliary pose models have no textures, as they are combined with the base model in order to generate the final rendition and the base model already includes all of the geometry. 276 | 277 | ## See also 278 | 279 | There are various open source projects providing info on Granny files: 280 | 281 | * [https://github.com/arves100/Granny2-research/wiki/File-Format-Documentation](https://github.com/arves100/Granny2-research/wiki/File-Format-Documentation) 282 | * [https://github.com/arves100/opengr2](https://github.com/arves100/opengr2) 283 | * [https://github.com/Karbust/gr2decode](https://github.com/Karbust/gr2decode) 284 | * [https://github.com/herenow/gr2-web](https://github.com/herenow/gr2-web) 285 | * [https://github.com/Norbyte/lslib/](https://github.com/Norbyte/lslib/) 286 | * [https://github.com/SWTOR-Slicers/Granny2-Plug-In-Blender-2.8x](https://github.com/SWTOR-Slicers/Granny2-Plug-In-Blender-2.8x) 287 | * [https://github.com/Helia01/GrannyMeshDumper](https://github.com/Helia01/GrannyMeshDumper) 288 | * [https://github.com/SiENcE/Iris1_DeveloperTools/tree/master/GrannyViewer](https://github.com/SiENcE/Iris1_DeveloperTools/tree/master/GrannyViewer) 289 | 290 | Some information about Oodle (from the people at RAD) can be found here: 291 | 292 | * [http://www.radgametools.com/oodle.htm](http://www.radgametools.com/oodle.htm) 293 | * [http://www.radgametools.com/oodlecompressors.htm](http://www.radgametools.com/oodlecompressors.htm 294 | ) 295 | 296 | Lastly, here's a description of the synthetic root bone concept and how it's used: 297 | 298 | * [http://hewiki.heroengine.com/wiki/Synthetic_Root_Bone](http://hewiki.heroengine.com/wiki/Synthetic_Root_Bone) 299 | * [http://hewiki.heroengine.com/wiki/Character_Animation_Movement](http://hewiki.heroengine.com/wiki/Character_Animation_Movement) 300 | -------------------------------------------------------------------------------- /Images/ACT_EmptyLayersExample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/ACT_EmptyLayersExample.png -------------------------------------------------------------------------------- /Images/GAT_WaterTileFlags.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GAT_WaterTileFlags.png -------------------------------------------------------------------------------- /Images/GND_AntiBleedingBufferArea.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_AntiBleedingBufferArea.bmp -------------------------------------------------------------------------------- /Images/GND_ColorMapOriginal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_ColorMapOriginal.png -------------------------------------------------------------------------------- /Images/GND_ColorMapPosterized.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_ColorMapPosterized.png -------------------------------------------------------------------------------- /Images/GND_DynamicLightsEstimation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_DynamicLightsEstimation.png -------------------------------------------------------------------------------- /Images/GND_DynamicLightsOriginalPosition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_DynamicLightsOriginalPosition.png -------------------------------------------------------------------------------- /Images/GND_SurfacesVisualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_SurfacesVisualization.png -------------------------------------------------------------------------------- /Images/GND_UnwrappedColorMap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_UnwrappedColorMap.png -------------------------------------------------------------------------------- /Images/GND_UnwrappedShadowMap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_UnwrappedShadowMap.png -------------------------------------------------------------------------------- /Images/GND_VertexConnections_ORIGINAL.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_VertexConnections_ORIGINAL.png -------------------------------------------------------------------------------- /Images/GND_VertexConnections_RIGHT.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_VertexConnections_RIGHT.png -------------------------------------------------------------------------------- /Images/GND_VertexConnections_WRONG.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GND_VertexConnections_WRONG.png -------------------------------------------------------------------------------- /Images/GR2_BaseSkeleton.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GR2_BaseSkeleton.png -------------------------------------------------------------------------------- /Images/GR2_OodleString.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/GR2_OodleString.png -------------------------------------------------------------------------------- /Images/WindowsSystemPalette.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/WindowsSystemPalette.png -------------------------------------------------------------------------------- /Images/coordinatesCorrectionEN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/coordinatesCorrectionEN.png -------------------------------------------------------------------------------- /Images/interpretingHeightVectors.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Images/interpretingHeightVectors.png -------------------------------------------------------------------------------- /PAL.MD: -------------------------------------------------------------------------------- 1 | # PAL Format Specification 2 | 3 | ## Overview 4 | 5 | PAL files are simple color palettes used to share sets of related colors without having to redefine them repeatedly, and thus primarily serve to reduce memory (and disk) usage. For that same reason, the files don't include any header but rather are just a concatenated list of RGB values. 6 | 7 | ## Prerequisites 8 | 9 | In order to understand the PAL file format, some familiarity with the following concepts is required: 10 | 11 | * [Color palettes](https://en.wikipedia.org/wiki/Palette_(computing)), a simple lookup table for RGB(A) colors 12 | 13 | 14 | ## Transparency 15 | 16 | Transparency is determined for each color according to its RGBA values, after loading the palette: 17 | 18 | transparentColor = { alpha = 0, red = 10, green = 10, blue = 10 } 19 | 20 | if color.red >= 0xFE and color.green < 0x04 and color.blue >= 0xFE then 21 | color = transparentColor 22 | else 23 | color.alpha = 0xFF 24 | endif 25 | 26 | This hack may be explained by the fact that not all of the bitmaps in the client use the exact same background color (there's different shades of pink). 27 | 28 | ## Packed Palettes 29 | 30 | Only the leftmost 5 bit of each color value are actually stored, and the rest is truncated. The alpha component is stored as a single bit flag (transparent or not), as defined by the conversion algorithm above. 31 | 32 | The implications are clear (pun intended): Palette colors can't use opacity since the alpha value is discarded, so they're either fully visible or entirely transparent based on the color used. 33 | 34 | Additionally, similar colors will become identical if they differ only in the last 3 bit. I'm not sure if the difference is perceptible or if those color values are even used at all, but it's something to keep in mind as the client won't be able to render them correctly. 35 | 36 | ## Layout 37 | 38 | There's very little to say: 39 | 40 | * All palettes contain 256 colors 41 | * Colors are given in 8-bit ARGB format 42 | * The entire palette is therefore 1024 bytes in size 43 | 44 | Since there's no header or versions, that's about all there is to this format. -------------------------------------------------------------------------------- /README.MD: -------------------------------------------------------------------------------- 1 | **This repository will no longer see updates.** My work continues here: [Ragnarok Research Lab](https://github.com/RagnarokResearchLab/) 2 | 3 | To see the latest results of my research into Ragnarok Online's file formats, visit [the new documentation website](https://ragnarokresearchlab.github.io/file-formats/). 4 | 5 | Note: I'll archive the repo after the migration of all relevant contents has been completed. 6 | 7 | --- 8 | 9 | # Ragnarok Online File Formats Manifest 10 | 11 | Gravity Co has used some fairly arcane file formats in creating their first MMORPG client, presumably supported by custom (in-house) or now-obsolete software. 12 | 13 | This repository contains my findings from researching how they work and how they can be used or modified, alongside sources and links that at times have been quite difficult to track down after all these years. 14 | 15 | Hopefully having it all collected in a git repository will allow the information to remain available for many years to come. Feel free to do with this as you please! 16 | 17 | Obvious disclaimer: I don't claim ownership of any findings and results of the hard work that others have put into analysing these file formats. Sometimes it's impossible to say who originally discovered the various details due to the passage of time and obscure references I had to consult, but I'll list my sources as best as possible. 18 | 19 | ## Status 20 | 21 | I've analysed and created tools to work with most of the common file formats by now. However, there are many unknowns and smaller details that may still need clearing up in order to understand the formats perfectly. 22 | 23 | Often these are likely to be just oddities stemming from the fact that Ragnarok Online uses the old Arcturus engine, so they might not be required to process the files and render their contents correctly. 24 | 25 | Newer (post-Renewal) changes may not be reflected in this specification, though I expect to look into them more once the basics are covered. 26 | 27 | ## Contributing 28 | 29 | Please open an issue or contact me directly via Discord (RDW#9823) if you can contribute in any way! Hopefully, with some help from the community most of the unknowns can be eliminated eventually. 30 | 31 | ## File Types Overview 32 | 33 | All of the file types come in different versions, because *of course they do*. Usually, newer versions add more features while many of the older versions are virtually unused, being remnants from the early days of RO development and Gravity's previous project, [Arcturus: The Curse and Loss of Divinity](https://tcrf.net/Arcturus:_The_Curse_and_Loss_of_Divinity). 34 | 35 | They used a custom, in-house engine that was later extended to create Ragnarok Online. It's also most probably why RO's file formats and architecture seem pretty odd, by modern standards. 36 | 37 | However, since even the earliest RO clients don't support all of the legacy formats, they will be of limited interest. As existing tools aren't really compatible, researching them further isn't something I'm currently planning on. 38 | 39 | ### World Data (Map files) 40 | 41 | The most complex data stored and processed by the client is arguably the world and environment data. There are several files that are combined to create a representation of the world (maps) and its parameters, which however are not perfectly understood at this time. 42 | 43 | There exist several third-party programs that appear to recreate a faithful copy of the original client's rendering output, so the information available should be enough to interpret the original world without visible deficits. 44 | 45 | | Extension | Interpretation | Contents | 46 | | --------- | --------------------------- | -------------------------------------------------------------- | 47 | | GND | Ground | Map geometry, texture references, lightmap data | 48 | | GAT | Altitude and Terrain | Terrain altitude and type (ground, cliff, water) | 49 | | RSW | World Resources | 3D objects, water, lights, audio sources, effect emitters | 50 | 51 | ### Sprites (2D Actors) 52 | 53 | Most characters, monsters, and NPCs are displayed as 2D sprites with their various animations defined as "animation clips", which consist of multiple images. One clip exists for each of the different view angles, separating a full circle into just eight basic directions. 54 | 55 | Which direction should be used is determined by the client based on the camera position, i.e. the angle between the camera and the monster's position, and the assumed direction it is currently facing (I think). The latter is probably derived from its movement pattern, since I didn't find any serverside code that would manage this. 56 | 57 | The information on how to display a given unit's ingame actions is embedded in special collection files that contain additional information, such as sound effects to be played as part of the animation. 58 | 59 | Animations are composed of simple image files that will be played as individual frames, consisting of one or several layers that can be arranged to display multiple sprites together, in a predefined order, and with various visual changes applied to them (such as rotation, scaling, transparency, and color shading). 60 | 61 | | Extension | Interpretation | Contents | 62 | | --------- | -------------------- | --------------------------------------------------------------------- | 63 | | SPR | Spritesheet | Sprite images as bitmaps or TGA, metadata and, optionally, an RGB palette | 64 | | ACT | Animation Collection | Keyframes, animation details, and, optionally, a list of sound effects to be played | 65 | | IMF | Interface Management File | Information used to render sprites on the UI (2D) layer (seems obsolete?)| 66 | 67 | ### Models (3D Objects and Actors) 68 | 69 | Very few 3D models have been used for actual monsters, and the formats are somewhat unfamiliar to me. Most 3D models are those placed on the map while rendering and represent objects in the environment, like trees or architecture. 70 | 71 | Since the format for these appears to be custom or vastly outdated, I haven't found any standard tools to edit them, but people have written software to at least display the models and render them as part of a given map. 72 | 73 | | Extension | Interpretation | Contents | 74 | | --------- | -------------------- | --------------------------------------------------------------------- | 75 | | RSM | Model | 3D models used for objects (and sometimes units) in the game world | 76 | | GR2 | GRANNY2 | 3D models stored in an ancient format once used by the [Granny3D](https://radgametools.com/granny3d) framework | 77 | | RSX | ?| Arcturus 3D model format? (doesn't appear to be used in RO) | 78 | 79 | ### Effects 80 | 81 | Effects come in several varieties, all of which are processed separately. Some effects simply use the existing ACT/SPR formats, some use a separate compiled (binary) format, some are hardcoded into the client executable and in later versions particle effects can also be generated from settings stored in specific Lua files. 82 | 83 | The hardcoded effects appear to be limited to effects involving 3D components, such as particles and geometric primitive shapes, though they will often use sprites and other textures as part of their animation. Purely sprite-based animations are provided as separate files, which are mostly understood by now. 84 | 85 | Particles are configured using the "effect tool" and only seem to be used in recent years, with individual configuration files defining effect emitters for a given map. 86 | 87 | | Extension | Interpretation | Contents | 88 | | --------- | -------------------- | --------------------------------------------------------------------- | 89 | | STR | ? | Compiled (binary) effects format | 90 | | EZV | ? | Raw (text-based) effect format? | 91 | 92 | ### Script and configuration files 93 | 94 | Client-side data and Mercenary/Homunculus AI is written in the Lua programming language and embedded in the client itself, or sometimes as text-files with roughly-CSV-like structure (where fields are separated by hash tags). 95 | 96 | Lua files are human-readable text files and thus easily understood, even if the creators clearly didn't have readability in mind. LUB files are precompiled Lua files and can be somewhat reversed. This allows one to read much of them, though some information is still lost in the process. 97 | 98 | Thankfully, LUB files are only used in Post-Renewal clients (or so it seems), which means that all the classic files are perfectly readable. Converters exist to restore the original table structure from LUB files, though I cannot vouch for their correctness. 99 | 100 | | Extension | Interpretation | Contents | 101 | | --------- | -------------------- | --------------------------------------------------------------------- | 102 | | LUA | Lua source code | Lua script file | 103 | | LUB | Lua bytecode | Precompiled Lua script | 104 | 105 | ### Audio files (Music and Sound Effects) 106 | 107 | Finally, something easy! These are just regular WAV or MP3 files, respectively, and can be played with any standard program. 108 | 109 | ### Interface files 110 | 111 | They are regular images (bitmaps). Nothing to be said about them, other than that the Korean/Unicode filenames don't normally translate properly on non-Korean Windows systems and therefore can end up as gibberish. 112 | 113 | Only 256 colours are supported and the first palette index is used as transparent background colour. Usually this is the oldschool "pink" with RGB code (``#FF00FF``), though similar colours can also be occur: 114 | 115 | > It's safe to clear anything with red and blue > 0xf0 and green < 0x10 116 | 117 | Guild emblems are stored as compressed bitmaps, in the [EBM](EBM.MD) format. 118 | 119 | ## Community Resources 120 | 121 | See [here](CommunityProjects.md) for a list of editing tools and client reimplementation projects that may help with understanding the file formats described in this documentation. 122 | 123 | ## Endianness 124 | 125 | All numbers found in RO's binary formats are stored as [little-endian](https://en.wikipedia.org/wiki/Endianness), which means that the individual bytes have to be "reversed" before interpreting the binary data as a number. -------------------------------------------------------------------------------- /RSM.MD: -------------------------------------------------------------------------------- 1 | # RSM Format Specification 2 | 3 | RSM files contain data used to render 3D objects in the game world. They are mostly reserved for things like buildings, trees, and even hunter traps. 4 | 5 | ## Prerequisites 6 | 7 | In order to understand the RSM file format, some familiarity with the following concepts is required: 8 | 9 | * Basic [string representations](https://www.cs.yale.edu/homes/aspnes/pinewiki/C(2f)Strings.html) (both counted and delimited strings are used) 10 | * [Polygon meshes](https://en.wikipedia.org/wiki/Polygon_mesh), used to represent objects in 3D space 11 | * [Smoothing groups](https://en.wikipedia.org/wiki/Smoothing_group), a way to implicitly define surface normals 12 | * 3D keyframe animation, where animation keyframes are stored as matrices 13 | 14 | ## Overview 15 | 16 | The RSM format seems to be the standard format for 3D models used by Gravity, and it supports most of the features one would expect from a 3D format: Geometry, texturing information, and even a keyframe-based animation system. 17 | 18 | Meshes are defined as a hierarchy of nodes, with transformations being applied to the tree in a cascading fashion in order to compute the final position of each contained object (in local space). These meshes are then positioned according to the instancing properties defined in the RSW file for a given map (in world space). 19 | 20 | It should be noted that, while models can be animated, they aren't used to represent moving agents in the game world, like players or enemies, i.e., they can't "move". Their use lies exclusively in representing decorative scene objects ("props" or "doodads"). 21 | 22 | Nowadays, there are two versions of RSM files, with major differences and incompatibilities between them. The newer models all use the ``.rsm2`` file extension, while the original versions always use ``.rsm``. 23 | 24 | It also appears that the older versions have been (repeatedly) altered, without updating the version number. This leads to conflicting information and a general sense of confusion when it comes to the exact details of the various properties; a situation that, unfortunately, I've not managed to completely untangle so far. 25 | 26 | ## Nodes 27 | 28 | Each model contains (at least) one root node, and an arbitrary number of child nodes. The latter can be nested multiple times, and usually define smaller parts of a larger model. For example, the root node of a "windmill" model could be the base of the building, while the individual wheels would be defined in separate child nodes. 29 | 30 | Often, these will be smaller moving parts that are animated independently and need to be transformed correctly to form a cohesive whole. In the original version, only one such construction could be stored in the RSM file, defined by its root node, but it seems RSM2 files also support storing several independent root nodes at once. 31 | 32 | --- 33 | 34 | The rest of this article is unfinished :( Proceed at your own peril. 35 | 36 | ## Animations 37 | 38 | Seems like they can be chained = concatenated matrix multiplications (just basic Linear Algebra/transformations, I guess)? 39 | 40 | * Animation keyframes 41 | 42 | Not the same kinds of keyframes as used by 2D animations, but rather "snapshots" that are then interpolated? 43 | 44 | > During disassembling original client from end of 2016 i found that all the info about rsm format that i had found is wrong. It seems that koreans removed position animation from 1.5. Okay, i written their behavior, but i still has extra 8 bytes at the end of file and i cannot figure out are they unused in original client or not. There's no extra bytes in 1.4 for example. 45 | 46 | Verify this? See Aesir source and compare the various implementations/do some testing? 47 | 48 | ## Transparency 49 | 50 | > I've used BrowEdit to compare my results and found an issue that was the same as in my project. Some models had wrong depth writing. Using the correct order, it is possible to render objects regardless of their transparency. You can try to open BrowEdit and compare dicastes01 with this image. 51 | 52 | What image? Did I save it somewhere? Hmm. Not sure what's with the depth writing, research later? 53 | 54 | > In Arcturus you could interact with certain 3D models (e.g. doors, chests), so C3dActor was a part of the game object hierarchy. In Ragnarok, all 3D models are just decoration and not part of the game play and aren't considered game objects. (The exceptions to the rule are Granny actors and traps, the latter of which are actually RSM models drawn by a Skill object.) 55 | 56 | RSM models drawn by a Skill object? That sounds so crazy, it might just be true. 57 | 58 | > Yeah you have to calculate bounding boxes to display rsm correctly on the map. 59 | 60 | Will have to see, so far I've only rendered the terrain (GND/ground mesh). The bounding box wasn't really needed for that. 61 | 62 | ## Bounding Boxes 63 | 64 | * Information about the bounding boxes, used for collision detection 65 | 66 | 67 | > Looking at your issue, you might want to take a look at the code here: https://github.com/Doddler/RagnarokRebuild/blob/master/RebuildClient/Assets/Scripts/MapEditor/Editor/RagnarokModelLoader.cs 68 | 69 | > Around line 340 is where it's centered. The short of it is that you have to recalculate the bounds for the model and it's children and recenter it on load. I do it myself by calculating the bounds of the full model and then, from it's local position, subtracting the min of the bounds y axis and the center of the bounds x and z axis (assuming y is up). 70 | > 71 | *Source: Doddler* 72 | 73 | ## Duplicate faces 74 | 75 | > Some RSM files apparently have duplicate faces 76 | 77 | Do they, now? What a surprise (...not). How are they treated by the client? 78 | 79 | # Shading 80 | 81 | > The RSM now support is natural shadding type (without, flat, or smooth) instead of using smooth shadding on all models, result: better render, faster loading. 82 | 83 | ## Layout 84 | 85 | **TODO: Add tables for the different versions** 86 | 87 | 88 | 89 | ### Version 1.1 90 | 91 | This is [claimed to exist](https://github.com/flaviojs/eathena-devel-FlavioJS/blob/master/client/file_formats/rsm.txt), though I've never seen it. 92 | 93 | ### Version 1.2 94 | 95 | This is claimed to: 96 | 97 | * Add the color parameter to texture vertices 98 | * Add the smoothing group property to triangle faces 99 | 100 | Since there are no earlier versions to compare this one to, this can't be verified. 101 | 102 | ### Version 1.3 103 | 104 | This is said to alter the format of the bounding boxes, adding an unknown boolean flag. It's sensible to assume that this would affect the collision detection somehow, most likely either simplifying it (for performance reasons) or making it more accurate. 105 | 106 | ### Version 1.4 107 | 108 | This is said to add transparency (flavioJS), but Tokei writes that it isn't used ingame. 109 | 110 | ### Version 1.5 111 | 112 | Adds scale keyframes to each node (appears to be unused) 113 | 114 | ### Version 2.2 115 | 116 | * Animation duration is now given in frames per second 117 | * Each model can have multipe root nodes 118 | * Adds translation key frames to each node 119 | * Strings are stored with their length instead of fixed-length with a null terminator 120 | * The initial placement is stored as a transformation matrix and baked into the animation keyframes, instead of being computed from the individual transformation components 121 | 122 | ### Version 2.3 123 | 124 | * Textures are referrenced for each mesh, not globally 125 | * Adds texture animations to each node 126 | 127 | ## See also 128 | 129 | * [Tokei's explanation of the RSM2 changes](https://rathena.org/board/topic/127587-rsm2-file-format/) -------------------------------------------------------------------------------- /RSW.MD: -------------------------------------------------------------------------------- 1 | # RSW Format Specification 2 | 3 | ## Overview 4 | 5 | RSW files essentially define the rendered scene for a given map. They include information about: 6 | 7 | * The map's water plane (in older versions) 8 | * Light sources and global illumination 9 | * Environmental audio sources 10 | * 3D models (architecture) 11 | * Particle effect emitters 12 | * The boundaries of the map's coordinate system (?) 13 | * A scene graph-like representation of the object hierarchy (tree of bounding boxes) 14 | 15 | ## Prerequisites 16 | 17 | In order to understand the RSW file format, some familiarity with the following concepts is recommended: 18 | 19 | * [Bounding Boxes](https://en.wikipedia.org/wiki/Minimum_bounding_box), which are often used for collision detection and [frustrum culling](https://en.wikipedia.org/wiki/Hidden-surface_determination) 20 | * [Quad Trees](https://en.wikipedia.org/wiki/Quadtree#Region_quadtree), a data structure that is here used for the purposes of culling objects from the scene 21 | * [Particle systems](https://en.wikipedia.org/wiki/Particle_system), since particle emitters are frequently encountered in RSW files 22 | 23 | If you want to understand the water plane's wave animations, try these links: 24 | 25 | * You'll need a decent understanding of [waveforms](https://en.wikipedia.org/wiki/Waveform), specifically [sine curves](https://en.wikipedia.org/wiki/Sine_wave) and the [unit circle](https://en.wikipedia.org/wiki/Unit_circle#Trigonometric_functions_on_the_unit_circle) 26 | * Some other terms may also pop up in this article: [Phase shift](https://en.wikipedia.org/wiki/Phase_(waves)#Phase_shift), [amplitude](https://en.wikipedia.org/wiki/Amplitude), and [wavelength](https://en.wikipedia.org/wiki/Wavelength#Sinusoidal_waves) 27 | * It would certainly help to know about keyframe animations and sampling (can't find a good link) 28 | 29 | ## The Water Plane 30 | 31 | In the most commonly-used versions of the format, information about the map's water can be found. This is used to build a special "water plane" mesh, the vertices of which are animated according to a waveform curve in order to simulate waves, which are always tied to the ground mesh's geometry. 32 | 33 | It should be noted that there is no actual fluid simulation or wind effect, or anything resembling a typical physics engine being used in RO; the systems are far more primitive and probably hand-rolled. 34 | 35 | As of the latest version, this information is no longer present. It is instead found in the maps' GND file. 36 | 37 | There's also a secondary "texture cycling animation", which simply replaces the texture image for each water surface with the next one, similar to how sprites are animated by cycling through a texture atlas. 38 | 39 | ### Water Types 40 | 41 | Each map has exactly one assigned "water type". These simply define what texture to use and provide the prefix to the water texture's file name, which is appended with the animation frame ID to determine which of the individual frames to display, while cycling through all 32 frames. 42 | 43 | Not all of them are obviously water; there's also some "mud" and "lava" textures that use the same mechanism (described below) to animate. 44 | 45 | ### Texture Cycling 46 | 47 | A moving water surface is simulated by increasing a cycling counter each frame and updating the water mesh's texture after it has been displayed for a sufficient duration, with texture indices cycling from 0 to 31. This number appears to be fixed, as all water types have exactly 32 frames. 48 | 49 | A full cycle then takes ``3 * 32 = 96 frames / (60 frames / second) = 1.6 seconds`` at the default cycling speed of ``3``, which is also set in the map's RSW file, assuming a fixed framerate of 60 FPS (which is clearly what they were counting on). I'm guessing this all would break down if the FPS drop. 50 | 51 | ### Ground Tiles and the Water Plane 52 | 53 | The water plane itself appears to not cover the entire map area, as one would expect from a naive approach. Instead, each visible ground tile (GND cube's TOP surface) is assigned a "water vertex", which is the position of the water plane for this corner of the tile. These vertices effectively form the water plane described above; which isn't necessarily a continuous surface. 54 | 55 | The X and Z coordinates are always pinned to the tile's X and Z coordinates, while the Y coordinate is used to define the water level at this corner of the tile. If the water is still, all four corners will be at the same height, which is exactly the water level as defined in the map's RSW file. 56 | 57 | ### Wave Cycles 58 | 59 | If there is a wave effect, the various wave parameters will be used to simulate physical waves in the following manner: 60 | 61 | * The "wind" is thought to always blow in direction (1,1), meaning from bottom left to top right (from southwest to northeast), in map coordinates, i.e. (x, z) 62 | * This means that the wave will hit the ground tile in this same direction, which is "faked" by setting the top left and bottom right corners to the same height and the bottom left and top right to a different one, each corresponding to the wave height at the respective time in the wave cycle 63 | * Calculate the height of the wave [crest and trough](https://en.wikipedia.org/wiki/Crest_and_trough) and map them to the tile's corners 64 | * What I call the "left" trough is [phase-shifted](https://upload.wikimedia.org/wikipedia/commons/9/92/Phase_shifter_using_IQ_modulator.gif) to the left of the crest, the "right" trough to the right 65 | * As the frames iterate through the wave cycle, the vertex heights are modified to display the water mesh's vertices in a "wavy" pattern 66 | 67 | ### Animation Sampling Basics 68 | 69 | This simulation updates every frame (so ideally, 60 times per second?). It is not triggered via a timer, so variances in frame rate should affect the speed of wave animations greatly. 70 | 71 | Each update will increase the "time" of the wave cycle by a small, fixed amount, until the full cycle has ran its course and is reset. Subsequently, the "water vertices" are updated, altering their height at each visible tile only. 72 | 73 | What tiles are visible appears to be determined by the same mechanism used to cull objects from the scene, by utilizing the quad tree of regions and their bounding boxes, as defined in the GND file. 74 | 75 | ### Waveform Sampling Algorithm 76 | 77 | The computation itself is a bit involved, but here is essentially how it works: 78 | 79 | 1. Compute the wave sampling offset ("animation frame index") 80 | 2. Compute the relative offsets of the wave troughs from the crest 81 | 3. Map the troughs and crest to the corners of each ground tile, accounting for its map position 82 | 4. Sample a sine curve at this offset to calculate the surface height at the crest and troughs 83 | 5. Scale the offsets and account for the plane's water level to determine the final height 84 | 85 | To help visualize this, one can imagine a repeating wave pattern moving over the water surface, in the implied wind direction of (1, 1), i.e. north-east, with each trough and crest being exactly one tile apart: 86 | 87 | ![Assets/Images/RSW_WaveTroughsAndCrest.png](Assets/Images/RSW_WaveTroughsAndCrest.png) 88 | 89 | *Pictured: A testament to my incredible graphs editing skills (waves advance in direction north-east)* 90 | 91 | The tiles are implied and form a single GND surface (4x4 GAT tiles): 92 | 93 | ![Assets/Images/RSW_WaveTroughsAndCrestWithGrid.png](Assets/Images/RSW_WaveTroughsAndCrestWithGrid.png) 94 | 95 | As the imagined wind blows (the wave animation cycle proceeds), the troughs and crests of the waves (red and orange lines, respectively) move towards the north-east and the water vertices are deformed at the tile corners according to the sampled height value. You'll have to imagine the waves in my illustration since this is all terribly low-budget, but I'm sure you'll see the pattern ingame this way. 96 | 97 | If that wasn't exactly clear, and I'm almost certain it wasn't, I recommend checking out some ingame maps and observing the water plane's movement as it interfaces with the landmass. 98 | 99 | In excruciating detail, the algorithm would look something like this (not sure if this actually helps...): 100 | 101 | * Compute the wave cycle offset, as coordinates on the unit circle (in degrees) 102 | * Starting at zero, compute the next sampling offset for the wave crest by adding the phase offset 103 | * This happens once per frame (60 times per second?); imagine a clock ticking once per frame 104 | * With each "tick" of the clock, the offset moves around the unit circle 105 | * Whenever the animation loops around (reaches 180 degree), simply reset to -180 degree 106 | * Offset this by the position (in map coordinates) of the GAT tile the water vertex is assigned to 107 | * Additionally, phase-shift the trough offsets to move them left/right of the crest 108 | * This can be imagined as two additional hands of the clock (left and right of the original one) 109 | * Modulate a sine wave with the parameters from the RSW file and the above cycle offsets 110 | * This is again computed separately for each of the four corners of the GAT tile 111 | * During the above step, the crest is mapped to the bottom-right and top-left corners 112 | * The left trough is mapped to the bottom-left corner, with an offset of -1 113 | * The right trough is mapped to the top-right corner, with an offset of 1 114 | * Lastly, the final phase shift is obtained by factoring in the surface curvature (from the RSW) 115 | * The curvature angle therefore acts as an amplifier to increase the phase shift (if nonzero) 116 | * If you now "unwrap" the unit circle into a line from -180 to 180, take the offsets as x-values 117 | * Multiply in the phase shift, add the tile position offsets and sample a standard sine curve 118 | * FInally, multiply by the amplitude parameter (scaling factor) to determine the wave's height 119 | * The resulting height values are the final offsets of the wave's crest and both troughs at the corner 120 | * Those sampled height offsets are added to the water plane's Y-offset (the baseline water level) 121 | 122 | All of this only applies to maps that actually have a wave animations. When the waveform amplitude (local rate of change per sampled interval) is zero, there is no animation and the water plane's surface remains still, with the height simply being the water level (wave offset is zero). 123 | 124 | ### Waveform Visualization 125 | 126 | Since this is all very difficult to understand without visualizing the waveforms and trying out different values, I've prepared a spreadsheet you can play around with: [Download it here](Assets/WaveCycleVisualizationSheet.xlsx) and give it a try! 127 | 128 | PS: Requires MS Excel :( I didn't think to use Google Sheets or LibreOffice in time, so here we are. 129 | 130 | *Disclaimer: I made this only for myself and didn't originally intend to publish it, but then figured I might as well, so don't expect too much. Also, yes, I know git wasn't made to store binary files, but this repository shouldn't be missing anything of relevance so I added it anyway.* 131 | 132 | To help illustrate the effect of adjusting the various waveform parameters and their impact on the computed water surface over time, here's some graphs (made with the above spreadsheet): 133 | 134 | ![Assets/Images/RSW_WaveOffsets_alberta.png](Assets/Images/RSW_WaveOffsets_alberta.png) 135 | *Pictured: Surface curvature over time for ``alberta.rsw`` (medium-sized waves are repeating every 2 seconds)* 136 | 137 | ![Assets/Images/RSW_WaveOffsets_comodo.png](Assets/Images/RSW_WaveOffsets_comodo.png) 138 | *Pictured: Surface curvature over time for ``comodo.rsw`` (large waves are cycling every 3 seconds)* 139 | 140 | ![Assets/Images/RSW_WaveOffsets_aldebaran.png](Assets/Images/RSW_WaveOffsets_aldebaran.png) 141 | *Pictured: Surface curvature over time for ``aldebaran.rsw`` (the water barely moves; each cycle takes 6 seconds)* 142 | 143 | It should be noted that the above graphs depict *only* the curvature, and not the final offset (height of the water plane's vertices). It is used as an input in the final step of the computation, as described above, but the water surface closely mirrors the sine curve as it's only scaled by the amplitude parameter afterwards. 144 | 145 | ### Renewal Water Changes 146 | 147 | As mentioned, in the latest (Renewal-only) RSW and GND versions there have been some significant changes. For details, check out the [GND specification](GND.MD), which includes my findings on the matter. 148 | 149 | In terms of wave cycles, I don't believe the fundamentals have changed. I'm assuming the wave animation mechanism simply extends to each of the water planes assigned to a map, i.e., there would be several animations running in parallel (if they aren't all identical). 150 | 151 | All of this still requires further research as it can't easily be verified without access to the kRO servers. 152 | 153 | ### Transparency 154 | 155 | Since the terrain beneath the water plane is always shining through, I'm guessing there's some transparency being used for the water planes. It's hard to tell how it works exactly, but my "best guess" estimate based on manual testing is that it could be around a 50% opacity (alpha value of ``0.5``): 156 | 157 | ![Assets/Images/RSW_WaterPlaneAlpha.png](Assets/Images/RSW_WaterPlaneAlpha.png) 158 | 159 | *Pictured: Overlaying water textures with zero and 50% transparency, respectively* 160 | 161 | While this seems like a "good enough" estimate, it's been claimed the actual transparency value used is `` 144 / 255 = 0.564705882`` (Source: Borf). 162 | 163 | ### Environmental Light Influences 164 | 165 | It looks like the environmental light may also influence the color of the water plane somehow. This is mostly noticeable when it's very dark, such as in the case of ``mag_dun01``: The "water" (lava) texture is quite bright, but the ingame rendition looks rather dim - in alignment with the rest of the scene. 166 | 167 | If there was indeed no such influence, the texture should be as bright as the textured ground mesh surfaces, where in reality it's similar in color to the "walls" and shadows (determined by ambient light). 168 | 169 | Therefore I'm (again, guessing here) that the ambient color of the scene light might be factored into the water plane's diffuse color, though more testing needs to be done to verify (or disprove) this. 170 | 171 | It's quite possible this is just another artifact of them applying some sort of filter. As the colors of all third-party-renditions I've seen look noticeably off, it has been postulated that there's some magnifying (or dampening) of the lighting, e.g. a bloom filter or some other post-processing effects. 172 | 173 | ### Texture Dimensions 174 | 175 | Another piece of evidence that suggests there is special handling for the "lava"-type water textures is the fact they are the only textures that are larger than the standard size of 128x128 pixels. 176 | 177 | While the regular water textures are mapped to a single GND surface (4x4 world units, at 32px each, for a total water segment size of 128x128 pixels), water of type 4 and 6 uses textures of size 256x256 pixels that are seemingly mapped to four GND surfaces (16x16 world units, at 32px each). 178 | 179 | ## Bounding Areas and Culling 180 | 181 | All scene objects are rendered (and culled) based on their position in the world. In order to speed up the computations required to determine whether an object is visible and must be rendered, a quad tree is used to store the bounding boxes for the given region. 182 | 183 | You can find an explanation [here](https://herc.ws/board/topic/9409-high-level-mappings-question-quadtree/#comment-55523). I believe this is already sufficient detail, so I won't repeat it all in this document. The only thing worth mentioning is that the four regions of the quad tree correspond to the quarters of the contained area in the usual order: 184 | 185 | 1. Bottom-left (southwest) region 186 | 2. Bottom-right (southeast) region 187 | 3. Top-left (northwest) region 188 | 4. Top-right (northeast) region 189 | 190 | To round it off, here's the bounding boxes (contained in the quad tree) of ``niflheim.rsw`` visualized: 191 | 192 | ![Assets/Images/RSW_QuadTreeVisualization.png](Assets/Images/RSW_QuadTreeVisualization.png) 193 | 194 | As you can see, all the scene objects (most importantly, the models) fit neatly into those areas. In this screenshot, only the leaf nodes (smallest regions) are rendered since it would otherwise quickly become impossible to see what's going on. 195 | 196 | Just keep in mind there are more regions than that in each quad tree, starting from the root node (encompasses the entire map) with the next level splitting the map into four areas, then sixteen, etcetera. 197 | 198 | This information is still of practical relevance in modern engines: There are some maps, like ``amatsu.rsw`` or ``gef_fild07.rsw``, where models have been placed outside of the boundaries of the top-level region. These have to be removed from the scene, because they look completely out of place. 199 | -------------------------------------------------------------------------------- /SPR.MD: -------------------------------------------------------------------------------- 1 | # SPR Format Specification 2 | 3 | ## Overview 4 | 5 | SPR files are a compiled (binary) form of a sprite sheet, used to store all images used to represent an ingame entity in a single file. As such, they're essentially a texture atlas and can be represented similarly after the individual images (frames) have been extracted. 6 | 7 | They encode the following information: 8 | 9 | * Pixel data (for one or several images) 10 | * Frame definitions 11 | * Palette information 12 | 13 | ## Prerequisites 14 | 15 | In order to understand the SPR file format, some familiarity with the following concepts is required: 16 | 17 | * [RGB](https://en.wikipedia.org/wiki/RGB_color_model) and [alpha channels](https://en.wikipedia.org/wiki/Alpha_compositing): A standard way to define colors and transparency 18 | * Sprite sheets (see [Texture Atlas](https://en.wikipedia.org/wiki/Texture_atlas)): A straightforward method to combine textures in a reversible way 19 | * [Palettes](https://en.wikipedia.org/wiki/Palette_(computing)), [bitmaps](https://en.wikipedia.org/wiki/BMP_file_format), and [indexed colors](https://en.wikipedia.org/wiki/Indexed_color): By using a lookup table, less space is used to store pixels 20 | * [Run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding): This basic lossless compression algorithm serves to reduce the image's file size 21 | 22 | ## Frames 23 | 24 | Each frame simply defines the pixel data, width, and height of a bitmap. The way these are defined differs, but at the end of the day the result is always an image that can be used to render a sprite in the game. 25 | 26 | **Indexed** frames use the color palette that is appended at the end of the file, with each pixel index referring to one of the 256 colors instead of the a RGB(A) value that represents the pixel color (and opacity). Old versions appear to use a predefined "system palette" instead, though more information is needed on the subject. 27 | 28 | **RGB** frames instead define the pixels as RGBA values. Transparency is only supported starting from version 2.0, with previous versions ignoring the alpha value. These frames always appear after the indexed color images (if any exist) and it is common to see very few to none of those, which is probably due to them taking up more space and only being required for transparency effects. 29 | 30 | ## Palettes 31 | 32 | All palettes contain 256 colors and are defined as RGBA values, i.e., 4 bytes per color for a total of 1024 bytes that are located at the end of the file. They're referred by the indexed-color pixels and serve as a Color Lookup Table (CLUT) when generating the actual images. The first color (at index 0) is set as the transparency color, i.e. it won't be shown in the game. 33 | 34 | It's not quite clear what *exact* palette SPR files of version 1.0 use, since they don't include the actual palette colors and are said to use the "system palette" instead. From my limited research, this most likely to refers to the [Windows 256 color standard palette](Images/WindowsSystemPalette.png) that was commonly used at the time of RO's development. 35 | 36 | Disclaimer: Since I haven't seen this version used anywhere, I can't guarantee that those colors are indeed accurate. 37 | 38 | ## Compression of similar (transparent) pixels 39 | 40 | In the later versions, indexed-color pixels that represent the invisible background are compressed using Run-Length Encoding (see above). This doesn't apply to the RGBA frames, however, which are not compressed. 41 | 42 | This simply means that all "runs" of multiple (more than two) zero bytes in the indexed-color frames are replaced with ``00 ??`` where ``??`` is the number of zeroes replaced (two zero bytes result in ``00``, i.e., a single zero byte). Decompressing then simply removes ``??`` and replaces it with the respective number of zero bytes again. 43 | 44 | Since it is almost guaranteed that many "runs" of the same color will appear in the transparent background and rarely, if ever, in the actual sprites, applying RLE to the remaining data wouldn't be helpful in reducing the file size (in fact, it would make it larger as each one-byte indexed-color pixel would be replaced by *two* bytes). 45 | 46 | > When a 0x00 byte is parsed, the next byte indicates the number of 0x00 bytes it decompresses to. (0x00 0x00 decompresses to a single 0x00 byte) 47 | 48 | ## Layout 49 | 50 | The file structure differs slightly with each file version, as newer versions added more features. Please consult the tables below for a detailed specification. 51 | 52 | It appears that the header encodes the version in reverse, i.e. ``53 50 01 02`` reads as ``SP 1.2`` but it's actually version 2.1 and not 1.2. 53 | 54 | ### Version 1.0 55 | 56 | In its oldest and most primitive form, a SPR file includes only the very basics required to display an image and the definition for the spritesheet's frames. 57 | 58 | | Field | Offset | Size | Type | Description and notes | 59 | | :-----------------: | :----: | :------: | :----: | :--------------------------------------------------------: | 60 | | Header | 0 | 2 bytes | char | "SP" as an ASCII-encoded string | 61 | | Version | 2 | 2 bytes | binary | Versioning information (Minor.Major) | 62 | | Indexed Frame Count | 4 | 2 bytes | int | The number of individual indexed-color images in the atlas | 63 | | Bitmaps | 6+ | variable | struct | The size depends on the frame count and number of pixels | 64 | 65 | I haven't seen this version myself, but it seems likely it would've been used in the early versions of the game; or perhaps by its predecessor Arcturus, which used the same engine. 66 | 67 | ### Version 1.1 68 | 69 | Rather than using the "system palette" (see above), a 256-color palette consisting of RGBA values is added to the end of the SPR file itself. 70 | 71 | | Field | Offset | Size | Type | Description and notes | 72 | | :-----------------: | :------: | :------: | :----: | :--------------------------------------------------------: | 73 | | Header | 0 | 2 bytes | char | "SP" as an ASCII-encoded string | 74 | | Version | 2 | 2 bytes | binary | Versioning information (Minor.Major) | 75 | | Indexed Frame Count | 4 | 2 bytes | int | The number of individual indexed-color images in the atlas | 76 | | Indexed Bitmaps | 6+ | variable | struct | The size depends on the frame count and number of pixels | 77 | | Palette | EOF-1024 | variable | struct | Listed in order ABGR, defines the CLUT (see above) | 78 | 79 | I have yet to see this version myself, so I can't confirm these details. 80 | 81 | ### Version 2.0 82 | 83 | This version introduces images with transparency, which are included as separate RGBA frames defining each pixel and no longer refer to the palette. 84 | 85 | | Field | Offset | Size | Type | Description and notes | 86 | | :-----------------: | :------: | :------: | :----: | :--------------------------------------------------------: | 87 | | Header | 0 | 2 bytes | char | "SP" as an ASCII-encoded string | 88 | | Version | 2 | 2 bytes | binary | Versioning information (Minor.Major) | 89 | | Indexed Frame Count | 4 | 2 bytes | int | The number of individual indexed-color images in the atlas | 90 | | RGBA Frame Count | 6 | 2 bytes | int | The number of individual RGBA images in the atlas | 91 | | Indexed Bitmaps | 8+ | variable | struct | The size depends on the frame count and number of pixels | 92 | | RGBA Bitmaps | 8+ | variable | struct | The size depends on the frame count and number of pixels | 93 | | Palette | EOF-1024 | variable | struct | Listed in order ABGR, defines the CLUT (see above) | 94 | 95 | I have yet to see this version myself, so I can't confirm these details. 96 | 97 | ### Version 2.1 98 | 99 | Similar to version 2.0, but now all background pixels of the indexed-color bitmaps (i.e., those using the palette color with index 0) are compressed using RLE. 100 | 101 | | Field | Offset | Size | Type | Description and notes | 102 | | :----------------------: | :------: | :------: | :----: | :--------------------------------------------------------: | 103 | | Header | 0 | 2 bytes | char | "SP" as an ASCII-encoded string | 104 | | Version | 2 | 2 bytes | binary | Versioning information (Minor.Major) | 105 | | Indexed Frame Count | 4 | 2 bytes | int | The number of individual indexed-color images in the atlas | 106 | | RGBA Frame Count | 6 | 2 bytes | int | The number of individual RGBA images in the atlas | 107 | | RLE-Encoded Indexed Bitmaps | 8+ | variable | struct | The size depends on the frame count and number of pixels | 108 | | RGBA Bitmaps | 8+ | variable | struct | The size depends on the frame count and number of pixels | 109 | | Palette | EOF-1024 | variable | struct | Listed in order ABGR, defines the CLUT (see above) | 110 | 111 | ## Tools 112 | 113 | There are a variety of tools available to parse SPR files and obtain the image data in its raw form: 114 | 115 | * [Tokei's GRF Viewer](https://rathena.org/board/topic/77080-grf-grf-editor): Allows opening of SPR/ACT even from the GRF. More importantly, it has great visualization for sprites and animations 116 | * [actOR2](https://ratemyserver.net/index.php?page=download_tool): Oldschool executable tool that allows editing SPR (and the associated ACT) files 117 | * [SprViewer](https://ratemyserver.net/index.php?page=download_tool): Minimalist oldschool executable that extracts SPR data, but doesn't support ACT 118 | 119 | After converting the binary data back to the original (bitmap) form, any image editing software can be used to modify them. If your goal is to create spritesheets, there's a few options that should work well enough: 120 | 121 | * [TexturePacker](https://www.codeandweb.com/texturepacker): By far the most-feature rich and easy-to-use offline tool, but it's very limited in its free edition. 122 | * There's also an online tool supporting the same format: https://www.codeandweb.com/free-sprite-sheet-packer -------------------------------------------------------------------------------- /STR.MD: -------------------------------------------------------------------------------- 1 | # STR Format Specification 2 | 3 | ## Overview 4 | 5 | STR files are a compiled (binary) form of animated, image-based effects used to display certain kinds of 2D effects in the game world. According to the author of RoBrowser, they are simply EZV compiled to binary (**TODO: Verify**). 6 | 7 | In conjunction with hardcoded 3D effects that use geometrical primitives and Direct3D textures directly, as well as regular sprites (stored in SPR/ACT files), different types of effects can be realized. 8 | 9 | They encode the following information: 10 | 11 | * References to texture files (bitmaps) 12 | * Data used to animate each layer 13 | * Animation properties applicable to all textures 14 | 15 | ## Prerequisites 16 | 17 | In order to understand the STR file format, some familiarity with the following concepts is required: 18 | 19 | * It's recommended to read the [SPR](SPR.MD) and [ACT](ACT.MD) specifications first, as sprite animations share many key concepts with STR effects 20 | * [Keyframe animation](https://en.wikipedia.org/wiki/Key_frame) is essentially what's being used by both sprites and effects, in a slightly different fashion 21 | 22 | ## Effects vs. Sprites 23 | 24 | While storing effects in a separate file format seems handy, they differ from SPR and ACT files (that can also be abused to create effects) in that they don't contain the actual image data. Instead, they reference files locally, i.e. inside the ``data/texture/effect`` folder. This is where both the STR files and the bitmaps they use are stored. 25 | 26 | From a technical POV, it appears that sprites are simply meshes rendered in billboard mode (always facing the camera) and then are modified to account for camera perspective, while STR effects are rendered either as projected textures (for AOE/ground effects) or similarly as "actors" in the game world but without the complex logic required to handle look/face directions. More research is needed to confirm this, however. 27 | 28 | ## Layout 29 | 30 | The file structure differs slightly with each file version, as newer versions added more features. Please consult the tables below for a detailed specification. 31 | 32 | **TODO: Note the version byte order** 33 | 34 | **TODO: Layout for the different versions** 35 | 36 | ## Tools 37 | 38 | There are only a few tools available to parse STR files, as far as I know: 39 | 40 | * [RO STR Viewer](https://herc.ws/board/topic/17268-rostrviewer-reupload): Looks a bit dated, but contains all the information as far as I can tell 41 | * https://github.com/skardach/ro-str-viewer: I actually didn't care enough to make it run, but an open-source reference implementation is always nice to have 42 | -------------------------------------------------------------------------------- /Videos/ACT_AnchorPointsVisualization.mkv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rdw-archive/RagnarokFileFormats/986c100346217a26eb581d4f66244db5569ef78b/Videos/ACT_AnchorPointsVisualization.mkv --------------------------------------------------------------------------------