Merge READMEs into the root README

Bring the wpak documentation into the root README and remove the
duplicate copy under mbEditorPro2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,7 @@
+.idea/
+cachefiles/
+__pycache__/
+*.wpak
+*.wpak.*
+*.old
+ARCANE_DUMP/

README.md

@@ -18,6 +18,7 @@ Team Magicbane presents to the community technology allowing MagicBox administra
 to fully modify their game using nothing more than notepad.exe. <br>
 
 - Tool directly converts Shadowbane .cache file data to and from a high fidelity pure JSON representation.
+- Shadowbane .wpak archives are decrypted to and from their plain text config files.
 - Media such as textures and sound are converted in-situ.
 - Enumerations are used throughout hiding from the user ugly signed ulong values.
 - Written in Python and open-source allowing modification and extension with ease.
@@ -49,6 +50,20 @@ mbEditorPro.py pack all
 New binary cache file(s) will be built from the JSON and output into the current directory. <br>
 Pack only reads from the un-numbered folders (e.g. COBJECTS). If a cache file already exists, a numbered copy is written instead (e.g. CObjects.cache.1). <br>
 
+### wpak files
+
+```
+mbEditorPro.py unpack wpak
+mbEditorPro.py unpack wpak Config
+mbEditorPro.py pack wpak
+mbEditorPro.py pack wpak Config
+```
+
+A .wpak is a zip archive whose entries are each Blowfish encrypted (.tga icons are stored unencrypted).
+Unpack decrypts each pak into **ARCANE_DUMP/<name>** (e.g. ARCANE_DUMP/Config), one file per entry.
+Pack encrypts each folder back into <name>.wpak. Name a single pak to work on just that one.
+The same numbering rules as the caches apply. <br>
+
 ### Limitations
 
 Hirelings, ZoneEvents and Mobile Inventory are not parsed or written as there were no <br>
@@ -65,10 +80,9 @@ The tool generating pure JSON has the following benefits:
 
 ### Future
 
-mbEditorPro will be the cornerstone of Magicbane's data project. The Team's goals are threefold:
+mbEditorPro will be the cornerstone of Magicbane's data project. The Team's goals are twofold:
 
 - Ship MagicBox 1.5 with both Aerynth and Vorringia mapsets.
-- Integrate Magicbane's wpak extractor technology into the tool.
 - Use data for a detailed audit of Magicbane's database.<br>
 
 ### More Information

mbEditorPro2.0/README.md

@@ -1,77 +0,0 @@
-```
- • ▌ ▄ ·. ▄▄▄▄·     ▄▄▄ .·▄▄▄▄  ▪  ▄▄▄▄▄      ▄▄▄       ▄▄▄·▄▄▄        
- ·██ ▐███▪▐█ ▀█▪    ▀▄.▀·██▪ ██ ██ •██  ▪     ▀▄ █·    ▐█ ▄█▀▄ █·▪     
- ▐█ ▌▐▌▐█·▐█▀▀█▄    ▐▀▀▪▄▐█· ▐█▌▐█· ▐█.▪ ▄█▀▄ ▐▀▀▄      ██▀·▐▀▀▄  ▄█▀▄ 
- ██ ██▌▐█▌██▄▪▐█    ▐█▄▄▌██. ██ ▐█▌ ▐█▌·▐█▌.▐▌▐█•█▌    ▐█▪·•▐█•█▌▐█▌.▐▌
- ▀▀  █▪▀▀▀·▀▀▀▀      ▀▀▀ ▀▀▀▀▀• ▀▀▀ ▀▀▀  ▀█▄▀▪.▀  ▀    .▀   .▀  ▀ ▀█▄▀▪
-               Magicbane Emulator Project © 2013 - 2022
-                          www.magicbane.com
-```
-
-# mbEditorPro v 2.0
-
-![Example JSON](https://cdn.discordapp.com/attachments/399623779531096074/1029063849640468550/unknown.png)<br>
-![Example Textures](https://cdn.discordapp.com/attachments/534260038039109632/1056584030310641696/image.png)<br>
-![Example Usage](https://cdn.discordapp.com/attachments/398522378545790976/1058787471544025138/image.png)<br>
-
-Team Magicbane presents to the community technology allowing MagicBox administrators<br>
-to fully modify their game using nothing more than notepad.exe. <br>
-
-- Tool directly converts Shadowbane .cache file data to and from a high fidelity pure JSON representation.
-- Media such as textures and sound are converted in-situ.
-- Enumerations are used throughout hiding from the user ugly signed ulong values.
-- Written in Python and open-source allowing modification and extension with ease.
-- Want to raise all weapon damage by x%; Just modify item values as they are being parsed!
-
-The tool built with Eldritch Technologies parses **all Shadowbane .cache files** giving the user complete control over the game<br>
-
-Team Magicbane aims to deliver you the best crackhead free MMO technology around; all for free!
-
-### Requirements:
-
-mbEditorPro requires pillow for image manipulation and tqdm for metrics.
-
-```$ pip install -r requirements.txt```
-
-### Usage:
-```
-mbEditorPro.py unpack cobjects
-mbEditorPro.py unpack all
-```
-JSON output will be placed into the folder **ARCANE_DUMP** <br>
-Unpack only reads the plain-named cache files (e.g. CObjects.cache). If a target folder already exists, a numbered folder is created instead (e.g. COBJECTS.1) so earlier output is never overwritten. <br>
-
-```
-mbEditorPro.py pack textures
-mbEditorPro.py pack all
-```
-
-New binary cache file(s) will be built from the JSON and output into the current directory. <br>
-Pack only reads from the un-numbered folders (e.g. COBJECTS). If a cache file already exists, a numbered copy is written instead (e.g. CObjects.cache.1). <br>
-
-### Limitations
-
-Hirelings, ZoneEvents and Mobile Inventory are not parsed or written as there were no <br>
-examples in the .cache files to begin with.  The game does seem to support them, however.
-
-### JSON
-
-The tool generating pure JSON has the following benefits:
-
-- It's ubiquitous and the tooling is mature.  The standard interop exchange format.
-- Human and machine-readable.
-- Ease of importing into MySQL or other databases.
-- Use as an intermediate representation to build a GUI World Editor or FBX importer.
-
-### Future
-
-mbEditorPro will be the cornerstone of Magicbane's data project. The Team's goals are threefold:
-
-- Ship MagicBox 1.5 with both Aerynth and Vorringia mapsets.
-- Integrate Magicbane's wpak extractor technology into the tool.
-- Use data for a detailed audit of Magicbane's database.<br>
-
-### More Information
-
-See the mbEditorPro2.0 Wiki for more detailed information.
-

mbEditorPro2.0/arcane/enums/arc_item.py

@@ -218,12 +218,13 @@ STRING_TO_ITEM_TYPE = {
     'REALMCHARTER': ITEM_TYPE_REALMCHARTER,
 }
 
-ITEM_USE_FLAGS_NONE = 0
-ITEM_USE_FLAGS_USE_SELF = 1 << 0
-ITEM_USE_FLAGS_USE_TARGET = 1 << 1
-ITEM_USE_FLAGS_EMPTY_DESTROY = 1 << 2
-ITEM_USE_FLAGS_EMPTY_NEWITEM = 1 << 3
-ITEM_USE_FLAGS_NUM = 4
+# Bit order from the client's use flag name table (sb.exe 0x16C9728)
+ITEM_USE_FLAGS_NONE = 1 << 0
+ITEM_USE_FLAGS_USE_SELF = 1 << 1
+ITEM_USE_FLAGS_USE_TARGET = 1 << 2
+ITEM_USE_FLAGS_EMPTY_DESTROY = 1 << 3
+ITEM_USE_FLAGS_EMPTY_NEWITEM = 1 << 4
+ITEM_USE_FLAGS_NUM = 5
 
 ITEM_USE_FLAGS_TO_STRING = {
     ITEM_USE_FLAGS_NONE: 'NONE',

mbEditorPro2.0/arcane/util/ArcWpak.py

@@ -0,0 +1,184 @@
+#  • ▌ ▄ ·. ▄▄▄▄·     ▄▄▄ .·▄▄▄▄  ▪  ▄▄▄▄▄      ▄▄▄       ▄▄▄·▄▄▄
+#  ·██ ▐███▪▐█ ▀█▪    ▀▄.▀·██▪ ██ ██ •██  ▪     ▀▄ █·    ▐█ ▄█▀▄ █·▪
+#  ▐█ ▌▐▌▐█·▐█▀▀█▄    ▐▀▀▪▄▐█· ▐█▌▐█· ▐█.▪ ▄█▀▄ ▐▀▀▄      ██▀·▐▀▀▄  ▄█▀▄
+#  ██ ██▌▐█▌██▄▪▐█    ▐█▄▄▌██. ██ ▐█▌ ▐█▌·▐█▌.▐▌▐█•█▌    ▐█▪·•▐█•█▌▐█▌.▐▌
+#  ▀▀  █▪▀▀▀·▀▀▀▀      ▀▀▀ ▀▀▀▀▀• ▀▀▀ ▀▀▀  ▀█▄▀▪.▀  ▀    .▀   .▀  ▀ ▀█▄▀▪
+#                Magicbane Emulator Project © 2013 - 2022
+#                           www.magicbane.com
+
+import os
+import time
+import zipfile
+
+from .ArcFileCache import next_available_path
+
+# A .wpak is a zip (client uses minizip, ArcWPakFile 0x54D3C0) whose entries are
+# each Blowfish CFB-64 encrypted with a zero IV (BF_cfb64_encrypt 0xCD9000),
+# except image entries.
+WPAK_KEY = bytes([
+    0x85, 0x71, 0x40, 0x3C, 0x14, 0x50, 0x0B, 0x52,
+    0x73, 0x2D, 0x10, 0x08, 0x63, 0x59, 0x5B, 0xAA,
+])  # hardcoded at 0x54DEC0
+
+WPAK_COMPRESS_LEVEL = 9  # zipOpenNewFileInZip level in ArcWPakFile writer
+
+# Map icon paks keep their .tga entries unencrypted (LoadMapZoneData 0xAACA60).
+WPAK_PLAIN_EXTENSIONS = ('.tga',)
+
+MASK32 = 0xFFFFFFFF
+
+
+def pi_fraction_words(count):
+    # Blowfish's initial P-array and S-boxes are the hex digits of pi's fraction.
+    guard_bits = 64
+    bits = count * 32 + guard_bits
+    one = 1 << bits
+
+    def arctan_inverse(x):
+        x_squared = x * x
+        term = one // x
+        total = term
+        sign = -1
+        n = 1
+        while term:
+            term //= x_squared
+            total += sign * (term // (2 * n + 1))
+            sign = -sign
+            n += 1
+        return total
+
+    pi = 4 * (4 * arctan_inverse(5) - arctan_inverse(239))
+    fraction = (pi - (3 << bits)) >> guard_bits
+
+    return [(fraction >> (32 * (count - 1 - i))) & MASK32 for i in range(count)]
+
+
+class ArcBlowfish:
+    p_array = None
+    s_boxes = None
+
+    def __init__(self, key):
+        words = pi_fraction_words(18 + 4 * 256)
+        self.p_array = words[:18]
+        self.s_boxes = [words[18 + 256 * i: 18 + 256 * (i + 1)] for i in range(4)]
+        self.set_key(key)
+
+    def set_key(self, key):
+        left = 0
+        right = 0
+
+        for i in range(18):
+            key_word = 0
+            for j in range(4):
+                key_word = (key_word << 8) | key[(i * 4 + j) % len(key)]
+            self.p_array[i] ^= key_word
+
+        for i in range(0, 18, 2):
+            left, right = self.encrypt_block(left, right)
+            self.p_array[i] = left
+            self.p_array[i + 1] = right
+
+        for s_box in self.s_boxes:
+            for i in range(0, 256, 2):
+                left, right = self.encrypt_block(left, right)
+                s_box[i] = left
+                s_box[i + 1] = right
+
+    def encrypt_block(self, left, right):
+        p_array = self.p_array
+        s0, s1, s2, s3 = self.s_boxes
+
+        for i in range(16):
+            left ^= p_array[i]
+            feistel = ((s0[left >> 24] + s1[(left >> 16) & 0xFF]) & MASK32) ^ s2[(left >> 8) & 0xFF]
+            right ^= (feistel + s3[left & 0xFF]) & MASK32
+            left, right = right, left
+
+        return right ^ p_array[17], left ^ p_array[16]
+
+    def keystream_block(self, iv):
+        left, right = self.encrypt_block(
+            int.from_bytes(iv[:4], 'big'),
+            int.from_bytes(iv[4:], 'big')
+        )
+        return left.to_bytes(4, 'big') + right.to_bytes(4, 'big')
+
+    def cfb64_decrypt(self, data):
+        output = bytearray(len(data))
+        iv = bytes(8)
+
+        for offset in range(0, len(data), 8):
+            keystream = self.keystream_block(iv)
+            cipher_block = data[offset:offset + 8]
+            for i, cipher_byte in enumerate(cipher_block):
+                output[offset + i] = cipher_byte ^ keystream[i]
+            iv = cipher_block
+
+        return bytes(output)
+
+    def cfb64_encrypt(self, data):
+        output = bytearray(len(data))
+        iv = bytes(8)
+
+        for offset in range(0, len(data), 8):
+            keystream = self.keystream_block(iv)
+            plain_block = data[offset:offset + 8]
+            for i, plain_byte in enumerate(plain_block):
+                output[offset + i] = plain_byte ^ keystream[i]
+            iv = bytes(output[offset:offset + 8])
+
+        return bytes(output)
+
+
+WPAK_CIPHER = ArcBlowfish(WPAK_KEY)
+
+
+def is_encrypted_entry(filename):
+    return not filename.lower().endswith(WPAK_PLAIN_EXTENSIONS)
+
+
+def load_wpak_file(filename):
+    entries = []
+
+    with zipfile.ZipFile(filename) as wpak_file:
+        for info in wpak_file.infolist():
+            if info.is_dir():
+                continue
+            data = wpak_file.read(info)
+            if is_encrypted_entry(info.filename):
+                data = WPAK_CIPHER.cfb64_decrypt(data)
+            entries.append([info.filename, data])
+
+    return entries
+
+
+def load_wpak_directory(directory):
+    entries = []
+
+    # Client finds entries by name (unzLocateFile 0xCD7920), so order only needs to be stable.
+    for filename in sorted(os.listdir(directory), key=str.upper):
+        filepath = os.path.join(directory, filename)
+        if not os.path.isfile(filepath):
+            continue
+        with open(filepath, 'rb') as fp:
+            entries.append([filename, fp.read()])
+
+    return entries
+
+
+def build_wpak_bytes(entries, wpak_file):
+    with zipfile.ZipFile(wpak_file, 'w') as archive:
+        for filename, data in entries:
+            info = zipfile.ZipInfo(filename, date_time=time.localtime()[:6])
+            info.compress_type = zipfile.ZIP_DEFLATED
+            if is_encrypted_entry(filename):
+                data = WPAK_CIPHER.cfb64_encrypt(data)
+            archive.writestr(info, data, compresslevel=WPAK_COMPRESS_LEVEL)
+
+
+def save_wpak_file(filename, entries):
+    filename = next_available_path(filename)
+    print('Packing to ' + filename)
+
+    with open(filename, 'wb') as wpak_file:
+        build_wpak_bytes(entries, wpak_file)

mbEditorPro2.0/arcane/util/__init__.py

@@ -7,4 +7,5 @@
 #                           www.magicbane.com
 
 from .ArcFileCache import load_cache_file, save_cache_file, next_available_path
+from .ArcWpak import load_wpak_file, load_wpak_directory, save_wpak_file, build_wpak_bytes
 from .ResStream import ResStream

mbEditorPro2.0/mbEditorPro.py

@@ -85,9 +85,25 @@ EDITOR_TARGETS = [
     'skeleton',
     'terrain',
     'render',
+    'wpak',
     'all'
 ]
 
+# Each pak unpacks to ARCANE_DUMP/<name> and packs back to <name>.wpak.
+WPAK_NAMES = [
+    'AerynthIcons',
+    'Config',
+    'DalgothIcons',
+    'Environment',
+    'GeneralItemTables',
+    'ItemTables',
+    'ModTables',
+    'ModTypeTables',
+    'Transitions',
+    'VendorENGLISH',
+    'VorringiaIcons'
+]
+
 
 def create_unpack_directory(base_directory):
     # Never unpack over an existing folder; use the next free numbered name.
@@ -841,6 +857,61 @@ def test_render():
             print(res_id, in_stream.buffer.tell(), out_stream.buffer.tell())
 
 
+def selected_wpak_names():
+    # An optional third argument picks a single pak, e.g. "unpack wpak Config".
+    return sys.argv[3:] or WPAK_NAMES
+
+
+def unpack_wpak():
+    for wpak_name in selected_wpak_names():
+        wpak_filename = wpak_name + '.wpak'
+        if not os.path.isfile(wpak_filename):
+            print('***Skipping: ' + wpak_filename + ' not found.')
+            continue
+
+        entries = load_wpak_file(wpak_filename)
+        directory = create_unpack_directory(os.path.join(TARGET_DIRECTORY, wpak_name))
+
+        with tqdm(total=len(entries)) as pBar:
+            for filename, data in entries:
+                filepath = os.path.join(directory, filename)
+                with open(filepath, 'wb') as fp:
+                    fp.write(data)
+                pBar.update(1)
+
+
+def pack_wpak():
+    for wpak_name in selected_wpak_names():
+        directory = os.path.join(TARGET_DIRECTORY, wpak_name)
+        if not os.path.isdir(directory):
+            print('***Skipping: ' + directory + ' not found.')
+            continue
+
+        entries = load_wpak_directory(directory)
+        save_wpak_file(wpak_name + '.wpak', entries)
+
+
+def test_wpak():
+    for wpak_name in selected_wpak_names():
+        wpak_filename = wpak_name + '.wpak'
+        if not os.path.isfile(wpak_filename):
+            print('***Skipping: ' + wpak_filename + ' not found.')
+            continue
+
+        print(wpak_filename)
+        entries = load_wpak_file(wpak_filename)
+        out_stream = io.BytesIO()
+
+        build_wpak_bytes(entries, out_stream)
+        repacked = dict(load_wpak_file(out_stream))
+
+        for filename, data in entries:
+            try:
+                assert repacked[filename] == data
+            except:
+                print(filename, len(data), len(repacked.get(filename, b'')))
+
+
 def unpack_all():
     for cache_type in EDITOR_TARGETS[:-1]:
         method = 'unpack_' + cache_type + '()'
@@ -882,6 +953,7 @@ def usage():
     print('')
     print('example:  unpack texture')
     print('example:  pack texture')
+    print('example:  unpack wpak Config')
     exit(1)