Posts on Technically some technical stuff.

Over-Engineered My Self-Hosting with Kubernetes

carpentier.pf@gmail.com (Pierre-François Carpentier) — Mon, 20 Apr 2026 12:55:15 +0100

Introduction

Too Honest For CV Driven Development

Working at a “Big Tech” company has its perks: because of the scale such company operates at, you typically don’t have to deal with basic services like DNS, authentication or CI/CD as these services are managed by dedicated core teams.

On one hand, it’s great: badly deployed by an overstretched dev or ops guy, these core services can easily become huge time & efficiency sinks distracting us from solving the problems our customers pay us for.

On the other, it can leave gaps in your CV, especially if like me, you have scruples about gratuitously over-engineering things just to learn new tools/frameworks on the job at your employer’s expense.

One such gap I currently have is deploying and managing Kubernetes clusters.

To be honest, I kind of avoided diving into K8s. Maybe it’s an unfounded bias on my part, but Kubernetes always felt a bit overcomplicated, clunky and not very pleasant to manage.

But, well, it’s what the cool kids are doing these days, so here we go.

Is complexity always justified?

What I Wanted Out Of This

By the end of this process, I wanted:

A mostly automated base KVM hypervisor deployment
A worker Kubernetes cluster, with all the control plane bits configured
HTTPS load balancer + DNS
CI/CD with Argo (and integration with GitHub)
Docker/Container Registry

This infrastructure should also be managed through the usual “configuration as code” tools, namely Ansible, Tofu, and a bit of scripting. I want to be able to delete and recreate it at will.

I’m making the actual code I’ve used available here in my GitHub. But be aware it is quite tightly coupled to my infrastructure, somewhat badly vibe-coded, and might not be easily reusable.

Kubernetes Basics

Choosing A Kubernetes Distribution

Like chocolate, k8s comes in various flavors. I contemplated deploying it on a traditional distribution, maybe dusting off my Gentoo skills for example.

But finally, I picked the easier path of using a specialized K8S distribution.

I considered the following ones:

Talos Linux is an immutable, API-driven distribution. It has no SSH access, no shell, and everything is configured through declarative configs and the talosctl CLI.
Flatcar Container Linux is a fork of the original CoreOS Container Linux backed by Kinvolk, now part of Microsoft. It’s a minimal and immutable distribution but with SSH access.
OpenShift and Fedora CoreOS are Red Hat’s successors to the original CoreOS.

I ended up choosing Talos Linux. It looked like the most common option on /r/kubernetes, and it’s not linked (yet) to the usual corporate vampires.

Kubernetes Base Architecture

Kubernetes has three main categories of components. First is the Control Plane, which coordinates the cluster. Second are the Workers, i.e. the nodes actually running stuff. The third and last category is the Cluster Add-Ons, enabling optional (but commonly deployed) things like public DNS record management, load-balancing or audit tools.

In the Control Plane, here are the main components:

etcd (third party): Consistent and highly-available key value store for all API server data & states.
kube-apiserver: The core component server that exposes the Kubernetes HTTP API.
kube-scheduler: Looks for Pods not yet bound to a node, and assigns each Pod (~container execution) to a suitable node.
kube-controller-manager: Runs the controller loops (replication, namespace, endpoint, etc).

On the Workers, you have:

kubelet: Ensures that Pods are running, including their containers.
container runtime (third party): Software responsible for running containers, in our case containerd, but others are possible

And here are some cluster Add-ons we will deploy (non-exhaustive):

Gateway API (third party): OSI Layer 4 and 7 load balancer to connect our Pods to the outside, here, we will use Traefik, but other implementations are available. It replaces the old Ingress Controllers.
ExternalDNS (third party): DNS record manager, which integrates with various DNS providers’ APIs (AWS Route53, GCP DNS, OVH, Gandi, RFC2136) and gives names to your exposed services.

Here is how everything fits together (simplified):

Kubernetes Architecture (simplified)

The cluster components talk to each other using HTTP and gRPC and usually authenticate each other using mutual TLS certificates.

In addition, Talos adds its own components (apid, machined, etc) to configure the cluster and manage its idiosyncrasies (custom init, etc).

Non-K8s Bits

The Metal + A Few Nuts & Bolts

First, there was the recommissioning of my old rig itself (i5 2500k/32GB DDR3) which required a bit of work, like installing some new storage with 3D printed adapters (5.25" to 3.5" + 3.5" to 2.5").

I also did some power consumption optimization (CPU down-clocking, removing/disabling unnecessary cards) to not have the thing draw ~120W continuously. It still draws ~50W however, which is… “not great, not terrible”… (I am considering replacing it with a refurb mini-pc or an old laptop to be honest).

After that, I installed the latest Debian and applied the following Ansible playbook to make a basic hypervisor out of it.

I also deployed an internal DNS server with TSIG/RFC 2136 dynamic zones on a SPARC V100 using OpenBSD for funsies.

And finally, I’ve created a Debian utility VM for support services like a Docker image registry or an LDAP server directory (VM created like in Cloud @Home and configured through this utility.yml Ansible playbook).

Bad Tools, Bad Worker

On our dev machine, we will need a few tools, namely:

Docker: Build, ship, and run containers.
OpenTofu (tofu): Open-source Terraform fork; describes and applies.
kubectl: CLI to administer and debug k8s.
talosctl: CLI for bootstrapping and operating Talos.
Helm: “Package Manager” used to install k8s applications and components.

Assuming you are using a deb-based distribution, here is how to install them (if you use macOS, Windows, or OS/2, I leave this exercise to the reader).

docker and kubectl are usually available in Debian/Ubuntu and derivative repositories.

apt install kubectl docker.io

For talosctl, I packaged it in my own repository, but you can directly grab it from Talos releases if you don’t trust me (protip: you shouldn’t):

# Get the architecture and version
. /etc/os-release
ARCH=$(dpkg --print-architecture)
wget -qO - https://kakwa.github.io/misc-pkg/GPG-KEY.pub | gpg --dearmor |sudo tee /etc/apt/keyrings/misc-pkg.gpg >/dev/null
sudo tee /etc/apt/sources.list.d/misc-pkg.sources >/dev/null <<EOF
Types: deb
URIs: https://kakwa.github.io/misc-pkg/deb.${VERSION_CODENAME}.${ARCH}/
Suites: ${VERSION_CODENAME}
Components: main
Signed-By: /etc/apt/keyrings/misc-pkg.gpg
Architectures: ${ARCH}
EOF
apt update

apt install talosctl

OpenTofu publishes tofu packages in their own repository:

curl -fsSL https://get.opentofu.org/opentofu.gpg | gpg --dearmor | sudo tee /etc/apt/keyrings/opentofu.gpg > /dev/null
curl -fsSL https://packages.opentofu.org/opentofu/tofu/gpgkey | gpg --dearmor | sudo tee /etc/apt/keyrings/opentofu-repo.gpg > /dev/null

sudo tee /etc/apt/sources.list.d/opentofu.sources > /dev/null <<EOF
Types: deb
URIs: https://packages.opentofu.org/opentofu/tofu/any/
Suites: any
Components: main
Signed-By: /etc/apt/keyrings/opentofu.gpg /etc/apt/keyrings/opentofu-repo.gpg
EOF
apt update

apt install tofu

And so does Helm:

curl -fsSL https://packages.buildkite.com/helm-linux/helm-debian/gpgkey | gpg --dearmor | sudo tee /etc/apt/keyrings/helm.gpg > /dev/null
sudo tee /etc/apt/sources.list.d/helm-stable-debian.sources > /dev/null <<EOF
Types: deb
URIs: https://packages.buildkite.com/helm-linux/helm-debian/any/
Suites: any
Components: main
Signed-By: /etc/apt/keyrings/helm.gpg
EOF
sudo apt update

sudo apt install helm

Deploy That Damn Cluster Already!

Like the Debian utility VM, I’ve also used Tofu to create the k8s/Talos cluster, except this time, we are “simply” creating nine nodes (3 control plane + 6 workers) instead of one (and I do mean “simply”, the for(each) loop really feels like cheating sometimes).

The full code is available on GitHub and leverages the KVM/libvirt, the Talos and the dns Tofu providers.

Be aware that unlike the Tofu code from Hyperscaler Cloud @Home, it’s more tied to my home network environment, and would need a fair bit of tweaking to run on your setup.

Talos Image Management

Talos has an Image Factory for creating custom images with specific versions, architectures, and extensions.

In my case, I left it nearly vanilla, but you can add things if you have special needs, for example, completely at random in these AI days, NVIDIA drivers for CUDA workloads.

To get the list of customization/versions, simply go through the form and grab the generated schematic at the end.

The schematic can be used to configure and download the Talos image in the Tofu provider, and register it in Libvirt:

# Create a schematic with custom extensions
resource "talos_image_factory_schematic" "this" {
 schematic = yamlencode({
 customization = {
 systemExtensions = {
 officialExtensions = [
 "siderolabs/binfmt-misc",
 ]
 }
 }
 })
}

# Build the image URL from the schematic
locals {
 talos_version = "v1.12.6"
 talos_image_url = "https://factory.talos.dev/image/${talos_image_factory_schematic.this.id}/${local.talos_version}/nocloud-amd64.qcow2"
}

# Download the image directly into libvirt
resource "libvirt_volume" "talos_base" {
 name = "talos-base.qcow2"
 pool = "mid-pool"
 create = {
 content = {
 url = local.talos_image_url
 }
 }
 target = {
 format = { type = "qcow2" }
 }
}

Cluster Network

It’s not k8s, but we need a network for our cluster.

Let’s create one quickly, behind a NAT and with DHCP:

resource "libvirt_network" "talos_network" {
 name = "talos-network"
 autostart = true
 forward = { mode = "nat" }
 bridge = {
 name = "virbr1"
 stp = "on"
 delay = "0"
 }
 ips = [{
 address = "192.168.100.1"
 netmask = "255.255.255.0"
 dhcp = {
 ranges = [{
 start = "192.168.100.50"
 end = "192.168.100.254"
 }]
 }
 }]
}

Control Plane Nodes

Our Kubernetes control plane will use 3 nodes with the minimal Talos specs, i.e. 2 cores+2GB RAM as we are fairly limited in space here.

For resiliency in production, this number is usually increased to 5. On paper this could be further increased to any odd value, but at the cost of latency. Underneath all that, the cluster states are backed by the raft-based, strongly consistent, etcd key/value store, which explains this behavior.

Here is the definition of the nodes:

locals {
 control_plane_nodes = {
 for i in range(3) : "talos-cp-${i + 1}" => {
 memory_mb = 2048
 vcpu = 2
 }
 }
}

To bootstrap the nodes, in particular the network configuration, we need to pass a few cloud-init parameters, which we can create with the libvirt_cloudinit_disk resource from the libvirt Tofu provider:

resource "libvirt_cloudinit_disk" "cp_seed" {
 for_each = local.control_plane_nodes

 name = "${each.key}-cloudinit"
 user_data = <<-EOF
 timezone: UTC
 EOF
 meta_data = <<-EOF
 instance-id: ${each.key}
 local-hostname: ${each.key}
 EOF
 network_config = <<-EOF
 version: 2
 ethernets:
 eth0:
 dhcp4: true
 EOF
}

# Convert cloud-init to ISO volume
resource "libvirt_volume" "cp_seed_volume" {
 for_each = local.control_plane_nodes
 name = "${each.key}-cloudinit.iso"
 pool = "slow-pool"
 create = {
 content = {
 url = "file://${libvirt_cloudinit_disk.cp_seed[each.key].path}"
 }
 }
}

We also need some disks for our nodes, using the Talos image as a base:

# Each node gets its own disk backed by the base image
resource "libvirt_volume" "cp_disk" {
 for_each = local.control_plane_nodes

 name = "${each.key}-disk.qcow2"
 pool = "mid-pool"
 capacity = 107374182400 # 100GB
 backing_store = {
 path = libvirt_volume.talos_base.path
 format = { type = "qcow2" }
 }
 target = {
 format = { type = "qcow2" }
 }
}

And finally, we can create the VMs themselves, reusing these disks:

resource "libvirt_domain" "control_plane" {
 for_each = local.control_plane_nodes

 name = each.key
 memory = each.value.memory_mb * 1024
 vcpu = each.value.vcpu
 running = true
 autostart = true

 os = {
 type = "hvm"
 type_arch = "x86_64"
 type_machine = "q35"
 }

 cpu = {
 mode = "host-passthrough"
 }

 devices = {
 disks = [
 {
 source = {
 volume = {
 pool = libvirt_volume.cp_disk[each.key].pool
 volume = libvirt_volume.cp_disk[each.key].name
 }
 }
 target = { dev = "vda", bus = "virtio" }
 driver = { type = "qcow2" }
 },
 {
 device = "cdrom"
 source = {
 volume = {
 pool = "slow-pool"
 volume = "${each.key}-cloudinit.iso"
 }
 }
 target = { dev = "sda", bus = "sata" }
 }
 ]
 interfaces = [
 {
 type = "network"
 model = { type = "virtio" }
 source = {
 network = {
 network = libvirt_network.talos_network.name
 }
 }
 wait_for_ip = { timeout = 300, source = "any" }
 }
 ]
 }
}

Worker Nodes

Similarly, we will deploy 6 nodes which will be used as workers.

Here is a rough outline of the code, but since it’s mostly identical to the control plane, this is a cut down version of the code as it’s just more of the same:

locals {
 worker_nodes = {
 for i in range(6) : "talos-worker-${i + 1}" => {
 [...]
 }
 }
}

resource "libvirt_cloudinit_disk" "worker_seed" {
 for_each = local.worker_nodes
 name = "${each.key}-cloudinit"
 [...]
}

# Convert cloud-init to ISO volume
resource "libvirt_volume" "worker_seed_volume" {
 for_each = local.worker_nodes
 name = "${each.key}-cloudinit.iso"
 [...]
}

# main worker VM disks
resource "libvirt_volume" "worker_disk" {
 for_each = local.worker_nodes
 [...]
}

resource "libvirt_domain" "workers" {
 for_each = local.worker_nodes
 [...]
}

Remember Remember, The IPs Of Our Cluster

Once we have our VMs, it would be nice to not have to go fishing for the IPs of our cluster nodes.

So, through a simple template like:

export CONTROL_PLANE_IP=(${join(" ", [for ip in control_plane_ips : "${"\""}${ip}${"\""}"])})
export WORKER_IP=(${join(" ", [for ip in worker_ips : "${"\""}${ip}${"\""}"])})
export CONTROL_PLANE_VIP="${control_plane_vip}"

And a bit of OpenTofu code leveraging the data.libvirt_domain_interface_addresses Tofu datasource:

# Recovery of the IPs
# might need some modification (FIXME hard coded indices are sketchy)
locals {
 provider_talos_cp_ips = {
 for k in keys(local.control_plane_nodes) :
 k => try(data.libvirt_domain_interface_addresses.control_plane[k].interfaces[1].addrs[0].addr, "")
 }
 talos_worker_ips = {
 for k in keys(local.worker_nodes) :
 k => try(data.libvirt_domain_interface_addresses.workers[k].interfaces[1].addrs[0].addr, "")
 }
}

# We also define a VIP for the cluster, to more easily access it in the future.
variable "control_plane_vip" {
 description = "Virtual IP for the Talos/Kubernetes control plane API"
 type = string
 default = "192.168.100.10"
}

# Use the template and the IPs to generate the talos-env.sh inventory file
resource "local_file" "env" {
 content = templatefile("${path.module}/env.tpl", {
 control_plane_ips = local.provider_talos_cp_ips
 worker_ips = local.talos_worker_ips
 control_plane_vip = var.control_plane_vip
 })
 filename = ".//talos-env.sh"
 file_permission = "0644"
}

We can let Tofu create an env file which once sourced, will provide convenient variables for the rest of this setup.

At Last, Kubernetes & Talos Setup

After a while, you should have a bunch of new VMs, booted-up into an unconfigured Talos.

If you use virt-manager, you should see a dashboard like that in the first tty of each VM:

Fresh Talos VM in maintenance mode before apply-config

Note that on all nodes, currently, Type is unknown, Stage is Maintenance, Cluster and various other parameters are n/a.

This means that right now, we have just a bunch of individual nodes not talking to each other.

Also note that the cluster is in a vulnerable state at this point, with anybody able to take control of your freshly provisioned nodes.

We must configure all the nodes in order to have a properly functioning and secured k8s cluster.

Generating The Cluster Config

So, the first step is to generate a configuration auto-magically using talosctl gen config

# Set some variables
source ./talos-env.sh
CLUSTER_NAME="kakwalab-talos-cluster"
BOOTSTRAP_CP="${CONTROL_PLANE_IP[0]}"
TEMP_ENDPOINT="https://${BOOTSTRAP_CP}:6443"

talosctl gen config "${CLUSTER_NAME}" "${TEMP_ENDPOINT}"

This creates a base configuration consisting of controlplane.yaml, worker.yaml, and talosconfig.

And for the most part, these mainly serve to create mTLS key pairs (the Certificate Authority) for authentication. And they also configure some internal network stuff for the cluster.

Configuration bootstrapping

Apply the generated config to all nodes (control planes first, then workers), then bootstrap the cluster once:

# If not already done
source ./talos-env.sh

# Apply control planes configuration
for ip in "${CONTROL_PLANE_IP[@]}"; do
 talosctl apply-config --insecure --nodes "${ip}" --file controlplane.yaml
done

# Apply workers configuration
for ip in "${WORKER_IP[@]}"; do
 talosctl apply-config --insecure --nodes "${ip}" --file worker.yaml
done

# Wait for nodes to come back up
sleep 120

# Bootstrap (only once, on the first control plane)
talosctl bootstrap --endpoints "${BOOTSTRAP_CP}" --nodes "${BOOTSTRAP_CP}"

While nodes restart and synchronize after apply-config, the nodes could stay in booting state for a few minutes:

Worker node still coming up after configuration

After the bootstrap, check the cluster health, and fetch the kubeconfig file for kubectl once it’s healthy:

talosctl health --endpoints "${BOOTSTRAP_CP}" --nodes "${BOOTSTRAP_CP}" \
 --control-plane-nodes "$(IFS=,; echo "${CONTROL_PLANE_IP[*]}")" \
 --worker-nodes "$(IFS=,; echo "${WORKER_IP[*]}")" \
 --wait-timeout 5m

talosctl kubeconfig . --endpoints "${BOOTSTRAP_CP}" --nodes "${BOOTSTRAP_CP}"

On a control plane node TTY, a healthy dashboard looks like this:

Healthy control plane after bootstrap

Workers should show a similar status in the dashboard:

Healthy worker node

From there, you should be able to run kubectl, and see your cluster:

export KUBECONFIG=$(pwd)/kubeconfig

kubectl get nodes

NAME STATUS ROLES AGE VERSION
talos-cp-1 Ready control-plane 65m v1.35.0
talos-cp-2 Ready control-plane 65m v1.35.0
talos-cp-3 Ready control-plane 65m v1.35.0
talos-worker-1 Ready <none> 64m v1.35.0
talos-worker-2 Ready <none> 64m v1.35.0
talos-worker-3 Ready <none> 64m v1.35.0
talos-worker-4 Ready <none> 64m v1.35.0
talos-worker-5 Ready <none> 64m v1.35.0
talos-worker-6 Ready <none> 64m v1.35.0

VIP for kubectl

It’s optional, but let’s add a virtual IP on the control plane nodes to ease management.

First, let’s patch the control plane node configuration to use this VIP:

for ip in "${CONTROL_PLANE_IP[@]}"; do
 talosctl patch mc --endpoints "${ip}" --nodes "${ip}" --patch "
machine:
 network:
 interfaces:
 - interface: eth0
 dhcp: true
 vip:
 ip: ${CONTROL_PLANE_VIP}
"
done

After VIP election, we can then set the control plane endpoint to it.

for ip in "${CONTROL_PLANE_IP[@]}"; do
 talosctl patch mc --endpoints "${ip}" --nodes "${ip}" --patch "
cluster:
 controlPlane:
 endpoint: https://${CONTROL_PLANE_VIP}:6443
"
done

And grab an updated kubeconfig

talosctl kubeconfig . --endpoints "${BOOTSTRAP_CP}" --nodes "${BOOTSTRAP_CP}"

(Not So) Optional Kubernetes Add-Ons

MetalLB

In the cloud, Kubernetes leverages the providers’ load balancer services such as NLB on AWS. But that is not something we have at home.

MetalLB fills that gap for on-premises deployments. In a simple deployment, it watches LoadBalancer services and assigns addresses from a pool you define to be used by said LoadBalancer services.

First we need a bit of configuration to tweak the MetalLB namespace permissions (aka PSS/PSA or Pod Security Standards).

metallb-namespace.yaml:

# PSA: MetalLB speaker/FRR need NET_ADMIN, hostNetwork, etc. — not compatible with restricted.
apiVersion: v1
kind: Namespace
metadata:
 name: metallb
 labels:
 pod-security.kubernetes.io/enforce: privileged
 pod-security.kubernetes.io/audit: privileged
 pod-security.kubernetes.io/warn: privileged

And from there, we need to declare the range of IPs we let MetalLB to play with:

metallb-lan.yaml

# Apply after the MetalLB Helm release is healthy. Namespace must match the chart install.
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
 name: lan-pool
 namespace: metallb
spec:
 addresses:
 - 192.168.1.48/28 # Modify with your own IPs (be cautious about collision, especially with DHCP ranges)
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
 name: lan-pool-l2
 namespace: metallb
spec:
 ipAddressPools:
 - lan-pool

export KUBECONFIG="$(pwd)/kubeconfig"
export METALLB_NAMESPACE="metallb"

kubectl apply -f ./metallb-namespace.yaml

helm repo add metallb https://metallb.github.io/metallb --force-update
helm repo update metallb
helm upgrade --install metallb metallb/metallb --namespace "$METALLB_NAMESPACE"

kubectl rollout status deployment/metallb-controller -n "$METALLB_NAMESPACE" --timeout=300s
kubectl rollout status daemonset/metallb-speaker -n "$METALLB_NAMESPACE" --timeout=300s

kubectl apply -f ./metallb-lan.yaml

Traefik Deployment

On top of MetalLB we can now deploy Traefik, which will handle our load balancing needs, both of TCP/UDP and HTTP directly.

We also need some permissions on the Traefik namespace, which we will set with a bit of YAML:

traefik-namespace.yaml:

# PSA baseline: avoids restricted seccomp warnings on Traefik controller without full privileged.
apiVersion: v1
kind: Namespace
metadata:
 name: traefik
 labels:
 pod-security.kubernetes.io/enforce: baseline
 pod-security.kubernetes.io/audit: baseline
 pod-security.kubernetes.io/warn: baseline

And from there, we configure a fairly generic Helm values for Traefik:

traefik-helm-values.yaml:

# Traefik as cluster ingress (Ingress + IngressRoute CRD).
ingressClass:
 enabled: true
 isDefaultClass: true

service:
 annotations:
 metallb.universe.tf/address-pool: lan-pool
 spec:
 type: LoadBalancer

providers:
 kubernetesCRD:
 enabled: true
 kubernetesIngress:
 enabled: true
 # Fill Ingress status.loadBalancer so ExternalDNS (and clients) see the Traefik LB IP/hostname.
 publishedService:
 enabled: true
 pathOverride: traefik/traefik

deployment:
 replicas: 1

We can then use these to create the Traefik namespace:

export KUBECONFIG="./kubeconfig"
export TRAEFIK_NAMESPACE="traefik"

kubectl apply -f ./traefik-namespace.yaml

And finally add the Helm chart repository and install or upgrade Traefik:

helm repo add traefik https://traefik.github.io/traefik-helm-chart --force-update
helm repo update traefik

helm upgrade --install traefik traefik/traefik \
 --namespace "$TRAEFIK_NAMESPACE" \
 -f ./traefik-helm-values.yaml

After the controller is up, check that the load balancer received an address from your pool:

kubectl -n "$TRAEFIK_NAMESPACE" get svc

From there, standard Ingress objects with ingressClassName: traefik (and optional Traefik annotations) are what connect hostnames to workloads—exactly what the first application chart below relies on.

DNS Management (ExternalDNS)

Thanks to Traefik, we have exposed and load-balanced our service, but usually you also want to give it names in DNS. While it could be managed separately from Kubernetes, it is nicer to have the cluster own those records and declare the desired hostname next to the Ingress in Helm. The usual tool for that is ExternalDNS, which can integrate with many DNS providers (Route53, Gandi, OVH, and so on).

In my case, I use the generic RFC2136/TSIG integration to let it manage records in a private int.kakwalab.ovh zone.

To enable ExternalDNS, start from a Helm values template with the parameters your server needs (zone name, key algorithm, DNS server IP and port, and so on):

provider:
 name: rfc2136

# Replace with your DNS server
extraArgs:
 - --rfc2136-host=192.168.1.25
 - --rfc2136-port=5353
 - --rfc2136-zone=int.kakwalab.ovh
 - --rfc2136-tsig-secret-alg=hmac-sha512

env:
 - name: EXTERNAL_DNS_RFC2136_TSIG_SECRET
 valueFrom:
 secretKeyRef:
 name: external-dns-rfc2136
 key: rfc2136-tsig-secret
 - name: EXTERNAL_DNS_RFC2136_TSIG_KEYNAME
 valueFrom:
 secretKeyRef:
 name: external-dns-rfc2136
 key: rfc2136-tsig-keyname

txtOwnerId: talos-home-tf

# Replace with your DNS zone
domainFilters:
 - int.kakwalab.ovh

sources:
 - service
 - ingress

policy: upsert-only

logLevel: info
interval: 1m

rbac:
 create: true

resources:
 requests:
 cpu: 50m
 memory: 64Mi
 limits:
 memory: 128Mi

From there, prepare a namespace and a secret with your TSIG key:

export KUBECONFIG="./kubeconfig"

export EXTERNAL_DNS_NAMESPACE="external-dns"

# TSIG credentials (replace with your own)
DNS_TSIG_KEY_SECRET="LTEzODcyLTE0NzIxCgLTEzODcyLTE0NzIxCgLTEzODcyLTE0NzIxCgQ=="
TSIG_KEYNAME="sec1_key"

# Create namespace for ExternalDNS
kubectl create namespace "$EXTERNAL_DNS_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -

# Create the service secrets
kubectl -n "$EXTERNAL_DNS_NAMESPACE" create secret generic external-dns-rfc2136 \
 --from-literal=rfc2136-tsig-secret="$DNS_TSIG_KEY_SECRET" \
 --from-literal=rfc2136-tsig-keyname="$TSIG_KEYNAME" \
 --dry-run=client -o yaml | kubectl apply -f -

And install with Helm:

# Add Helm Repository
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/ >/dev/null
helm repo update external-dns >/dev/null

# Install it, using the upstream helm template and our configuration
helm upgrade --install external-dns external-dns/external-dns \
 --namespace "$EXTERNAL_DNS_NAMESPACE" \
 -f "./external-dns-helm-values.yaml"

A Few Closing Items

Deploying A Private Docker Registry

To store and distribute our app container image, I’ve also deployed a private Docker Registry on the utility VM.

The deployment is done through this Ansible docker_registry/ role.

The gist being that it deploys a docker-registry package I’ve built, configures plus enables the service and puts an nginx in front of it for Auth Basic and TLS.

For TLS, it uses a proper Let’s Encrypt certificate as loading a custom, self-signed, CA in Talos seems tedious.

As it’s a non-public service, the certificate was obtained using the DNS-01 challenge with certbot’s OVH plugin (my DNS provider).

For our app, this is where our Kubernetes cluster will download its container images

Our First App!

Now that we have a cluster and its surrounding infrastructure, let’s finally do something useful.

As it happened to be, I was getting a bit annoyed by the prevalence of AI posts in my Hacker News RSS feed.

Through a bit of Q&D vibe-coding on hnrss doing some keyword matching (words like ai, llm, anthropic…), I’ve created a fork adding /ai and /noai endpoints separating AI and non AI content.

This filtering works great, but now, we must host this thing, and well… we have a brand new Kubernetes cluster.

This service is actually kind of ideal for k8s since it doesn’t have a persistent layer.

In addition to the filtering mods, I’ve added the bits necessary for Kubernetes, namely:

a Dockerfile to create a container,
a small script to publish our container image into our registry
a minimal Helm Chart

To deploy it on our cluster, do the following.

Clone the repository and enter it:

git clone https://github.com/kakwa/hnrss-ai-filtering.git
cd hnrss-ai-filtering

Set a few env vars for docker and kubectl/helm:

export REGISTRY_USER=R_USER
export REGISTRY_PASSWORD=R_PASSWORD
export REGISTRY_HOST=registry.example.com
export REGISTRY=${REGISTRY_HOST}
export KUBECONFIG=$HOME/.kubeconfig

Publishing a new version of the docker image in our registry:

./scripts/publish-image.sh

Creating or updating the service definition:

# Create a namespace for the service (if necessary)
kubectl create namespace hnrss

# Go and tweak the helm configuration
cd helm/
vim values.yaml # in particular, tweak the DNS domain.

# Apply helm configuration (creation and updates)
helm upgrade --install hnrss ./ --namespace hnrss --set registryAuth.password="$REGISTRY_PASSWORD"

Rolling out a new version after publishing in the docker registry:

kubectl rollout restart deployment -n hnrss

Conclusion

So, was this exercise worth it?

… Well… kind of… but not completely.

For my personal stuff, I’m much more in the “just use a VPS” camp. Just to illustrate, the hnrss app took me 15 minutes to deploy on my VPS. In k8s, with all the Helm template and Docker stuff, it was closer to two hours (granted, I was also fixing the last cluster setup issues).

In a professional context, while I do appreciate what K8s brings (clean decoupling between infrastructure and app + easy instrumentation enabling stuff like CI/CD), I just want to remain being a user of the thing, not administer it.

I like things often considered boring, LDAP and IAM topics for example, but here, I think I’ve reached my limits. There are simply too many moving pieces and too much magic glue going on. Kubernetes does solve problems, but it really feels like a tedious and clunky solution. To the point I’m wondering if something a bit more opinionated, complete out of the box and simpler will not pop-up and replace it in the future.

Just to illustrate, this already lengthy deployment is in fact far from complete. From the goals I set out at the beginning, I’m still missing the CI/CD part (for example ArgoCD). And beyond that, there are many other subjects which would need to be tackled in a production & non-solo environment:

RBAC and access control, like described in Kubernetes RBAC and authentication (OIDC, webhooks, group claims).
Management UI, for example Headlamp
Monitoring, probably levering Prometheus
Logging: probably leveraging Fluentd for log forwarding, and either Elasticsearch or ClickHouse for indexing.
Persistent volumes, could for example leverage iSCSI volumes
Project templates

But it was not lost time either. Even if right now, I’m kind of fed up with it, I will probably revisit it in the future and do the CI/CD stuff (easily bootstrapping a new service is simply too appealing).

It also gave me an appreciation for Kubernetes operators. Now I get why it’s a specialized and hard-to-fill role.

And lastly, discovering what you want and don’t want to do before fully committing to it is always valuable. Managing Kubernetes clusters all day long is simply not for me.

Relevant Links

Kubernetes Documentation: The star of the show.
Talos Linux: API-driven, immutable OS for Kubernetes.
Talos Documentation: Official documentation for Talos.
Talos Image Factory: Custom Talos images builder.
OpenTofu: Open-source infrastructure as code tool (Terraform Fork).
kubectl: CLI tool for k8s administration.
Helm: “package manager” & service specification for k8s
OpenTofu libvirt provider: KVM/libvirt provider for Tofu.
OpenTofu Talos provider: Talos provider for Tofu.
MetalLB: K8S On-prem IPs management add-on for supporting LoadBalancer.
Traefik: K8S LoadBalancer/Service Gateway add-on.
ExternalDNS: K8S add-ons to manage DNS records.
Argo CD: GitOps continuous delivery for K8S applications.
Headlamp: Management UI for K8s.
Fluentd: Unified logging layer.
home.tf: The Ansible and OpenTofu code used in this article.

Hyperscaler Cloud @Home

carpentier.pf@gmail.com (Pierre-François Carpentier) — Fri, 17 Apr 2026 12:55:15 +0100

Introduction

These days, at work, we don’t really have to care about server racking, cabling, powering, and the many other *ings required when dealing with hardware.

Logistic companies like Amazon/AWS have managed to hide it all behind convenient APIs, and frankly, they do it better than we could.

That’s only valid in a professional context however. At home, unless you are /r/wallstreetbets inclined financially, putting your credit card into Vercel, AWS or GCP is a bit risky for your bank account (see these billing horror stories).

Homelabing with an old rig in the closet or a fancy 10" rack of refurb mini-PCs often are more sensible options.

Cloud @Home

But at first glance, it comes at a cost: manual setup is in and things are no longer abstracted away…

… or so it seems…

In reality, we can actually re-use at home most of the same tools available for the Cloud(™).

This post will present one of them: Terraform/Tofu and show how it can manage a basic Libvirtd/KVM hypervisor.

Locally Sourced Tofu

Making a hypervisor out of a Debian install is extremely easy: install Debian, and run apt install libvirt-daemon.

But past that, wouldn’t it be nice to be able to manage our libvirtd/kvm hypervisor with Tofu/HCL like we would on AWS?

Well, thanks to dmacvicar’s libvirt provider, we can, and here is how.

Notes:

Don’t hesitate to play with this code. You can download the full main.tf, and with a few tweaks, you should be able to apply it on your own hypervisor.

To apply this configuration, run tofu init (once) and tofu apply as root in the same dir as main.tf.

OpenTofu can be installed from the official website.

Libvirt Provider Setup

First, declare the libvirt provider and point it at the local hypervisor:

terraform {
 required_providers {
 libvirt = {
 source = "dmacvicar/libvirt"
 version = "~> 0.9.7"
 }
 }
}

provider "libvirt" {
 uri = "qemu:///system"
}

Notes: qemu:///system connects to the local hypervisor. You can also point it at a remote host over SSH with qemu+ssh://<USER>@<HOST>/system. But be aware that it could make image management fiddly (local FS vs remote FS).

Networks

We probably need some LAN segments to put our VMs in, at least a private network and also, usually, a bridge to our home network.

The libvirt provider can manage these networks through its libvirt_network resources.

Here is private NATed network declaration:

locals {
 network_name = "homelab-nat"
 network_cidr = "192.168.150.0/24"
}

resource "libvirt_network" "MY_NETWORK" {
 name = local.network_name
 autostart = true
 forward = { mode = "nat" }
 bridge = {
 name = "virbr1"
 stp = "on"
 delay = "0"
 }
 ips = [{
 address = cidrhost(local.network_cidr, 1)
 netmask = cidrnetmask(local.network_cidr)
 dhcp = {
 ranges = [{
 start = cidrhost(local.network_cidr, 50)
 end = cidrhost(local.network_cidr, 254)
 }]
 }
 }]
}

Outside of Tofu & libvirt we will also need to create a br0 NIC bridge for direct access to our home network.

To create it temporarily:

ip link add name br0 type bridge
ip link set br0 up

To make it persistent on Debian, edit /etc/network/interfaces and add something like:

auto br0
iface br0 inet dhcp
 bridge_ports eth0
 bridge_stp off
 bridge_fd 0

Replace eth0 with your actual physical NIC.

Storage Pools

Before creating any disks/volumes, we need storage pools first. A pool is just a directory that libvirt manages.

At its simplest, a declaration looks like that:

resource "libvirt_pool" "MY_DISK_POOL" {
 name = "MY_DISK_POOL"
 type = "dir"
 target = {
 path = "/var/lib/libvirt/images/MY_DISK_POOL"
 }
}

Base VM Image

We could install the VM using a modified .iso or with pxe netboot setup, like we would with bare metal machines.

But, since we are already past the bare metal, a more convenient option is to directly download and use official images from Debian, Rocky or your preferred distribution.

By just pointing at the URL, the libvirt Tofu module can download and register this base image in our hypervisor’s libvirt:

locals {
 debian_image_url = "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2"
}

resource "libvirt_volume" "debian_base" {
 name = "debian-base.qcow2"
 pool = libvirt_pool.MY_DISK_POOL.name
 create = {
 content = {
 url = local.debian_image_url
 }
 }
 target = {
 format = { type = "qcow2" }
 }
}

And from there, with the backing_store directive, this image can be used as the foundation for multiple VMs.

Cloud Init

Cloud images usually ship with cloud-init, which we can leverage to bootstrap the VMs.

At this stage, it’s advisable to do as little as possible, but we will typically want to configure:

some network settings
a Salt/Puppet/Chef agent or some SSH credentials for Ansible
usually, a default account for troubleshooting

One of the nice feature of this tofu/libvirt provider is its ability to create cloud-init payloads/volumes.

With tofu apply, this will create a small (< 1MB) .iso volume which can be attached to the VM cdrom, and consumed at first boot:

# Cloud-init configuration for the YOUR_VM VM
resource "libvirt_cloudinit_disk" "YOUR_VM_seed" {
 name = "YOUR_VM-cloudinit"

 user_data = <<-EOF
 #cloud-config
 users:
 - name: admin
 groups: [sudo]
 shell: /bin/bash
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3... your-key-here

 write_files:
 - path: /etc/sudoers.d/admin
 permissions: '0440'
 content: |
 %admin ALL=(ALL:ALL) NOPASSWD: ALL

 chpasswd:
 list: |
 root:password
 expire: false

 ssh_pwauth: true

 packages:
 - openssh-server
 - qemu-guest-agent

 runcmd:
 - systemctl enable --now qemu-guest-agent

 timezone: UTC
 EOF

 meta_data = <<-EOF
 instance-id: YOUR_VM
 local-hostname: YOUR_VM
 EOF

 network_config = <<-EOF
 version: 2
 ethernets:
 enp1s0:
 addresses:
 - 192.168.1.13/24
 gateway4: 192.168.1.254
 nameservers:
 addresses:
 - 192.168.1.25
 enp2s0:
 dhcp4: true
 EOF
}

# create cloud-init to ISO volume
resource "libvirt_volume" "YOUR_VM_seed_volume" {
 name = "YOUR_VM-cloudinit.iso"
 pool = libvirt_pool.MY_DISK_POOL.name
 create = {
 content = {
 url = "file://${libvirt_cloudinit_disk.YOUR_VM_seed.path}"
 }
 }
}

Notes: we also install and enable qemu-guest-agent here. Installing it helps a lot with IP management in the libvirt provider.

VM Creation

Once we have the networks, the pools, the base OS image and some cloud-init configuration, we can at last create the VM itself.

Let’s start with the main disk:

resource "libvirt_volume" "YOUR_VM_disk" {
 name = "YOUR_VM-disk.qcow2"
 pool = libvirt_pool.MY_DISK_POOL.name
 capacity = 53687091200 # 50GB

 # Reuse the cloud image we got previously
 backing_store = {
 path = libvirt_volume.debian_base.path # Base VM Image we downloaded earlier.
 format = { type = "qcow2" }
 }
 target = {
 format = { type = "qcow2" }
 }
}

And from there, we can actually create the VM:

resource "libvirt_domain" "YOUR_VM" {
 name = "YOUR_VM"
 type = "kvm"
 memory = 1048576 # 1GB (1024 * 1024)
 vcpu = 2
 running = true
 autostart = true

 os = {
 type = "hvm"
 type_arch = "x86_64"
 type_machine = "q35"

 # Weirdly in bios mode, virtio on disks was not working.
 # The VM was not booting completely.
 # So EFI it is.
 firmware = "efi"
 }

 cpu = {
 mode = "host-passthrough"
 }

 # Necessary for EFI
 features = {
 acpi = true
 apic = {}
 }

 devices = {
 pci = [
 { type = "pci", model = "pcie-root" }
 ]
 disks = [
 {
 source = {
 volume = {
 # our disk
 pool = libvirt_volume.YOUR_VM_disk.pool
 volume = libvirt_volume.YOUR_VM_disk.name
 }
 }
 target = { dev = "vda", bus = "virtio" }
 driver = { type = "qcow2" }
 },
 {
 device = "cdrom"
 source = {
 volume = {
 pool = libvirt_pool.MY_DISK_POOL.name
 volume = libvirt_volume.YOUR_VM_seed_volume.name # Use the cloud-init iso/image we defined earlier
 }
 }
 target = { dev = "sda", bus = "sata" }
 }
 ]

 # Two network interfaces: bridge br0 + MY_NETWORK
 interfaces = [
 {
 type = "bridge"
 model = { type = "virtio" }
 source = { bridge = { bridge = "br0" } }
 wait_for_ip = { timeout = 300, source = "any" }
 },
 {
 type = "network"
 model = { type = "virtio" }
 source = {
 network = {
 network = libvirt_network.MY_NETWORK.name
 }
 }
 wait_for_ip = { timeout = 300, source = "any" }
 }
 ]

 # Channel for qemu-guest-agent communication between VM and hypervisor
 # mostly to report back the VM's IPs
 channels = [
 {
 target = { virt_io = { name = "org.qemu.guest_agent.0" } }
 source = { unix = { mode = "bind" } }
 }
 ]

 # Useful to get a console without vnc/spice
 # run > sudo screen `virsh ttyconsole YOUR_VM`
 consoles = [
 {
 type = "pty"
 target = { type = "serial", port = "0" }
 }
 ]

 # GUI/spice troubleshooting (through virt-manager for example)
 videos = [
 { model = { type = "virtio", heads = 1, primary = "yes" } }
 ]
 graphics = [
 {
 spice = {
 autoport = "no"
 port = 5930
 listen = "127.0.0.1"
 listeners = [
 {
 type = "address"
 address = {
 address = "127.0.0.1"
 }
 }
 ]
 }
 }
 ]
 }
}

DNS Records

Lastly, as it’s nicer to deal with proper hostname rather than IP addresses, let’s configure a Tofu DNS provider.

First, you need to configure have a DNS server allowing RFC 2136 dynamic updates to a zone.

I will not detail this configuration much here, but with Bind/Named, it could be done like that:

First, generate a key:

$ dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST example.com.
# it creates two files:
$ ls K*
Kexample.com.+165+27879.key Kexample.com.+165+27879.private

# content
$ cat K*.private
[...]
Key: dGVzdC1leGFtcGxlLXRzaWctc2VjcmV0
[...]

Second, allow the zone to be updated by said key:

key example.com. {
 algorithm hmac-sha512;
 secret "odGVzdC1leGFtcGxlLXRzaWctc2VjcmV0";
};

zone "example.com" {
 type master;
 file "/var/lib/bind/db.example.com";
 allow-update { key "example.com."; };
};

And check that the bind user is able to write the /var/lib/bind/ directory and the zone file.

Once you have your DNS server setup, you can again use Tofu and its dns provider to manage the DNS records:

# datasource in the libvirt provider to recover the IP addresses
data "libvirt_domain_interface_addresses" "YOUR_VM" {
 domain = libvirt_domain.YOUR_VM.name
 source = "agent"
}

# set some variables,
# also the libvirt_domain_interface_addresses data might need some filtering there
# (ex: pick the correct nic, filter out localhost, etc)
locals {
 dns_zone = "example.com."
 my_hosts = {
 "your-vm" = data.libvirt_domain_interface_addresses.YOUR_VM.interfaces[2].addrs[0].addr
 }
}

provider "dns" {
 update {
 server = "192.168.100.100" # Replace with your DNS server 
 port = 53
 key_name = "tofu-homelab." # Replace with your zone
 key_algorithm = "hmac-sha512"
 key_secret = "dGVzdC1leGFtcGxlLXRzaWctc2VjcmV0" # base64 TSIG secret from your nameserver config
 }
}

# loop over your VMs
resource "dns_a_record_set" "lan" {
 for_each = local.my_hosts
 zone = local.dns_zone
 name = each.key
 addresses = [each.value]
 ttl = 300
}

# CNAME/Alias example
resource "dns_cname_record" "alias" {
 zone = local.dns_zone
 name = "alias-myvm"
 cname = "myvm.${local.dns_zone}"
 ttl = 300
}

Generating Ansible Inventory

The last bit I like to do is to have Tofu generate an Ansible inventory file.

This enables me to bridge the .tf/infrastructure creation step and the server configuration step more naturally.

In terms of implementation, it’s nothing fancy, just a basic template leveraging the output of the other resources:

locals {
 your_vm_ip = data.libvirt_domain_interface_addresses.YOUR_VM.interfaces[2].addrs[0].addr
}

resource "local_file" "inventory" {
 content = <<-EOF
 [debian_vms]
 your-vm ansible_host=${local.your_vm_ip} ansible_user=${local.debian_admin_user}
 EOF
 filename = "../ansible/inventory/inventory.txt"
 file_permission = "0644"
}

Closing Thoughts

Honestly, after struggling a bit (mostly my fault, I was using an old and buggy version for a while, make sure you are at least on v0.9), even if it’s a little finicky at times, I was surprised how well I managed to get it working.

Being able to nuke entire stacks and re-create them at will, with just a tofu destroy;tofu apply is extremely convenient and strangely satisfying.

Plus, the base hypervisor is extremely simple: just a base headless Debian with libvirtd installed.

Sure, I could have installed Proxmox, use one of the Proxmox Tofu Providers, and have a more AWS/GCP-like experience. But, in truth, this simple setup is already more than enough to cosplay as your typical “latte drinking bigtech/GAFAM SRE/DevSecOps/CloudEng”.

It surely helped me a lot when deploying my @home k8s cluster.

Once again, don’t hesitate to download the full main.tf, and play with it.

Kudos to Duncan Mac-Vicar P. for implementing and sharing this provider.

PCIe x16 to x1 GPU Adapter - Feels wrong, but does work

carpentier.pf@gmail.com (Pierre-François Carpentier) — Fri, 02 Jan 2026 18:29:53 +0100

Why Would You Do That?

I have an old PC tower lying around that I use as a lab/KVM hypervisor from time to time. The thing is completely headless like a server should… maybe a bit too headless actually.

The motherboard doesn’t have any video port, and the two PCIEx16 ports are used by some old OCZ Velodrives storage cards, so there is no way to plug a screen.

To a degree, it’s possible to make do with the serial port, but it gets old really fast when trying to troubleshoot pre-grub boot issues or reinstalling the OS.

The Setup

I vaguely remembered seeing some mining rigs using PCIe risers and even horror stories like dremeling some of the lines off the cards directly. So, I tried my luck with one of these adapters from Amazon.

Next to the card I had lying around (an old Nvidia GT 710), it looks like this:

The card & the adapter

With the added height, the stock bracket no longer fitted, so I designed a new one.

I started from this great collection (from an absolute chad also using FreeCAD).

From there, the design was fairly straightforward, using a photo as reference for drawing the offset cutouts.

new bracket

With the new bracket, things looked like that.

PCIEx1 GPU, Shiny!

And here how it looks with all the cards back in the case:

Yep, that’s full

Some Closing Thoughts

It feels like it should not work, but it does!

That being said, be aware of the slot power limits. I’m not sure why (since additional PCIe lines don’t add power pins), but PCIe x1 slots can draw significantly less power than PCIe x16 slots. Consequently, I would only do it with the lowest power possible GPUs, like an Nvidia GT 710 or an AMD R5.

The cards also need to be low profile for this assembly to fit in the case.

With these caveats in place, this hack does work, however wrong it feels, and is definitely more convenient than having to open the case and juggle cards around whenever this machine is misbehaving.

In the unlikely case you are in the same boat (no video port, or some for AI setup maybe?), it’s definitely an option worth trying, as long as you are not hoping to play Crysis on the poor thing.

GANSS GS87 - Keyboard Factory Reset

carpentier.pf@gmail.com (Pierre-François Carpentier) — Mon, 24 Nov 2025 21:00:37 +0100

Just a really quick one. I’m kind of cheap, so years ago, I bought a Hello Ganss GS87-C mechanical keyboard.

Hello Ganss

Never heard about this brand before or since, but it has served me well enough. Not sure if it has genuine MX Brown Cherries, but these, along with the overall build quality, are good enough.

However, recently, my Alt and Super/Win keys got swapped accidentally and I struggled a bit to solve the issue as I thought initially it was some kind of short in the matrix.

In the end, it was just a setting issue… at the keyboard level, and with a bit of research, I managed to find this manual, and in particular:

Shortcut	Description
`FN` + `Caps`	Switch the position of Caps and Left Ctrl
`FN` + `Win_L`	Lock/Unlock WIN key
`FN` + `A`	Hold down for 3s – Windows system support (Default)
`FN` + `S`	Hold down for 3s – Mac system support
`FN` + `Tab`	On/off the macro functions in the driver
`FN` + `Del`	Ins
`FN` + `PgUp`	Home
`FN` + `PgDw`	End
`FN` + `N`	Hold down for 3s – Power saving function off/on (default off). The indicator flashes once (off) or three times (on)
`FN` + `Space`	Hold down for 5s – Reset to factory settings

FN + Space did the trick… If you told me 20 years ago that, I would need to do a factory reset on a freaking run of the mill keyboard…

I hope it could help someone, or at least feed the AI that helps the same someone.

Nut Embedding: Threading for People Too Cheap to Buy Inserts

carpentier.pf@gmail.com (Pierre-François Carpentier) — Tue, 07 Oct 2025 02:27:43 +0200

Introduction

While the usual way to add threads to a 3D printing part is to use threaded inserts, another often overlooked option is nut embedding.

Inserts usually are the best option, but for onesies/twosies it can be annoying to have to buy and wait a whole bag of those to arrive, especially considering the average tinkerer most likely already has the perfectly sized nut in their collection.

In such cases, nut embedding is a good alternative, here’s how it works:

Leave a pocket in the model for the nut (with appropriate tolerances).
In the slicer, add a pause at the top of the pocket layer.
During the print, when the pause is reached, manually insert the nut into the pocket.
Resume the print and let the printer cover the nut.

How To Do It

Part Design - A Sacrifice To Bridge The Void

Designing the pocket is straightforward: measure your nut and create a pocket with appropriate clearance on each side (+0.5mm on the nut dimensions should do it).

In addition, I recommend adding one thin (~0.2mm) “sacrificial layer” directly into the model.

This sacrificial layer is here to ease bridging over the nut once the latter as been inserted. This layer will be punched through when screwing in the bolt for the first time.

3D model with pocket and sacrificial layer

Some slicers fill-in closed cavities by default. No doubt there is an option somewhere to disable it, but instead of searching for it, an easy hack is to insert a really tiny (~1µm) hole in the sacrificial layer.

Add a tiny 1µm hole to prevent the slicer from filling the void

Here how it looks in 3D:

Loading 3D model...

3D knob model with embedding pocket

Slicing, Printing and Seasoning

Slicer

Once your model is ready, import it into your slicer, slice it, and add a PAUSE at the top of the nut pocket (the start of the sacrificial layer).

The process varies between slicers, but here’s how to do it in PrusaSlicer:

Click Slice now.
Using the vertical slider, navigate to the sacrificial layer.
Right-click on the + icon and select Add Pause.
Export the G-code.

Adding a pause in PrusaSlicer

Note 1: The PAUSE occurs at the start of the selected layer. Don’t be confused by the default horizontal cursor position, which is set at the end of the layer.

Note 2: Make sure your toolhead moves out of the way, either through your printer’s PAUSE macro or by using G-code like:

M25 ; pause SD print
G91 ; switch to relative positioning
G1 X25 Y25 Z25 F3000 ; move nozzle +25mm in X, Y, and Z (away from print)
M0 ; wait for user to press button / resume
G1 X-25 Y-25 Z-25 F3000 ; move nozzle back -25mm to original position
G90 ; return to absolute positioning
M24 ; resume SD print

Or, more simply, if your printer supports it:

M600 X25 Y25 Z25 ; "color change" without retraction.

Printing

Once you have your G-code, start the print and wait.

When the PAUSE occurs, insert the nut carefully (be cautious not to detach the part from the build plate… don’t ask me how I know…).

Insert the nut during the pause

Then resume the print. The printer will add the sacrificial layer and bridge over the nut.

Sacrificial layer being printed over the nut

Let the print complete. If everything goes well, you’ll end up with a clean part and an invisible embedded nut.

Finished part with embedded nut

Conclusion

Embedding nuts in prints is a useful technique that’s definitely handy in a pinch. It can even be marginally more resistant than inserts in some situation.

Plus, this technique isn’t limited to nuts. It can be leveraged in many other situations, such as adding mass to a part, inserting magnets, or adding rebar for reinforcement.

Hope you found this technique interesting, and happy printing.

My Silly Sun Server - Part 3/3: The Software

carpentier.pf@gmail.com (Pierre-François Carpentier) — Wed, 01 Oct 2025 00:01:49 +0200

The Software Side

In Part 1/3 (intro) and Part 2/3 (hardware), we dealt with the hardware side, and rebuilt our cute V100 into a more manageable format.

Now it’s time to resuscitate this server.

For that we will need to interact with the server’s firmware (LOMlite2 & Open Firmware), set up a netboot server, install a usable OS, and finally, configure some services.

LOMlite2

Presentation

LOM stands for Lights Out Management. It fulfills the same role as IPMI compatible devices.

It’s a small Baseboard Management Controller (BMC), similar to HP’s iLO or Dell’s iDRAC, monitoring the hardware (fans, PSUs), setting the boot sequence, and of course, turning it off and on again.

On this V100, we have the LOMlite2 version, which is only accessible through serial (bigger & newer servers, like V210s or T2000s, have ALOM & ILOM with network & telnet/SSH capabilities).

Serial Cabling

To access LOM, you need one of these blue ‘Cisco’ cables with DB9 + RJ45 connectors and a serial adapter:

Blue Cisco console cable with USB serial adapter

Connect it to the upper port on the server, and use your favorite serial terminal software.

Upper RJ45 serial console (LOM) on the Sun V100 rear panel

A SPARC of Life!

The serial connection settings are the common 9600 baud, no parity, one stop bit and full duplex mode (should be the default of your preferred software).

Because I’m lazy, I’m using good old screen but you could use something else like minicom.

screen /dev/ttyUSB0

You should see something like the following when plugging in the server, or at least a lom> prompt:

LOMlite starting up.

CPU type: H8/3437S, mode 3
Ram-test: 2048 bytes OK
Initialising i2c bus: OK
Searching for EEPROMs: 50(cfg) 
I2c eeprom @50: OK
i2c bus speed code 01... OK
Probing for lm80s: none
Probing for lm75s: 48
System functions: PSUs fans breakers rails gpio temps host CLI ebus clock 

LOMlite console
lom>
LOM event: +0h0m0s LOM booted
lom>

Switching Between `ok>`/`OS` & `lom>` Prompts

By default the Serial Port is shared between the lom> prompt and the main server ok (Open Firmware)/OS shell prompt.

Here are a few useful instructions from Sun documentation to navigate between these:

There are three prompts.

 ok> -------------------- (normal prompt when the OS is not running)
 lom> -------------------- (available whether OS is running or not)
 # -------------------- (the OS prompt)

To move between the "ok>" prompt and the "lom>" prompt, type:
 ok> #. There must be less than 1 second between the "#" and "."
 lom> ------> This is the prompt you get

To move between the "lom>" prompt and the "ok>" prompt type:
 lom> console

Factory EEPROM Reset

I initially had issues getting to Open Firmware. In its past life, it seems this server was configured to not share LOM and tty on the same serial port.

I was getting LOM access via the upper port all right, but the Open Firmware/OS on the lower port remained silent.

lom> break
Console not shared.

lom> console -f
Console not shared.

I settled on attempting a factory reset, but it proved somewhat challenging. I tried to use procedures like this one, without luck.

I also tried using the bootmode -u and console -f flags, again, without luck.

I was about to give up when I stumbled upon this email from a former Sun employee. (Thanks Mr. Andy for posting this on a random mailing list 16 years ago, the Internet is truly amazing sometimes).

So, as indicated by this mail, I ran the following commands to reset the eeprom:

lom> set extra-cmds on

Extra commands are reserved for SUN service personnel.
Unauthorised use invalidates machine service warranty.

lom> eepromreset

lom> reset -l

And bingo, I was in business.

I was getting the ok> prompt and could switch back and forth between lom> and ok>

However, when trying to boot net, I was getting:

ok boot net
Fast Data Access MMU Miss

The following reset seems to have solved the issue:

ok set-defaults
Setting NVRAM parameters to default values.
ok reset-all

I’m kind of curious which other magic commands are available.

Open Firmware

Open Firmware is the Forth-powered (a fun little language) boot firmware used by Sun servers and other non‑x86 platforms from that period, like PowerPC Macs and IBM POWER servers. FYI, Sun had its own implementation: OpenBoot.

Useful Commands

Here are a few useful commands:

Main help:

ok help

Enter 'help command-name' or 'help category-name' for more help
(Use ONLY the first word of a category description) 
Examples: help select -or- help line
 Main categories are: 
[...]
System and boot configuration parameters
[...]
nvramrc (making new commands permanent)

Sub-help (note: need to use lowercase):

ok help system

devalias - Display all device aliases
devalias <name> <value> - Create or change a device alias
printenv Show all configuration parameters
 numbers are shown in decimal
setenv <name> <value> Change a configuration parameter
 changes are permanent but only take effect after a reset
 Examples:
 setenv input-device ttya - use ttya input next time
 setenv screen-#rows 0x1e - use 30 rows of display ( hex 1e )
 setenv boot-device net - specify network as boot device
 setenv auto-boot? false - disable automatic boot
set-defaults Revert to factory configuration
See also: nvramrc

List device aliases (boot devices):

ok devalias

dload /pci@1f,0/ethernet@c:,
net0 /pci@1f,0/ethernet@c
net /pci@1f,0/ethernet@c
cdrom /pci@1f,0/ide@d/cdrom@3,0:f
disk3 /pci@1f,0/ide@d/disk@3,0
disk2 /pci@1f,0/ide@d/disk@2,0
disk1 /pci@1f,0/ide@d/disk@1,0
disk0 /pci@1f,0/ide@d/disk@0,0
[...]

Get the current environment variables:

ok printenv

auto-boot? true true
watchdog-reboot? false false
diag-file 
diag-device net net
boot-file 
boot-device disk0 disk net
local-mac-address? false false
net-timeout 0 0
ansi-terminal? true true
screen-#columns 80 80
screen-#rows 34 34
silent-mode? false false

Set the boot device (persistent):

ok setenv boot-device disk0
boot-device = disk0

ok reset

One-time boot of another device:

ok boot net

Netboot & OS Install

Open Firmware on these machines is able to boot over the network a bit like PXE.

Ad‑hoc netboot lab setup for Open Firmware testing

The major difference is the lack of DHCP support: it instead relies on RARP (static MAC -> IP mapping).

Also, the netboot server needs to be on the same LAN subnet (or plugged directly into) as our cute Sun server. And the netboot server must also be a TFTP server hosting the boot file at a set location (derived from the IP we give to our Sun server).

Netboot Server Setup - The Annoying & Legacy Way

Let’s set up a netboot server (Debian/Ubuntu based) as our Sunny God intended.

First, get a root terminal:

# pick your poison
sudo -i
# or
su -

Install the necessary software:

apt install atftpd rarpd

# Set Server NIC & IPs
export BOOT_SERVER_NIC=enp0s25
export SUN_V100_IP=172.24.42.51

# Must be .150 (?)
export BOOT_SERVER_IP=172.24.42.150

Download & put the boot image in the correct TFTP location (IP address in hex):

cd /srv/tftp/
wget https://technically.kakwalab.ovh/files/ofwboot.kakwa.test

# calculate the file name expected by Open Firmware (IP address in hexadecimal)
arg="`echo ${SUN_V100_IP} | sed 's/\./ /g'`"
img_name=`printf "%.2X%.2X%.2X%.2X\n" $arg`

# create hardlink to it
ln -f ofwboot.kakwa.test ${img_name}

Start the TFTP server:

systemctl start atftpd.service

Set the server IP:

ip addr add ${BOOT_SERVER_IP}/24 dev ${BOOT_SERVER_NIC}

Launch rarpd in the foreground:

rarpd -e -dv ${BOOT_SERVER_NIC}

From the LOM connected console, start the V100:

# Set bootmode to ok/ofw prompt
# Note: if nothing is installed, defaults to boot net anyway
lom> bootmode forth
lom> reset
# if necessary
lom> poweron

After a few minutes, you should get the following prompt:

LOM event: +3h36m30s host power on
Aborting startup sequence because of lom bootmode "forth".
Input and output on ttya.
Type fexit to resume normal startup sequence.
Type help for more information
ok

From the ok> prompt, enter the following to initiate netbooting:

ok boot net
Timeout waiting for ARP/RARP packet

You should see log messages like on your rarpd server:

rarpd[16222]: RARP request from 00:03:ba:5b:ae:b3 on enp0s25
rarpd[16222]: not found in /etc/ethers

Replace with your MAC address:

export SUN_V100_MAC="00:03:ba:5b:ae:b3"

# Normalize MAC (uppercase, no colons)
MAC_UPPER=$(echo "$SUN_V100_MAC" | tr '[:lower:]' '[:upper:]')
MAC_NOPUNCT=$(echo "$MAC_UPPER" | tr -d ':')

# Generate a hostname for rarp & ethers/hosts mapping
# Note: could be any name, just avoid collisions
HOSTNAME="sparc-${MAC_NOPUNCT}"

# Make sure ethers file exists
touch /etc/ethers

# Add to /etc/ethers if not already present
grep -q -F "$MAC_UPPER $HOSTNAME" /etc/ethers || \
 echo "$MAC_UPPER $HOSTNAME" | sudo tee -a /etc/ethers

# Add to /etc/hosts if not already present
grep -q -F "$SUN_V100_IP $HOSTNAME" /etc/hosts || \
 echo "$SUN_V100_IP $HOSTNAME" | sudo tee -a /etc/hosts

After a while, the server should get an IP and retrieve the boot file:

Boot device: /pci@1f,0/ethernet@c File and args: 
27a00 >> kakwa's OFW BOOT 1.29
Using RARP protocol: ip address: 172.24.42.1, netmask: 255.255.0.0, server: 172.24.42.150
TFTP IP address: 172.24.42.150
Fast Data Access MMU Miss

It works! Not sure if we have created Paradise or Hell, however…

Netboot Server Setup - The Modern & Masochistic Way

In truth, I’m an atheist, I don’t believe in God, even the Sunnier ones.

And why is that? Well, this setup is really messy, and I’m kind of sorry if you read through it… or worse, if you actually tried to implement it. Also, spoiler, for our NetBSD/OpenBSD netbooting target, even more services are required.

So I’ve committed blasphemy and created my own all-in-one Go netboot server directly providing the RARP + TFTP combo (plus, spoiler, also BOOTP + NFSv2 + HTTP).

I must confess I’ve sinned even more by letting the twin d(a)emons Claude & ChatGPT do most of the work, especially the network protocol implementation. So, expect a few bugs.

On top of that, this netboot server is, by design, very limited. It only provides a single bootstrap path/set of boot files and should only be used within a dedicated LAN segment. Don’t rely on this server if you are still bootstrapping hundreds of SPARC servers like in the good old Jumpstart days. However, for the onesies/twosies like here, don’t hesitate to give it a try.

But enough about why my masochistic tendencies led me to develop a full netboot server for such a dead platform. Here’s how to set up this damn piece of software.

Building the server:

# build dependencies
sudo apt install golang git make

# clone sources
git clone https://github.com/kakwa/ofw-install-server
cd ofw-install-server/
make

# check the help
./ofw-install-server -h

If required, setup your nic

# NIC & IP to use for the boot server
export BOOT_SERVER_NIC="enp0s25"
export BOOT_SERVER_IP="172.24.42.150"

# Configure the NIC if necessary
sudo ip addr add "${BOOT_SERVER_IP}/24" dev "${BOOT_SERVER_NIC}"

Also, optionally, make the deploy server a router:

export WAN_NIC="wlp3s0"

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -F
iptables -t nat -F
iptables -A FORWARD -i ${BOOT_SERVER_NIC} -s ${BOOT_SERVER_IP}/24 -j ACCEPT
iptables -A FORWARD -i ${WAN_NIC} -d ${BOOT_SERVER_IP}/24 -j ACCEPT
iptables -t nat -A POSTROUTING -o ${WAN_NIC} -j MASQUERADE
iptables -t nat -A POSTROUTING -o ${WAN_NIC} -j MASQUERADE

Start the server with the TFTP & RARP module enabled:

# Retrieve something to boot:
wget https://technically.kakwalab.ovh/files/ofwboot.kakwa.test

# Start the server
sudo ./ofw-install-server -iface "${BOOT_SERVER_NIC}" -rarp \
 -tftp -tftp-file ./ofwboot.kakwa.test

Netbooting Debian 6

Just for reference, here is how to get and netboot the last working Debian installer:

# Recover the boot.img file from the netboot installer `.deb` package.
wget https://archive.debian.org/debian/pool/main/d/debian-installer-netboot-images/debian-installer-6.0-netboot-sparc_20110106.squeeze4.b6_all.deb
mkdir debtmp && dpkg-deb -x *.deb debtmp/
find ./debtmp/ -name 'boot.img' -exec cp {} ./boot-debian-6.img \;
rm -rf -- debian-installer-6.0-netboot-sparc_20110106.squeeze4.b6_all.deb ./debtmp/

# Start the server with the Debian boot image
sudo ./ofw-install-server -iface ${BOOT_SERVER_NIC} -rarp \
 -tftp -tftp-file ./boot-debian-6.img

From there, you should be able to install Debian 6, albeit with a few headaches like pointing to the archive.debian.org repository or handling expired GPG keys.

Note: there is a Debian 7 version of the installer, but it kernel-oopsed on me.

`*BSD` == More Netboot Setup…

For NetBSD/OpenBSD, this is “slightly” more complex than the Debian case:

Open Firmware first RARPs & TFTP‑loads a tiny second‑stage loader (ofwboot.net).
That loader then speaks BOOTPARAMS and/or BOOTP to learn its IP and the NFS root path
It then mounts that root over good old NFSv2,
And finally loads the kernel & ramdisk from there.

Yes: RARP + TFTP + BOOTP + NFSv2 on the same LAN segment…

Oh, and to add a bit of fun, NFSv2 is kind of obsolete (removed from Debian since 2022-03-13 in nfs-utils 1:2.6.1-1~exp1).

But once again, with a bit of vibe coding and a few tweaks, we are able to add these services to our custom netboot server.

OpenBSD Install

In the end, I opted for NetBSD so I didn’t fully install OpenBSD. But given I managed to start the installer, I’m fairly confident I would be able to install it if needed.

Also, just for kicks, given all the services we have in our netboot server, I’ve taken the liberty to add one last bit: an HTTP server to serve an OpenBSD autoinstall configuration file.

Lastly, it’s worth mentioning that the OpenBSD version of ofwboot.net gave me quite a lot of headaches. I never quite managed to feed it the NFS server IP and file path/name given by BOOTP (typo? bug? wrong BOOTP option? magic values?).

I also tried to tweak the OpenBSD ofwboot.net, but without luck. If you want to give it a try, the code is available here and can be built under Linux (see README.md instructions). I also tried to make it use RARP and TFTP for the second boot stage instead of NFS+BOOTP.

In the end, I chose to use the NetBSD’s ofwboot.net version for both NetBSD and OpenBSD and it seems to work fine. FYI, both versions come from the same source, but the NetBSD one seems marginally more modern.

But enough said, here’s the setup:

# Tweak it to the latest versions
export OPENBSD_VERSION="7.7"
export NETBSD_VERSION="10.1"

# If you want to try your luck with the OpenBSD ofwboot.net
# wget "https://ftp.openbsd.org/pub/OpenBSD/${OPENBSD_VERSION}/sparc64/ofwboot.net"

# Recover bootloader from NetBSD
wget https://cdn.netbsd.org/pub/NetBSD/NetBSD-${NETBSD_VERSION}/sparc64/installation/netboot/ofwboot.net
# Recover OpenBSD install RamDisk
wget "https://ftp.openbsd.org/pub/OpenBSD/${OPENBSD_VERSION}/sparc64/bsd.rd"
# Recover an autoinstall file
wget https://raw.githubusercontent.com/kakwa/silly-sun-server/f881b5427f1b103411285ac169dbb102ad37f874/misc/openbsd-install.conf

Start the install server:

sudo ./ofw-install-server -rarp -tftp -tftp-file ./ofwboot.net -bootp -nfs -nfs-file ./bsd.rd -http -http-file ./openbsd-install.conf

NetBSD Install

So in the end, I finally settled on installing NetBSD:

Recover & prepare the files:

# Tweak it to the latest version
export NETBSD_VERSION="10.1"

wget "https://cdn.netbsd.org/pub/NetBSD/NetBSD-${NETBSD_VERSION}/sparc64/installation/netboot/ofwboot.net"
wget "https://cdn.netbsd.org/pub/NetBSD/NetBSD-${NETBSD_VERSION}/sparc64/binary/kernel/netbsd-INSTALL.gz"
gunzip netbsd-INSTALL.gz

Start the server:

sudo ./ofw-install-server -rarp -tftp -tftp-file ./ofwboot.net -bootp -nfs -nfs-file ./netbsd-INSTALL

One Last Bit

And then I just did this final tweak in the ok prompt to make it boot on disk persistently:

ok setenv boot-device disk0
reset

Not sure why the default boot-device config (disk net) didn’t work. Maybe the disk dev alias was missing?

But frankly, by this point, I could not care less.

This cute little Sun server is finally working!

A Bit of NetBSD SysAdmin

I will not delve too deep into software configuration as this is not a blog post about NetBSD administration.

But nonetheless, I will present a few useful bits.

Hardware Monitoring

netra-x1# envstat
 Current CritMax WarnMax WarnMin CritMin Unit
[admtemp0]
 internal: 37.000 95.000 -55.000 degC
 external: 58.000 105.000 -55.000 degC
[lom0]
 Fault LED: FALSE
 Alarm1: FALSE
 Alarm2: FALSE
 Alarm3: TRUE

SSHD

Enabling SSH access, because network access is nicer than serial:

grep -q '^sshd=YES' /etc/rc.conf || echo 'sshd=YES' >> /etc/rc.conf
/etc/rc.d/sshd start

NTP

On such an old server, the clock might be out of date, leading to weird errors:

# Force synchronization
ntpdate 2.netbsd.pool.ntp.org

# Enable NTP
grep -q '^ntpd=YES' /etc/rc.conf || echo 'ntpd=YES' >> /etc/rc.conf
/etc/rc.d/ntpd start

Getting A Package Manager

Install pkgin:

pkg_add https://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/`uname -m`/`uname -r`/All/pkgin

Update the index:

pkgin update

Search for packages:

pkgin search neovim

Install packages:

pkgin install neovim

Configuring Services

Now that I had a working server, I used Ansible to configure a bunch of services:

Reverse proxies + basic auth for my 3D printers & other IoTs.
A bit of public static hosting.
A personal FreshRSS instance.

I managed to get everything working, but my NetBSD experience has been a bit rough around the edges.

In particular, the binary packages are rather inconsistent, and frequently have dependencies/linking issues or are really outdated. I had to fall back on pkgsrc quite often, and well… compiling big projects like php or nginx feels like torturing this poor old machine. Given I’ve already angered our Machine Overlords (sketchy story involving compiling Gentoo on a PowerBook Titanium 1 GHz… inside a fridge), it’s something I would prefer to avoid.

If you are in the same SPARCy boat, I’ve published the resulting binary packages here.

But I managed to make everything work in the end. Here are my Ansible playbook & roles if you are interested. They are a bit buggy and not fully idempotent, but they can help you get started if you want to work on NetBSD, even with other CPU architectures.

Lastly, as a final touch, I thought it was fitting to host a copy of the Sun System Handbook on this server.

Conclusion

It was a long project, lasting several months. The CAD/case design part in particular took me quite a while, which was not unexpected given my starting skills. Yet, I’m really pleased with the result, and I’ve learned a lot, from drawing properly constrained sketches to part assembly. Firing up FreeCAD is no longer a dreaded experience, and I’m now much quicker at designing accurate parts.

It also led me to numerous side projects, like playing with USB-C PD, rebuilding my Ender 3 with a new board and Klipper, or learning about laser cutting.

On the software side, it made me discover NetBSD and in the end, I managed to host everything I wanted on this small server.

I hope this cute thing will serve me well enough for at least a few years. SPARC may be on borrowed time, and things may get harder and harder to run, but I trust the NetBSD folks to not drop it soon (they have a reputation to uphold after all).

In any case, I had a lot of fun doing this project, and I hope you had some of this fun reading it.

Prepare for trouble, make it double

My Silly Sun Server - Part 2/3: The Hardware

carpentier.pf@gmail.com (Pierre-François Carpentier) — Tue, 30 Sep 2025 23:01:49 +0200

A Hardware Reinvention

In the introduction (Part 1/3), we outlined the four goals we have for our server.

In this part, we will deal with goals 1 and 2, i.e., the hardware rework — PSU, hard drive, cooling, and creating a custom enclosure that nods to Sun’s original design.

New Parts!

The Shopping List

Here is the shopping list to reach our goals:

The original IDE hard drives are too big and noisy, so I’m replacing them with a SanDisk 64 GB CF card + adapter thingy (CF cards are generally pin-compatible with IDE, giving good odds that it will work with decent performance).
The original PSU is quite bulky, but relatively small at 80 W. So I will also try my luck with a PicoPSU + external 12 V power brick.
Let’s also try our luck with a GaN USB‑C charger + USB‑C PD trigger board + Wide Input 12–25 V PicoPSU. On paper, it could provide us with a great little PSU instead of a rather large and sketchy black brick from an unknown manufacturer.
The original 40 × 40 mm 12 V fans are getting the Noctua treatment.

I also want to design a new and far more compact case for our cute server.

If you ask: yes, all that is not cheap, but saving money isn’t the goal here. If you need a budget‑friendly option, a second‑hand Dell/HP/Lenovo micro PC or an ARM SBC is far cheaper, reliable, efficient, and powerful.

Hard Drive

The original Sun V100 came with a pair of 3.5" IDE hard drives. These were a bit too bulky, and depending on the brand, to noisy.

So I chose to replace them with CompactFlash Card. CF cards are generally pin compatible with PATA/IDE, and only requires very basic and mostly passive adapter which makes then decent (and compact) replacement for old spinning rust.

Hard Drive replacement

The only unknown is how long these CF cards will last when subject to more regular IOs. If it becomes an issue, I might just bit the bullet, and buy a rather expensive industrial SLC variant.

PSU

The Sun V100 uses an 80 W PSU. It has the usual Molex IDE and 20‑pin ATX connectors of the PCs from its era. While it could probably be made quieter with the Noctua treatment, it’s simply too bulky.

So I chose to replace it. I initially tried to use a 120 W 12 V PicoPSU board combined with a USB‑C PD trigger board and a GaN charger.

Unfortunately I didn’t read the fine print closely enough. While USB‑C Power Delivery does define a 12 V level, it’s optional and seems uncommon — at least in the very scientific sample size of two chargers I had on hand.

USB-C Trigger Board

Worse, despite being set to 12 V, when I plugged the trigger board into the PSU, it delivered 15 V… Lesson learned: always check the output voltage on these.

So it was back to a cheap no‑name 12 V brick. But this option is still not quite right. The server often fails to start (switch-on current surge?), and need to be unplugged/plugged several times to get it working.

I will need to revisit the PSU part, this time a multi‑voltage model PicoPSU. Being able to use USB-C would be really nice, both to get quality chargers and maybe to use USB-C power banks as makeshift UPS.

But for now, let’s forge ahead.

Case Modeling

Caveman Meets CAD

I’m a firm believer in open source (and maybe a bit of a masochist), so FreeCAD it is, even if it can be a tad frustrating at times.

Truth be told, FreeCAD is the only CAD software I’ve ever dabbled in, and even then mostly in a “smash some basic shapes together and call it a model” kind of way.

I’m kind of CAD‑illiterate, and my experience so far has felt like being a caveman wandering into a machinist’s workshop with all its lathes, mills, saws, and drills… only to end up bashing them together like rocks.

This project is actually a good excuse to finally learn CAD properly and move beyond my caveman methods.

Side note: If you are also on that path, I highly recommend the great & numerous tutorials from Mango Jelly’s YouTube channel.

Panels

In the end, I settled on the following dimension: 254x220x44.50 mm. It leaves enough room for the main board and its RAM sticks plus a 30mm buffer on the front for fans and a PicoPSU board.

The most complex part to sketch-out was the back panel and the IO ports. But using a scanner, I managed to get an accurate layout.

Is it physical nmap?

From there, I was able to use this scan to sketch out the panel:

Using the scan for the design

The other panels, apart maybe from the fan cutouts, were far simpler to model: a rectangle with four chamferred holes and some recesses for the top panel.

Initially, due to the lack of space (254mm width for a 250mm board), I tried to use laser cut 2mm PMMA (aka plexiglas). But it proved to be too thin, flexible and britle:

2 mm PMMA cracked – too flimsy

In the end, I switched back to 3mm PLA printed panels, with reliefs in places for the board and ram sticks.

Switching to 3mm + reliefs

This option is far more rigid, and I might do something similar to the top and bottom panels (these are still in 2mm PMMA).

Air Duct

Silencing our little beast requires reworking the CPU heatsink.

In particular, the CPU FAN needs to go and fortunately it (and its plastic shroud can easily be detached from the heatsink itself.

But we need to replace it and have some forced draft on the heatsink to keep our venerable server cool.

To do so, I’ve designed an air duct that fits on the heatsink on one end, and to a case mounted 40mm fan on the other.

This was by far the most complex pieces I designed for this project, and it even required mocking the main board to check the fit, specially given the limited space, and the need to leave room for the cabling.

I ended up with a three-part design, constituted of separate top and bottom parts (for easy printing), and a fan shroud, designed to simply slide, and also adapt to different fan depths.

Here’s how it looks in CAD:

Final Fan Duct Design

Bezel

In all honesty, I don’t quite like the V100 front bezel. I don’t know, maybe it’s a bit too stubby, or too cheap? I much prefer the V210/V240 front, so I chose to model my bezel after it:

V100 Front Bezel

V210 Front Bezel

To get the basic shape, I imported the front image, resized to fit 254 × 44.50 mm, and then sketched out the front and back outline:

Sketches for bezel

From there, it was just a matter of lofting the two sketches:

Loft between the two sketches

And finally, adding the remaining features (holes, fillets, cut‑out for the logo):

Bezel final design

I’m quite happy with the result. I’m actually wondering if the concept could be extended to other servers (Dell, HP, Fujitsu, IBM) to pimp out 10‑inch racks.

Logo

The process to create the logo is a bit tedious, mainly due to FreeCAD performance issues, but works as follows:

I started by downloading the Sun logo in the SVG format from wikipedia:

Then, opened it in Inkscape and cleaned-it up (closing path in particular)
I opened it in FreeCAD
Switched to the Draft Workbench
Select all the parts -> Modification -> Draft to Sketch to transform it into a sketch
Closed the sketches whenever necessary

From there, I created a body, a base support (ellipse), ticked Allow Compound in the body properties, padded the ellipse at 1mm, padded the sketch of the logo at 2mm, and added an outter border also at 2mm.

To print it, I did a M600 filament/color change mid-print. Here is a (PrusaSlicer article on how to do it).

And here is the result (printed with a 0.25 nozzle):

3D‑printed Sun logo with filament color change

Let’s Build

Final Design & Assembly

Finally I created the remaining bits (stanoffs, corner brackets, etc), and used the rather new FreeCAD assembly workbench.

It worked and enabled me to validate and fix numerous dimension & positioning mistakes (holes missaligned, cutouts in the wrong place, parts colliding, etc).

The Workbench was a really valuable tool, but quite unstable (save as often as possible). Performance also left something to be desired (I fairness, I didn’t exactly helped it when for completeness, I added all the nuts and bolts to the assembly).

But enough complaining, here how the design looks:

Final Case Design

And now, it’s time to actually build the thing!

After a bit of 3D printers go BRRRRH, I was left with this collection of part:

All the parts

I started with the CPU duct assembly by fitting the inserts. Here, we must install them carefully, as they play double duty: fixation point for the heatsink and used to join the bottom and top parts of the air duct:

double duty inserts: attach to heatsink and join upper and lower parts

From there, we just screw the assembly from below the heatsink.

Heatsink attachement

After that, I just fixed the last part of the air duct to the 40mm FAN, itself screwed to the front panel alongside its twin.

Front panel with fan

And then it was the tedious job of installing the inserts into all the brackets and stand-offs.

5 inserts done, 46 to go

From there, and with a few screws, things started to take shape.

Partially assembled case

And with a bit of epoxy glue to fix the standoffs to the bottom panel (and also to the top for the CF card), the boards were finally secured inside the case, and with a bit of cable management, things looked like that:

final assembly

Time to close it up and admire the result.

Look At This Beauty!

Overall, I’m really happy with the result. This new case is much more compact, quiet, and fit to be used in my small apartment.

Plus, it still looks like a Sun Server:

Silly Sun Server - Front

Round the back, the I/O cut‑outs also fit nicely:

Silly Sun Server - Back

In case that interests you, I’ve made the CAD files for this project available on Github.

To be honest, it’s not perfect. The top/bottom panels still flex a bit and the CPU temperature is somewhat high at ~70 °C.

On the power‑consumption side, it’s “not great — not terrible”. This server draws 20–25 W, vs the typical 5–10 W of mini‑PC or laptop.

But these are far from deal‑breakers, and now this server can actually become useful… if we manage to install something onto it.

Believe it or not, it’s exactly what we will address in the next post (Part 3/3) (huge plot twist, I know…).

My Silly Sun Server - Part 1/3: Introduction

carpentier.pf@gmail.com (Pierre-François Carpentier) — Wed, 27 Aug 2025 19:01:49 +0200

Obsolete Tech In A Modern Age

Something Old Is New Again

One of my pet peeves is to bring new life into old hardware.

Here, I’m not thinking in a retro‑computing kind of way. Software is not exactly like fine wine, in my opinion—it usually doesn’t age well.

Also, old software tends to be isolated in its own bubble—amusing, but unless you are George R. R. Martin rocking WordStar 4.0 on MS‑DOS, it’s not really useful.

I prefer the challenge of trying to install modern software on these antiquities, make them able to interact with the internet, and do something useful with them.

Sure, it will not run a Kubernetes cluster, but these old machines generally still have enough oomph for lightweight use cases such as:

MPD/Music server (for example, coupled with an analog hi‑fi system)
Basic web hosting (blog, RSS aggregator, etc)
Mail server
Test/lab machines

The Little Server That Could (Not)

Enter the Sun V100, the 2001 entry‑level server from Sun Microsystems.

Sun V100 1U Server

It boasts impressive specs such as:

UltraSPARC‑IIe CPU @ 548 MHz
No GPU whatsoever (not great for AI, I guess :p)
2 GB RAM (if maxed out)
2 × 80 GB IDE hard drives
Two 100 Mbit/s NICs
LOMlite management over serial
Second serial port

In truth, this beast is a little asthmatic, even by 2001 standards. Still, the SPARC CPU is interesting, especially for testing endianness and memory alignment issues. And unlike most servers of that era, it doesn’t need a huge amount of power (~15 W TDP).

I got a few of these years ago for cheap, but, in truth, never did anything really useful with them. While not as bad as other monsters from the era, these cuties are simply a bit too big and loud. So, they sat in my cupboard for ages, in the dark, lonely, unused and far away from the information highways.

But let’s try to change that, have a fun project, and maybe learn a few things along the way.

A Two-Front Project

So yes, I aim to do something with this cute little server from another time. Not very rational and probably a bit challenging—but that’s what makes it fun!

But enough said! Now, we need to address the two broad complaints we have about this relic:

Make this server way smaller and quieter
Find and install a modern OS + up‑to‑date software to make it useful

The Hardware Side

Make It Smaller

This (not so) tiny server is well… a server. If it fits onto a small and standard 1U height (44.50 mm), it’s also kind of big in the other dimensions: 19"/482.60 mm by 17.55"/445 mm.

Fortunately, once opened, the server looks like this:

Sun Fire V100 with top cover removed – original layout

This cute beast could probably be a lot more compact. The main board, including the RAM sticks (low‑height variant), is 250 × 190 mm. If we ditch the original (and noisy) hard drives and PSU and cheat a little, we could even make it fit into a 254 mm (10") case and mount it in one of these fancy small 10‑inch racks… if we also manage to make it short enough (less than 300 mm).

This gives goal number one:

make it fit into 254 × 250 × 44 mm.

And also, if possible, as goal number one‑bis:

retain the original “Sun vibe” look.

Make It Quieter

Aside from that, we also need to silence this small beast.

In particular, we need to take care of these little bastards:

Original 40×40 mm 12 V fan

And also their lord and master:

Original CPU 12 V fan

Fortunately, we have options here—especially one that begins with an N.

Since I live in a small apartment, noise levels need to be reasonable—roughly at small NAS or fan‑cooled Wi‑Fi router levels.

That’s goal number two:

bring the noise down to something fit for a living space.

The Software Side

Operating System Choice

The sad reality is SPARC is a dying architecture. So our choices, as of 2025, are limited:

(Open)Solaris -> dead
Illumos (OpenSolaris successor) -> Support is being dropped
Linux -> SPARC64 is still supported by the kernel, but hardly any mainstream distribution supports it (Debian unofficially and Gentoo).
FreeBSD -> Support dropped with FreeBSD 13

This leaves more or less two choices:

Since burning CDs is kind of annoying, and given we will probably ditch the CD drive anyway, we will need to net‑install the OS.

That’s goal number three:

Deploy a net‑install server (similar to a JumpStart setup) and install NetBSD and/or OpenBSD.

Do Something Useful

Here, nothing fancy: we will go for basic web hosting (nginx, Let’s Encrypt, maybe a bit of Postgres & PHP), but since I’m an infra guy who likes infrastructure‑as‑code, let’s set goal number four:

Create a bit of Ansible to do something with our new toy.

The Next Steps

So to recap, we have the following four goals:

Make the server quieter
Make the server smaller
Net‑install NetBSD and/or OpenBSD
Configure some basic web hosting

Next, we’ll focus on the hardware track (Part 2/3) (1 and 2): taming the noise and re‑housing the V100 into a compact 10‑inch chassis while keeping the original Sun aesthetic.

After that, we’ll dive into the software track (Part 3/3) (3 and 4): net‑installing NetBSD/OpenBSD and automating a minimal web stack (nginx, Let’s Encrypt, maybe a dash of Postgres/PHP) with Ansible.

Reversing WoWs Resource Format - Part 5/5: Tidying-Up The Project

carpentier.pf@gmail.com (Pierre-François Carpentier) — Wed, 27 Aug 2025 00:04:00 +0200

Tidying-Up The Project

In the last part we got a first rough implementation.

In this final part, we will clean up things, correct a few shortcuts we took, and create an actual project out of this.

Creating Unit Tests

I did this project right around the release of ChatGPT 3.5. Initially I didn’t plan to add unit tests. But after giving the struct definitions and the format documentation, ChatGPT was able to generate test cases which, while not completely functional, were close enough to start with. It seems evident now, but at the time I was kind of blown away by it.

In the end, you can probably thank our AI overlords for the unit tests of this project. And it was a lifesaver when I significantly reworked the data loading from mmap to a proper unpacking taking endianness into account.

Also, I’ve used the usual suspects of GitHub Actions + CUnit + lcov for CI and code coverage measurement.

Fuzzing

C being the both-ways shotgun it is, you are most likely to get things wrong, especially in the unhappy paths.

In that regard, leveraging fuzzing & AFL++ greatly helps in catching memory issues. It works by taking a collection of valid input files (here, the .idx files) and tweaking them to try triggering crashes.

Here is the gist of using it:

Install AFL++ and build with AFL++ Instrumentation:

# Debian/Ubuntu
apt install afl-clang

cmake -DCMAKE_C_COMPILER=afl-clang -DCMAKE_CXX_COMPILER=afl-clang++ .
make

Run with a collection of valid .idx files

INDEX_DIR="/path/to/WoWs/bin/6831266/idx/"
afl-fuzz -i "$INDEX_DIR" -o ./out -t 10000 -- ./wows-depack-cli -i '@@'

Crashes go to out/crashes/ and can then be investigated using gdb, and potentially used as test/non-regression tests.

API Documentation

Here, we simply call good old Doxygen to the rescue.

The Doxygen annotations are easy to write these days using LLMs: if your naming scheme is decent enough, simply feeding the header definitions (structs and functions) will get you 90% of the way there. Add a few fixes, and you are in business.

Prettier docs are also slightly more likely to be read, so I’m using this nice theme. Just point to it in Doxyfile.in (HTML_EXTRA_STYLESHEET).

And lastly, to keep it up to date, I simply combined GitHub Actions & GitHub Pages to maintain and publish it.

Proper Unpacking

Initially, I did the unpacking using mmap + “casting” to structs. While it works, it’s a bit dangerous as endianness can become an issue, as is data alignment in structs (forces #pragma pack 1 which might not work on every architecture).

So I significantly reworked the project to properly read the file field by field, handling endianness along the way. It was a bit painful to do (having unit tests helped a lot in avoiding regressions there), but now the project is much cleaner on that front.

Annex 1 - A Few Links

Game: WoWs
Closed Source Utility: wowsunpack.exe
My Project: wows-depack (GitHub)
Similar Project in Rust: wowsunpack (GitHub)

Annex 2 - Misc Reverse Engineering Tips

Data Type Identification

These are more rule-of-thumb patterns and need to be used while looking at the surrounding data.

Unsigned integers

32-bit integers generally look like: 02 2F 00 00: higher bytes tend to be 00, lower bytes tend to be used.

The same thing applies to 64-bit integers.

They are typically used to describe the following elements:

counts
size
offsets

Offsets tend to have values divided by 4 or 8 (32 or 64 bits blocks), they also tend to be 64 bits these days (size_t).

32-bit low-value integers tend to be counts or string sizes.

Float

32-bit floats generally have all 4 bytes used, with nearly no zero nibbles, for example: b1 61 33 78.

These are difficult to distinguish from random ints at a glance; they need to be parsed and checked to see if the values are consistent (similar to other fields, within set boundaries, etc.).

RGBA

RGBA look like: 7f 7f fe 00 or 00 ff fe 00. It’s an array of 4 bytes {R,G,B,A} which tend to have recurring values, and often with 1 or 2 bytes in the extreme (00 or ff).

example:

0000b9a0 7f 7f fe 00 82 4b f0 3e 11 f7 02 3f 98 11 9e bf |.....K.>...?....|
0000ba00 7f 7f 00 00 67 cf c5 3e 10 f7 02 3f 98 11 9e bf |....g..>...?....|
0000ba10 00 7f 7f 00 67 cf c5 3e 15 f7 02 3f 36 e0 8c bf |....g..>...?6...|

Here 7f 7f fe 00, 7f 7f 00 00 and 00 7f 7f 00 are like RGBA values (the rest being floats).

Strings

A bunch of printable characters, often 00-terminated.

00015280 7f fe 7f 00 43 4d 5f 50 41 5f 75 6e 69 74 65 64 |....CM_PA_united|
00015290 2e 61 72 6d 6f 72 00 |.armor.|

Tools

File

Just a very simple utility to check for known file signatures:

file *

Strings

Tool to try extracting the strings contained in a given file:

strings -n <MIN_STR_LENGTH> <FILE>

There will be a bit of noise (increasing MIN_STR_LENGTH reduces it), but it should give you interesting human-readable strings contained in a given file.

Hexdump

hexdump is my go-to tool for investigating binary data. I especially like the hexdump -C FILE | less combo:

hexdump -C <FILE> | less

If you are investigating a specific section of a file, you can start at a given offset with the -s <SKIPPED_BYTES> option; this will make things easier to read and navigate and help determine the data alignment:

hexdump -s <SKIPPED_BYTES> -C <FILE> | less

To get a general feel, don’t hesitate to loop over files and display the first bits of a collection:

find ./ -name '*.geometry' | while read file;
do
 hexdump -C $file | head -n 6;
done

ImHex

While I’ve not used it here, you should give a try to ImHex. I’ve used it in subsequent works, and it’s an amazing tool that greatly helps in determining and validating the data structure of binary files.

Annex 3 - File Specification

General Information

The WoWs resources are packed into something similar to a .zip archive (WoTs and WoWPs actually use ZIP files).

Each archive is actually separated into two files:

a .pkg containing the compressed files concatenated together. This file is in the res_packages/ directory.
a .idx containing the index of the files contained in the .pkg and their metadata (name, path, type, etc). This file is located in the ./bin/<build_number>/idx/ directory.

Convention

A byte/8 bits is represented as follows:

+====+
| XX |
+====+

A variable length field (ex: strings) is represented as follows:

|~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
| Field |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~|

The boundary between two fields is marked as follows:

...=++=...
 ||
...=++=...

Misc

Integers (offset and size in particular) are Big Endian.

Strings seem limited to ASCII and are \0 terminated.

Index file

General Layout

The index file is composed of 5 sections:

|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| ^
| Header | } index header (number of files, pointers to sections, etc)
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| v

|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| ^
| File metadata 1 | |
|----------------------------| |
| [...] | } metadata section (pointer to name, type, etc)
|----------------------------| |
| File metadata Nfd | |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| v

|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| ^
| File Name 1 | |
|----------------------------| |
| [...] | } file names (`\0` separated strings)
|----------------------------| |
| File Name Nfd | |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| v

|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| ^
| File `.pkg` pointers 1 | |
|----------------------------| |
| [...] | } pkg pointers section in the `.pkg` file (offsets)
|----------------------------| |
| File `.pkg` pointers Nf | |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| v

|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| ^
| Footer | } index footer (corresponding `.pkg` file name)
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~| V

Layout

+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
| MA | MA | MA | MA || 00 | 00 | 00 | 02 || ID | ID | ID | ID || 40 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
|<----- magic ----->||<--- endianess --->||<------- id ------>||<--- unknown_2 --->|
| 32 bits || 32 bits || 32 bits || 32 bits |

+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
| FD | FD | FD | FD || FO | FO | FO | FO || 01 | 00 | 00 | 00 || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
|<- file_dir_count >||<-- file_count --->||<-------------- unknown_3 ------------->|
| 32 bits || 32 bits || 64 bits |

+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
| HS | HS | HS | HS | HS | HS | HS | HS || OF | OF | OF | OF | OF | OF | OF | OF |
+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
|<------------- header_size ------------>||<------- offset_idx_data_section ------>|
| 64 bits || 64 bits |

+====+====+====+====+====+====+====+=====+
| OE | OE | OE | OE | OE | OE | OE | OE |
+====+====+====+====+====+====+====+=====+
|<----- offset_idx_footer_section ------>|
| 64 bits |

Field descriptions

Field	size	Description
`magic`	32 bits	Magic bytes, always “ISFP”
`endianess`	32 bits	Endianess marker, always 0x2000000 if LE
`id`	32 bits	Unsure, unique per index file, not referenced anywhere else
`unknown_2`	32 bits	Unknown, always 0x40, maybe some offset
`file_dir_count`	32 bits	Number of files + directories (Nfd), also number of entries in the metadata section and the file names section
`file_count`	32 bits	Number of files (Nf), also the number of entries in the file pkg pointers section
`unknown_3`	64 bits	Unknown, always ‘1’, maybe the number of `.pkg` file the index file references (the format hints that it might be supported, but it’s not used)
`header_size`	64 bits	Most likely the header size, always 40
`offset_idx_data_section`	64 bits	Offset to the pkg data section, the offset is computed from `file_plus_dir_count` so `0x10` needs to be added
`offset_idx_footer_section`	64 bits	Offset to the footer section, the offset is computed from `file_plus_dir_count` so `0x10` needs to be added

File Metadata

This section is repeated for each file and directory (header->file_dir_count).

Layout

+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
| NS | NS | NS | NS | NS | NS | NS | NS || OF | OF | OF | OF | OF | OF | OF | OF |
+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
|<---------- file_name_size ------------>||<-------- offset_idx_file_name -------->|
| 64 bits || 64 bits |

+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
| UN | UN | UN | UN | UN | UN | UN | UN || T2 | T2 | T2 | T2 | T2 | T2 | T2 | T2 |
+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
|<----------------- id ----------------->||<------------- parent_id ------------->|
| 64 bits || 64 bits |

[...repeat...]

Field descriptions

Field	Size	Description
`file_name_size`	64 bits	Size of the file name string
`offset_idx_file_name`	64 bits	Offset from the start of the current metadata record to the start of the file name string
`id`	64 bits	Unique ID of the metadata record
`parent_id`	64 bits	ID of the potential parent record (in particular, a directory record)

File names section

This section is just \0 separated list of strings:

Layout

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+====+
| file name string | 00 |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+====+
[...repeat...]

File “.pkg” Pointers

This section is repeated for each file (header->file_count).

Layout

+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
| UO | UO | UO | UO | UO | UO | UO | UO || UT | UT | UT | UT | UT | UT | UT | UT |
+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
|<----------- metadata_id ------------->||<------------- footer_id -------------->|
| 64 bits || 64 bits |

+====+====+====+====+====+====+====+====++====+====+====+====++====+====+====+====+
| OF | OF | OF | OF | OF | OF | OF | OF || T1 | T1 | T1 | T1 || T2 | T2 | T2 | T2 |
+====+====+====+====+====+====+====+====++====+====+====+====++====+====+====+====+
|<--------- offset_pkg_data ----------->||<---- type_1 ----->||<----- type_2 ---->|
| 64 bits || 32 bits || 32 bits |

+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
| OE | OE | OE | OE || ID | ID | ID | ID | ID | ID | ID | ID || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
|<- size_pkg_data ->||<------------ id_pkg_data ------------>||<---- padding ---->|
| 32 bits || 64 bits || 32 bits |
[...repeat...]

Field Descriptions

Field	Size	Description
`metadata_id`	64 bits	ID of the corresponding metadata entry
`footer_id`	64 bits	ID of the footer entry (only one entry possible in practice)
`offset_pkg_data`	64 bits	Offset to the compressed data from the start of the `.pkg` file
`type_1`	32 bits	Compression param 1 (`0x0` for uncompressed, `0x5` for deflate)
`type_2`	32 bits	Compression param 2 (`0x0` for uncompressed, `0x1` for deflate)
`size_pkg_data`	32 bits	Size of the compressed data section in the `.pkg` file
`id_pkg_data`	64 bits	ID of the data section in the `.pkg` file
`padding`	32 bits	Always `0x00000000`

Layout

+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
| UO | UO | UO | UO | UO | UO | UO | UO || U3 | U3 | U3 | U3 | U3 | U3 | U3 | U3 |
+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
|<--------- pkg_file_name_size -------->||<-------------- unknown_7 ------------->|
| 64 bits || 64 bits |

+====+====+====+====+====+====+====+====++~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...
| UT | UT | UT | UT | UT | UT | UT | UT || pkg_file_name_string
+====+====+====+====+====+====+====+====++~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...
|<----------------- id ----------------->|
| 64 bits |

Field descriptions

Field	Size	Description
`pkg_file_name_size`	64 bits	Size of the corresponding `.pkg` file name string
`unknown_7`	64 bits	unknown, looks like an ID
`id`	64 bits	ID of the footer entry

PKG format

PKG Entry

The .pkg format is rather simple, it’s bunch of concatenated compressed (RFC 1951/Deflate) or raw/uncompressed data blobs (one for each file) separated by an ID.

Layout

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
| |
| Compressed (RFC 1951/Deflate) or Raw Data |
| |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
| 00 | 00 | 00 | 00 || XX | XX | XX | XX | XX | XX | 00 | 00 || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
|<--- padding_1 --->||<---------------- id ----------------->||<--- padding_2 --->|
| 32 bits || 64 bits || 32 bits |

[...repeat...]

Field descriptions

Field	Size	Description
`padding_1`	32 bits	Always `0x00000000`
`id_pkg_data`	64 bits	ID of the data blob
`padding_2`	32 bits	Always `0x00000000`

Reversing WoWs Resource Format - Part 4/5: Reading Everything

carpentier.pf@gmail.com (Pierre-François Carpentier) — Tue, 26 Aug 2025 00:03:00 +0200

The Implementation

In the last part, we discovered and got a fairly good idea of the metadata/IDX format.

In this part, we will create a rough implementation to extract the content.

Data Structures

First, define C structures matching our reverse-engineered format:

// INDEX file header
typedef struct {
 char magic[4];
 uint32_t unknown_1;
 uint32_t id;
 uint32_t unknown_2;
 uint32_t file_plus_dir_count;
 uint32_t file_count;
 uint64_t unknown_3;
 uint64_t header_size;
 uint64_t offset_idx_data_section;
 uint64_t offset_idx_footer_section;
} WOWS_INDEX_HEADER;

Parser Implementation

Note: These examples omit error handling for clarity. Production code requires bounds checking and validation.

File Mapping

Memory map the index file:

// Open the index file
int fd = open(args.input, O_RDONLY);

// Recover the file size
struct stat s;
fstat(fd, &s);
size_t index_size = s.st_size;

// Map the whole content in memory
char *index_content = mmap(0, index_size, PROT_READ, MAP_PRIVATE, fd, 0);

The second step is to have an entry point to actually parse the thing:

 WOWS_CONTEXT context;
 context.debug = true;

 // Start the parsing
 return wows_parse_index(index_content, index_size, &context);

Here, I pass the memory-mapped content of the index, its size (will be used in the future to avoid overflows), and a context, which will be used to pass parsing options and maintain “states” in the parsing if necessary.

Parsing the Header Section

int wows_parse_index(char *contents, size_t length, WOWS_CONTEXT *context) {
 // header section
 WOWS_INDEX_HEADER *header = (WOWS_INDEX_HEADER *)contents;

We can print it with a few printf:

int print_header(WOWS_INDEX_HEADER *header) {
 printf("Index Header Content:\n");
 printf("* magic: %.4s\n", (char *)&header->magic);
 printf("* unknown_1: 0x%x\n", header->unknown_1);
 [...]
 return 0;
}

Output:

Index Header Content:
* magic: ISFP
* unknown_1: 0x2000000
* id: 0xb4399d91
* unknown_2: 0x40
* file_plus_dir_count: 311
* file_count: 284
* unknown_3: 1
* header_size: 40
* offset_idx_data_section: 0x3bf6
* offset_idx_footer_section: 0x7136

Metadata entries

Then, we can do a bunch of pointer arithmetic operations to extract the other sections of the index file:

 // Recover the start of the metadata array
 WOWS_INDEX_METADATA_ENTRY *metadatas;
 metadatas =
 (WOWS_INDEX_METADATA_ENTRY *)(contents + sizeof(WOWS_INDEX_HEADER));

Then, we do something with these sections, like for example:

 // Parse & print each entry in the metadata section
 for (i = 0; i < header->file_plus_dir_count; i++) {
 if (context->debug) {
 print_metadata_entry(&metadatas[i], i);
 }
 }

With print_metadata_entry looking like this:

int print_metadata_entry(WOWS_INDEX_METADATA_ENTRY *entry, int index) {
 printf("Metadata entry %d:\n", index);
 printf("* file_type: %lu\n", entry->file_type_1);
 printf("* offset_idx_file_name: 0x%lx\n", entry->offset_idx_file_name);
 printf("* unknown_4: 0x%lx\n", entry->unknown_4);
 printf("* file_type_2: 0x%lx\n", entry->file_type_2);
 return 0;
}

Re-Evaluating some of the fields meaning:

Once done, it gives us a more comfortable read:

[...]
Metadata entry 0:
* file_type: 14
* offset_idx_file_name: 0x26e0
* unknown_4: 0x93b6404fba9a0c8f
* file_type_2: 0xdbb1a1d1b108b927

Metadata entry 1:
* file_type: 19
* offset_idx_file_name: 0x26ce
* unknown_4: 0xc7f7d0284a87ec8f
* file_type_2: 0x74d821503e1beba4

Metadata entry 2:
* file_type: 18
* offset_idx_file_name: 0x26c1
* unknown_4: 0x6b4f2cace7a270ad
* file_type_2: 0xdbb1a1d1b108b927

Metadata entry 3:
[...]

Metadata entry 310:
* file_type: 19
* offset_idx_file_name: 0x14fb
* unknown_4: 0xce7afff48d1bd174
* file_type_2: 0x74d821503e1beba4

This permits us to review our previous reverse and right away there are two interesting things to note:

There was a bit of an unknown regarding the number of metadata chunks: was it file_count or file_plus_dir_count? Now we are more certain it’s file_plus_dir_count as it’s the larger value. If it was not, we would try to parse a section past the metadatas as metadata with funky results (garbage or crash). This is not the case.
file_type in metadata is not a file type/enum. The values are small but quite varied; it’s more likely the length of the file name.

Lets check with the last entry:

Metadata entry 310:
* file_type: 19
[...]

kakwa@linux 6775398/idx » hexdump -C system_data.idx | less

[...]
00003be0 61 72 69 61 74 69 6f 6e 5f 64 75 6d 6d 79 2e 64 |ariation_dummy.d|
00003bf0 64 73 00 77 61 76 65 73 5f 68 65 69 67 68 74 73 |ds.waves_heights|
00003c00 30 2e 64 64 73 00 8f 0c 9a ba 4f 40 b6 93 70 11 |0.dds.....O@..p.|
00003c10 03 07 0d 33 ed 77 00 00 00 00 00 00 00 00 05 00 |...3.w..........|

The last file name is waves_heights0.dds, 18 characters long, with the \0, we have our 19 value.

So let’s rename this field.

Now that we have fixed that, we can recover the file names of each entry:

char *filename = (char *)entry;
filename += entry->offset_idx_file_name;

printf("* filename: %.*s\n", (int)entry->file_name_size, filename);

Nice:

Metadata entry 0:
* file_name_size: 14
* offset_idx_file_name: 0x26e0
* unknown_4: 0x93b6404fba9a0c8f
* file_type_2: 0xdbb1a1d1b108b927
* filename: KDStorage.bin

Metadata entry 1:
* file_name_size: 19
* offset_idx_file_name: 0x26ce
* unknown_4: 0xc7f7d0284a87ec8f
* file_type_2: 0x74d821503e1beba4
* filename: waves_heights1.dds

We have file names and directory names, for example:

[...]
Metadata entry 10:
* file_name_size: 11
* offset_idx_file_name: 0x2654
* unknown_4: 0x46c008ccf65395e0
* file_type_2: 0x46e29969bd85cf06
* filename: space_defs

Metadata entry 11:
* file_name_size: 13
* offset_idx_file_name: 0x263f
* unknown_4: 0x7213702d5e6899e0
* file_type_2: 0x13d93873302ed14c
* filename: aid_null.dds

Metadata entry 12:
* file_name_size: 16
* offset_idx_file_name: 0x262c
* unknown_4: 0xa1a829d8713f89e0
* file_type_2: 0xdbb1a1d1b108b927
* filename: camouflages.xml
[...]

There is something which should enable us to differentiate between the two, maybe one of the unknown fields.

Also, we still need to figure out how the directory system works:

How directories & subdirectories are composed (to get <dir>/<sub dir>/<sub sub dir>/ paths)
How the path goes back to the root (/)

We will not look at it here, but that’s something to keep in mind.

Extracting footer information:

WOWS_INDEX_FOOTER *footer = (WOWS_INDEX_FOOTER *)(contents + header->offset_idx_footer_section);
print_footer(footer);

The results were incorrect due to an offset miscalculation.

Index Footer Content:
* size_pkg_file_name: 50b0bd0300002d0b
* unknown_7: 0xe967
* unknown_6: 0x15

A file name size of 50b0bd0300002d0b? I don’t think so.

So let’s look at it more closely.

In the header, we have:

Index Header Content:
[...]
* offset_idx_footer_section: 0x7136
[...]

The hexdump gives:

kakwa@linux 6775398/idx » hexdump -s 6 -C system_data.idx| less

[...]
00007116 21 67 ac 70 22 ec ca b8 70 11 03 07 0d 33 ed 77 |!g.p"...p....3.w|
00007126 28 f9 15 0a 00 00 00 00 05 00 00 00 01 00 00 00 |(...............|
00007136 0b 2d 00 00 03 bd b0 50 67 e9 00 00 00 00 00 00 |.-.....Pg.......|
00007146 15 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00 |................|
00007156 70 11 03 07 0d 33 ed 77 73 79 73 74 65 6d 5f 64 |p....3.wsystem_d|
00007166 61 74 61 5f 30 30 30 31 2e 70 6b 67 00 |ata_0001.pkg.|

If our previous interpretation was correct, a simple offset from the start of the index file should be 0x7146, not 0x7136.

Maybe we are missing some fields in the footer, but given the previous 128 bits at offset 0x7136 really look like the end of a pkg metadata entry, I doubt it.

A more plausible explanation is that the offset is relative to the header id field at 0x10. Maybe the magic + unknown_1 bits, i.e., the first 128 bits, are considered to be a separate section.

Anyway, let’s just offset by 128 bits.

#define MAGIC_SECTION_OFFSET sizeof(uint32_t) * 4

// Get the footer section
WOWS_INDEX_FOOTER *footer = (WOWS_INDEX_FOOTER *)(contents + header->offset_idx_footer_section + MAGIC_SECTION_OFFSET);

That’s better:

Index Footer Content:
* size_pkg_file_name: 23
* unknown_7: 0x18
* unknown_6: 0xb5a4fa9349d9fd0d

We can also recover the pkg file name as follows:

char *pkg_filename = (char *)footer;
pkg_filename += sizeof(WOWS_INDEX_FOOTER);
printf("* pkg filename: %.*s\n",
 (int)footer->size_pkg_file_name, pkg_filename);

Safety Considerations

The current implementation lacks bounds checking and trusts all offsets—this needs fixing in production code.

PKG Data Entries

Similar process for the data section:

And lets add the 128 bits from the start (hexdump gives us 0x3c06, which is again a 0x10 difference with 0x3bf6).

 // Get pkg data pointer section
 WOWS_INDEX_DATA_FILE_ENTRY *data_file_entry =
 (WOWS_INDEX_DATA_FILE_ENTRY *)(contents +
 header->offset_idx_data_section +
 MAGIC_SECTION_OFFSET);

From there, we are not sure if we have header->file_plus_dir_count or header->file_count entries. The latter seems more likely as this section points to the pkg files, but that’s not a given.

Also, we are unsure how one entry there is paired with a metadata entry. Maybe the order is simply the same in this array, maybe the matching is done through one of the unknown field.

But first, lets dump the content with some printf:

Data file entry [0]:
* unknown_5: 0x93b6404fba9a0c8f
* unknown_6: 0x77ed330d07031170
* offset_pkg_data_chunk: 0x0
* type_1: 0x5
* type_2: 0x1
* size_pkg_data_chunk: 0x21f5
* id_pkg_data_chunk: 0x366c
* padding: 0x4a87ec8f

Data file entry [1]:
* unknown_5: 0x77ed330d07031170
* unknown_6: 0x5ef9b1e
* offset_pkg_data_chunk: 0x100000005
* type_1: 0x11515
* type_2: 0x97637703
* size_pkg_data_chunk: 0x2ab3e
* id_pkg_data_chunk: 0x6b4f2cace7a270ad
* padding: 0x7031170
[...]

Humm, that doesn’t look righ… Why is the padding not the expected 0x0? Also why the first entry looks mostly ok, except the padding, and the rest is garbage.

First, I double-check the WOWS_INDEX_DATA_FILE_ENTRY field sizes, and it was ok.

Then, I remembered that compilers can add padding to have all the fields properly aligned in memory, this helps with performances.

To avoid that, we need to add:

#pragma pack(1)

Now the output looks like that:

* unknown_5: 0x93b6404fba9a0c8f
* unknown_6: 0x77ed330d07031170
* offset_pkg_data_chunk: 0x0
* type_1: 0x5
* type_2: 0x1
* size_pkg_data_chunk: 0x21f5
* id_pkg_data_chunk: 0x366c5c4500bf
* padding: 0x0

Data file entry [1]:
* unknown_5: 0xc7f7d0284a87ec8f
* unknown_6: 0x77ed330d07031170
* offset_pkg_data_chunk: 0x5ef9b1e
* type_1: 0x5
* type_2: 0x1
* size_pkg_data_chunk: 0x11515
* id_pkg_data_chunk: 0x2ab3e97637703
* padding: 0x0

Data file entry [2]:
* unknown_5: 0x6b4f2cace7a270ad
* unknown_6: 0x77ed330d07031170
* offset_pkg_data_chunk: 0x2205
* type_1: 0x5
* type_2: 0x1
* size_pkg_data_chunk: 0x1cb
* id_pkg_data_chunk: 0xcadc1deb96d
* padding: 0x0

That’s much better.

But this small issue raises a number of issues with my method of parsing. Casting to structs comes with numerous issues, from overflows to endianness.

This is not that critical here since we are just trying to have a rough prototype, but on more critical software, that’s not a good idea.

After this prototype, it might be a good idea to start learning Rust ^^.

Also, if we try to parse header->file_plus_dir_count entries, we get the following:

Data file entry [283]:
* unknown_5: 0xb8caec2270ac6721
* unknown_6: 0x77ed330d07031170
* offset_pkg_data_chunk: 0xa15f928
* type_1: 0x5
* type_2: 0x1
* size_pkg_data_chunk: 0x2d0b
* id_pkg_data_chunk: 0xe96750b0bd03
* padding: 0x0

Data file entry [284]:
* unknown_5: 0x15
* unknown_6: 0x18
* offset_pkg_data_chunk: 0x77ed330d07031170
* type_1: 0x74737973
* type_2: 0x645f6d65
* size_pkg_data_chunk: 0x5f617461
* id_pkg_data_chunk: 0x676b702e31303030
* padding: 0x0

Data file entry [285]:
* unknown_5: 0x0
* unknown_6: 0x0
* offset_pkg_data_chunk: 0x0
* type_1: 0x0
* type_2: 0x0
* size_pkg_data_chunk: 0x0
* id_pkg_data_chunk: 0x0
* padding: 0x0

The entry 283 is OK. This is the 284th entry since we start at 0, which is exactly header->file_count. The next one has weird values and the rest just 0.

So header->file_count is indeed the number of entries in this section.

Entry Matching

The fact that on one side we have header->file_count and on the other header->file_plus_dir_count means it’s not a simple index matching.

Lets investigate the unknown fields:

[...]
Data file entry [279]:
* unknown_5: 0xce7afff48d1bd174
* unknown_6: 0x77ed330d07031170
[...]

Data file entry [280]:
* unknown_5: 0x199e99feb0c986f8
* unknown_6: 0x77ed330d07031170
[...]

unknown_6 is always the same, not really interesting.

unknown_5 on the contrary is specific to each entry:

kakwa@linux GitHub/wows-depack (main) » ./wows-depack-cli \
 -i ~/Games/World\ of\ Warships/bin/6775398/idx/system_data.idx | \
 grep 'unknown_5' | sort | uniq -c
[...]
 1 * unknown_5: 0x14b002d7c2835863
 1 * unknown_5: 0x15a7b41a61f65f9c
 1 * unknown_5: 0x15fcab5401f27f56
 1 * unknown_5: 0x18a0d0dc4b05f8fa
 1 * unknown_5: 0x192a05120f00553e
[...]

The values, however, are present two times—one in the metadata entry, the other in the data file entry:

Metadata entry [72]:
[...]
* unknown_4: 0x1011b17d9304bb39
[...]

Data file entry [65]:
[...]
* unknown_5: 0x1011b17d9304bb39
[...]

So the link is established through these fields. These are simply random, unique IDs for each entry.

In fact unknown_4 and unknown_5 are not the only fields leveraging this.

Looking at file_type_2 values, we get something like that:

kakwa@linux GitHub/wows-depack (main *) » ./wows-depack-cli \
 -i ~/Games/World\ of\ Warships/bin/6775398/idx/system_data.idx | \
 grep '0x937f155e4baaf562\|filename:' | grep -A 1 '0x937f155e4baaf562'

[...]
--
* file_type_2: 0x937f155e4baaf562
* filename: LowerAftTrans.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: MidBarbette.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: Bulkhead.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: MidBelt.dds
--
[...]
--
* unknown_4: 0x937f155e4baaf562
* filename: armour
--
[...]
--
* file_type_2: 0x937f155e4baaf562
* filename: Bottom.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: ConstrBig.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: ConstrMid.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: ConstrSm.dds
--
* file_type_2: 0x937f155e4baaf562
* filename: DoubleBottom.dds
--
[...]

Ok, file_type_2 is not a file type at all; it’s the id (unknown_4 right now) of just one node that really looks like a directory.

file_type_2 should probably be renamed parent_id or something.

Also, unknown_6 follows the same logic: it’s the id of the footer entry (side note: maybe the format supports having one index for several files).

Small Tangent

By this point, I was a bit intrigued by the type_1 and type_2 fields in the pkg pointer sections.

kakwa@linux GitHub/wows-depack (main) » for i in ~/Games/World\ of\ Warships/bin/6775398/idx/*;\
do
 ./wows-depack-cli -i "$i" | grep 'type_[12]:';
done | sort -n | uniq -c | sort -n

 57265 * type_1: 0x0
 57265 * type_2: 0x0
 232089 * type_1: 0x5
 232089 * type_2: 0x1

Ok, it seems that (type_1, type_2) can either have the (0x0, 0x0) values, or the (0x5, 0x1) values. In most cases, it’s the latter.

Looking at a few (0x0, 0x0), a common file with such values are .png.

It’s a bit of a wild guess, but these might be compression levels. (0x0, 0x0), i.e. no compression would be logical for .png as these files are already compressed. Compressing them would actually only cost CPU resources with no space gains.

For now, lets just keep that in mind, we will revisit it later.

Glueing the entries together

So, we have metadata entries which can be linked together, we have pkg pointer entries which are linked to metadata entries and footer entries.

It’s time to link all that together.

To do that, the obvious choice is to feed these IDs into an hash map, this will make look-ups easier and quicker.

Once done, the output now looks like that:

File entry [259]:
* metadata_id: 0xc90b3d356989c551
* footer_id: 0x77ed330d07031170
* offset_pkg_data: 0x5d30db8
* type_1: 0x5
* type_2: 0x1
* size_pkg_data: 0x89667
* id_pkg_data: 0xaab740ef4f6a6
* padding: 0x0
* file_name_size: 18
* offset_idx_file_name: 0x164e
* id: 0xc90b3d356989c551
* parent_id: 0xeb7ddcfb5178376
* filename: snow_tiles_ah.dds
parent [1]:
* file_name_size: 8
* offset_idx_file_name: 0x19cf
* id: 0xeb7ddcfb5178376
* parent_id: 0xb220a7743c83e638
* filename: weather
parent [2]:
* file_name_size: 5
* offset_idx_file_name: 0x16a3
* id: 0xb220a7743c83e638
* parent_id: 0x3837637adc4586b1
* filename: maps
parent [3]:
* file_name_size: 7
* offset_idx_file_name: 0x1746
* id: 0x3837637adc4586b1
* parent_id: 0xdbb1a1d1b108b927
* filename: system

This entry is for the following path: /system/maps/weather/snow_tiles_ah.dds

This should be enough now to start thinking about the actual tool and how it will be implemented.

Another tangent

The goal is in the end is to parse all the index files, so it got me curious about the IDs across the different files.

Looking at the dumps, content is a fairly common directory name, present in a lot of the index files.

And if we look at these records, we get:

kakwa@linux GitHub/wows-depack (main *%) » for i in ~/Games/World\ of\ Warships/bin/6775398/idx/*;\
do \
 echo $i; \
 ./wows-depack-cli -i "$i" | grep -B 5 '* filename: content$'; \
done | less

[...]
/home/kakwa/Games/World of Warships/bin/6775398/idx/camouflage.idx
parent [5]:
* file_name_size: 8
* offset_idx_file_name: 0xecae
* id: 0xa33046442d8327fc
* parent_id: 0xdbb1a1d1b108b927
* filename: content
--
parent [4]:
* file_name_size: 8
* offset_idx_file_name: 0xecae
* id: 0xa33046442d8327fc
* parent_id: 0xdbb1a1d1b108b927
* filename: content
--
parent [5]:
* file_name_size: 8
* offset_idx_file_name: 0xecae
* id: 0xa33046442d8327fc
* parent_id: 0xdbb1a1d1b108b927
* filename: content
[...]

Interestingly id is always 0xa33046442d8327fc (and also parent_id is 0xdbb1a1d1b108b927). This will make implementation a bit easier.

However, it raises an interesting question: how this id is generated? Is it completely random? Or is it derived from the path/name?

It’s not really critical to read files, but might be important to write content if we ever get to that.

File System Tree

Build a tree structure using:

HashMap for fast ID lookups (hashmap.c)
Inode types: files and directories
Tree construction: Start with PKG data chunks (files), resolve names via metadata, build directory hierarchy using parent_id relationships

The result: a complete archive tree with root node and children.

With a bit more work, adding a path/tree printer function, I now get something like that:

/postfx_animations.xml
/settings/Default_v3.settings
/settings/Default_v1.settings
/settings/Default_v2.settings
/scripts/user_data_object_defs/Barge.def
/scripts/user_data_object_defs/SpatialUIDebugTool.def
/scripts/user_data_object_defs/FogPoint.def
/scripts/user_data_object_defs/StaticSoundEmitter.def
[...]
/helpers/maps/green_hemisphere.dds
/helpers/maps/lev_dirt01.dds
/helpers/maps/fat_disc_quarter.dds
/helpers/maps/lev_grass_01.dds
/helpers/maps/lev_grassflowers.dds
/helpers/maps/red_ruler.dds
/helpers/maps/hemisphere.dds
/helpers/maps/disc_quarter.dds
/helpers/maps/red_hemisphere_ring.dds
/server_stats.xml

Or that in (ugly) tree form:

-./
 |-* postfx_animations.xml
 |--settings/
 | |-* Default_v3.settings
 | |-* Default_v1.settings
 | |-* Default_v2.settings
 |--scripts/
 | |--user_data_object_defs/
 | | |-* Barge.def
 | | |-* SpatialUIDebugTool.def
 | | |-* FogPoint.def
 | | |-* StaticSoundEmitter.def
 | | |-* Minefield.def
 | | |-* SoundedEffect.def
 | | |-* SquadronReticleTool.def
 | | |-* Trigger.def
 | | |-* WayPoint.def
[...]

Recap (Part 4)

We defined the C structs for the metadata/index
We resolved a few loose ends: filename lengths, parent IDs, unique identifiers for linking
We implemented a file tree using hashmaps and parent-child relationships
We identified compression type patterns

In the next and last part we will tie things together and wrap up the implementation.

Reversing WoWs Resource Format - Part 3/5: Dissecting The Index

carpentier.pf@gmail.com (Pierre-François Carpentier) — Mon, 25 Aug 2025 12:02:00 +0200

Index File Reverse Engineering

In the last part, we found the location of the index and determine the general layout.

Now, let’s do the detailed reverse engineering.

Metadata Chunk Format

The first section contains file metadata entries:

kakwa@linux 6775398/idx » hexdump -C system_data.idx | less

00000000 49 53 46 50 00 00 00 02 91 9d 39 b4 40 00 00 00 |ISFP......9.@...|
00000010 37 01 00 00 1c 01 00 00 01 00 00 00 00 00 00 00 |7...............|
00000020 28 00 00 00 00 00 00 00 f6 3b 00 00 00 00 00 00 |(........;......|
00000030 36 71 00 00 00 00 00 00 0e 00 00 00 00 00 00 00 |6q..............|
00000040 e0 26 00 00 00 00 00 00 8f 0c 9a ba 4f 40 b6 93 |.&..........O@..|
00000050 27 b9 08 b1 d1 a1 b1 db 13 00 00 00 00 00 00 00 |'...............|
00000060 ce 26 00 00 00 00 00 00 8f ec 87 4a 28 d0 f7 c7 |.&.........J(...|
00000070 a4 eb 1b 3e 50 21 d8 74 12 00 00 00 00 00 00 00 |...>P!.t........|
00000080 c1 26 00 00 00 00 00 00 ad 70 a2 e7 ac 2c 4f 6b |.&.......p...,Ok|
00000090 27 b9 08 b1 d1 a1 b1 db 0e 00 00 00 00 00 00 00 |'...............|
000000a0 b3 26 00 00 00 00 00 00 4e 84 a5 6a 94 dc 1f 7f |.&......N..j....|
000000b0 62 f5 aa 4b 5e 15 7f 93 10 00 00 00 00 00 00 00 |b..K^...........|
000000c0 a1 26 00 00 00 00 00 00 4e b0 fe 23 62 40 a5 65 |.&......N..#b@.e|
000000d0 27 b9 08 b1 d1 a1 b1 db 20 00 00 00 00 00 00 00 |'....... .......|
000000e0 91 26 00 00 00 00 00 00 8e c7 6a 58 7c 86 62 33 |.&........jX|.b3|
000000f0 27 b9 08 b1 d1 a1 b1 db 0f 00 00 00 00 00 00 00 |'...............|
00000100 91 26 00 00 00 00 00 00 8e 43 3d e9 cf 49 52 a4 |.&.......C=..IR.|
00000110 62 f5 aa 4b 5e 15 7f 93 16 00 00 00 00 00 00 00 |b..K^...........|
00000120 80 26 00 00 00 00 00 00 0e 3c 9a 6d 22 de 7b da |.&.......<.m".{.|
00000130 4e b0 fe 23 62 40 a5 65 0b 00 00 00 00 00 00 00 |N..#b@.e........|
00000140 76 26 00 00 00 00 00 00 0e 48 58 ea 50 44 1a 47 |v&.......HX.PD.G|
00000150 df 61 50 67 c7 3a dd 7a 13 00 00 00 00 00 00 00 |.aPg.:.z........|
00000160 61 26 00 00 00 00 00 00 e0 9f c3 bd d2 12 20 04 |a&............ .|
00000170 09 28 f1 df 2d 04 93 de 0b 00 00 00 00 00 00 00 |.(..-...........|

00002690 a4 eb 1b 3e 50 21 d8 74 0c 00 00 00 00 00 00 00 |...>P!.t........|
000026a0 21 15 00 00 00 00 00 00 00 40 cc c7 49 c4 54 09 |!........@..I.T.|
000026b0 a4 eb 1b 3e 50 21 d8 74 14 00 00 00 00 00 00 00 |...>P!.t........|
000026c0 0d 15 00 00 00 00 00 00 ce 6a 28 bc cf e7 79 c8 |.........j(...y.|
000026d0 a4 eb 1b 3e 50 21 d8 74 1a 00 00 00 00 00 00 00 |...>P!.t........|
000026e0 01 15 00 00 00 00 00 00 6c c0 c9 f7 7e 00 03 05 |........l...~...|
000026f0 a4 eb 1b 3e 50 21 d8 74 13 00 00 00 00 00 00 00 |...>P!.t........|
00002700 fb 14 00 00 00 00 00 00 74 d1 1b 8d f4 ff 7a ce |........t.....z.|
00002710 a4 eb 1b 3e 50 21 d8 74 4b 44 53 74 6f 72 61 67 |...>P!.tKDStorag|
00002720 65 2e 62 69 6e 00 77 61 76 65 73 5f 68 65 69 67 |e.bin.waves_heig|
00002730 68 74 73 31 2e 64 64 73 00 61 6e 69 6d 61 74 65 |hts1.dds.animate|
00002740 64 4d 69 73 63 73 2e 78 6d 6c 00 4c 6f 77 65 72 |dMiscs.xml.Lower|
00002750 44 65 63 6b 2e 64 64 73 00 63 6f 6d 6d 61 6e 64 |Deck.dds.command|

Staring at the hexdump long enough and we can start to see some patterns.

At regular intervals, every 256 bits, we get a 64-bit integer with a relatively low value, hinting at individual metadata sets of 256 bits.

This looks suspiciously like some kind of constant enum coded into a 64-bit integer. My best guess right now would be some kind of file type code, itself defined as a constant in the game engine like this:

#define FILE_TYPE_1 0x01
#define FILE_TYPE_2 0x02
// [...]

Let’s call it file_type for now.

Looking at the end of the first chunk (right before we get KDStorage.bin), we have to go back 4 x 64 bits to get something that looks like a file_type.

This means file_type is the first field in the 256-bit structure.

Let’s look at the next 64 bits: ce 26 00 00 00 00 00 00, c1 26 00 00 00 00 00 00, 80 26 00 00 00 00 00 00, etc. These values are again rather small.

Also these values, at least for the first ones, are suspiciously close to 00002718, i.e. right where the section containing the file names starts.

The last ones, 01 15 00 00 00 00 00 00, fb 14 00 00 00 00 00 00, etc., are smaller, and suspiciously, they have similar values to the length of the file name section (approximately 0x00003c00 - 0x00002710 = 0x0000014f0).

The second field is then probably some kind of offset. Most likely from the start of one 256-bit chunk to the start of one of the file names.

Looking at other .idx files seems to confirm that.

Let’s call it offset for now.

Next, let’s look at the remaining 128 bits.

8f 0c 9a ba 4f 40 b6 93 | 27 b9 08 b1 d1 a1 b1 db
8f ec 87 4a 28 d0 f7 c7 | a4 eb 1b 3e 50 21 d8 74
ad 70 a2 e7 ac 2c 4f 6b | 27 b9 08 b1 d1 a1 b1 db
4e 84 a5 6a 94 dc 1f 7f | 62 f5 aa 4b 5e 15 7f 93
4e b0 fe 23 62 40 a5 65 | 27 b9 08 b1 d1 a1 b1 db
8e c7 6a 58 7c 86 62 33 | 27 b9 08 b1 d1 a1 b1 db

First thing to note: all the bits are used, which disqualifies offsets or simple enum IDs like before.

Right now, we are not even sure these 128 bits are part of one 128-bit field (for example a hash), two 64-bit integers, four 32-bit integers, or any combination of 16, 32 or 64 bits that ends up making a 128-bit chunk.

Looking at it more closely, the first 64 bits looks rather random, the last 64 bits however? we see quite a few values repeating themselves (ex: 27 b9 08 b1 d1 a1 b1 db).

Let’s check the whole file:

kakwa@linux 6775398/idx » hexdump -C system_data.idx | grep '00 00 00 00 00 00 00' | \
 sed 's/^..........//' | sed 's/ .*//' | sort | uniq -c | sort -n

# note: first number is the number of occurrences
 1 00 00 00 00 00 00 00 af
 1 1c 6a 8d 7f df 8e b3 35
 1 28 00 00 00 00 00 00 00
 1 36 71 00 00 00 00 00 00
 1 37 01 00 00 1c 01 00 00
 1 7d e3 1c a4 35 3e 98 8b
 1 e0 95 53 f6 cc 08 c0 46
 2 12 46 58 36 c8 ec 47 8b
 2 4e b0 fe 23 62 40 a5 65
 2 d0 d7 a5 ce a8 86 0e ae
 3 1a aa c7 3c 4e 76 ad 94
 3 4c d1 2e 30 73 38 d9 13
 3 4c f0 ea c1 d5 5a 8d 12
 3 88 57 fc 1c 72 f3 84 fa
 3 ac 82 d5 f6 9e db 47 f9
 3 b1 86 45 dc 7a 63 37 38
 3 fc 27 83 2d 44 46 30 a3
 6 76 83 17 b5 cf dd b7 0e
 8 06 cf 85 bd 69 99 e2 46
 9 09 28 f1 df 2d 04 93 de
 10 a4 eb 1b 3e 50 21 d8 74
 10 d7 22 2f fc 0a 67 7a 0d
 14 aa db f0 18 01 89 b6 d8
 15 df 61 50 67 c7 3a dd 7a
 18 19 ac 65 3f 91 78 97 dc
 19 38 e6 83 3c 74 a7 20 b2
 19 59 dc e0 43 fc 88 b7 7c
 23 d3 9e 86 23 25 42 27 45
 24 0e 48 58 ea 50 44 1a 47
 33 d7 19 f3 03 3e 6e 59 03
 34 27 b9 08 b1 d1 a1 b1 db
 38 62 f5 aa 4b 5e 15 7f 93

Indeed, the repetitions are quite frequent, and running the same command on other files yields roughly the same values:

kakwa@linux 6775398/idx » hexdump -C particles.idx | grep '00 00 00 00 00 00 00' | \
 sed 's/^..........//' | sed 's/ .*//' | sort | uniq -c | sort -n

 1 27 b9 08 b1 d1 a1 b1 db
 1 28 00 00 00 00 00 00 00
 1 8e ae 00 00 00 00 00 00
 1 bd 01 00 00 b7 01 00 00
 4 ca 2c b7 97 24 b8 1c 86
 5 1f 19 a6 0c a2 9b 7f b3
 16 94 a1 23 f4 c5 41 b8 42
 20 91 cc 55 52 25 2a 42 d4
 81 de 3e 45 0e 99 dc 30 14
 317 66 52 00 d6 89 64 1d 2e

Moreover, sound_music.idx, which as the name implies probably only contains sound files, returns mostly one type:

kakwa@linux 6775398/idx » hexdump -C sound_music.idx | grep '00 00 00 00 00 00 00' | \
 sed 's/^..........//' | sed 's/ .*//' | sort | uniq -c | sort -n

 513 93 63 67 56 c2 97 75 69

Note: these commands are by no means accurate; they are likely to catch garbage and miscount. But these are quick and dirty ways to validate hypotheses.

So it seems we are dealing with another file_type field. Let’s call it file_type2, and rename the first one file_type1.

So in the end, making a few assumptions for now, we have figured out the rough format of this section:

+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
| T1 | T1 | T1 | T1 | T1 | T1 | T1 | T1 || OF | OF | OF | OF | OF | OF | OF | OF |
+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
|<---------- file type 1 -------------->||<------------ offset ----------------->|
| 64 bits || 64 bits |
+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
| UN | UN | UN | UN | UN | UN | UN | UN || T2 | T2 | T2 | T2 | T2 | T2 | T2 | T2 |
+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
|<------------ unknown ---------------->||<---------- file type 2 -------------->|
| 64 bits || 64 bits |

Header Section Analysis

The .idx file header contains:

00000000 49 53 46 50 00 00 00 02 91 9d 39 b4 40 00 00 00 |ISFP......9.@...|
00000010 37 01 00 00 1c 01 00 00 01 00 00 00 00 00 00 00 |7...............|
00000020 28 00 00 00 00 00 00 00 f6 3b 00 00 00 00 00 00 |(........;......|
00000030 36 71 00 00 00 00 00 00 0e 00 00 00 00 00 00 00 |6q..............|
00000040 e0 26 00 00 00 00 00 00 8f 0c 9a ba 4f 40 b6 93 |.&..........O@..|
00000050 27 b9 08 b1 d1 a1 b1 db 13 00 00 00 00 00 00 00 |'...............|
00000060 ce 26 00 00 00 00 00 00 8f ec 87 4a 28 d0 f7 c7 |.&.........J(...|
00000070 a4 eb 1b 3e 50 21 d8 74 12 00 00 00 00 00 00 00 |...>P!.t........|
00000080 c1 26 00 00 00 00 00 00 ad 70 a2 e7 ac 2c 4f 6b |.&.......p...,Ok|
00000090 27 b9 08 b1 d1 a1 b1 db 0e 00 00 00 00 00 00 00 |'...............|

These first bytes don’t look like the section previously mentioned; we have a magic number (ISFP), and then the content doesn’t look like a section at first (too many low-value 32-bit integers).

This means we most likely have a header section containing things like:

magic numbers
types
sizes
number of entries/files

The first thing to determine is the size of the header. Looking at it, the first section starts at 0x38 (recognisable by the full 64-bit integers).

This means the header is 7 × 64 bits.

Let’s analyze the content.

Looking at all the files, for the first 128 bits, we get:

kakwa@linux 6775398/idx » for i in *;do hexdump -C $i | head -n 1;done

[...]
00000000 49 53 46 50 00 00 00 02 1b 73 f9 d5 40 00 00 00 |ISFP.....s..@...|
00000000 49 53 46 50 00 00 00 02 78 f0 2c 09 40 00 00 00 |ISFP....x.,.@...|
00000000 49 53 46 50 00 00 00 02 b8 fe ba b9 40 00 00 00 |ISFP........@...|
00000000 49 53 46 50 00 00 00 02 06 24 fa 2d 40 00 00 00 |ISFP.....$.-@...|
00000000 49 53 46 50 00 00 00 02 1e 7e f6 d9 40 00 00 00 |ISFP.....~..@...|
00000000 49 53 46 50 00 00 00 02 dd 21 74 c2 40 00 00 00 |ISFP.....!t.@...|
00000000 49 53 46 50 00 00 00 02 33 28 63 bd 40 00 00 00 |ISFP....3(c.@...|
00000000 49 53 46 50 00 00 00 02 cb 5c e2 0d 40 00 00 00 |ISFP.....\..@...|
00000000 49 53 46 50 00 00 00 02 cb e8 8a fd 40 00 00 00 |ISFP........@...|
00000000 49 53 46 50 00 00 00 02 6e 04 b1 62 40 00 00 00 |ISFP....n..b@...|
00000000 49 53 46 50 00 00 00 02 15 1c a2 f9 40 00 00 00 |ISFP........@...|
[...]

As we can see, the 1st, 2nd, and 4th 32-bit chunks are always the same, and looking at the values, we have respectively:

a magic number (ISFP),
00 00 00 02 which is rather weird (it could be some kind of ID if we were little-endian, but the format is big-endian). Maybe it is actually part of the magic number. As it doesn’t vary, it’s not too important for the task at hand here.
40 00 00 00 which, like type 1, looks like a low-value enum, and given its position in the index file, within the header, we are most likely dealing with an archive type. Again, as it doesn’t vary, it’s not really important.

The 3rd 32-bit integer uses all the available bits, so it’s unlikely a size. Maybe it’s a CRC32 or a unique ID for the archives.

Edit: the 40 00 00 00 value upon closer inspection might not be an archive type; its value is 64 in decimal, which might be a header size, or simply storing the size of an integer.

So we have:

+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
| MA | MA | MA | MA || 00 | 00 | 00 | 02 || ID | ID | ID | ID || 40 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
|<----- magic ----->||<----- ???? ------>||<---- id/crc ----->||<----- ??????? --->|

Now, let’s look at the next 128 bits (second line in the hexdump).

kakwa@linux 6775398/idx » for i in *;do hexdump -C $i | head -n 2 | tail -n 1;done

00000010 bd 01 00 00 b7 01 00 00 01 00 00 00 00 00 00 00 |................|
00000010 35 01 00 00 1d 01 00 00 01 00 00 00 00 00 00 00 |5...............|
00000010 5b 00 00 00 57 00 00 00 01 00 00 00 00 00 00 00 |[...W...........|
00000010 5c 00 00 00 58 00 00 00 01 00 00 00 00 00 00 00 |\...X...........|
00000010 10 18 00 00 ff 17 00 00 01 00 00 00 00 00 00 00 |................|
00000010 29 3b 00 00 0b 3b 00 00 01 00 00 00 00 00 00 00 |);...;..........|
00000010 04 02 00 00 02 02 00 00 01 00 00 00 00 00 00 00 |................|
00000010 1e 05 00 00 c2 03 00 00 01 00 00 00 00 00 00 00 |................|
00000010 a5 01 00 00 36 01 00 00 01 00 00 00 00 00 00 00 |....6...........|
00000010 13 04 00 00 fb 02 00 00 01 00 00 00 00 00 00 00 |................|
00000010 79 03 00 00 a9 02 00 00 01 00 00 00 00 00 00 00 |y...............|

So here, we recognize two 32-bit integers due to the 00 00, and then either a fixed 64-bit integer with always a 01 00 00 00 00 00 00 00 value, or something like two 32-bit integers with value 01 00 00 00 and 00 00 00 00 (as the value never varies, again, it’s not that important).

Let’s try to determine the two 32-bit values. Let’s look at one of the files in particular:

kakwa@linux 6775398/idx » hexdump -C system_data.idx | less

[...]
00000010 37 01 00 00 1c 01 00 00 01 00 00 00 00 00 00 00 |7...............|
[...]

The first value is 37 01 00 00, i.e. converted to decimal, 311. Doing a strings system_data.idx > listing and removing the garbage (ex: w6~n) as best as possible, plus the .pkg file name, only keeping files and directory names, we get 310 entries, a remarkably close value.

Looking at other files, story is similar, this field roughly matches the number of strings we get from strings (never perfectly however, but if the names are too short, strings will ignore them, most likely explaining the small delta we have each time).

Consequently we can deduce it’s most likely the number of entries (files and directories) in the index file.

Next, we have 1c 01 00 00, i.e. converted to decimal, 284. This value is suspiciously close to the previous value. As we have both directories and file names, this number probably represents the number of items that are actual files.

Let’s validate that:

# Q&D filtering out names without an extension (no '.')
kakwa@linux 6775398/idx » cat listing | grep '\.' | wc -l

Bingo, we have the exact number we were looking for.

The last 64 bits could simply be ignored for now since they always have the same value.

So we have:

+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
| FD | FD | FD | FD || FI | FI | FI | FI || 01 | 00 | 00 | 00 || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
|<file + dir count >||<-- file count --->||<-------------- ???? ------------------>|

Next 128 bits:

kakwa@linux 6775398/idx » for i in *;do hexdump -C $i | head -n 3 | tail -n 1;done | less

[...]
00000020 28 00 00 00 00 00 00 00 58 8d 00 00 00 00 00 00 |(.......X.......|
00000020 28 00 00 00 00 00 00 00 7c 3f 02 00 00 00 00 00 |(.......|?......|
00000020 28 00 00 00 00 00 00 00 77 2f 00 00 00 00 00 00 |(.......w/......|
00000020 28 00 00 00 00 00 00 00 bc e9 01 00 00 00 00 00 |(...............|
00000020 28 00 00 00 00 00 00 00 c6 ea 02 00 00 00 00 00 |(...............|
00000020 28 00 00 00 00 00 00 00 a9 0f 00 00 00 00 00 00 |(...............|
00000020 28 00 00 00 00 00 00 00 c4 22 00 00 00 00 00 00 |(........"......|
00000020 28 00 00 00 00 00 00 00 b6 2b 00 00 00 00 00 00 |(........+......|
00000020 28 00 00 00 00 00 00 00 d6 41 00 00 00 00 00 00 |(........A......|
00000020 28 00 00 00 00 00 00 00 fd 21 00 00 00 00 00 00 |(........!......|
00000020 28 00 00 00 00 00 00 00 e1 25 00 00 00 00 00 00 |(........%......|
00000020 28 00 00 00 00 00 00 00 f5 31 00 00 00 00 00 00 |(........1......|
00000020 28 00 00 00 00 00 00 00 1e 42 00 00 00 00 00 00 |(........B......|
00000020 28 00 00 00 00 00 00 00 70 42 02 00 00 00 00 00 |(.......pB......|
[...]

So here, we have two 64-bit integers. The first one is 28 00 00 00 00 00 00 00 and always has the same value. Not sure what it represents; the value is somewhat close to the header size in bytes: 40 for this value, 56 for the full header size.

Maybe the header could vary in size in certain situations, and this represents its size minus some fixed part (like the first 16 bytes/128 bits). I’m also kind of betting that if the previous 64-bit integer (01 00 00 00 00 00 00 00) changes, this will also change.

But as it never varies in the set of index files we have here, we cannot really make any deduction, only guesses. So once again, let’s ignore it.

At this point, the idea of downloading other Wargaming games like World of Tanks or World of Warplanes popped up; maybe this will give complementary information regarding the unknown fields that start to pile up.

But let’s continue for now.

EDIT: It’s no help; World of Tanks and World of Warplanes simply pack their resources in .zip files… It’s a wild guess, but I kind of expect WoWs to be the same in the future. The WoWs packing format feels, in fact, somewhat legacy, custom, and far less efficient than a standard run-of-the-mill .zip file. Not to mention using .zip files means removing one bit of code to maintain.

The next value is again a 64-bit integer; it changes between each file.

Let’s focus on one file:

[...]
00000020 28 00 00 00 00 00 00 00 f6 3b 00 00 00 00 00 00 |(........;......|
[...]

At first, I thought it might be a file size, but quickly checking the index file size, I got:

index file size: 29043
value of this field in decimal: 15350

Checking another file, I got 77461 and 44807.

So no, it’s not the index size. However it is suspiciously ~1/2 of the file size, and after having stared at hexdumps for hours, I had another idea.

The third chunk of the file is right after the bundle of dir/file name strings which varies in length wildly (i.e., it’s not a fixed length or a multiple of a fixed length).

We probably need an offset pointing to the start of this section in the header.

And sure enough, looking where the bundle of strings stops, we get:

00003bd0 6f 69 73 65 2e 64 64 73 00 73 70 61 63 65 5f 76 |oise.dds.space_v|
00003be0 61 72 69 61 74 69 6f 6e 5f 64 75 6d 6d 79 2e 64 |ariation_dummy.d|
00003bf0 64 73 00 77 61 76 65 73 5f 68 65 69 67 68 74 73 |ds.waves_heights|
00003c00 30 2e 64 64 73 00 8f ec 87 4a 28 d0 f7 c7 70 11 |0.dds....J(...p.|
00003c10 03 07 0d 33 ed 77 1e 9b ef 05 00 00 00 00 05 00 |...3.w..........|

Okay, the string bundle ends at 00003c05; that’s quite near 3bf6, so this is certainly the offset to this third section or the end of the bundle of strings.

Most likely, the offset is not from the start of the file but from a specific point in the header (this field? end of header?); that’s why we get a -15 difference (0x00003c05 - 0x3bf6 = 0xF = 15). This -15 value is constant between files.

So we have:

+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
| HS | HS | HS | HS | HS | HS | HS | HS || OF | OF | OF | OF | OF | OF | OF | OF |
+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
|<------------ header size (?) -------->||<---- offset third section start ----->|

Last 64 bits:

kakwa@linux 6623042/idx » for i in *;do hexdump -C $i | head -n 4 | tail -n 1;done | less

[...]
00000030 40 af 03 00 00 00 00 00 1b 00 00 00 00 00 00 00 |@...............|
00000030 54 8d 1f 00 00 00 00 00 21 00 00 00 00 00 00 00 |T.......!.......|
00000030 8e ae 00 00 00 00 00 00 17 00 00 00 00 00 00 00 |................|
00000030 02 81 00 00 00 00 00 00 13 00 00 00 00 00 00 00 |................|
00000030 c9 24 00 00 00 00 00 00 1f 00 00 00 00 00 00 00 |.$..............|
[...]

So, we have a 64-bit integer, which is a relatively low value. This means it’s most likely a size or an offset.

If we pick one:

kakwa@linux 6623042/idx » hexdump -C system_data.idx | less

[...]
00000030 36 71 00 00 00 00 00 00 13 00 00 00 00 00 00 00 |6q..............|
[...]

We can see that the 36 71 00 00 00 00 00 00, 28982 once converted to decimal, is remarkably close to the file size (29043 bytes).

From there, we can guess it might be three things:

the actual index file size
an offset to something at the end of the file
pointer to the end of the third section

Let’s note that for now, and figure out the finer details at implementation time.

So, to recap, here is the header section format:

+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
| MA | MA | MA | MA || 00 | 00 | 00 | 02 || ID | ID | ID | ID || 40 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
|<----- magic ----->||<----- ???? ------>||<---- id/crc ----->||<----- ??????? --->|

+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
| FD | FD | FD | FD || FI | FI | FI | FI || 01 | 00 | 00 | 00 || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====++====+====+====+====++====+====+====+====+
|<file + dir count >||<-- file count --->||<-------------- ???? ------------------>|

+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
| HS | HS | HS | HS | HS | HS | HS | HS || OF | OF | OF | OF | OF | OF | OF | OF |
+====+====+====+====+====+====+====+=====++=====+====+====+====+====+====+====+====+
|<------------ header size (?) --------->||<----- offset third section start ----->|

+====+====+====+====+====+====+====+=====+
| OE | OE | OE | OE | OE | OE | OE | OE |
+====+====+====+====+====+====+====+=====+
|<---- offset third section end -------->|

Filename Section

Not much to say there.

Here’s what it looks like:

kakwa@linux 6775398/idx » hexdump -C system_data.idx| less

[...]
00002700 fb 14 00 00 00 00 00 00 74 d1 1b 8d f4 ff 7a ce |........t.....z.|
00002710 a4 eb 1b 3e 50 21 d8 74 4b 44 53 74 6f 72 61 67 |...>P!.tKDStorag|
00002720 65 2e 62 69 6e 00 77 61 76 65 73 5f 68 65 69 67 |e.bin.waves_heig|
00002730 68 74 73 31 2e 64 64 73 00 61 6e 69 6d 61 74 65 |hts1.dds.animate|
[...]
000028c0 74 73 2e 62 69 6e 00 6d 69 73 63 53 65 74 74 69 |ts.bin.miscSetti|
000028d0 6e 67 73 2e 78 6d 6c 00 64 61 6d 61 67 65 5f 64 |ngs.xml.damage_d|
000028e0 65 63 5f 32 5f 64 2e 64 64 32 00 64 61 6d 61 67 |ec_2_d.dd2.damag|
000028f0 65 5f 64 65 63 5f 31 5f 65 2e 64 64 73 00 63 72 |e_dec_1_e.dds.cr|
[...]
00003be0 61 72 69 61 74 69 6f 6e 5f 64 75 6d 6d 79 2e 64 |ariation_dummy.d|
00003bf0 64 73 00 77 61 76 65 73 5f 68 65 69 67 68 74 73 |ds.waves_heights|
00003c00 30 2e 64 64 73 00 8f 0c 9a ba 4f 40 b6 93 70 11 |0.dds.....O@..p.|
00003c10 03 07 0d 33 ed 77 00 00 00 00 00 00 00 00 05 00 |...3.w..........|
00003c20 00 00 01 00 00 00 f5 21 00 00 bf 00 45 5c 6c 36 |.......!....E\l6|

So it’s a bunch of \0 separated strings. The only thing interesting to note is that it’s not a fixed size section.

PKG Pointer Section

Here what it looks like:

kakwa@linux 6775398/idx » hexdump -C system_data.idx| less

[...]
00003bf0 64 73 00 77 61 76 65 73 5f 68 65 69 67 68 74 73 |ds.waves_heights|
00003c00 30 2e 64 64 73 00 8f 0c 9a ba 4f 40 b6 93 70 11 |0.dds.....O@..p.|
00003c10 03 07 0d 33 ed 77 00 00 00 00 00 00 00 00 05 00 |...3.w..........|
00003c20 00 00 01 00 00 00 f5 21 00 00 bf 00 45 5c 6c 36 |.......!....E\l6|
00003c30 00 00 00 00 00 00 8f ec 87 4a 28 d0 f7 c7 70 11 |.........J(...p.|
00003c40 03 07 0d 33 ed 77 1e 9b ef 05 00 00 00 00 05 00 |...3.w..........|
00003c50 00 00 01 00 00 00 15 15 01 00 03 77 63 97 3e ab |...........wc.>.|
00003c60 02 00 00 00 00 00 ad 70 a2 e7 ac 2c 4f 6b 70 11 |.......p...,Okp.|
00003c70 03 07 0d 33 ed 77 05 22 00 00 00 00 00 00 05 00 |...3.w."........|
00003c80 00 00 01 00 00 00 cb 01 00 00 6d b9 de c1 ad 0c |..........m.....|
00003c90 00 00 00 00 00 00 8e c7 6a 58 7c 86 62 33 70 11 |........jX|.b3p.|
00003ca0 03 07 0d 33 ed 77 e0 23 00 00 00 00 00 00 05 00 |...3.w.#........|
00003cb0 00 00 01 00 00 00 65 00 00 00 f1 d2 87 5a d2 00 |......e......Z..|
00003cc0 00 00 00 00 00 00 8e 43 3d e9 cf 49 52 a4 70 11 |.......C=..IR.p.|
00003cd0 03 07 0d 33 ed 77 4d 1f 6a 05 00 00 00 00 05 00 |...3.wM.j.......|
00003ce0 00 00 01 00 00 00 bb 19 00 00 f7 53 4a b1 38 ab |...........SJ.8.|
00003cf0 00 00 00 00 00 00 0e 3c 9a 6d 22 de 7b da 70 11 |.......<.m".{.p.|
00003d00 03 07 0d 33 ed 77 55 24 00 00 00 00 00 00 05 00 |...3.wU$........|
00003d10 00 00 01 00 00 00 72 0d 00 00 83 0a 72 88 a3 5c |......r.....r..\|
00003d20 00 00 00 00 00 00 98 45 00 7a 16 6e 84 21 70 11 |.......E.z.n.!p.|
00003d30 03 07 0d 33 ed 77 73 ce cb 06 00 00 00 00 05 00 |...3.ws.........|
00003d40 00 00 01 00 00 00 f9 b6 b7 00 ff 7e f7 7b 5b 30 |...........~.{[0|
00003d50 b8 00 00 00 00 00 98 51 4a 00 2c 12 71 ad 70 11 |.......QJ.,.q.p.|
00003d60 03 07 0d 33 ed 77 7e 63 75 05 00 00 00 00 05 00 |...3.w~cu.......|
[...]
00007100 00 00 01 00 00 00 6f 09 00 00 fc 56 94 f8 9a 37 |......o....V...7|
00007110 00 00 00 00 00 00 21 67 ac 70 22 ec ca b8 70 11 |......!g.p"...p.|
00007120 03 07 0d 33 ed 77 28 f9 15 0a 00 00 00 00 05 00 |...3.w(.........|
00007130 00 00 01 00 00 00 0b 2d 00 00 03 bd b0 50 67 e9 |.......-.....Pg.|
00007140 00 00 00 00 00 00 15 00 00 00 00 00 00 00 18 00 |................|
00007150 00 00 00 00 00 00 70 11 03 07 0d 33 ed 77 73 79 |......p....3.wsy|
00007160 73 74 65 6d 5f 64 61 74 61 5f 30 30 30 31 2e 70 |stem_data_0001.p|
00007170 6b 67 00 |kg.|
00007173
(END)

This section contains 384-bit records (48 bytes each) linking files to their data in .pkg files through IDs and offsets.

Each record contains metadata IDs, PKG file offsets, data sizes, and compression types.

# let's skip the first 6 bytes to align hexdump output
kakwa@linux 6775398/idx » hexdump -s 6 -C system_data.idx| less

[...]
00003be6 6f 6e 5f 64 75 6d 6d 79 2e 64 64 73 00 77 61 76 |on_dummy.dds.wav|
00003bf6 65 73 5f 68 65 69 67 68 74 73 30 2e 64 64 73 00 |es_heights0.dds.|
00003c06 8f 0c 9a ba 4f 40 b6 93 70 11 03 07 0d 33 ed 77 |....O@..p....3.w|
00003c16 00 00 00 00 00 00 00 00 05 00 00 00 01 00 00 00 |................|
00003c26 f5 21 00 00 bf 00 45 5c 6c 36 00 00 00 00 00 00 |.!....E\l6......|
00003c36 8f ec 87 4a 28 d0 f7 c7 70 11 03 07 0d 33 ed 77 |...J(...p....3.w|
00003c46 1e 9b ef 05 00 00 00 00 05 00 00 00 01 00 00 00 |................|
00003c56 15 15 01 00 03 77 63 97 3e ab 02 00 00 00 00 00 |.....wc.>.......|
00003c66 ad 70 a2 e7 ac 2c 4f 6b 70 11 03 07 0d 33 ed 77 |.p...,Okp....3.w|
[...]

Much nicer!

With that, we immediately notice a cycle, with the 70 11 03 07 0d 33 ed 77 value.

There are 6 × 64 = 384 bits between each 70 11 03 07 0d 33 ed 77 value.

Let’s try to confirm that with the IDs.

We are spotting the bf 00 45 5c 6c 36 seen previously in the .pkg file. 384 bits later, we see 03 77 63 97 3e ab 02.

With a bit of digging (it’s right in the middle of the .pkg file, and this ID is not neatly in a 64-bit aligned chunk because of the variable size of the DEFLATE blocks), we indeed find it.

We have indeed a 384-bit cycle, which neatly fits in 3 lines of hexdump!

So each of these is one record:

00003c06 8f 0c 9a ba 4f 40 b6 93 70 11 03 07 0d 33 ed 77 |....O@..p....3.w|
00003c16 00 00 00 00 00 00 00 00 05 00 00 00 01 00 00 00 |................|
00003c26 f5 21 00 00 bf 00 45 5c 6c 36 00 00 00 00 00 00 |.!....E\l6......|

00003c36 8f ec 87 4a 28 d0 f7 c7 70 11 03 07 0d 33 ed 77 |...J(...p....3.w|
00003c46 1e 9b ef 05 00 00 00 00 05 00 00 00 01 00 00 00 |................|
00003c56 15 15 01 00 03 77 63 97 3e ab 02 00 00 00 00 00 |.....wc.>.......|

00003c66 ad 70 a2 e7 ac 2c 4f 6b 70 11 03 07 0d 33 ed 77 |.p...,Okp....3.w|
00003c76 05 22 00 00 00 00 00 00 05 00 00 00 01 00 00 00 |."..............|
00003c86 cb 01 00 00 6d b9 de c1 ad 0c 00 00 00 00 00 00 |....m...........|

00003c96 8e c7 6a 58 7c 86 62 33 70 11 03 07 0d 33 ed 77 |..jX|.b3p....3.w|
00003ca6 e0 23 00 00 00 00 00 00 05 00 00 00 01 00 00 00 |.#..............|
00003cb6 65 00 00 00 f1 d2 87 5a d2 00 00 00 00 00 00 00 |e......Z........|

00003cc6 8e 43 3d e9 cf 49 52 a4 70 11 03 07 0d 33 ed 77 |.C=..IR.p....3.w|
00003cd6 4d 1f 6a 05 00 00 00 00 05 00 00 00 01 00 00 00 |M.j.............|
00003ce6 bb 19 00 00 f7 53 4a b1 38 ab 00 00 00 00 00 00 |.....SJ.8.......|

All these records are from the same file,

let’s grab a few from other files:

File 2:

0000f6fd 58 fe 65 b4 59 b6 b0 77 a4 8c 78 6a 58 aa 65 84 |X.e.Y..w..xjX.e.|
0000f70d 00 00 00 00 00 00 00 00 05 00 00 00 01 00 00 00 |................|
0000f71d bc 99 05 00 f4 3a 67 8b 80 00 08 00 00 00 00 00 |.....:g.........|

0000f72d a0 e5 c7 22 cc 49 d3 31 a4 8c 78 6a 58 aa 65 84 |...".I.1..xjX.e.|
0000f73d 3e 9e 7e 05 00 00 00 00 05 00 00 00 01 00 00 00 |>.~.............|
0000f74d 79 c6 03 00 dc b8 5f 80 80 00 08 00 00 00 00 00 |y....._.........|

File 3:

00002d0c ed cf 33 f8 a5 94 53 56 0d a7 9c b9 bf 60 f5 3e |..3...SV.....`.>|
00002d1c 40 a8 08 07 00 00 00 00 00 00 00 00 00 00 00 00 |@...............|
00002d2c 38 ab 00 00 5d cf 4e b6 38 ab 00 00 00 00 00 00 |8...].N.8.......|

00002d3c ed ea da 52 4e 8f 70 ed 0d a7 9c b9 bf 60 f5 3e |...RN.p......`.>|
00002d4c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00002d5c 98 00 00 00 a4 63 f2 9b 98 00 00 00 00 00 00 00 |.....c..........|

Lets study:

00003c06 8f 0c 9a ba 4f 40 b6 93 70 11 03 07 0d 33 ed 77 |....O@..p....3.w|
00003c16 00 00 00 00 00 00 00 00 05 00 00 00 01 00 00 00 |................|
00003c26 f5 21 00 00 bf 00 45 5c 6c 36 00 00 00 00 00 00 |.!....E\l6......|

For the first 128 bits we get:

00003c06 8f 0c 9a ba 4f 40 b6 93 70 11 03 07 0d 33 ed 77 |....O@..p....3.w|

From the fact the last 64 bits are constant, we can deduce we have probably two 64 bits integers in the first 128 bits

Once again, these are using all the available bits and seem rather random. It’s difficult to link these to their role, so… lets simply ignore these for now.

+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
| UO | UO | UO | UO | UO | UO | UO | UO || UT | UT | UT | UT | UT | UT | UT | UT |
+====+====+====+====+====+====+====+====++====+====+====+====+====+====+====+====+
|<------------ unknown 1 -------------->||<------------- unknow 2 -------------->|
| 64 bits || 64 bits |

Next 64 bits, we have a low value 64 bits integer, so most likely an offset. It is also suspiciously at 0x0 for the first record which is also the first data chunk in the .pkg file.

So, it’s most likely the start of a data chunk in a .pkg file. Looking at other records confirms that.

Next, we have two extremely low value 32 bits (0x5 and 0x1). So once again, most likely some kind of enum, lets call them type1 and type2.

+====+====+====+====+====+====+====+====++====+====+====+====++====+====+====+====+
| OF | OF | OF | OF | OF | OF | OF | OF || T1 | T1 | T1 | T1 || T2 | T2 | T2 | T2 |
+====+====+====+====+====+====+====+====++====+====+====+====++====+====+====+====+
|<----------start offset pkg ---------->||<---- type 1 ----->||<----- type 2 ---->|
| 64 bits || 32 bits || 32 bits |

For the Last 128 bits, we get the following:

00003c26 f5 21 00 00 bf 00 45 5c 6c 36 00 00 00 00 00 00 |.!....E\l6......|

So, here, we see our .pkg ID (bf 00 45 5c 6c 36) right in the middle. Given its size, this ID is probably stored on 64 bits.

After that, last 32 bits, we get a bunch of 00, maybe a reserved field, but more likely some kind of padding.

Before that, we get a low value 32 bits integer. When comparing with the .pkg file, f5 21 00 00 is the offset where the first data chunk ends.

So it’s the end offset. But 32 bits seems rather small to store such offset (specially given the start offset is 64 bits. Also, for other data chunks, this doesn’t line-up.

However, it could very much be a relative offset (to the start of the data chunk).

Lets validate that with the second record:

00003c36 8f ec 87 4a 28 d0 f7 c7 70 11 03 07 0d 33 ed 77 |...J(...p....3.w|
00003c46 1e 9b ef 05 00 00 00 00 05 00 00 00 01 00 00 00 |................|
00003c56 15 15 01 00 03 77 63 97 3e ab 02 00 00 00 00 00 |.....wc.>.......|

More hexdump! (searching the data chunk using the 03 77 63 97 3e ab 02 ID and the start offset 1e 9b ef 05 00 00 00 00, or 0x05ef9b1e once we take endianess into account):

kakwa@linux World of Warships/res_packages » hexdump -C system_data_0001.pkg | less

[...]
05ef9b10 00 00 17 4b 28 42 80 40 00 00 00 00 00 00 8c 7d |...K(B.@.......}|
05ef9b20 3b 50 5d d9 b6 dd 7d b6 3e 76 15 b2 5f 20 89 04 |;P]...}.>v.._ ..|
05ef9b30 55 bd 00 44 82 aa ec 7a 9c bd f7 79 45 57 39 40 |U..D...z...yEW9@|
05ef9b40 90 d0 4e 8c c0 01 9d 21 48 e8 0c 41 a2 ce 10 24 |..N....!H..A...$|
[...]
05f0b010 d1 d7 52 b0 de cc fa 9c 2b 8d b5 57 a4 02 ff 1a |..R.....+..W....|
05f0b020 43 5d 2d 43 3f cc d1 0b f8 88 92 a8 9f 5a 70 64 |C]-C?........Zpd|
05f0b030 46 fb 7f 00 00 00 00 03 77 63 97 3e ab 02 00 00 |F.......wc.>....|
05f0b040 00 00 00 94 b7 67 90 db 68 9e e6 d9 1f 36 62 6f |.....g..h....6bo|
05f0b050 6f 22 76 f6 62 e3 62 77 67 7a 7a ba ab bb 8c aa |o"v.b.bwgzz.....|
05f0b060 4a 52 95 5c ca a5 cf 64 d2 24 bd 27 41 d0 00 04 |JR.\...d.$.'A...|
[...]

We indeed find the start of our data chunk at 05ef9b1e (8c after a bunch of 00 on the first line).

And looking for the 00 00 00 00 ID ID [...] pattern in between the data chunks, we can determine the end of this chunk to be at 0x05f0b032.

Doing 0x05f0b032 - 0x05ef9b1e, we get 0x11514, that’s almost our 15 15 01 00 once we swap endianess, and add 1.

+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
| OE | OE | OE | OE || ID | ID | ID | ID | ID | ID | ID | ID || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
|<-- offset end --->||<------------- ID '.pkg' ------------->||<---- padding ---->|
| 32 bits || 64 bits || 32 bits |

So to recap, we have:

+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
| UO | UO | UO | UO | UO | UO | UO | UO || UT | UT | UT | UT | UT | UT | UT | UT |
+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
|<------------ unknown 1 -------------->||<-------------- unknown 2 ------------->|
| 64 bits || 64 bits |
+====+====+====+====+====+====+====+====++====+====+====+====++====+====+====+====+
| OF | OF | OF | OF | OF | OF | OF | OF || T1 | T1 | T1 | T1 || T2 | T2 | T2 | T2 |
+====+====+====+====+====+====+====+====++====+====+====+====++====+====+====+====+
|<--------- start offset pkg ---------->||<---- type 1 ----->||<----- type 2 ---->|
| 64 bits || 32 bits || 32 bits |
+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
| OE | OE | OE | OE || ID | ID | ID | ID | ID | ID | ID | ID || 00 | 00 | 00 | 00 |
+====+====+====+====++====+====+====+====+====+====+====+====++====+====+====+====+
|<-- offset end --->||<------------- ID '.pkg' ------------->||<---- padding ---->|
| 32 bits || 64 bits || 32 bits |

The Last Bits

So, okay, we have the core of the 3 parts of the file.

But we can see the last few bits don’t follow this pattern (especially with the .pkg file name), which means we have a footer:

# Last block
00007116 21 67 ac 70 22 ec ca b8 70 11 03 07 0d 33 ed 77 |!g.p"...p....3.w|
00007126 28 f9 15 0a 00 00 00 00 05 00 00 00 01 00 00 00 |(...............|
00007136 0b 2d 00 00 03 bd b0 50 67 e9 00 00 00 00 00 00 |.-.....Pg.......|
# Footer
00007146 15 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00 |................|
00007156 70 11 03 07 0d 33 ed 77 73 79 73 74 65 6d 5f 64 |p....3.wsystem_d|
00007166 61 74 61 5f 30 30 30 31 2e 70 6b 67 00 |ata_0001.pkg.|

If we look at the content, we have 3 × 64 bits before the name starts (73 79 73 74 65 6d 5f 64 for system_d)

Looking at it more closely, given all the 00, it seems we have three 64-bit integers there.

15 00 00 00 00 00 00 00
18 00 00 00 00 00 00 00
70 11 03 07 0d 33 ed 77

70 11 03 07 0d 33 ed 77 is the “unknown 2” we saw previously, so still no luck, but let’s name it the same way.

18 00 00 00 00 00 00 00 seems to have the same value across all files, so probably not that important.

The only one that varies across files is the 15 00 00 00 00 00 00 00, but always in the same kind of values around 0x15. In decimal it’s 21.

Strangely, system_data_0001.pkg is 20 chars long, 21 if we include the \0 at the end.

So we can deduce it’s actually the .pkg file name string size.

So we have:

+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
| UO | UO | UO | UO | UO | UO | UO | UO || U3 | U3 | U3 | U3 | U3 | U3 | U3 | U3 |
+====+====+====+====+====+====+====+====++=====+====+====+====+====+====+====+====+
|<--------- size pkg file name -------->||<-------------- unknown 3 ------------->|
| 64 bits || 64 bits |
+=====+====+====+====+====+====+====+====+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...
| UT | UT | UT | UT | UT | UT | UT | UT | file name string
+=====+====+====+====+====+====+====+====+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...
|<-------------- unknown 2 ------------->|
| 64 bits |

Recap (Part 3)

Mapped overall .idx layout into 4 parts:
- Header: magic ISFP, counts (entries and files), header-size-like field, offsets to start/end of the third section, and a per-file id/crc-like value.
- Names: \0-separated file and directory names.
- Records section (per-record 384 bits): two 64-bit unknowns (likely ids, one enabling parent/child hierarchy), 64-bit start offset in .pkg, 2×32-bit types, 32-bit relative end size, 64-bit .pkg chunk id, and padding.
- Footer: .pkg filename length, an unknown 64-bit value, a repeated 64-bit constant, then the .pkg filename string.
Established the link between names and .pkg chunks via IDs and offsets, enough to reconstruct directories and locate data.

In the next part of this series we will start the actual implementation.

Reversing WoWs Resource Format - Part 2/5: Looking For The Metadata

carpentier.pf@gmail.com (Pierre-François Carpentier) — Mon, 25 Aug 2025 00:02:00 +0200

Searching & Reading The Metadata

In Part 1, we discovered:

Data lives in res_packages/ as custom .pkg archives.
Each .pkg is a sequence of DEFLATE-compressed blobs separated by 64-bit IDs with zero padding.
No file names inside .pkg; names/paths must exist elsewhere (in indexes).

Now we need to find where and how the file and directory structure is stored.

Back To File Exploration

The metadata hopefully isn’t embedded in executables. Let’s search:

# List all files
# then remove uninteresting bits like replays, crashes, DLLs, logs, .pkg
# or CEF stuff (embedded Chrome used for the armory, inventory, dockyard and clan base))

kakwa@linux Games/World of Warships » find ./ -type f | grep -v cef | grep -v replays \
 | grep -v crashes | grep -v '.pkg' | grep -v '.dll' | grep -v '.log' \
 | grep -v '\.exe' | less

[...]
./bin/6775398/res/texts/pl/LC_MESSAGES/global.mo
./bin/6775398/res/texts/zh_tw/LC_MESSAGES/global.mo
./bin/6775398/res/texts/fr/LC_MESSAGES/global.mo
./bin/6775398/res/texts/zh_sg/LC_MESSAGES/global.mo
./bin/6775398/res/camerasConsumer.xml
./bin/6775398/bin32/paths.xml
./bin/6775398/bin32/Licenses.txt
./bin/6775398/bin32/monitor_config.json
./bin/6775398/bin64/paths.xml
./bin/6775398/bin64/Licenses.txt
./bin/6775398/bin64/monitor_config.json
[...]
./bin/6775398/idx/spaces_dock_dry.idx
./bin/6775398/idx/spaces_dock_hsf.idx
./bin/6775398/idx/spaces_sea_hope.idx
./bin/6775398/idx/vehicles_pve.idx
./bin/6775398/idx/vehicles_level8_fr.idx
./bin/6775398/idx/vehicles_level5_panasia.idx
[...]

The .idx files look promising, especially their names match the .pkg files:

spaces_dock_hsf.idx → spaces_dock_hsf_0001.pkg
vehicles_level9_ned.idx → vehicles_level9_ned_0001.pkg

Checking We Have The Metadata

Let’s take a look:

kakwa@linux bin/6775398/idx » strings -n 5 system_data.idx

#%B'E
E)zj'
FM'lb
%}n:b
(	?A+
c|'lY
zc78'
tKDStorage.bin
waves_heights1.dds
animatedMiscs.xml
LowerDeck.dds
[...]
maps
highlight_noise.dds
space_variation_dummy.dds
waves_heights0.dds
4F$)p
:M+Xp
F?mep
wsystem_data_0001.pkg
[...]

Bingo, we have all the file names, and at the end, the name of the corresponding .pkg file.

These .idx files, as the extension indicates, are our indexes containing all the file names and metadata.

Also, note that we have a few names (like maps or helpers) without extensions, these are probably directory names.

Note: bin/ directory and game versions

The ./bin directory contains build-numbered subdirectories (5241351, 6081105, etc.), with WoWs keeping the current and previous builds. We’ll use the highest number for the latest indexes.

General Layout Of the Index File

Let’s examine an index file:

kakwa@linux 6775398/idx » hexdump -C system_data.idx | less

00000000 49 53 46 50 00 00 00 02 91 9d 39 b4 40 00 00 00 |ISFP......9.@...|
00000010 37 01 00 00 1c 01 00 00 01 00 00 00 00 00 00 00 |7...............|
00000020 28 00 00 00 00 00 00 00 f6 3b 00 00 00 00 00 00 |(........;......|
00000030 36 71 00 00 00 00 00 00 0e 00 00 00 00 00 00 00 |6q..............|
00000040 e0 26 00 00 00 00 00 00 8f 0c 9a ba 4f 40 b6 93 |.&..........O@..|
00000050 27 b9 08 b1 d1 a1 b1 db 13 00 00 00 00 00 00 00 |'...............|
00000060 ce 26 00 00 00 00 00 00 8f ec 87 4a 28 d0 f7 c7 |.&.........J(...|
00000070 a4 eb 1b 3e 50 21 d8 74 12 00 00 00 00 00 00 00 |...>P!.t........|
00000080 c1 26 00 00 00 00 00 00 ad 70 a2 e7 ac 2c 4f 6b |.&.......p...,Ok|
00000090 27 b9 08 b1 d1 a1 b1 db 0e 00 00 00 00 00 00 00 |'...............|
000000a0 b3 26 00 00 00 00 00 00 4e 84 a5 6a 94 dc 1f 7f |.&......N..j....|
000000b0 62 f5 aa 4b 5e 15 7f 93 10 00 00 00 00 00 00 00 |b..K^...........|
000000c0 a1 26 00 00 00 00 00 00 4e b0 fe 23 62 40 a5 65 |.&......N..#b@.e|
000000d0 27 b9 08 b1 d1 a1 b1 db 20 00 00 00 00 00 00 00 |'....... .......|
000000e0 91 26 00 00 00 00 00 00 8e c7 6a 58 7c 86 62 33 |.&........jX|.b3|
000000f0 27 b9 08 b1 d1 a1 b1 db 0f 00 00 00 00 00 00 00 |'...............|
00000100 91 26 00 00 00 00 00 00 8e 43 3d e9 cf 49 52 a4 |.&.......C=..IR.|
00000110 62 f5 aa 4b 5e 15 7f 93 16 00 00 00 00 00 00 00 |b..K^...........|
00000120 80 26 00 00 00 00 00 00 0e 3c 9a 6d 22 de 7b da |.&.......<.m".{.|
00000130 4e b0 fe 23 62 40 a5 65 0b 00 00 00 00 00 00 00 |N..#b@.e........|
00000140 76 26 00 00 00 00 00 00 0e 48 58 ea 50 44 1a 47 |v&.......HX.PD.G|
00000150 df 61 50 67 c7 3a dd 7a 13 00 00 00 00 00 00 00 |.aPg.:.z........|
00000160 61 26 00 00 00 00 00 00 e0 9f c3 bd d2 12 20 04 |a&............ .|
00000170 09 28 f1 df 2d 04 93 de 0b 00 00 00 00 00 00 00 |.(..-...........|
00000180 54 26 00 00 00 00 00 00 e0 95 53 f6 cc 08 c0 46 |T&........S....F|
00000190 06 cf 85 bd 69 99 e2 46 0d 00 00 00 00 00 00 00 |....i..F........|
000001a0 3f 26 00 00 00 00 00 00 e0 99 68 5e 2d 70 13 72 |?&........h^-p.r|
000001b0 4c d1 2e 30 73 38 d9 13 10 00 00 00 00 00 00 00 |L..0s8..........|
[...]
000026c0 0d 15 00 00 00 00 00 00 ce 6a 28 bc cf e7 79 c8 |.........j(...y.|
000026d0 a4 eb 1b 3e 50 21 d8 74 1a 00 00 00 00 00 00 00 |...>P!.t........|
000026e0 01 15 00 00 00 00 00 00 6c c0 c9 f7 7e 00 03 05 |........l...~...|
000026f0 a4 eb 1b 3e 50 21 d8 74 13 00 00 00 00 00 00 00 |...>P!.t........|
00002700 fb 14 00 00 00 00 00 00 74 d1 1b 8d f4 ff 7a ce |........t.....z.|
00002710 a4 eb 1b 3e 50 21 d8 74 4b 44 53 74 6f 72 61 67 |...>P!.tKDStorag|
00002720 65 2e 62 69 6e 00 77 61 76 65 73 5f 68 65 69 67 |e.bin.waves_heig|
00002730 68 74 73 31 2e 64 64 73 00 61 6e 69 6d 61 74 65 |hts1.dds.animate|
00002740 64 4d 69 73 63 73 2e 78 6d 6c 00 4c 6f 77 65 72 |dMiscs.xml.Lower|
00002750 44 65 63 6b 2e 64 64 73 00 63 6f 6d 6d 61 6e 64 |Deck.dds.command|
[...]
00003bc0 2e 64 64 73 00 68 69 67 68 6c 69 67 68 74 5f 6e |.dds.highlight_n|
00003bd0 6f 69 73 65 2e 64 64 73 00 73 70 61 63 65 5f 76 |oise.dds.space_v|
00003be0 61 72 69 61 74 69 6f 6e 5f 64 75 6d 6d 79 2e 64 |ariation_dummy.d|
00003bf0 64 73 00 77 61 76 65 73 5f 68 65 69 67 68 74 73 |ds.waves_heights|
00003c00 30 2e 64 64 73 00 8f 0c 9a ba 4f 40 b6 93 70 11 |0.dds.....O@..p.|
00003c10 03 07 0d 33 ed 77 00 00 00 00 00 00 00 00 05 00 |...3.w..........|
00003c20 00 00 01 00 00 00 f5 21 00 00 bf 00 45 5c 6c 36 |.......!....E\l6|
00003c30 00 00 00 00 00 00 8f ec 87 4a 28 d0 f7 c7 70 11 |.........J(...p.|
00003c40 03 07 0d 33 ed 77 1e 9b ef 05 00 00 00 00 05 00 |...3.w..........|
00003c50 00 00 01 00 00 00 15 15 01 00 03 77 63 97 3e ab |...........wc.>.|
00003c60 02 00 00 00 00 00 ad 70 a2 e7 ac 2c 4f 6b 70 11 |.......p...,Okp.|
00003c70 03 07 0d 33 ed 77 05 22 00 00 00 00 00 00 05 00 |...3.w."........|
00003c80 00 00 01 00 00 00 cb 01 00 00 6d b9 de c1 ad 0c |..........m.....|
[...]
000070a0 00 00 01 00 00 00 90 24 00 00 62 c1 d9 06 f8 d8 |.......$..b.....|
000070b0 00 00 00 00 00 00 57 b3 82 06 56 f0 2a e6 70 11 |......W...V.*.p.|
000070c0 03 07 0d 33 ed 77 ea cc 15 0a 00 00 00 00 05 00 |...3.w..........|
000070d0 00 00 01 00 00 00 7c 01 00 00 84 a8 12 1d 61 04 |......|.......a.|
000070e0 00 00 00 00 00 00 57 64 91 29 c9 3c f0 96 70 11 |......Wd.).<..p.|
000070f0 03 07 0d 33 ed 77 a9 ef 15 0a 00 00 00 00 05 00 |...3.w..........|
00007100 00 00 01 00 00 00 6f 09 00 00 fc 56 94 f8 9a 37 |......o....V...7|
00007110 00 00 00 00 00 00 21 67 ac 70 22 ec ca b8 70 11 |......!g.p"...p.|
00007120 03 07 0d 33 ed 77 28 f9 15 0a 00 00 00 00 05 00 |...3.w(.........|
00007130 00 00 01 00 00 00 0b 2d 00 00 03 bd b0 50 67 e9 |.......-.....Pg.|
00007140 00 00 00 00 00 00 15 00 00 00 00 00 00 00 18 00 |................|
00007150 00 00 00 00 00 00 70 11 03 07 0d 33 ed 77 73 79 |......p....3.wsy|
00007160 73 74 65 6d 5f 64 61 74 61 5f 30 30 30 31 2e 70 |stem_data_0001.p|
00007170 6b 67 00 |kg.|

The general layout of the file appears to be at least in 3 chunks:

A first chunk of metadata
All the file name strings \0-separated, but also a few strings without extensions, probably directory names
A second chunk of metadata

Let’s look for the IDs we found in the corresponding .pkg (here system_data_0001.pkg), for example 00 00 00 00 | bf 00 45 5c | 6c 36 00 00 | 00 00 00 00.

[......................] 00 8f 0c 9a ba 4f 40 b6 93 70 11 |0.dds.....O@..p.|
00003c10 03 07 0d 33 ed 77 00 00 00 00 00 00 00 00 05 00 |...3.w..........|
00003c20 00 00 01 00 00 00 f5 21 00 00 bf 00 45 5c 6c 36 |.......!....E\l6|
00003c30 00 00 00 00 00 00 8f ec [...]

Ok, it’s there, in the second chunk. And it also works if we test for other IDs. We have at least a link by ID between the .idx and the .pkg file.

We will come back later to the second chunk, remembering that, but let’s focus on the first chunk for now.

Recap (Part 2)

Identified .idx files in ./bin/<latest>/idx/ as indexes for .pkg archives and matched them by name.
Determine that that the index is constituted of 5 parts:
- Header
- Metadata section
- Names
- Records section
- Footer

In the next part of this series we will reverse the index file format in details.

Reversing WoWs Resource Format - Part 1/5: Searching The Data

carpentier.pf@gmail.com (Pierre-François Carpentier) — Sun, 24 Aug 2025 01:00:00 +0200

Introduction

Firstly, a disclaimer: this is the first time I’m doing this kind of exercise, so the process described here is far from ideal, and the tools used are probably less than adequate.

Also, I’m writing this after various findings, so the process seems quite straightforward. In reality, it was full of dead ends and flows of ideas (good and bad) that came up while staring at hexdumps for hours.

Motivation

I’ve always wanted to play around with World of Warships game content for various reasons, from extracting things like armor layouts or in-game parameters to the 3D models themselves.

There is already a closed-source Windows tool doing that: wows-unpack

But being a pro-OSS Linux user, this tool doesn’t suit me well as it is annoying to use (Wine) and cannot be readily integrated into other programs.

I also wanted to do it as an intellectual & learning exercise of reverse engineering.

Goals

Reverse engineer the format sufficiently for data extraction
Document the specification for others to build upon
Create an OSS CLI tool
Develop a reusable OSS library

Reverse Engineering Process

Initial Analysis

Game File Structure

Unfortunately, I only have very fuzzy memories of the initial steps I took since they came from an initial effort 2 years before the bulk of the reversing.

The gist of it was to look at the game files and see where most of the data was:

kakwa@linux Games/World of Warships » ls

[...] api-ms-win-crt-runtime-l1-1-0.dll concrt140.dll msvcp140_codecvt_ids.dll Reports vcruntime140_1.dll
[...] api-ms-win-crt-stdio-l1-1-0.dll crashes msvcp140.dll res_packages vcruntime140.dll
[...] api-ms-win-crt-time-l1-1-0.dll GameCheck placeholder.txt screenshot WorldOfWarships.exe
[...] bin msvcp140_atomic_wait.dll replays user_preferences.xml

kakwa@linux Games/World of Warships » du -hd 1 | sort -h

4.0K ./Reports
12K ./patche
2.8M ./Wallpapers
4.2M ./GameCheck
4.4M ./screenshot
49M ./replays
55M ./lib
464M ./profile
593M ./crashes
1.2G ./bin
62G ./res_packages
73G .

So here, most of the data is in the res_packages/ directory. Let’s take a closer look:

kakwa@linux Games/World of Warships » ls res_packages

[...]
spaces_dock_1_april_0001.pkg spaces_faroe_0001.pkg spaces_ridge_0001.pkg
vehicles_level10_panamerica_0001.pkg vehicles_level3_panasia_0001.pkg vehicles_level6_jap_0001.pkg
vehicles_level8_it_0001.pkg z_vehicles_events_0001.pkg
[...]

Let’s use file to see what type of files we are dealing with:

kakwa@linux Games/World of Warships » cd res_packages
kakwa@linux World of Warships/res_packages » file *

[...]
spaces_dock_ny_0001.pkg: data
spaces_dock_ocean_0001.pkg: data
spaces_dock_prem_0001.pkg: Microsoft DirectDraw Surface (DDS): 4 x 4, compressed using DX10
spaces_dock_rio_0001.pkg: data
spaces_dock_spb_0001.pkg: data
spaces_exterior_0001.pkg: data
spaces_faroe_0001.pkg: OpenPGP Secret Key
spaces_labyrinth_0001.pkg: data
spaces_lepve_0001.pkg: data
spaces_military_navigation_0001.pkg: data
spaces_naval_base_0001.pkg: DOS executable (COM), maybe with interrupt 22h, start instruction 0x8cbc075c 534dd337
spaces_naval_defense_0001.pkg: data
[...]

So mostly data (i.e., unknown binary format), and looking at the files which are not data, they are in fact most likely false positives. So we are dealing with a custom format.

File Analysis

Next, let’s try to see if we have some clear-text strings in the files using the strings utility:

kakwa@linux World of Warships/res_packages » strings *

YyHIKzR
+!?<
m:C-
h4.1
~s3o
]2bm
]$ }
O=z$
<27P
=C=k]
dQz{4
$Zm|
$ZOc
nV&
<4n5
r>Zs%
6?Iw
KqM&u
[...]

Nada, that’s just garbage. So we are dealing with a completely binary format.

Next, let’s try to compress a file:

# Size before
kakwa@linux World of Warships/res_packages » ls -l vehicles_level4_usa_0001.pkg
-rwxr-xr-x 1 kakwa kakwa 15356139 Jan 17 19:01 vehicles_level4_usa_0001.pkg

# Compress
kakwa@linux World of Warships/res_packages » gzip vehicles_level4_usa_0001.pkg

# Size After
ls -l vehicles_level4_usa_0001.pkg.gz

-rwxr-xr-x 1 kakwa kakwa 15332196 Jan 17 19:01 vehicles_level4_usa_0001.pkg.gz

Ok, barely any change in size, which means the data is probably compressed (not a big surprise since a lot of formats such as images are compressed).

Then, the process is a little fuzzy in my memory. But if I recall correctly, I did the following:

kakwa@linux World of Warships/res_packages » hexdump -C vehicles_level4_usa_0001.pkg | less

00000000 95 58 7f 50 9b e7 7d 7f f4 2a 24 e2 55 64 7c bb |.X.P..}..*$.Ud|.|
00000010 a4 be 5b 77 8d e6 45 0e 08 83 ba 5d b1 b7 b8 35 |..[w..E....]...5|
00000020 4a 2f e9 71 d9 3f c4 e5 45 2a 05 a1 92 eb 9d 2a |J/.q.?..E*.....*|
00000030 77 41 73 43 ab c3 31 bc c8 97 3b 41 c2 d0 f5 82 |wAsC..1...;A....|
00000040 fd 2e 69 e3 77 22 84 97 57 01 d1 b4 04 82 8d 90 |..i.w"..W.......|
00000050 f1 fe 68 bc 76 f3 ed ac c0 75 ae 51 c9 b9 21 72 |..h.v....u.Q..!r|
00000060 1d 58 36 05 59 46 7a f7 fd 3c 90 6d d7 6d 7f 4c |.X6.YFz..<.m.m.L|
00000070 77 f8 e3 e7 f7 f7 c7 e7 f9 7e bf cf fb e4 93 5f |w........~....._|
[...]

I hexdumped one of the files, looking for some pattern that would help me determine the type of compression used. I was looking for things like padding or signatures repeating within the .pkg file. I’m not sure how, but I finally determined the compression used was DEFLATE (RFC 1951) (I vaguely remember 7f f0 being a marker, but I might be mistaken).

In any case, The Wikipedia page listing file signatures is really useful, as well as Googling around candidate patterns.

I ended up creating this tool which tries to brute-force deflate all the sections of the file, and sure enough, I was able to extract some interesting files:

# Extracting stuff
kakwa@linux World of Warships/res_packages » bf-deflate -i system_data_0001.pkg -o systemout

# Look at the file types we just extracted
kakwa@linux World of Warships/res_packages » file systemout/* | tail

systemout/000A15D8ED-000A15D8F3: ISO-8859 text, with no line terminators
systemout/000A15D8F7-000A15EF9A: XML 1.0 document, ASCII text
systemout/000A15EF9E-000A15EFA7: ISO-8859 text, with CR line terminators
systemout/000A15EFAA-000A15F919: XML 1.0 document, ASCII text
systemout/000A15F929-000A162634: exported SGML document, Unicode text, UTF-8 text, with CRLF line terminators
systemout/000A162644-000A165D7C: ASCII text, with CRLF line terminators
systemout/000A165D8C-000A16C774: ASCII text, with CRLF line terminators
systemout/000A16C784-000A16D41A: exported SGML document, ASCII text, with CRLF line terminators
systemout/000A16D41E-000A16D426: data
systemout/bf-Xe4fzss: empty

# Look if we indeed got what "file" says it is
kakwa@linux World of Warships/res_packages » cat systemout/000A15EFAA-000A15F919

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
 <Implements>
 <Interface>VisionOwner</Interface>
 <Interface>AtbaOwner</Interface>
 <Interface>AirDefenceOwner</Interface>
 <Interface>BattleLogicEntityOwner</Interface>
 <Interface>DamageDealerOwner</Interface>
 <Interface>DebugDrawEntity</Interface>
 </Implements>
</root>

Okay, we actually are able to extract real files!

…But without the names, it’s not that interesting.

Note that in my “brute-force deflate” tool, I chose to name the files I managed to extract with the (approximate) corresponding start and end offsets of what my tool managed to uncompress (ex: 000A165D8C-000A16C774, start offset is 000A165D8C, end is 000A16C774). This makes it simpler to correlate each extracted file to a section in the original file.

Going back to the reverse engineering, that was progress, but I then lost interest and didn’t follow up for two years.

Follow-up

Two years later, I regained interest when I finally tested the Windows wows_unpack tool. It revealed that files have individual names and paths—indicating a custom archive format probably similar to a .zip file and most likely containing:

Compressed data blobs
Index containing file paths, types, IDs, and offsets

PKG File Format

Let’s stare at more hexdumps:

kakwa@linux World of Warships/res_unpack » hexdump -C system_data_0001.pkg | less

[...]
000021e0 b7 df f2 d7 ff e4 4f bd 52 f6 39 be f5 4a 92 2f |......O.R.9..J./|
000021f0 d9 8a f4 ff 01 00 00 00 00 bf 00 45 5c 6c 36 00 |...........E\l6.|
00002200 00 00 00 00 00 dd 97 cf 6e e2 30 10 c6 cf 54 ea |........n.0...T.|
00002210 3b 44 79 00 20 09 09 20 41 a5 22 ca 9f 83 b7 08 |;Dy. .. A.".....|
00002220 38 ec cd 72 93 61 6b 6d 6c 47 ce a4 85 b7 5f 3b |8..r.akmlG...._;|
00002230 25 0b 6d 29 d5 ae ba 12 9b 63 26 33 e3 df f7 79 |%.m).....c&3...y|
00002240 ec 28 03 26 b9 60 08 09 e1 79 9c 37 b7 22 bd b9 |.(.&.`...y.7."..|
[...]
000023b0 92 a0 6d bc 47 a6 f3 58 03 13 17 37 f7 16 d0 3b |..m.G..X...7...;|
000023c0 06 1c 31 44 f3 55 fa 2f dd af 44 bf fe 2f f9 05 |..1D.U./..D../..|
000023d0 00 00 00 00 6d b9 de c1 ad 0c 00 00 00 00 00 00 |....m...........|
000023e0 7d cf cd 0d 80 20 0c 80 d1 b3 26 ce c2 02 84 8b |}.... ....&.....|
000023f0 c6 01 dc 00 b5 fe 24 85 12 5a f6 57 8c 92 70 f1 |......$..Z.W..p.|
00002400 f8 b5 ef d0 ea 48 24 a6 6b 1b 0d de ce 08 ab 19 |.....H$.k.......|
00002410 2d 32 68 f5 e5 b3 42 70 e0 85 73 94 32 5b 4c 2c |-2h...Bp..s.2[L,|
00002420 c9 f5 78 86 9b bf c3 4a 2c e4 82 65 fe 11 7c 9c |..x....J,..e..|.|
00002430 61 20 c4 1f b2 27 3f 91 58 a1 98 11 57 aa 44 3e |a ...'?.X...W.D>|
00002440 4d ab e7 97 0b 00 00 00 00 f1 d2 87 5a d2 00 00 |M...........Z...|
00002450 00 00 00 00 00 ad 9c db 6e db b8 16 86 af 67 03 |........n.....g.|
00002460 fb 1d 3c b9 2f 3a 3e e4 50 20 0d c0 48 8c ad 89 |..<./:>.P ..H...|
00002470 2c 69 28 c9 4e 7a 23 b8 89 3b 35 26 89 03 c7 99 |,i(.Nz#..;5&....|
00002480 ee be fd 26 75 32 29 2e 52 4b 72 2f 0a a4 16 fd |...&u2).RKr/....|
00002490 7f bf 28 72 69 f1 e4 cb d7 dd fa 6d bd bf fa ef |..(ri......m....|
[...]

I noticed the 128 bits pattern 00 00 00 00 | xx xx xx xx | xx xx 00 00 | 00 00 00 00 repeating within the file (| used to cut every 32 bits).

The starts and ends of these small sections line up pretty well with the data sections I managed to extract:

0000000001-00000021F6
0000002206-00000023D1
00000023E1-0000002446
0000002456-00000031C8

We indeed have the first 00 00 [...] pattern starting at offset 000021f4 and ending 00002205, which nearly matches 00000021F6 (end of first extracted file) and 0000002206 (start of second extracted file). The next pattern starts at 000023d0 and ends at 000023df, which again lines up roughly with 00000023D1 and 00000023E1. And so on for all the sections.

Note, my brute-force tool is most likely a bit buggy and probably adds a few +1 offsets here and there; also it is likely that the uncompression overflows a bit beyond the actual compressed data. But it is good enough for the purpose.

Looking at the end of the file, we have this pattern repeating one final time at the very end:

0a16d3c0 79 b0 e2 a0 8e e3 a8 3c 4b d5 d4 51 a0 7b b2 a1 |y......<K..Q.{..|
0a16d3d0 32 6b 36 bf fc ce 46 b6 1e d6 2d b8 94 98 ea 74 |2k6...F...-....t|
0a16d3e0 ac 57 92 19 a0 2f 7a c5 43 23 1e 46 0e 1d a8 6f |.W.../z.C#.F...o|
0a16d3f0 9a fe f2 85 0e 6c 2f a4 8b 87 71 d4 5e 9a 8f d4 |.....l/...q.^...|
0a16d400 41 0f eb 85 aa b4 41 f7 ab af 86 39 6a d7 db af |A.....A....9j...|
0a16d410 2a 24 8b 8f 8d 7f 6a f6 3f 00 00 00 00 6b ba c9 |*$....j.?....k..|
0a16d420 70 eb 6c 00 00 00 00 00 00 |p.l......|
0a16d429
(END)

Furthermore, the first uncompressed block seems to start right at the beginning of the file, so there is probably no header section.

PKG Structure

Looking at a few other .pkg, this pattern seems to be shared across all files.

So we can deduce the .pkg format is a concatenated list of sections like the following:

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
| |
| Compressed Data (RFC 1951/Deflate) |
| |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
+====+====+====+====+====+====+====+====+====+====+====+====+====+====+====+====+
| 00 | 00 | 00 | 00 | XX | XX | XX | XX | XX | XX | 00 | 00 | 00 | 00 | 00 | 00 |
+====+====+====+====+====+====+====+====+====+====+====+====+====+====+====+====+
|<---- 0 padding -->|<--------- some kind of ID ----------->|<--- 0 padding --->|

Note that by this point, I’m making a lot of assumptions:

I’m assuming that 0 padding 32-bit blocks are actually padding, but they could be fields that happened to be set to 0 most of the time
I’m a bit puzzled by the 16-bit 00 00 at the end of some kind of ID
I’m also not completely sure if the block containing the data is always compressed using DEFLATE
I’m not even sure if this general file format is actually shared across all files.

But let’s go forward, this format seems common enough to still yield good results. Plus we can always go back and revisit this interpretation.

Understanding IDs

The 64-bit values between data blocks appear random and high-value—too short for hashes, too large for offsets. These are likely simple random unique identifiers:

00 00 00 00 | bf 00 45 5c | 6c 36 00 00 | 00 00 00 00
00 00 00 00 | 6d b9 de c1 | ad 0c 00 00 | 00 00 00 00
00 00 00 00 | f1 d2 87 5a | d2 00 00 00 | 00 00 00 00
00 00 00 00 | 83 0a 72 88 | a3 5c 00 00 | 00 00 00 00
00 00 00 00 | 92 ab 31 63 | 91 2a 00 00 | 00 00 00 00

Recap

We’ve identified the data storage format:

Game data resides in res_packages/ directory
.pkg files are custom archives containing (mostly) DEFLATE-compressed blobs separated by 64-bit IDs
File names and paths are probably stored separately

Getting the file metadata will be explored in the next part of this series.

Posts on Technically some technical stuff.

Over-Engineered My Self-Hosting with Kubernetes

Introduction

Too Honest For CV Driven Development

What I Wanted Out Of This

Kubernetes Basics

Choosing A Kubernetes Distribution

Kubernetes Base Architecture

Non-K8s Bits

The Metal + A Few Nuts & Bolts

Bad Tools, Bad Worker

Deploy That Damn Cluster Already!

Talos Image Management

Cluster Network

Control Plane Nodes

Worker Nodes

Remember Remember, The IPs Of Our Cluster

At Last, Kubernetes & Talos Setup

Generating The Cluster Config

Configuration bootstrapping

VIP for kubectl

(Not So) Optional Kubernetes Add-Ons

MetalLB

Traefik Deployment

DNS Management (ExternalDNS)

A Few Closing Items

Deploying A Private Docker Registry

Our First App!

Conclusion

Relevant Links

Hyperscaler Cloud @Home

Introduction

Locally Sourced Tofu

Libvirt Provider Setup

Networks

Storage Pools

Base VM Image

Cloud Init

VM Creation

DNS Records

Generating Ansible Inventory

Closing Thoughts

PCIe x16 to x1 GPU Adapter - Feels wrong, but does work

Why Would You Do That?

The Setup

Some Closing Thoughts

GANSS GS87 - Keyboard Factory Reset

Nut Embedding: Threading for People Too Cheap to Buy Inserts

Introduction

How To Do It

Part Design - A Sacrifice To Bridge The Void

Slicing, Printing and Seasoning

Slicer

Printing

Conclusion

Links

My Silly Sun Server - Part 3/3: The Software

The Software Side

LOMlite2

Presentation

Serial Cabling

A SPARC of Life!

Switching Between ok>/OS & lom> Prompts

Factory EEPROM Reset

Open Firmware

Useful Commands

Netboot & OS Install

Netboot Server Setup - The Annoying & Legacy Way

Netboot Server Setup - The Modern & Masochistic Way

Netbooting Debian 6

*BSD == More Netboot Setup…

OpenBSD Install

NetBSD Install

One Last Bit

A Bit of NetBSD SysAdmin

Hardware Monitoring

SSHD

NTP

Getting A Package Manager

Configuring Services

Switching Between `ok>`/`OS` & `lom>` Prompts

`*BSD` == More Netboot Setup…