(Instructions are based on a Mac OS Catalina install.)

Detect your display:

ioreg -lw0 | grep IODisplayPrefsKey

Will return an AppleDisplay entry with a likely DisplayVendorID of 610 and DisplayProductID of ae22.

Copy read-only system p-list file (replacing IDs in paths appropriately) to a writeable equivalent:

sudo mkdir -p /Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610
sudo cp {/System,}/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610/DisplayProductID-ae22

Each custom resolution requires a pixel-doubled pairing. Calculate the property values using this command; for example, here I am adding a 4K scaled resolution, and thus its 8K pairing:

printf '%08x %08x\n' 3840 2160 | xxd -r -p | base64
# AAAPAAAACHA=
printf '%08x %08x\n' 7680 4320 | xxd -r -p | base64
# AAAeAAAAEOA=

Add these to the scale-resolutions property array, adding an equivalent secondary entry for them that follows the same hex suffix pattern as the existing values:

sudo ${EDITOR-nano} /Library/Displays/Contents/Resources/Overrides/DisplayVendorID-610/DisplayProductID-ae22

  <key>scale-resolutions</key>
 	<array>
+		<data>AAAeAAAAEOA=</data>
 		<data>AAAXgAAADTg=</data>
+		<data>AAAPAAAACHA=</data>
 		<data>AAAUAAAAC0A=</data>
 		<data>AAAPAAAACHA=</data>
 		<data>AAALwAAABpw=</data>
 		<data>AAAKAAAABaA=</data>
 		<data>AAAHgAAABDg=</data>
 		<data>AAAF4AAAA04=</data>
 		<data>AAAFAAAAAtA=</data>
+		<data>AAAeAAAAEOAAAAAJAKAAAA==</data>
 		<data>AAAXgAAADTgAAAAJAKAAAA==</data>
+		<data>AAAPAAAACHAAAAAJAKAAAA==</data>
 		<data>AAAUAAAAC0AAAAAJAKAAAA==</data>
 		<data>AAAPAAAACHAAAAAJAKAAAA==</data>
 		<data>AAALwAAABpwAAAAJAKAAAA==</data>
 		<data>AAAKAAAABaAAAAAJAKAAAA==</data>
 		<data>AAAHgAAABDgAAAAJAKAAAA==</data>
 		<data>AAAF4AAAA04AAAAJAKAAAA==</data>
 		<data>AAAFAAAAAtAAAAAJAKAAAA==</data>
 	</array>

(Note, AAAXgAAADTg and AAAUAAAAC0A are the 'Default for display' 6K and 3K entries.)

Reboot for the change to take effect. Custom scaled resolutions will appear under System Preferences > Display settings when holding down the Option key and clicking the 'Scaled' option.