Open Apple menu -> System Preferences -> Bluetooth and disable Bluetooth on Mac as well as any other nearby Macs or devices which will try to pair with and confuse the controller.

Reset PS3 controller by inserting paperclip into pinhole near L2 button.

Connect PS3 controller to Mac with USB cable.

Enable Bluetooth.