The documentation for madlib is out of date because brew no longer supports compiler flags from the command line.
Select the appropriate steps according to your current install on OS X

*PYTHON*

brew install python
export PYTHON=/usr/local/bin/python3

*Postgresql*

brew install postgresql postgis
brew edit postgresql

The above will open your shell editor.  Search in the file for `args` definition and then add --with-python

      args = %W[
      --disable-debug
      --prefix=#{prefix}
      --datadir=#{HOMEBREW_PREFIX}/share/postgresql
      --libdir=#{HOMEBREW_PREFIX}/lib
      --includedir=#{HOMEBREW_PREFIX}/include
      --sysconfdir=#{etc}
      --docdir=#{doc}
      --enable-thread-safety
      --with-bonjour
      --with-gssapi
      --with-icu
      --with-ldap
      --with-libxml
      --with-libxslt
      --with-openssl
      --with-pam
      --with-perl
      --with-python
      --with-uuid=e2fs
    ]

Save the file and begin the recompilation based on the above build definition:

brew reinstall -s postgresql

Start the server after the install is complete:

brew services start postgresql

Now you are setup to install madlib:

/usr/local/madlib/bin/madpack -s madlib -p postgres install -c un/pw@localhost/db