I had to change the rvmsudo script (around line 82 where it filters out environment variables) to allow JAVA_HOME through by adding it to the end of the list.