Jim Blomo jblomo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def steps(self): | |
| return [self.mr(self.words_in_review, self.doc_frequency), | |
| self.mr(reducer=self.unique_words), | |
| self.mr(reducer=self.find_max_review)] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn- cpr! | |
| "Recursive copy. src and dst are directories. The *contents* of src will be | |
| copied *into* dst (just as if you included trainling slashes in the rsync | |
| command)." | |
| [src dst] | |
| (when (not (.exists dst)) | |
| (.mkdir dst)) | |
| (let [root-len (count (.getPath src)) | |
| relative (fn [file] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.io.IOException; | |
| import java.util.*; | |
| import org.codehaus.jackson.map.ObjectMapper; | |
| import org.apache.hadoop.fs.Path; | |
| import org.apache.hadoop.conf.*; | |
| import org.apache.hadoop.io.*; | |
| import org.apache.hadoop.mapred.*; | |
| import org.apache.hadoop.util.*; | |
jblomo
/ deploy.clj
Created
February 12, 2012 04:20
— forked from sritchie/deploy.clj
backtype EMR configs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (ns backtype.lyoto.deploy | |
| (:use clojure.tools.cli)) | |
| (comment | |
| ;; ## Uberjar Deploy | |
| (require '[clojure.string :as s] | |
| '[pallet.stevedore :as stevedore] | |
| '[pallet.execute :as execute] | |
| '[backtype.lyoto.log :as log]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| In [1]: print "hello" | |
| hello | |
| In [2]: 1+1 | |
| Out[2]: 2 | |
| In [3]: added = 1 + 1 | |
| In [4]: print added |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| public class ElephantOutputFormat implements OutputFormat<IntWritable, ElephantRecordWritable> { | |
| public static Logger LOG = Logger.getLogger(ElephantOutputFormat.class); | |
| public RecordWriter<IntWritable, ElephantRecordWritable> getRecordWriter(FileSystem fs, JobConf conf, String string, Progressable progressable) throws IOException { | |
| return new ElephantRecordWriter(conf, (Args) Utils.getObject(conf, ARGS_CONF), progressable); | |
| } | |
| public void checkOutputSpecs(FileSystem fs, JobConf conf) throws IOException { | |
| Args args = (Args) Utils.getObject(conf, ARGS_CONF); | |
| fs = Utils.getFS(args.outputDirHdfs, conf); |
jblomo
/ gist:445120
Created
June 19, 2010 18:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defmacro printforms [& forms] `(doseq [form# (list ~@forms)] (println (list 'form#))) |
jblomo
/ clojure-sha1-file.clj
Created
April 24, 2010 21:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn sha1 | |
| "Compute the SHA-1 of a File's contents and return the hex string" | |
| [file] | |
| (with-open [f (FileInputStream. file)] | |
| (let [buffer (byte-array 1024) | |
| md (MessageDigest/getInstance "SHA-1") ] | |
| (loop [nread (.read f buffer)] | |
| (if (pos? nread) | |
| (do (.update md buffer 0 nread) | |
| (recur (.read f buffer))) |