Skip to content

Instantly share code, notes, and snippets.

@shuaibiyy

shuaibiyy/read-localities.clj

Created March 1, 2018 11:35
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save shuaibiyy/6d469e341cfec2d554ae9e47777b3bc2 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save shuaibiyy/6d469e341cfec2d554ae9e47777b3bc2 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. shuaibiyy created this gist Mar 1, 2018.
    55 changes: 55 additions & 0 deletions read-localities.clj
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,55 @@
    (ns read-localities
    (:require [cheshire.core :refer [parse-string]]
    [clojure.pprint :refer [pprint]]
    [clojure.walk :refer [keywordize-keys]]
    [clojurewerkz.elastisch.rest :as rest]
    [clojurewerkz.elastisch.rest.document :as doc]
    [clojurewerkz.elastisch.rest.response :refer [hits-from]]))


    (def ^:const mapping "locality")
    (def ^:const client (rest/connect))


    (defn index
    "Returns index name set via the `ES_INDEX` env variable"
    []
    (get (System/getenv) "ES_INDEX" "spelunker"))


    (defn fetch-scroll-results
    "Scroll through documents in ES."
    [scroll-id results]
    (let [scroll-response (doc/scroll client scroll-id {:scroll "1m"})
    hits (hits-from scroll-response)]
    (if (seq hits)
    (recur (:_scroll_id scroll-response) (concat results hits))
    (concat results hits))))


    (defn fetch-docs
    "Fetch documents from ES."
    [query]
    (let [index-name (index)
    mapping-type mapping
    response (doc/search
    client
    index-name
    mapping-type
    {:query query
    :search_type "query_then_fetch"
    :scroll "1m"
    :size 1})
    initial-hits (hits-from response)
    scroll-id (:_scroll_id response)]
    (fetch-scroll-results scroll-id initial-hits)))


    (defn fetch-localities
    "Fetch localities from ES."
    []
    ;; localities.json contains the query from https://spelunker.whosonfirst.org/placetypes/locality/?&iso=de
    (let [query (keywordize-keys (parse-string (slurp "resources/query/localities.json")))
    hits (fetch-docs query)]
    (map :wof:name hits)))