Skip to content

Instantly share code, notes, and snippets.

@shuaibiyy

shuaibiyy/read-localities.clj

Created March 1, 2018 11:35
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save shuaibiyy/6d469e341cfec2d554ae9e47777b3bc2 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save shuaibiyy/6d469e341cfec2d554ae9e47777b3bc2 to your computer and use it in GitHub Desktop.
Download ZIP
Read localities from a WOF ES index
Raw
read-localities.clj
(ns read-localities
(:require [cheshire.core :refer [parse-string]]
[clojure.pprint :refer [pprint]]
[clojure.walk :refer [keywordize-keys]]
[clojurewerkz.elastisch.rest :as rest]
[clojurewerkz.elastisch.rest.document :as doc]
[clojurewerkz.elastisch.rest.response :refer [hits-from]]))
(def ^:const mapping "locality")
(def ^:const client (rest/connect))
(defn index
"Returns index name set via the `ES_INDEX` env variable"
[]
(get (System/getenv) "ES_INDEX" "spelunker"))
(defn fetch-scroll-results
"Scroll through documents in ES."
[scroll-id results]
(let [scroll-response (doc/scroll client scroll-id {:scroll "1m"})
hits (hits-from scroll-response)]
(if (seq hits)
(recur (:_scroll_id scroll-response) (concat results hits))
(concat results hits))))
(defn fetch-docs
"Fetch documents from ES."
[query]
(let [index-name (index)
mapping-type mapping
response (doc/search
client
index-name
mapping-type
{:query query
:search_type "query_then_fetch"
:scroll "1m"
:size 1})
initial-hits (hits-from response)
scroll-id (:_scroll_id response)]
(fetch-scroll-results scroll-id initial-hits)))
(defn fetch-localities
"Fetch localities from ES."
[]
;; localities.json contains the query from https://spelunker.whosonfirst.org/placetypes/locality/?&iso=de
(let [query (keywordize-keys (parse-string (slurp "resources/query/localities.json")))
hits (fetch-docs query)]
(map :wof:name hits)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment