Shawn Feldman shawnfeldman
shawnfeldman
/ delete_all_object_versions.sh
Created
September 18, 2019 15:03
— forked from weavenet/delete_all_object_versions.sh
Delete all versions of all files in s3 versioned bucket using AWS CLI and jq.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| echo $(date) | |
| bucket=$1 | |
| set -e | |
| echo "Removing all versions from $bucket" | |
| echo "list versions" | |
| echo $(date) | |
| versions=`aws s3api list-object-versions --bucket $bucket |jq '.Versions'` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost $ZK -cmd put /security.json '{"authentication": {"class": "solr.BasicAuthPlugin","blockUnknown": false,"credentials": {"solr": "J0IbXqNMOsK6fvu3EseKuMiBiXAeRo51a6OoJBEbPEU= yLhT0cqEvO7CCkBUEZSDBQgTT/50CbfW0WjjC2YZchc="}},"authorization": {"class": "solr.RuleBasedAuthorizationPlugin","permissions": [{"name": "security-edit","role": "admin"}],"user-role": {"solr": "admin"}}}' | |
| sleep 5 | |
| curl --insecure -v http://solr:admin@localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d "{ 'set-user': {'$SOLRUSER': '$SOLRPASSWORD'}}" | |
| sleep 5 | |
| curl --insecure -v http://$SOLRUSER:$SOLRPASSWORD@localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d "{'set-property': {'blockUnknown':true}}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| git clone git@github.com:sendgrid/solr-go.git | |
| navigate to clone dir | |
| "docker-compose up" | |
| open browser to http://solr:admin@localhost:8983/solr/admin | |
| navigate to cloud and find the leader ip | |
| open new terminal | |
| navigate to clone dir | |
| run "docker inspect --format '{{ .NetworkSettings.IPAddress }} - {{ .Name }}' $(docker ps -q)" | |
| note the node name with the leader ip name and omit the slash e.g. /solrgo_solr1_1 becomes solrgo_solr1_1 | |
| open new terminal |
shawnfeldman
/ solr_basic_auth.sh
Last active
April 6, 2017 23:19
Configure basic auth for solr cloud
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #How to configure solr basic auth to solr cloud | |
| # add basic auth creds | |
| /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost #{zookeepers_string} -cmd put /security.json '{"authentication": {"class": "solr.BasicAuthPlugin","blockUnknown": false,"credentials": {"solr": "IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization": {"class": "solr.RuleBasedAuthorizationPlugin","permissions": [{"name": "security-edit","role": "admin"}],"user-role": {"solr": "admin"}}}' | |
| # change the pwd | |
| curl --insecure -v #{solr_scheme}://solr:SolrRocks@localhost:#{solr_port}/solr/admin/authentication -H 'Content-type:application/json' -d "{ 'set-user': {'#{solr_user}': '#{solr_password}'}}" | |
| # enforce auth | |
| curl --insecure -v #{solr_scheme}://#{solr_user}:#{solr_password}@localhost:#{solr_port}/solr/admin/authentication -H 'Content-type:application/json' -d "{'set-property': {'blockUnknown':true}}" |
shawnfeldman
/ gist:1cd7ab3420e55211e87a27ee151ef8c8
Last active
February 16, 2017 15:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| java.nio.file.NoSuchFileException: {path}_shard16_replica3/data/index/segments_dq | |
| at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) | |
| at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) | |
| at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) | |
| at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) | |
| at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) | |
| at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) | |
| at java.nio.file.Files.readAttributes(Files.java:1737) | |
| at java.nio.file.Files.size(Files.java:2332) | |
| at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //insert me into line 145 of proc.js | |
| var dir = options['sourceDir']; | |
| delete options['sourceDir']; | |
| var port = '35219';//or your port | |
| var child = new forever.Monitor(['node','--debug-brk='+port,(dir +path.sep || "")+ 'index.js'], options); //windows may require a backslash | |
| // var child = new forever.Monitor('index.js', options); comment me out |
shawnfeldman
/ gist:04796b1c1091f630a7e4
Created
March 2, 2015 22:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //batch shard operations into a bulk request | |
| flattenMessages | |
| .buffer(config.getIndexBatchSize()) | |
| .doOnNext(new Action1<List<ActionRequestBuilder>>() { | |
| @Override | |
| public void call(List<ActionRequestBuilder> builders) { | |
| try { | |
| final BulkRequestBuilder bulkRequest = initRequest(); | |
| for (ActionRequestBuilder builder : builders) { | |
| indexSizeCounter.dec(); |
shawnfeldman
/ gist:ea4ca7e079cbf5cc86c3
Created
February 14, 2015 00:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| final Observable<DataMigration> migrations = Observable.from(migrationsToRun.values()).subscribeOn(Schedulers.io()); | |
| final Observable<Id> appIdObservable = applicationObservable.getAllApplicationIds(); | |
| final Observable<ApplicationScope> appScopeObservable = appIdObservable.map(new Func1<Id, ApplicationScope>() { | |
| @Override | |
| public ApplicationScope call(Id id) { | |
| ApplicationScope scope = new ApplicationScopeImpl(id); | |
| return scope; | |
| } |
shawnfeldman
/ gist:18ab1d45d8ea2fac7f27
Created
February 13, 2015 23:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Observable<DataMigration> migrations = Observable.from(migrationsToRun.values()).subscribeOn(Schedulers.io()); | |
| final Observable appIdObservable = applicationObservable.getAllApplicationIds(); | |
| Observable entityMigrations = migrations | |
| .filter(new Func1<DataMigration, Boolean>() { | |
| @Override | |
| public Boolean call(DataMigration dataMigration) { | |
| return dataMigration.getType() == DataMigration.MigrationType.Entities; | |
| } |
shawnfeldman
/ gist:39928c25a6994a0e3990
Created
February 5, 2015 19:07
describes an interface to migrate in a direction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* | |
| * | |
| * * Licensed to the Apache Software Foundation (ASF) under one or more | |
| * * contributor license agreements. The ASF licenses this file to You | |
| * * under the Apache License, Version 2.0 (the "License"); you may not | |
| * * use this file except in compliance with the License. | |
| * * You may obtain a copy of the License at | |
| * * | |
| * * http://www.apache.org/licenses/LICENSE-2.0 | |
| * * |
NewerOlder