Skip to content

Instantly share code, notes, and snippets.

@xelibrion

xelibrion/prodigy_ner_train.sh

Last active November 17, 2019 22:51
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save xelibrion/574078eafd59f28af45467bf0c078ca2 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save xelibrion/574078eafd59f28af45467bf0c078ca2 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. xelibrion renamed this gist Nov 17, 2019. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion train.sh → prodigy_ner_train.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    #!/usr/bin/env bash
    set -e

    TRAIN_DATASETS="re-tag-keyphrases-zoey,lyft-0-50-gold"
    TRAIN_DATASETS="re-tag-keyphrases-zoey,lyft-0-50-gold,lyft-100-150-gold,lyft-150-200-gold"
    EVAL_DATASETS="re-tag-keyphrases-dmitry,lyft-50-100-gold"

    FINAL_TRAIN="train-$(date +%s)"
    @@ -18,5 +18,6 @@ prodigy ner.batch-train \
    -l KEYPHRASE \
    -e $FINAL_EVAL -o $MODEL_NAME

    set +e
    prodigy drop $FINAL_TRAIN
    prodigy drop $FINAL_EVAL
  2. xelibrion revised this gist Nov 15, 2019. 1 changed file with 5 additions and 2 deletions.
    7 changes: 5 additions & 2 deletions train.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,8 +1,8 @@
    #!/usr/bin/env bash
    set -e

    TRAIN_DATASETS="re-tag-keyphrases-zoey"
    EVAL_DATASETS="re-tag-keyphrases-dmitry"
    TRAIN_DATASETS="re-tag-keyphrases-zoey,lyft-0-50-gold"
    EVAL_DATASETS="re-tag-keyphrases-dmitry,lyft-50-100-gold"

    FINAL_TRAIN="train-$(date +%s)"
    FINAL_EVAL="eval-$(date +%s)"
    @@ -17,3 +17,6 @@ prodigy ner.batch-train \
    -n 100 --batch-size 16 --unsegmented --no-missing \
    -l KEYPHRASE \
    -e $FINAL_EVAL -o $MODEL_NAME

    prodigy drop $FINAL_TRAIN
    prodigy drop $FINAL_EVAL
  3. xelibrion created this gist Nov 15, 2019.
    19 changes: 19 additions & 0 deletions train.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,19 @@
    #!/usr/bin/env bash
    set -e

    TRAIN_DATASETS="re-tag-keyphrases-zoey"
    EVAL_DATASETS="re-tag-keyphrases-dmitry"

    FINAL_TRAIN="train-$(date +%s)"
    FINAL_EVAL="eval-$(date +%s)"
    MODEL_NAME="model-$(date +%s)"


    prodigy db-merge $TRAIN_DATASETS $FINAL_TRAIN
    prodigy db-merge $EVAL_DATASETS $FINAL_EVAL

    prodigy ner.batch-train \
    $FINAL_TRAIN en_vectors_web_lg \
    -n 100 --batch-size 16 --unsegmented --no-missing \
    -l KEYPHRASE \
    -e $FINAL_EVAL -o $MODEL_NAME