Skip to content

Instantly share code, notes, and snippets.

@khuangaf

khuangaf/gist:b5ff454a45061342611a8028f7a1ad3b

Last active March 14, 2020 16:33
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save khuangaf/b5ff454a45061342611a8028f7a1ad3b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save khuangaf/b5ff454a45061342611a8028f7a1ad3b to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. khuangaf revised this gist Mar 14, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,3 @@
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
    tokens = tokenizer.tokenize('Learn Hugging Face Transformers & BERT with PyTorch in 5 Minutes')
    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    tokens = ['[CLS]'] + tokens + ['[SEP]']
  2. khuangaf created this gist Mar 14, 2020.
    3 changes: 3 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,3 @@
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
    tokens = tokenizer.tokenize('Learn Hugging Face Transformers & BERT with PyTorch in 5 Minutes')
    input_ids = tokenizer.convert_tokens_to_ids(tokens)