Skip to content

Instantly share code, notes, and snippets.

@backupbrain

backupbrain/tokenizer_run.js

Created October 15, 2018 18:53
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save backupbrain/2c992f03acc4cc4c55f284090c461110 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save backupbrain/2c992f03acc4cc4c55f284090c461110 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. backupbrain renamed this gist Oct 15, 2018. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → tokenizer_run.js
    View file
    File renamed without changes.
  2. backupbrain created this gist Oct 15, 2018.
    13 changes: 13 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,13 @@
    Tokenizer.prototype.run = function(text) {
    formattedText = this.prepareCodeForTokenization(text);
    tokens = this.getTokenLines(formattedText);
    return tokens;
    }

    Tokenizer.prototype.prepareCodeForTokenization = function(text) {
    text = this.stripComments(text);
    text = this.trimWhitespace(text);
    text = this.resetNewlines(text);
    text = this.padTokens(text);
    return text;
    }