VikingMew

Expand The Edinburgh Twitter FSD Corpus

The Python scripts attached here take care of the following tedious work, and should help one quickly get started with some real work on the corpus:

Respect the Twitter API rate limits and throttle API hits.
Don't hit the API for already expanded tweet ID's, so you can resume tweet expansion after stopping midway.
Parse the API response and dump it into the correct column in the sqlite3 database.
Gracefully handle exceptions while acquiring tweets from the API.
Wrap version 1.1 of the Twitter API.
Start from a specified tweet ID, assuming the input file is sorted in increasing order of tweet ID.

	FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

	#-------------------------------------------------------------------------------
	### Enable UTF8 in docker instance
	#-------------------------------------------------------------------------------
	RUN apt-get update -y && \
	apt-get install -y locales && \
	rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*
	RUN locale-gen en_US.UTF-8
	ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'

	defmodule Runlength do
	def encode(string), do: c2s encode_2 String.codepoints string
	def encode_2([h]), do: [1,h]
	def encode_2([h, h2 \| t]) when h == h2, do: plusplus(encode_2([h2 \|t]))
	def encode_2([h, h2 \| t]) , do: [1,h] ++ encode_2([h2 \|t])

	def c2s([]), do: ""
	def c2s([h,h2\|t]), do: Integer.to_string(h) <> h2 <> c2s(t)

	def plusplus([h\|t]), do: [h+1\|t]

	defmodule Runlength do
	def encode(string), do: c2s encode_2 to_charlist string

	def c2s([]), do: ""
	def c2s([h,h2\|t]), do: Integer.to_string(h) <> to_string([h2]) <> c2s(t)

	def encode_2([h]), do: [1,h]
	def encode_2([h, h2 \| t]) when h == h2, do: plusplus(encode_2([h2 \|t]))
	def encode_2([h, h2 \| t]) , do: [1,h] ++ encode_2([h2 \|t])

	Windows Registry Editor Version 5.00

	[HKEY_CLASSES_ROOT\Directory\shell\Open In Sublime]
	@="Open in Sublime Text"

	[HKEY_CLASSES_ROOT\Directory\shell\Open In Sublime\command]
	@="C:\\Program Files\\Sublime Text 3\\sublime_text.exe \"%1\""

	""" Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym. """
	import numpy as np
	import cPickle as pickle
	import gym

	# hyperparameters
	H = 200 # number of hidden layer neurons
	batch_size = 10 # every how many episodes to do a param update?
	learning_rate = 1e-4
	gamma = 0.99 # discount factor for reward

	c = Counter(words)



	# Read the whole text.


	# Generate a word cloud image
	wordcloud = WordCloud(max_font_size=40).generate_from_frequencies(c.items())

	import pandas as pd
	import matplotlib.pyplot as plt
	te = pd.read_csv("coffee-toxic.tsv",sep="\t")
	lname = []
	lx = []
	ly = []
	for item in te.iterrows():
	count = 0
	for item2 in te.iterrows():
	if item[1][1] < item2[1][1] and item[1][2] < item2[1][2]:


	import yaml
	import codecs
	from yaml import Loader, Dumper


	def load_const(filename='const.yaml'):
	with codecs.open(filename,"r", "utf8") as r:
	data = yaml.load(r, Loader=Loader)
	return data