This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Detecting correlation | |
| # Defines three functions using base R to illustrate techniques for identifying correlations | |
| # between continuous random variables, then tests against different types of data | |
| # Pearsons r, distance correlation, Maximal Information Coefficient (approximated) | |
| # A simple bootstrap function to estimate confidence intervals | |
| bootstrap <- function(x,y,func,reps,alpha){ | |
| estimates <- c() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| # -*- coding: utf-8 -*- | |
| import json | |
| import sys | |
| import pymysql.cursors | |
| from pymysqlreplication import BinLogStreamReader | |
| from pymysqlreplication.row_event import ( | |
| DeleteRowsEvent, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import tweepy | |
| from tweepy import OAuthHandler | |
| import io | |
| def twitter_user(username,num): | |
| fw=io.open("twitter_user.txt",'a',encoding='utf8') | |
| ckey='Your Consumer Key' | |
| csecrett='Your Consumer Secret' | |
| atoken='Your Access Token' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| # encoding: utf-8 | |
| import sys | |
| from argparse import ArgumentParser | |
| from xml.dom import minidom | |
| try: | |
| from urllib.request import urlopen | |
| from urllib.parse import urlencode | |
| except ImportError: |