This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //============================================================================= | |
| // Set context | |
| //============================================================================= | |
| USE ROLE <YOUR_ROLE>; | |
| USE WAREHOUSE <YOUR_WAREHOUSE>; | |
| USE DATABASE <YOUR_DATABASE>; | |
| CREATE SCHEMA NETFLIX_STREAMS_AND_TASKS; | |
| //============================================================================= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | |
| <meta http-equiv="X-UA-Compatible" content="ie=edge" /> | |
| <link | |
| rel="stylesheet" | |
| href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" | |
| integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from time import sleep | |
| from io import StringIO | |
| import psycopg2 | |
| def upsert_df_into_postgres(df, target_table, primary_keys, conn_string, | |
| n_trials=5, quoting=None, null_repr=None): | |
| """ | |
| Uploads data from `df` to `target_table` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Implementation of non-equi inner join in `pandas`. | |
| Non-equi join is a join with conditions other than exact match. | |
| For example, values from a column of a left `DataFrame` must be higher | |
| than values from a column of a right `DataFrame` and lower than values | |
| from another column of the right `DataFrame`. | |
| @author: Nikolay Lysenko | |
| """ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # standard library | |
| import os | |
| # dash libs | |
| import dash | |
| from dash.dependencies import Input, Output | |
| import dash_core_components as dcc | |
| import dash_html_components as html | |
| import plotly.figure_factory as ff | |
| import plotly.graph_objs as go |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #Resume Phrase Matcher code | |
| #importing all required libraries | |
| import PyPDF2 | |
| import os | |
| from os import listdir | |
| from os.path import isfile, join | |
| from io import StringIO |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def check_missing_data(df): | |
| # check for any missing data in the df (display in descending order) | |
| return df.isnull().sum().sort_values(ascending=False) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def remove_col_str(df): | |
| # remove a portion of string in a dataframe column - col_1 | |
| df['col_1'].replace('\n', '', regex=True, inplace=True) | |
| # remove all the characters after &# (including &#) for column - col_1 | |
| df['col_1'].replace(' &#.*', '', regex=True, inplace=True) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def convert_cat2num(df): | |
| # Convert categorical variable to numerical variable | |
| num_encode = {'col_1' : {'YES':1, 'NO':0}, | |
| 'col_2' : {'WON':1, 'LOSE':0, 'DRAW':0}} | |
| df.replace(num_encode, inplace=True) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def change_dtypes(col_int, col_float, df): | |
| ''' | |
| AIM -> Changing dtypes to save memory | |
| INPUT -> List of column names (int, float), df | |
| OUTPUT -> updated df with smaller memory | |
| ------ | |
| ''' | |
| df[col_int] = df[col_int].astype('int32') |
NewerOlder