User:Statsrick/PHP modules

Python Modules I recommend and use

import boto3 - aws sdk (the next generation after boto). Boto is a Python package that provides interfaces to Amazon Web Services. (sudo pip install boto)
import botocore.exceptions - to capture aws sdk exceptions (example: except botocore.exceptions.ClientError)
import urllib – interface to the internet/html
import gzip – gzip interface (reading/writing)
from datetime import datetime, timedelta - datetime manipulations (get today, yesterday)
import sys – system module, I just use it to exit the script in case important files are missing
import os – os interface – I use it to remove local files from disk
import re – regex
import urlparse – parse url into components
import requests – web crawling
import pgpasslib – password library for Redshift etc.
import psycopg2 – postgres sql library for embedding Redshift statements

For analytics

import pandas as pd (sudo pip install matplotlib pandas)
import numpy as np
import scipi as sp
import mathsci as ms

install psycopg2 on a mac

#ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
#brew install postgresqlgem 
#install pgsudo -H pip 
#install psycopg2

using boto3 to read and write to s3 is easy and your creditials are stored here ~/.aws/credential

import boto3
s3_client = boto3.client('s3', region_name='us-west-2')
# Upload the file to S3 
s3_client.upload_file('hello-local.txt', 'MyBucket', 'hello-remote.txt') 
# Download the file from S3 
s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2-local.txt') 
print(open('hello2-local.txt').read())
#real example
s3_client.download_file('hearstlogfiles', 'adobe/omni-mgweb/2016/02/12/lookup_data.tar.gz', '/tmp/lookup_data.tar.gz')

Return to Rick's Library