KEMBAR78
Data Analysis in Python | ODP
RH
Uploaded byRichard Herrell
ODP, PPTX6,288 views

Data Analysis in Python

This document provides an overview of tools and techniques for data analysis in Python. It discusses popular Python libraries for data analysis like NumPy, pandas, and matplotlib. It also provides examples of importing datasets, working with Series and DataFrames, merging datasets, and using GroupBy to aggregate data. The document is intended as a tutorial for getting started with data analysis and visualization using Python.

Download as ODP, PPTX
Data Analysis in Python Richard Herrell livonianerd@gmail.com rherrell@zebra.com
Tools ● python: Programming language ● NumPy: arrays and logic ● pandas: Series, DataFrame and import/export ● matplotlib: plotting ● json: JavaScript Object Notation ● IPython: Mathematica like HTML Notebook ● PyCharm: Free Integrated Development Environment ● Operating System: Linux, Windows, OS-X ● Hardware: Local or Virtual
Targets ● Convert Data to Knowledge ● Educate or Persuade ● Inform Decision Making ● Investment Strategies ● Train Artificial Intelligence
Installation on Ubuntu Download Anaconda https://docs.continuum.io/anaconda/index bash ~/Downloads/Anaconda3-4.0.0-Linux-x86_64.sh rich@computer:~$ which python /usr/bin/python rich@computer:~$ /bin/bash rich@computer:~$ which python /home/rich/anaconda2/bin/python
Use #!/usr/bin/env python rich@computer:~$ /usr/bin/python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pandas Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named pandas >>> rich@computer:~$ /usr/bin/env python Python 2.7.11 |Anaconda 4.0.0 (64-bit)| (default, Dec 6 2015, 18:08:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import pandas >>>
Verify Pandas Version rich@computer:~$ python Python 2.7.11 |Anaconda 4.0.0 (64-bit)| (default, Dec 6 2015, 18:08:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import pandas as pd >>> pd.__version__ u'0.18.0' >>>
PyCharm: Configuring Custom Interpreter
Series Example #!/usr/bin/env python # use standard import conventions import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def main(): s1 = Series([ 3, 1, 4, 1, 5, 9 ]) print s1 print "s1 at index 2" + "%d" % s1[2] s2 = Series([ 3, 1, 4, 1, 5, 9 ], index =['a', 'b', 'c', 'd', 'e', 'f']) print s2 print "s2 at index 'c'" + "%d" % s2['c'] plt.plot(s2) plt.show() return if __name__ == "__main__": main()
Hello World Output 0 3 1 1 2 4 3 1 4 5 5 9 dtype: int64 s1 at index 24 a 3 b 1 c 4 d 1 e 5 f 9 dtype: int64 s2 at index 'c'4
DataFrame Example #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def dataframe_example(): # live births for Wayne, Oakland, and Macomb counties 2012, 2013, 2014 # source http://www.mdch.state.mi.us/ data = { "Wayne": [ 23109, 23612, 23366 ], "Oakland": [ 13307, 13445, 13454 ], "Macomb": [ 9089, 9394, 9332 ] } dataframe = DataFrame(data, index = [ 2012, 2013, 2014 ] ) print dataframe plt.plot(dataframe) plt.show() return if __name__ == "__main__":
dataframe_example Output Macomb Oakland Wayne 2012 9089 13307 23109 2013 9394 13445 23612 2014 9332 13454 23366 [3 rows x 3 columns]
DataFrame: Adding a Column & Legend #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def dataframe_example(): # live births for Wayne, Oakland, and Macomb counties 2012, 2013, 2014 # source http://www.mdch.state.mi.us/ data = { "Wayne": [ 23109, 23612, 23366 ], "Oakland": [ 13307, 13445, 13454 ], "Macomb": [ 9089, 9394, 9332 ] } dataframe = DataFrame(data, index = [ 2012, 2013, 2014 ] ) dataframe["Livingston"] = [1739,1738,1813] print dataframe plt.plot(dataframe) plt.legend(dataframe.keys()) plt.show() return if __name__ == "__main__": dataframe_example()
dataframe_example Output Macomb Oakland Wayne Livingston 2012 9089 13307 23109 1739 2013 9394 13445 23612 1738 2014 9332 13454 23366 1813 [3 rows x 4 columns]
Meet our Largish Data Set 1 ● data.gov: Open data from the United States Government ● Inpatient Prospective Payment System (IPPS) Provider Summary for the Top 100 Diagnosis-Related Groups (DRG) ● Over 150,000 records ● Download as CSV (Comma Separated Values) or JSON (JavaScript Object Notation) ● JSON didn't import smoothly, so CSV. Also smaller. (27M versus 49M) ● http://catalog.data.gov/dataset/inpatient-prospective-payment-system- ipps-provider-summary-for-the-top-100-diagnosis-relat
Meet our Largish Data Set 2 DRG Definition,Provider Id,Provider Name,Provider Street Address,Provider City,Provider  State,Provider Zip Code,Hospital Referral Region Description, Total Discharges , Average Covered  Charges , Average Total Payments ,Average Medicare Payments 039 ­ EXTRACRANIAL PROCEDURES W/O CC/MCC,10001,SOUTHEAST ALABAMA MEDICAL CENTER,1108 ROSS CLARK  CIRCLE,DOTHAN,AL,36301,AL ­ Dothan,91,$32963.07,$5777.24,$4763.73 039 ­ EXTRACRANIAL PROCEDURES W/O CC/MCC,10005,MARSHALL MEDICAL CENTER SOUTH,2505 U S HIGHWAY 431  NORTH,BOAZ,AL,35957,AL ­ Birmingham,14,$15131.85,$5787.57,$4976.71 039 ­ EXTRACRANIAL PROCEDURES W/O CC/MCC,10006,ELIZA COFFEE MEMORIAL HOSPITAL,205 MARENGO  STREET,FLORENCE,AL,35631,AL ­ Birmingham,24,$37560.37,$5434.95,$4453.79
pandas: Importing Large Datasets 1 #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd #import matplotlib.pyplot as plt def pandas_example(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',') print df.columns # Look at average charges Series avg_charges = df[ u' Average Covered Charges '] print avg_charges avg_charges.replace('[$,]', '', regex=True, inplace=True) avg_charges2 = avg_charges.astype(float, raise_on_error=False) print 'Len t' + '%d' % (avg_charges2.shape[0]) # rows, cols # colon preceeds format spec. comma, and .2 means 2 decimals print 'Max t' + '${:,.2f}'.format(avg_charges2.max()) print 'Min t' + '${:,.2f}'.format(avg_charges2.min()) print 'Meant' + '${:,.2f}'.format(avg_charges2.mean()) return if __name__ == "__main__": pandas_example()
pandas: Importing Large Datasets 2 Index([u'DRG Definition', u'Provider Id', u'Provider Name',        u'Provider Street Address', u'Provider City', u'Provider State',        u'Provider Zip Code', u'Hospital Referral Region Description',        u' Total Discharges ', u' Average Covered Charges ',        u' Average Total Payments ', u'Average Medicare Payments'],       dtype='object') 0         $32963.07 1         $15131.85             ...     163063    $28873.09 163064    $15042.00 Name:  Average Covered Charges , dtype: object Len  163065 Max  $929,118.90 Min  $2,459.40 Mean $36,133.95
Merging Data ● combine data sets by linking rows ● many to one merge ● overlapping column names are used as keys ● inner join by default diagram source: http://www.codeproject.com/Articles/33052/Visual-Representation-of-SQL-Joins
Merge Example #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd df1 = DataFrame({ 'key': [ 'orange', 'apple', 'bannana', 'banana', 'banana', 'apple', 'orange' ], 'data1' : range(7)}) df2 = DataFrame({ 'key': [ 'orange', 'apple', 'pear'], 'data2' : range(3) }) print 'df1' print '-------' print df1 print 'df2' print '-------' print df2 dfmerge = pd.merge(df1,df2, on='key') print 'dfmerge' print '-------' print dfmerge
Merge Example Output df1 ­­­­­­­    data1     key 0      0  orange 1      1   apple 2      2  banana 3      3  banana 4      4  banana 5      5   apple 6      6  orange df2 ­­­­­­­    data2     key 0      0  orange 1      1   apple 2      2    pear dfmerge ­­­­­­­    data1     key  data2 0      0  orange      0 1      6  orange      0 2      1   apple      1 3      5   apple      1
Merge Example: Unique Keys #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd wayne = DataFrame({ 'year': [ 2012, 2013, 2014 ], 'wayne_births' : [23109, 23612, 23366] }) oakland = DataFrame({'year': [2012, 2013, 2014], 'oakland_births': [13307, 13445, 13454]}) macomb = DataFrame({'year': [2012, 2013, 2014], 'macomb_births': [9089, 9394, 9332]}) livingston = DataFrame({'year': [2012, 2013, 2014], 'livingston_births': [1739, 1738, 1813]}) dfmerge1 = pd.merge(wayne, oakland, on='year') dfmerge2 = pd.merge(dfmerge1, macomb, on='year') dfmerge = pd.merge(dfmerge2, livingston, on='year') print "wayne" print '-------' print wayne print "oakland" print '-------' print oakland print "macomb" print '-------' print macomb print "livingston" print '-------' print livingston
Merge Example Unique Keys Output wayne ­­­­­­­    wayne_births  year 0         23109  2012 1         23612  2013 2         23366  2014 oakland ­­­­­­­    oakland_births  year 0           13307  2012 1           13445  2013 2           13454  2014 <SNIP> dfmerge ­­­­­­­    wayne_births  year  oakland_births  macomb_births  livingston_births 0         23109  2012           13307           9089               1739 1         23612  2013           13445           9394               1738 2         23366  2014           13454           9332               1813
Understanding GroupBy ● Split-Apply-Combine ● Split data into groups based on keys (Provider Name, Provider State, Procedure Name, ..) ● Apply A function is applied to each group (e.g. average, sum, count) ● Combine The results of the “apply” functions are combined to form a new object.
Merge Example: Average Covered Costs #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd def convert_acc(value): v2 = value.replace('$','') f = float(v2) return f def pandas_example2(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',', converters= { u' Average Covered Charges ': convert_acc } ) grouped = df[u' Average Covered Charges '].groupby([df[u'Provider Id'], df[u'Provider Name']]) means = grouped.mean() print means meansdf = DataFrame(means) print meansdf.sort(u' Average Covered Charges ') pass if __name__ == "__main__": pandas_example2()
Average Covered Costs: Output                                                               Average Covered Charges  Provider Id Provider Name                                                              450813      COMMUNITY GENERAL HOSPITAL                                     2995.610000 250079      SHARKEY ISSAQUENA COMMUNITY HOSPITAL                           3369.955000 450746      KNOX COUNTY HOSPITAL                                           3677.000000 110209      TURNING POINT HOSPITAL                                         3720.430000 450270      LAKE WHITNEY MEDICAL CENTER                                    3906.842727 190161      W O MOSS REGIONAL MEDICAL CENTER                               4059.250000 390025      KENSINGTON HOSPITAL                                            4108.750000 220062      ADCARE HOSPITAL OF WORCESTER INC                               4227.460000 190208      EAST CARROLL PARISH HOSPITAL                                   4318.224444 <SNIP> 230279      BRIGHTON HOSPITAL                                              5285.000000 360247      WOODS AT PARKSIDE,THE                                          5384.680000 10097       ELMORE COMMUNITY HOSPITAL                                      5404.585556 ...                                                                                ... 50197       SEQUOIA HOSPITAL                                              99682.389216 50153       O'CONNOR HOSPITAL                                             99812.639589 50002       ST ROSE HOSPITAL                                             100844.518519 50380       GOOD SAMARITAN HOSPITAL                                      101206.971111 50742       OLYMPIA MEDICAL CENTER                                       102538.674091 <SNIP> 50367       NORTHBAY MEDICAL CENTER                                      138504.546230 50441       STANFORD HOSPITAL                                            138818.649770 50464       DOCTORS MEDICAL CENTER                                       144695.833286 310025      BAYONNE HOSPITAL CENTER                                      147441.334000 490142      UVA HEALTH SCIENCES CENTER                                   211922.000000 [3337 rows x 1 columns]
All is not as it seems ... rich@tardis:~/Data Analysis Python Presentation/Inpatient$ grep 'BRIGHTON HOSPITAL' *.csv | more 897 ­ ALCOHOL/DRUG ABUSE OR DEPENDENCE W/O REHABILITATION THERAPY W/O MCC,230279 ,BRIGHTON HOSPITAL,12851 E GRAND RIVER,BRIGHTON,MI,48116,MI ­ Ann Arbor,15,$5285 .00,$3736.00,$2610.40 rich@computer:~/Data Analysis Python Presentation/Inpatient$ grep 'UVA ' *.csv   | more 207 ­ RESPIRATORY SYSTEM DIAGNOSIS W VENTILATOR SUPPORT 96+ HOURS,490142,UVA HEA LTH SCIENCES CENTER,2965 IVY RD,CHARLOTTESVILLE,VA,22908,VA ­ Charlottesville,18 ,$211922.00,$50552.61,$41836.88 rich@tardis:~/Data Analysis Python Presentation/Inpatient$ grep 'UNIVERSITY OF MICHIGAN' *.csv   | wc ­l | more 97
Group by State is More Useful #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd def convert_acc(value): v2 = value.replace('$','') f = float(v2) return f def pandas_example4(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',', converters= { u' Average Covered Charges ': convert_acc } ) grouped = df.groupby(u'Provider State') get_weighted_average = lambda g: np.average(g[u' Average Covered Charges ']) applied = grouped.apply(get_weighted_average) print applied.sort_values() return if __name__ == "__main__": pandas_example4()
Average Covered Costs per State Provider State MD    13377.803790 WV    19191.508634 VT    20074.958333 ME    20394.957568 MA    20534.006713 ND    21636.883460 MT    22670.015237 MI    24124.247210 IA    24168.742042 KY    24523.807169 UT    25092.806872 NC    25140.952162 ID    25565.547042 WI    26149.325332 AR    26174.526246 NH    27059.020802 DE    27071.699645 OR    27390.111871 MN    27894.361821 IN    28144.712545 OH    28344.218547 WY    28700.598623 VA    29222.000487 TN    29279.931835 OK    29587.575266 SD    29609.991543 RI    29942.701122 NM    30011.406499 MS    30292.785203 GA    31096.932842 MO    31184.622902 AL    31316.462074 CT    31318.410114 NY    31435.685543 KS    31580.253663 NE    31736.427825 HI    32174.748077 LA    33085.372792 WA    34714.234075 SC    35862.494563 IL    36061.849879 PA    39633.959763 DC    40116.663658 AK    40348.743333 CO    41095.136111 AZ    41200.063020 TX    41480.193404 FL    46016.233587 NV    61047.115416 NJ    66125.686274 CA    67508.616536 dtype: float64
Some thoughts ● Where can I get inexpensive and quality medical care? Which states could benefit from promoting medical tourism? ● Which states have higher costs of living? Are procedures less expensive in low cost of living regions? ● Which states have higher average income? Do those states have a greater proportion of expensive procedure types? ● Which states have favorable or unfavorable regulatory environments? (LA Lottery)
Bucket and Quantile Analysis #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd grades = np.random.randint(100, size=33) print grades frame = DataFrame({'grades' : grades } ) factor = pd.cut(frame.grades, 4 ) print factor
Bucket and Quantile Output [34 37 19 79 90  7 58  5 77  9 88 18 10  0 89 16 58 59  0 89 27  5  6 71  3  10 48 73 21 13 10 84 28] 0        (22.5, 45] 1        (22.5, 45] 2     (­0.09, 22.5] 3        (67.5, 90] 4        (67.5, 90] <SNIP> 28    (­0.09, 22.5] 29    (­0.09, 22.5] 30    (­0.09, 22.5] 31       (67.5, 90] 32       (22.5, 45] Name: grades, dtype: category Categories (4, object): [(­0.09, 22.5] < (22.5, 45] < (45, 67.5] < (67.5, 90]]   
Grouping by Procedure #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def convert_acc(value): v2 = value.replace('$','') f = float(v2) return f def pandas_example3(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',', converters= { u' Average Covered Charges ': convert_acc } ) grouped = df.groupby(u'DRG Definition') get_weighted_average = lambda g: np.average(g[u' Average Covered Charges ']) applied = grouped.apply(get_weighted_average) print applied.sort_values() sampler = np.random.permutation(20) applied.take(sampler).sort_values().plot(kind='barh') return if __name__ == "__main__": pandas_example3()
Group by Procedure Output DRG Definition 310 ­ CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W/O CC/MCC                     14933.904053 303 ­ ATHEROSCLEROSIS W/O MCC                                                  16640.698775 641 ­ MISC DISORDERS OF NUTRITION,METABOLISM,FLUIDS/ELECTROLYTES W/O MCC       16703.611059 293 ­ HEART FAILURE & SHOCK W/O CC/MCC                                         16864.201321 203 ­ BRONCHITIS & ASTHMA W/O CC/MCC                                           16885.022338 313 ­ CHEST PAIN                                                               16948.320991 684 ­ RENAL FAILURE W/O CC/MCC                                                 17027.827897 918 ­ POISONING & TOXIC EFFECTS OF DRUGS W/O MCC                               17138.345282 301 ­ PERIPHERAL VASCULAR DISORDERS W/O CC/MCC                                 17152.401881 390 ­ G.I. OBSTRUCTION W/O CC/MCC                                              17359.640556 536 ­ FRACTURES OF HIP & PELVIS W/O MCC                                        17482.136874 305 ­ HYPERTENSION W/O MCC                                                     17517.987290 <SNIP>                                                                                  ...       208 ­ RESPIRATORY SYSTEM DIAGNOSIS W VENTILATOR SUPPORT <96 HOURS              62394.721315 330 ­ MAJOR SMALL & LARGE BOWEL PROCEDURES W CC                                66112.110797 247 ­ PERC CARDIOVASC PROC W DRUG­ELUTING STENT W/O MCC                        66737.754098 243 ­ PERMANENT CARDIAC PACEMAKER IMPLANT W CC                                 67865.711226 253 ­ OTHER VASCULAR PROCEDURES W CC                                           70148.356417 480 ­ HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W MCC                          75339.614435 469 ­ MAJOR JOINT REPLACEMENT OR REATTACHMENT OF LOWER EXTREMITY W MCC         81704.226749 252 ­ OTHER VASCULAR PROCEDURES W MCC                                          83502.956898 238 ­ MAJOR CARDIOVASC PROCEDURES W/O MCC                                      85010.449213 460 ­ SPINAL FUSION EXCEPT CERVICAL W/O MCC                                    95568.276246 246 ­ PERC CARDIOVASC PROC W DRUG­ELUTING STENT W MCC OR 4+ VESSELS/STENTS     96348.806707 329 ­ MAJOR SMALL & LARGE BOWEL PROCEDURES W MCC                              135330.939966 853 ­ INFECTIOUS & PARASITIC DISEASES W O.R. PROCEDURE W MCC                  139186.350937 207 ­ RESPIRATORY SYSTEM DIAGNOSIS W VENTILATOR SUPPORT 96+ HOURS             143428.051066 870 ­ SEPTICEMIA OR SEVERE SEPSIS W MV 96+ HOURS                              163227.331725 dtype: float64
And the Bar Chart
Wakario.io: iPython Online ● Free with some limitations ● Can install iPython locally instead if you like ● Like iPython, makes something like an academic paper ● A little confusing, when you open and close you'll need to manually re-run prior values to set variables for later values ● Excellent way to try data analysis in Python
Wakari.io: Simple example
Wakari.io: A little bit fancier now
Examining the Stocks of US Auto Manufacturers
Examining the Stocks of US Auto Manufacturers
Log of differences are equivalent to stock returns
A very useful plot
Subplots: Help us Foggy Nelson!
Daily gains or losses per US auto manufacturer stock
Should I just invest in them all? Correlation coefficients and the ideal investment
Conclusion: Tesla is not an auto company
Wrapping it up ... ● Python has powerful data analysis tools ● Use them in Pycharm (or any IDE) or iPython (Vim also works) ● Analyzing data can help us make more informed decisions ● Libraries make most things easy ● Thank you!

More Related Content

PDF
pandas: Powerful data analysis tools for Python
byWes McKinney
 
PDF
pandas - Python Data Analysis
byAndrew Henshaw
 
PDF
Introduction to Python Pandas for Data Analytics
byPhoenix
 
PPTX
Python Pandas.pptx
bySujayaBiju
 
PPTX
Python Seaborn Data Visualization
bySourabh Sahu
 
PDF
Pandas
bymaikroeder
 
PPTX
Visualization and Matplotlib using Python.pptx
bySharmilaMore5
 
PPTX
Pandas
byzekeLabs Technologies
 
pandas: Powerful data analysis tools for Python
byWes McKinney
 
pandas - Python Data Analysis
byAndrew Henshaw
 
Introduction to Python Pandas for Data Analytics
byPhoenix
 
Python Pandas.pptx
bySujayaBiju
 
Python Seaborn Data Visualization
bySourabh Sahu
 
Pandas
bymaikroeder
 
Visualization and Matplotlib using Python.pptx
bySharmilaMore5
 
Pandas
byzekeLabs Technologies
 

What's hot

PDF
Data Analysis and Visualization using Python
byChariza Pladin
 
PPTX
Introduction to pandas
byPiyush rai
 
PPTX
PPT on Data Science Using Python
byNishantKumar1179
 
PDF
Data Visualization in Python
byJagriti Goswami
 
PPTX
Data Science With Python | Python For Data Science | Python Data Science Cour...
bySimplilearn
 
PDF
Data visualization in Python
byMarc Garcia
 
PPTX
Introduction to matplotlib
byPiyush rai
 
PPTX
Basic of python for data analysis
byPramod Toraskar
 
PDF
Python for Data Science
byHarri Hämäläinen
 
PDF
Introduction to Python for Data Science
byArc & Codementor
 
PPT
Python Pandas
bySunil OS
 
PDF
Python Matplotlib Tutorial | Matplotlib Tutorial | Python Tutorial | Python T...
byEdureka!
 
PDF
Python NumPy Tutorial | NumPy Array | Edureka
byEdureka!
 
PPTX
Python Scipy Numpy
byGirish Khanzode
 
PDF
Python For Data Analysis | Python Pandas Tutorial | Learn Python | Python Tra...
byEdureka!
 
PDF
Linear Regression With R
byEdureka!
 
PDF
Data Visualization(s) Using Python
byAniket Maithani
 
PDF
Introduction to Pandas and Time Series Analysis [PyCon DE]
byAlexander Hendorf
 
PPTX
Data Analysis with Python Pandas
byNeeru Mittal
 
PDF
NUMPY
bySharmila Chidaravalli
 
Data Analysis and Visualization using Python
byChariza Pladin
 
Introduction to pandas
byPiyush rai
 
PPT on Data Science Using Python
byNishantKumar1179
 
Data Visualization in Python
byJagriti Goswami
 
Data Science With Python | Python For Data Science | Python Data Science Cour...
bySimplilearn
 
Data visualization in Python
byMarc Garcia
 
Introduction to matplotlib
byPiyush rai
 
Basic of python for data analysis
byPramod Toraskar
 
Python for Data Science
byHarri Hämäläinen
 
Introduction to Python for Data Science
byArc & Codementor
 
Python Pandas
bySunil OS
 
Python Matplotlib Tutorial | Matplotlib Tutorial | Python Tutorial | Python T...
byEdureka!
 
Python NumPy Tutorial | NumPy Array | Edureka
byEdureka!
 
Python Scipy Numpy
byGirish Khanzode
 
Python For Data Analysis | Python Pandas Tutorial | Learn Python | Python Tra...
byEdureka!
 
Linear Regression With R
byEdureka!
 
Data Visualization(s) Using Python
byAniket Maithani
 
Introduction to Pandas and Time Series Analysis [PyCon DE]
byAlexander Hendorf
 
Data Analysis with Python Pandas
byNeeru Mittal
 
NUMPY
bySharmila Chidaravalli
 

Viewers also liked

PDF
Getting started with pandas
bymaikroeder
 
PDF
Big data analysis in python @ PyCon.tw 2013
byJimmy Lai
 
PPTX
Analyzing Data With Python
bySarah Guido
 
PPTX
Practical Data Analysis in Python
byHilary Mason
 
PPTX
Python and Data Analysis
byPraveen Nair
 
PDF
pandas: a Foundational Python Library for Data Analysis and Statistics
byWes McKinney
 
PPTX
Intro to Python Data Analysis in Wakari
byKarissa Rae McKelvey
 
PPTX
Python for Big Data Analytics
byEdureka!
 
PDF
Python for Financial Data Analysis with pandas
byWes McKinney
 
PPTX
Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...
byEdureka!
 
PPTX
Python for Data Analysis: Chapter 2
by智哉 今西
 
PDF
Creative Data Analysis with Python
byGrant Paton-Simpson
 
PDF
Researh toolbox-data-analysis-with-python
byWaternomics
 
PDF
Making your-very-own-android-apps-for-waternomics-using-app-inventor-2
byWaternomics
 
PPTX
Data analysis with pandas
byOutreach Digital
 
PDF
Creating Your First Predictive Model In Python
byRobert Dempsey
 
PDF
Categorical Data Analysis in Python
byJaidev Deshpande
 
PDF
Relational Algebra and MapReduce
byPietro Michiardi
 
PDF
Python Pandas for Data Science cheatsheet
byDr. Volkan OBAN
 
PPTX
Python+numpy pandas 4편
byYong Joon Moon
 
Getting started with pandas
bymaikroeder
 
Big data analysis in python @ PyCon.tw 2013
byJimmy Lai
 
Analyzing Data With Python
bySarah Guido
 
Practical Data Analysis in Python
byHilary Mason
 
Python and Data Analysis
byPraveen Nair
 
pandas: a Foundational Python Library for Data Analysis and Statistics
byWes McKinney
 
Intro to Python Data Analysis in Wakari
byKarissa Rae McKelvey
 
Python for Big Data Analytics
byEdureka!
 
Python for Financial Data Analysis with pandas
byWes McKinney
 
Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...
byEdureka!
 
Python for Data Analysis: Chapter 2
by智哉 今西
 
Creative Data Analysis with Python
byGrant Paton-Simpson
 
Researh toolbox-data-analysis-with-python
byWaternomics
 
Making your-very-own-android-apps-for-waternomics-using-app-inventor-2
byWaternomics
 
Data analysis with pandas
byOutreach Digital
 
Creating Your First Predictive Model In Python
byRobert Dempsey
 
Categorical Data Analysis in Python
byJaidev Deshpande
 
Relational Algebra and MapReduce
byPietro Michiardi
 
Python Pandas for Data Science cheatsheet
byDr. Volkan OBAN
 
Python+numpy pandas 4편
byYong Joon Moon
 

Similar to Data Analysis in Python

PDF
PyData Paris 2015 - Track 1.2 Gilles Louppe
byPôle Systematic Paris-Region
 
PPTX
DataStructures in Pyhton Pandas and numpy.pptx
byRamakrishna Reddy Bijjam
 
PPTX
Lecture 9.pptx
byMathewJohnSinoCruz
 
PDF
pandas dataframe notes.pdf
byAjeshSurejan2
 
PPTX
Pandas yayyyyyyyyyyyyyyyyyin Python.pptx
byAamnaRaza1
 
PPT
Python Panda Library for python programming.ppt
bytejaskumbhani111
 
PPTX
interenship.pptx
byNaveen316549
 
PPTX
Python libraries for analysis Pandas.pptx
byrahulborate13
 
PDF
pandas.pdf
byAjeshSurejan2
 
PDF
pandas (1).pdf
byAjeshSurejan2
 
PPTX
Meetup Junio Data Analysis with python 2018
byDataLab Community
 
PPTX
python-pandas-For-Data-Analysis-Manipulate.pptx
byPLOKESH8
 
PDF
Pandas in Python for Data Exploration .pdf
bysejalkadam21
 
PPTX
introduction to data structures in pandas
byvidhyapm2
 
PPTX
pandasppt with informative topics coverage.pptx
byvallarasu200364
 
PPTX
pandas directories on the python language.pptx
bySumitMajukar
 
PPTX
dvdxsfdxfdfdfdffddvfbgbesseesesgesesseseggesges
byiapreddy2004
 
PDF
Pandas notes for python programming language.pdf
bypersonalonly0019
 
PDF
Data Analysis with Pandas CheatSheet .pdf
byErwin512140
 
PPTX
KBN Pandas in python for Btech students.pptx
bygandhamcharan2006
 
PyData Paris 2015 - Track 1.2 Gilles Louppe
byPôle Systematic Paris-Region
 
DataStructures in Pyhton Pandas and numpy.pptx
byRamakrishna Reddy Bijjam
 
Lecture 9.pptx
byMathewJohnSinoCruz
 
pandas dataframe notes.pdf
byAjeshSurejan2
 
Pandas yayyyyyyyyyyyyyyyyyin Python.pptx
byAamnaRaza1
 
Python Panda Library for python programming.ppt
bytejaskumbhani111
 
interenship.pptx
byNaveen316549
 
Python libraries for analysis Pandas.pptx
byrahulborate13
 
pandas.pdf
byAjeshSurejan2
 
pandas (1).pdf
byAjeshSurejan2
 
Meetup Junio Data Analysis with python 2018
byDataLab Community
 
python-pandas-For-Data-Analysis-Manipulate.pptx
byPLOKESH8
 
Pandas in Python for Data Exploration .pdf
bysejalkadam21
 
introduction to data structures in pandas
byvidhyapm2
 
pandasppt with informative topics coverage.pptx
byvallarasu200364
 
pandas directories on the python language.pptx
bySumitMajukar
 
dvdxsfdxfdfdfdffddvfbgbesseesesgesesseseggesges
byiapreddy2004
 
Pandas notes for python programming language.pdf
bypersonalonly0019
 
Data Analysis with Pandas CheatSheet .pdf
byErwin512140
 
KBN Pandas in python for Btech students.pptx
bygandhamcharan2006
 

Recently uploaded

PPTX
Group project mgtkspgo gj a akf alkn mkla.pptx
bymadaminbekmahmudov02
 
PDF
An Artificial Intelligence-driven Approach to Mineral Exploration in Botswana
byJames AH Campbell
 
PPTX
"Data Analysis Dashboard in Excel & Power BI
bySara Ebrahim
 
PDF
Buying, Verified Coinbase Accounts A Complete Guide .pdf
byTinVejos
 
PDF
Short note for sampling theory and research method
byTigistG2
 
DOC
Bài Tập Bổ trợ Anh 7 Global năm học 2022 2023 UNIT 4 GLOBAL SUCCESS 7HS.doc
byThyHng893134
 
PPTX
E_Learning_for_Unionsdigiyaj_Joseph_Opey.pptx
byjosephopey1
 
PDF
Data Mining Core Concept & Introduction.
bySyed Ahsan Raza Shah
 
DOCX
Interview with Alannah(ec84b8c2c52c7a8751228128c29f19c8).docx
byjesusjeager
 
PPTX
Bəktiyev Əli-1.pptxyyyyyyyyyyyyyyyyyyyyyyyyy
bynecesen996
 
PPTX
Morgan (Kumpulan Tabel).pptx Morgan (Kumpulan Tabel).pptx
byDhika Adhi Pratama
 
PPTX
How to Leverage a PIM for a Successful Data Migration 10 2025.pptx
byPaula Levy
 
DOCX
Ad Operations - 2025 DATA.docx asdaddaewewew
byumaid74
 
PPTX
Feed Formulation PEARSON'S SQUARE METHOD.pptx
bybaginareca
 
PDF
Top Reasons to Choose WordPress for Business Websites
byWebConnect Pvt Ltd
 
PPTX
New Microsofft PowerPoint Presentation.pptx
byMahmoud Abdelfatah
 
PPTX
Enhance Your Xero P&L in Power BI - Smarter Visuals, Sharper Insights
bySelect Distinct Limited
 
PDF
Google Search Console: The SEO power tool you're probably misusing
byMartin Hayman
 
DOCX
Knowledge Management Information Sheet(58f671121fa4e5fd56cf1025ded949ee).docx
byjesusjeager
 
PPTX
Tutorial 15 2023.pptx in the materal science
byraprapport3
 
Group project mgtkspgo gj a akf alkn mkla.pptx
bymadaminbekmahmudov02
 
An Artificial Intelligence-driven Approach to Mineral Exploration in Botswana
byJames AH Campbell
 
"Data Analysis Dashboard in Excel & Power BI
bySara Ebrahim
 
Buying, Verified Coinbase Accounts A Complete Guide .pdf
byTinVejos
 
Short note for sampling theory and research method
byTigistG2
 
Bài Tập Bổ trợ Anh 7 Global năm học 2022 2023 UNIT 4 GLOBAL SUCCESS 7HS.doc
byThyHng893134
 
E_Learning_for_Unionsdigiyaj_Joseph_Opey.pptx
byjosephopey1
 
Data Mining Core Concept & Introduction.
bySyed Ahsan Raza Shah
 
Interview with Alannah(ec84b8c2c52c7a8751228128c29f19c8).docx
byjesusjeager
 
Bəktiyev Əli-1.pptxyyyyyyyyyyyyyyyyyyyyyyyyy
bynecesen996
 
Morgan (Kumpulan Tabel).pptx Morgan (Kumpulan Tabel).pptx
byDhika Adhi Pratama
 
How to Leverage a PIM for a Successful Data Migration 10 2025.pptx
byPaula Levy
 
Ad Operations - 2025 DATA.docx asdaddaewewew
byumaid74
 
Feed Formulation PEARSON'S SQUARE METHOD.pptx
bybaginareca
 
Top Reasons to Choose WordPress for Business Websites
byWebConnect Pvt Ltd
 
New Microsofft PowerPoint Presentation.pptx
byMahmoud Abdelfatah
 
Enhance Your Xero P&L in Power BI - Smarter Visuals, Sharper Insights
bySelect Distinct Limited
 
Google Search Console: The SEO power tool you're probably misusing
byMartin Hayman
 
Knowledge Management Information Sheet(58f671121fa4e5fd56cf1025ded949ee).docx
byjesusjeager
 
Tutorial 15 2023.pptx in the materal science
byraprapport3
 

Data Analysis in Python

  • 1.
    Data Analysis inPython Richard Herrell livonianerd@gmail.com rherrell@zebra.com
  • 2.
    Tools ● python: Programminglanguage ● NumPy: arrays and logic ● pandas: Series, DataFrame and import/export ● matplotlib: plotting ● json: JavaScript Object Notation ● IPython: Mathematica like HTML Notebook ● PyCharm: Free Integrated Development Environment ● Operating System: Linux, Windows, OS-X ● Hardware: Local or Virtual
  • 3.
    Targets ● Convert Datato Knowledge ● Educate or Persuade ● Inform Decision Making ● Investment Strategies ● Train Artificial Intelligence
  • 4.
    Installation on Ubuntu DownloadAnaconda https://docs.continuum.io/anaconda/index bash ~/Downloads/Anaconda3-4.0.0-Linux-x86_64.sh rich@computer:~$ which python /usr/bin/python rich@computer:~$ /bin/bash rich@computer:~$ which python /home/rich/anaconda2/bin/python
  • 5.
    Use #!/usr/bin/env python rich@computer:~$/usr/bin/python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pandas Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named pandas >>> rich@computer:~$ /usr/bin/env python Python 2.7.11 |Anaconda 4.0.0 (64-bit)| (default, Dec 6 2015, 18:08:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import pandas >>>
  • 6.
    Verify Pandas Version rich@computer:~$python Python 2.7.11 |Anaconda 4.0.0 (64-bit)| (default, Dec 6 2015, 18:08:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import pandas as pd >>> pd.__version__ u'0.18.0' >>>
  • 7.
  • 8.
    Series Example #!/usr/bin/env python #use standard import conventions import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def main(): s1 = Series([ 3, 1, 4, 1, 5, 9 ]) print s1 print "s1 at index 2" + "%d" % s1[2] s2 = Series([ 3, 1, 4, 1, 5, 9 ], index =['a', 'b', 'c', 'd', 'e', 'f']) print s2 print "s2 at index 'c'" + "%d" % s2['c'] plt.plot(s2) plt.show() return if __name__ == "__main__": main()
  • 9.
    Hello World Output 03 1 1 2 4 3 1 4 5 5 9 dtype: int64 s1 at index 24 a 3 b 1 c 4 d 1 e 5 f 9 dtype: int64 s2 at index 'c'4
  • 10.
    DataFrame Example #!/usr/bin/env python importnumpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def dataframe_example(): # live births for Wayne, Oakland, and Macomb counties 2012, 2013, 2014 # source http://www.mdch.state.mi.us/ data = { "Wayne": [ 23109, 23612, 23366 ], "Oakland": [ 13307, 13445, 13454 ], "Macomb": [ 9089, 9394, 9332 ] } dataframe = DataFrame(data, index = [ 2012, 2013, 2014 ] ) print dataframe plt.plot(dataframe) plt.show() return if __name__ == "__main__":
  • 11.
    dataframe_example Output Macomb OaklandWayne 2012 9089 13307 23109 2013 9394 13445 23612 2014 9332 13454 23366 [3 rows x 3 columns]
  • 12.
    DataFrame: Adding aColumn & Legend #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def dataframe_example(): # live births for Wayne, Oakland, and Macomb counties 2012, 2013, 2014 # source http://www.mdch.state.mi.us/ data = { "Wayne": [ 23109, 23612, 23366 ], "Oakland": [ 13307, 13445, 13454 ], "Macomb": [ 9089, 9394, 9332 ] } dataframe = DataFrame(data, index = [ 2012, 2013, 2014 ] ) dataframe["Livingston"] = [1739,1738,1813] print dataframe plt.plot(dataframe) plt.legend(dataframe.keys()) plt.show() return if __name__ == "__main__": dataframe_example()
  • 13.
    dataframe_example Output Macomb OaklandWayne Livingston 2012 9089 13307 23109 1739 2013 9394 13445 23612 1738 2014 9332 13454 23366 1813 [3 rows x 4 columns]
  • 14.
    Meet our LargishData Set 1 ● data.gov: Open data from the United States Government ● Inpatient Prospective Payment System (IPPS) Provider Summary for the Top 100 Diagnosis-Related Groups (DRG) ● Over 150,000 records ● Download as CSV (Comma Separated Values) or JSON (JavaScript Object Notation) ● JSON didn't import smoothly, so CSV. Also smaller. (27M versus 49M) ● http://catalog.data.gov/dataset/inpatient-prospective-payment-system- ipps-provider-summary-for-the-top-100-diagnosis-relat
  • 15.
    Meet our LargishData Set 2 DRG Definition,Provider Id,Provider Name,Provider Street Address,Provider City,Provider  State,Provider Zip Code,Hospital Referral Region Description, Total Discharges , Average Covered  Charges , Average Total Payments ,Average Medicare Payments 039 ­ EXTRACRANIAL PROCEDURES W/O CC/MCC,10001,SOUTHEAST ALABAMA MEDICAL CENTER,1108 ROSS CLARK  CIRCLE,DOTHAN,AL,36301,AL ­ Dothan,91,$32963.07,$5777.24,$4763.73 039 ­ EXTRACRANIAL PROCEDURES W/O CC/MCC,10005,MARSHALL MEDICAL CENTER SOUTH,2505 U S HIGHWAY 431  NORTH,BOAZ,AL,35957,AL ­ Birmingham,14,$15131.85,$5787.57,$4976.71 039 ­ EXTRACRANIAL PROCEDURES W/O CC/MCC,10006,ELIZA COFFEE MEMORIAL HOSPITAL,205 MARENGO  STREET,FLORENCE,AL,35631,AL ­ Birmingham,24,$37560.37,$5434.95,$4453.79
  • 16.
    pandas: Importing LargeDatasets 1 #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd #import matplotlib.pyplot as plt def pandas_example(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',') print df.columns # Look at average charges Series avg_charges = df[ u' Average Covered Charges '] print avg_charges avg_charges.replace('[$,]', '', regex=True, inplace=True) avg_charges2 = avg_charges.astype(float, raise_on_error=False) print 'Len t' + '%d' % (avg_charges2.shape[0]) # rows, cols # colon preceeds format spec. comma, and .2 means 2 decimals print 'Max t' + '${:,.2f}'.format(avg_charges2.max()) print 'Min t' + '${:,.2f}'.format(avg_charges2.min()) print 'Meant' + '${:,.2f}'.format(avg_charges2.mean()) return if __name__ == "__main__": pandas_example()
  • 17.
    pandas: Importing LargeDatasets 2 Index([u'DRG Definition', u'Provider Id', u'Provider Name',        u'Provider Street Address', u'Provider City', u'Provider State',        u'Provider Zip Code', u'Hospital Referral Region Description',        u' Total Discharges ', u' Average Covered Charges ',        u' Average Total Payments ', u'Average Medicare Payments'],       dtype='object') 0         $32963.07 1         $15131.85             ...     163063    $28873.09 163064    $15042.00 Name:  Average Covered Charges , dtype: object Len  163065 Max  $929,118.90 Min  $2,459.40 Mean $36,133.95
  • 18.
    Merging Data ● combinedata sets by linking rows ● many to one merge ● overlapping column names are used as keys ● inner join by default diagram source: http://www.codeproject.com/Articles/33052/Visual-Representation-of-SQL-Joins
  • 19.
    Merge Example #!/usr/bin/env python importnumpy as np from pandas import DataFrame, Series import pandas as pd df1 = DataFrame({ 'key': [ 'orange', 'apple', 'bannana', 'banana', 'banana', 'apple', 'orange' ], 'data1' : range(7)}) df2 = DataFrame({ 'key': [ 'orange', 'apple', 'pear'], 'data2' : range(3) }) print 'df1' print '-------' print df1 print 'df2' print '-------' print df2 dfmerge = pd.merge(df1,df2, on='key') print 'dfmerge' print '-------' print dfmerge
  • 20.
  • 21.
    Merge Example: UniqueKeys #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd wayne = DataFrame({ 'year': [ 2012, 2013, 2014 ], 'wayne_births' : [23109, 23612, 23366] }) oakland = DataFrame({'year': [2012, 2013, 2014], 'oakland_births': [13307, 13445, 13454]}) macomb = DataFrame({'year': [2012, 2013, 2014], 'macomb_births': [9089, 9394, 9332]}) livingston = DataFrame({'year': [2012, 2013, 2014], 'livingston_births': [1739, 1738, 1813]}) dfmerge1 = pd.merge(wayne, oakland, on='year') dfmerge2 = pd.merge(dfmerge1, macomb, on='year') dfmerge = pd.merge(dfmerge2, livingston, on='year') print "wayne" print '-------' print wayne print "oakland" print '-------' print oakland print "macomb" print '-------' print macomb print "livingston" print '-------' print livingston
  • 22.
    Merge Example UniqueKeys Output wayne ­­­­­­­    wayne_births  year 0         23109  2012 1         23612  2013 2         23366  2014 oakland ­­­­­­­    oakland_births  year 0           13307  2012 1           13445  2013 2           13454  2014 <SNIP> dfmerge ­­­­­­­    wayne_births  year  oakland_births  macomb_births  livingston_births 0         23109  2012           13307           9089               1739 1         23612  2013           13445           9394               1738 2         23366  2014           13454           9332               1813
  • 23.
    Understanding GroupBy ● Split-Apply-Combine ●Split data into groups based on keys (Provider Name, Provider State, Procedure Name, ..) ● Apply A function is applied to each group (e.g. average, sum, count) ● Combine The results of the “apply” functions are combined to form a new object.
  • 24.
    Merge Example: AverageCovered Costs #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd def convert_acc(value): v2 = value.replace('$','') f = float(v2) return f def pandas_example2(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',', converters= { u' Average Covered Charges ': convert_acc } ) grouped = df[u' Average Covered Charges '].groupby([df[u'Provider Id'], df[u'Provider Name']]) means = grouped.mean() print means meansdf = DataFrame(means) print meansdf.sort(u' Average Covered Charges ') pass if __name__ == "__main__": pandas_example2()
  • 25.
    Average Covered Costs:Output                                                               Average Covered Charges  Provider Id Provider Name                                                              450813      COMMUNITY GENERAL HOSPITAL                                     2995.610000 250079      SHARKEY ISSAQUENA COMMUNITY HOSPITAL                           3369.955000 450746      KNOX COUNTY HOSPITAL                                           3677.000000 110209      TURNING POINT HOSPITAL                                         3720.430000 450270      LAKE WHITNEY MEDICAL CENTER                                    3906.842727 190161      W O MOSS REGIONAL MEDICAL CENTER                               4059.250000 390025      KENSINGTON HOSPITAL                                            4108.750000 220062      ADCARE HOSPITAL OF WORCESTER INC                               4227.460000 190208      EAST CARROLL PARISH HOSPITAL                                   4318.224444 <SNIP> 230279      BRIGHTON HOSPITAL                                              5285.000000 360247      WOODS AT PARKSIDE,THE                                          5384.680000 10097       ELMORE COMMUNITY HOSPITAL                                      5404.585556 ...                                                                                ... 50197       SEQUOIA HOSPITAL                                              99682.389216 50153       O'CONNOR HOSPITAL                                             99812.639589 50002       ST ROSE HOSPITAL                                             100844.518519 50380       GOOD SAMARITAN HOSPITAL                                      101206.971111 50742       OLYMPIA MEDICAL CENTER                                       102538.674091 <SNIP> 50367       NORTHBAY MEDICAL CENTER                                      138504.546230 50441       STANFORD HOSPITAL                                            138818.649770 50464       DOCTORS MEDICAL CENTER                                       144695.833286 310025      BAYONNE HOSPITAL CENTER                                      147441.334000 490142      UVA HEALTH SCIENCES CENTER                                   211922.000000 [3337 rows x 1 columns]
  • 26.
    All is notas it seems ... rich@tardis:~/Data Analysis Python Presentation/Inpatient$ grep 'BRIGHTON HOSPITAL' *.csv | more 897 ­ ALCOHOL/DRUG ABUSE OR DEPENDENCE W/O REHABILITATION THERAPY W/O MCC,230279 ,BRIGHTON HOSPITAL,12851 E GRAND RIVER,BRIGHTON,MI,48116,MI ­ Ann Arbor,15,$5285 .00,$3736.00,$2610.40 rich@computer:~/Data Analysis Python Presentation/Inpatient$ grep 'UVA ' *.csv   | more 207 ­ RESPIRATORY SYSTEM DIAGNOSIS W VENTILATOR SUPPORT 96+ HOURS,490142,UVA HEA LTH SCIENCES CENTER,2965 IVY RD,CHARLOTTESVILLE,VA,22908,VA ­ Charlottesville,18 ,$211922.00,$50552.61,$41836.88 rich@tardis:~/Data Analysis Python Presentation/Inpatient$ grep 'UNIVERSITY OF MICHIGAN' *.csv   | wc ­l | more 97
  • 27.
    Group by Stateis More Useful #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd def convert_acc(value): v2 = value.replace('$','') f = float(v2) return f def pandas_example4(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',', converters= { u' Average Covered Charges ': convert_acc } ) grouped = df.groupby(u'Provider State') get_weighted_average = lambda g: np.average(g[u' Average Covered Charges ']) applied = grouped.apply(get_weighted_average) print applied.sort_values() return if __name__ == "__main__": pandas_example4()
  • 28.
    Average Covered Costsper State Provider State MD    13377.803790 WV    19191.508634 VT    20074.958333 ME    20394.957568 MA    20534.006713 ND    21636.883460 MT    22670.015237 MI    24124.247210 IA    24168.742042 KY    24523.807169 UT    25092.806872 NC    25140.952162 ID    25565.547042 WI    26149.325332 AR    26174.526246 NH    27059.020802 DE    27071.699645 OR    27390.111871 MN    27894.361821 IN    28144.712545 OH    28344.218547 WY    28700.598623 VA    29222.000487 TN    29279.931835 OK    29587.575266 SD    29609.991543 RI    29942.701122 NM    30011.406499 MS    30292.785203 GA    31096.932842 MO    31184.622902 AL    31316.462074 CT    31318.410114 NY    31435.685543 KS    31580.253663 NE    31736.427825 HI    32174.748077 LA    33085.372792 WA    34714.234075 SC    35862.494563 IL    36061.849879 PA    39633.959763 DC    40116.663658 AK    40348.743333 CO    41095.136111 AZ    41200.063020 TX    41480.193404 FL    46016.233587 NV    61047.115416 NJ    66125.686274 CA    67508.616536 dtype: float64
  • 29.
    Some thoughts ● Wherecan I get inexpensive and quality medical care? Which states could benefit from promoting medical tourism? ● Which states have higher costs of living? Are procedures less expensive in low cost of living regions? ● Which states have higher average income? Do those states have a greater proportion of expensive procedure types? ● Which states have favorable or unfavorable regulatory environments? (LA Lottery)
  • 30.
    Bucket and QuantileAnalysis #!/usr/bin/env python import numpy as np from pandas import DataFrame, Series import pandas as pd grades = np.random.randint(100, size=33) print grades frame = DataFrame({'grades' : grades } ) factor = pd.cut(frame.grades, 4 ) print factor
  • 31.
    Bucket and QuantileOutput [34 37 19 79 90  7 58  5 77  9 88 18 10  0 89 16 58 59  0 89 27  5  6 71  3  10 48 73 21 13 10 84 28] 0        (22.5, 45] 1        (22.5, 45] 2     (­0.09, 22.5] 3        (67.5, 90] 4        (67.5, 90] <SNIP> 28    (­0.09, 22.5] 29    (­0.09, 22.5] 30    (­0.09, 22.5] 31       (67.5, 90] 32       (22.5, 45] Name: grades, dtype: category Categories (4, object): [(­0.09, 22.5] < (22.5, 45] < (45, 67.5] < (67.5, 90]]   
  • 32.
    Grouping by Procedure #!/usr/bin/envpython import numpy as np from pandas import DataFrame, Series import pandas as pd import matplotlib.pyplot as plt def convert_acc(value): v2 = value.replace('$','') f = float(v2) return f def pandas_example3(): # source: data.gov df = pd.read_table("/home/rich/Data Analysis Python Presentation/Inpatient/Inpatient_Prospective_Payment_System__IPPS__Provider_Summary _for_the_Top_100_Diagnosis-Related_Groups__DRG__-_FY2011.csv", sep=',', converters= { u' Average Covered Charges ': convert_acc } ) grouped = df.groupby(u'DRG Definition') get_weighted_average = lambda g: np.average(g[u' Average Covered Charges ']) applied = grouped.apply(get_weighted_average) print applied.sort_values() sampler = np.random.permutation(20) applied.take(sampler).sort_values().plot(kind='barh') return if __name__ == "__main__": pandas_example3()
  • 33.
    Group by ProcedureOutput DRG Definition 310 ­ CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W/O CC/MCC                     14933.904053 303 ­ ATHEROSCLEROSIS W/O MCC                                                  16640.698775 641 ­ MISC DISORDERS OF NUTRITION,METABOLISM,FLUIDS/ELECTROLYTES W/O MCC       16703.611059 293 ­ HEART FAILURE & SHOCK W/O CC/MCC                                         16864.201321 203 ­ BRONCHITIS & ASTHMA W/O CC/MCC                                           16885.022338 313 ­ CHEST PAIN                                                               16948.320991 684 ­ RENAL FAILURE W/O CC/MCC                                                 17027.827897 918 ­ POISONING & TOXIC EFFECTS OF DRUGS W/O MCC                               17138.345282 301 ­ PERIPHERAL VASCULAR DISORDERS W/O CC/MCC                                 17152.401881 390 ­ G.I. OBSTRUCTION W/O CC/MCC                                              17359.640556 536 ­ FRACTURES OF HIP & PELVIS W/O MCC                                        17482.136874 305 ­ HYPERTENSION W/O MCC                                                     17517.987290 <SNIP>                                                                                  ...       208 ­ RESPIRATORY SYSTEM DIAGNOSIS W VENTILATOR SUPPORT <96 HOURS              62394.721315 330 ­ MAJOR SMALL & LARGE BOWEL PROCEDURES W CC                                66112.110797 247 ­ PERC CARDIOVASC PROC W DRUG­ELUTING STENT W/O MCC                        66737.754098 243 ­ PERMANENT CARDIAC PACEMAKER IMPLANT W CC                                 67865.711226 253 ­ OTHER VASCULAR PROCEDURES W CC                                           70148.356417 480 ­ HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W MCC                          75339.614435 469 ­ MAJOR JOINT REPLACEMENT OR REATTACHMENT OF LOWER EXTREMITY W MCC         81704.226749 252 ­ OTHER VASCULAR PROCEDURES W MCC                                          83502.956898 238 ­ MAJOR CARDIOVASC PROCEDURES W/O MCC                                      85010.449213 460 ­ SPINAL FUSION EXCEPT CERVICAL W/O MCC                                    95568.276246 246 ­ PERC CARDIOVASC PROC W DRUG­ELUTING STENT W MCC OR 4+ VESSELS/STENTS     96348.806707 329 ­ MAJOR SMALL & LARGE BOWEL PROCEDURES W MCC                              135330.939966 853 ­ INFECTIOUS & PARASITIC DISEASES W O.R. PROCEDURE W MCC                  139186.350937 207 ­ RESPIRATORY SYSTEM DIAGNOSIS W VENTILATOR SUPPORT 96+ HOURS             143428.051066 870 ­ SEPTICEMIA OR SEVERE SEPSIS W MV 96+ HOURS                              163227.331725 dtype: float64
  • 34.
  • 35.
    Wakario.io: iPython Online ●Free with some limitations ● Can install iPython locally instead if you like ● Like iPython, makes something like an academic paper ● A little confusing, when you open and close you'll need to manually re-run prior values to set variables for later values ● Excellent way to try data analysis in Python
  • 36.
  • 37.
    Wakari.io: A littlebit fancier now
  • 38.
    Examining the Stocksof US Auto Manufacturers
  • 39.
    Examining the Stocksof US Auto Manufacturers
  • 40.
    Log of differencesare equivalent to stock returns
  • 41.
  • 42.
    Subplots: Help usFoggy Nelson!
  • 43.
    Daily gains orlosses per US auto manufacturer stock
  • 44.
    Should I justinvest in them all? Correlation coefficients and the ideal investment
  • 45.
    Conclusion: Tesla isnot an auto company
  • 46.
    Wrapping it up... ● Python has powerful data analysis tools ● Use them in Pycharm (or any IDE) or iPython (Vim also works) ● Analyzing data can help us make more informed decisions ● Libraries make most things easy ● Thank you!