Newbe Dev Stack

Answer : The seaborn equivalent of df.boxplot() is sns.boxplot(x="variable", y="value", data=pd.melt(df)) Complete example: import numpy as np; np.random.seed(42) import pandas as pd import matplotlib.pyplot as plt import seaborn as sns df = pd.DataFrame(data = np.random.random(size=(4,4)), columns = ['A','B','C','D']) sns.boxplot(x="variable", y="value", data=pd.melt(df)) plt.show() This works because pd.melt converts a wide-form dataframe A B C D 0 0.374540 0.950714 0.731994 0.598658 1 0.156019 0.155995 0.058084 0.866176 2 0.601115 0.708073 0.020584 0.969910 3 0.832443 0.212339 0.181825 0.183405 to long-form variable value 0 A 0.374540 1 A 0.156019 2 A 0.601115 3 A 0.832443 4 B 0.950714 5 B 0.155995 6 B 0.708073 7 B 0.212339 8 C 0.731994 9 C ...

Answer : copying data from your link and running df = pd.read_clipboard() then using your code df = df.replace(np.nan,0) df = df.groupby(['home_team'])['arrests'].mean() df.plot.bar() Good one by @piRSuared, and I just buitified his answer :) ## referenced to the answer by @piRSquared df = df.replace(np.nan,0) df = df.groupby(['home_team'])['arrests'].mean() ax = df.plot(kind='bar', figsize=(10,6), color="indigo", fontsize=13); ax.set_alpha(0.8) ax.set_title("My Bar Plot", fontsize=22) ax.set_ylabel("Some Heading on Y-Axis", fontsize=15); plt.show()

Answer : A helper function for appending DataFrame to existing Excel file: def append_df_to_excel(filename, df, sheet_name='Sheet1', startrow=None, truncate_sheet=False, **to_excel_kwargs): """ Append a DataFrame [df] to existing Excel file [filename] into [sheet_name] Sheet. If [filename] doesn't exist, then this function will create it. Parameters: filename : File path or existing ExcelWriter (Example: '/path/to/file.xlsx') df : dataframe to save to workbook sheet_name : Name of sheet which will contain DataFrame. (default: 'Sheet1') startrow : upper left cell row to dump data frame. Per default (startrow=None) calculate the last row in the existing DF and write to the next row... truncate_sheet : truncate (remove and recreate) [sheet_name] before ...

Answer : I think you can use concat : print pd.concat([t1, t2, t3, t4, t5]) Maybe you can ignore_index : print pd.concat([t1, t2, t3, t4, t5], ignore_index=True) More info in docs. Have you simply tried using a list as argument of append? Or am I missing anything? import numpy as np import pandas as pd dates = np.asarray(pd.date_range('1/1/2000', periods=8)) df1 = pd.DataFrame(np.random.randn(8, 4), index=dates, columns=['A', 'B', 'C', 'D']) df2 = df1.copy() df3 = df1.copy() df = df1.append([df2, df3]) print df

Answer : Try this: import pandas as pd import numpy as np from pandas import * >>> L = [4, nan ,6] >>> df = Series(L) >>> df 0 4 1 NaN 2 6 >>> if(pd.isnull(df[1])): print "Found" Found >>> if(np.isnan(df[1])): print "Found" Found STEP 1.) df[df.isnull().any(1)] ----> Will give you dataframe with rows and column, if any value there is nan. STEP 2.) this will give you location in dataframe where exactly value is nan. then you could do if(**df.iloc[loc_row,loc_colum]==np.nan**): print"your code here" You can use "isnull" with "at" to check a specific value in a dataframe. For example: import pandas as pd import numpy as np df = pd.DataFrame([[np.nan, 2], [1, 3], [4, 6]], columns=['A', 'B']) Yeilds: A B 0 NaN 2 1 1.0 3 2 4.0 6 To check the values: pd.isnull(df.at[0,'A']) -> True pd.isnu...

Answer : pandas.HDFStore.put() has parameter append (which defaults to False ) - that instructs Pandas to overwrite instead of appending. So try this: store = pd.HDFStore('test.h5') store.append('name_of_frame', ohlcv_candle, format='t', data_columns=True) we can also use store.put(..., append=True) , but this file should also be created in a table format: store.put('name_of_frame', ohlcv_candle, format='t', append=True, data_columns=True) NOTE: appending works only for the table ( format='t' - is an alias for format='table' ) format. tohlcv_candle.to_hdf('test.h5',key='this_is_a_key', append=True, mode='r+', format='t') You need to pass another argument append=True to specify that the data is to be appended to existing data if found under that key, instead of over-writing it. Without this, the default is False and if it encounters an existing table under 'this_is_a_key...

Answer : TL;DR I believe you're seriously underestimating memory requirements. Even assuming that data is fully cached, storage info will show only a fraction of peak memory required for bringing data back to the driver. First of all Spark SQL uses compressed columnar storage for caching. Depending on the data distribution and compression algorithm in-memory size can be much smaller than the uncompressed Pandas output, not to mention plain List[Row] . The latter also stores column names, further increasing memory usage. Data collection is indirect, with data being stored both on the JVM side and Python side. While JVM memory can be released once data goes through socket, peak memory usage should account for both. Plain toPandas implementation collects Rows first, then creates Pandas DataFrame locally. This further increases (possibly doubles) memory usage. Luckily this part is already addressed on master (Spark 2.3), with more direct approach using Arrow serialization...

Answer : Let's try: import pandas as pd import numpy as np import networkx as nx import matplotlib.pyplot as plt df = pd.DataFrame({'number':['123','234','345'],'contactnumber':['234','345','123'],'callduration':[1,2,4]}) df G = nx.from_pandas_edgelist(df,'number','contactnumber', edge_attr='callduration') durations = [i['callduration'] for i in dict(G.edges).values()] labels = [i for i in dict(G.nodes).keys()] labels = {i:i for i in dict(G.nodes).keys()} fig, ax = plt.subplots(figsize=(12,5)) pos = nx.spring_layout(G) nx.draw_networkx_nodes(G, pos, ax = ax, labels=True) nx.draw_networkx_edges(G, pos, width=durations, ax=ax) _ = nx.draw_networkx_labels(G, pos, labels, ax=ax) Output: Do not agree with what has been said. In the calcul of different metrics that takes into account the weight of each edges like the pagerank or the betweeness centrality your weight would...

Answer : With newer Pandas versions, if someone is interested, here a slightly different solution with Pandas only: ax = data.plot(kind='hist',subplots=True,sharex=True,sharey=True,title='My title') You can use suptitle() : import pylab as pl from pandas import * data = DataFrame(np.random.randn(500).reshape(100,5), columns=list('abcde')) axes = data.hist(sharey=True, sharex=True) pl.suptitle("This is Figure title") I found a better way: plt.subplot(2,3,1) # if use subplot df = pd.read_csv('documents',low_memory=False) df['column'].hist() plt.title('your title') It is very easy, display well at the top, and will not mess up your subplot.

Answer : You can create additional dictionary with mapping: from sklearn import preprocessing le = preprocessing.LabelEncoder() le.fit(data['name']) le_name_mapping = dict(zip(le.classes_, le.transform(le.classes_))) print(le_name_mapping) {'Tom': 0, 'Nick': 1, 'Kate': 2} The best way of doing this can be to use label encoder of sklearn library. Something like this: from sklearn import preprocessing le = preprocessing.LabelEncoder() le.fit(["paris", "paris", "tokyo", "amsterdam"]) list(le.classes_) le.transform(["tokyo", "tokyo", "paris"]) list(le.inverse_transform([2, 2, 1])) A simple & elegant way to do the same. cat_list = ['Sun', 'Sun', 'Wed', 'Mon', 'Mon'] encoded_data, mapping_index = pd.Series(cat_list).factorize() and you are done , check below print(encoded_data) print(mapping_index) print(mapping_index.get_loc(...

Answer : Add a new pandas.Series using pandas.DataFrame.append(). If you wish to specify the name (AKA the "index") of the new row, use: df.append(pandas.Series(name='NameOfNewRow')) If you don't wish to name the new row, use: df.append(pandas.Series(), ignore_index=True) where df is your pandas.DataFrame. You can add it by appending a Series to the dataframe as follows. I am assuming by blank you mean you want to add a row containing only "Nan". You can first create a Series object with Nan. Make sure you specify the columns while defining 'Series' object in the -Index parameter. The you can append it to the DF. Hope it helps! from numpy import nan as Nan import pandas as pd >>> df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'], ... 'B': ['B0', 'B1', 'B2', 'B3'], ... 'C': ['C0', 'C...

Answer : Use groupby and agg , and aggregate only unique values by calling Series.unique : df.astype(str).groupby('prop1').agg(lambda x: ','.join(x.unique())) prop2 prop3 prop4 prop1 K20 12,1,66 travis,leo 10.0,4.0 L30 3,54,11,10 bob,john 11.2,10.0 df.astype(str).groupby('prop1', sort=False).agg(lambda x: ','.join(x.unique())) prop2 prop3 prop4 prop1 L30 3,54,11,10 bob,john 11.2,10.0 K20 12,1,66 travis,leo 10.0,4.0 If handling NaNs is important, call fillna in advance: import re df.fillna('').astype(str).groupby('prop1').agg( lambda x: re.sub(',+', ',', ','.join(x.unique())) ) prop2 prop3 prop4 prop1 K20 12,1,66 travis,leo 10.0,4.0 L30 3,54,11,10 bob,john 11.2,10.0

Answer : Pandas timestamp differences returns a datetime.timedelta object. This can easily be converted into hours by using the *as_type* method, like so import pandas df = pandas.DataFrame(columns=['to','fr','ans']) df.to = [pandas.Timestamp('2014-01-24 13:03:12.050000'), pandas.Timestamp('2014-01-27 11:57:18.240000'), pandas.Timestamp('2014-01-23 10:07:47.660000')] df.fr = [pandas.Timestamp('2014-01-26 23:41:21.870000'), pandas.Timestamp('2014-01-27 15:38:22.540000'), pandas.Timestamp('2014-01-23 18:50:41.420000')] (df.fr-df.to).astype('timedelta64[h]') to yield, 0 58 1 3 2 8 dtype: float64 This was driving me bonkers as the .astype() solution above didn't work for me. But I found another way. Haven't timed it or anything, but might work for others out there: t1 = pd.to_datetime('1/1/2015 01:00') t2 = pd.to_datetime('1/1/2015 03:30') print pd.Timedelta(t2...

Answer : The other answers are quite technical and hard to understand. I was in your position before so I'll explain in simple terms . Airflow can do anything . It has BashOperator and PythonOperator which means it can run any bash script or any Python script. It is a way to organize (setup complicated data pipeline DAGs), schedule, monitor, trigger re-runs of data pipelines, in a easy-to-view and use UI. Also, it is easy to setup and everything is in familiar Python code. Doing pipelines in an organized manner (i.e using Airflow) means you don't waste time debugging a mess of data processing ( cron ) scripts all over the place. Apache Beam is a wrapper for the many data processing frameworks (Spark, Flink etc.) out there. The intent is so you just learn Beam and can run on multiple backends (Beam runners). If you are familiar with Keras and TensorFlow/Theano/Torch, the relationship between Keras and its backends is similar to the relationship between Beam and its...