How do I refer to the index of my Pandas dataframe?

I have a Pandas dataframe where I have designated some of the columns as indices: planets_dataframe.set_index([‘host’,’name’], inplace=True) and would like to be able to refer to these indices in a variety of contexts. Using the name of an index works fine in queries planets_dataframe.query(‘host == “PSR 1257 12″‘) but results in an error if try … Read more

pandas.concat of multiple data frames using only common columns

I have multiple pandas data frame objects cost1, cost2, cost3 …. They have different column names (and number of columns) but have some in common. Number of columns is fairly large in each data frame, hence handpicking the common columns manually will be painful. How can I append rows from all of these data frames … Read more

How to convert a case-class-based RDD into a DataFrame?

The Spark documentation shows how to create a DataFrame from an RDD, using Scala case classes to infer a schema. I am trying to reproduce this concept using sqlContext.createDataFrame(RDD, CaseClass), but my DataFrame ends up empty. Here’s my Scala code: // sc is the SparkContext, while sqlContext is the SQLContext. // Define the case class … Read more

Pandas, group by count and add count to original dataframe?

When trying to count rows with similar ‘kind’ in data frame: import pandas as pd items = [(‘aaa’,’aaa text 1′), (‘aaa’,’aaa text 2′), (‘aaa’,’aaa text 3′), (‘bb’, ‘bb text 1’), (‘bb’, ‘bb text 2’), (‘bb’, ‘bb text 3’), (‘bb’, ‘bb text 4’), (‘cccc’,’cccc text 1′), (‘cccc’,’cccc text 2′), (‘dd’, ‘dd text 1’), (‘e’, ‘e text … Read more

dataframe to dict such that one column is the key and the other is the value [duplicate]

This question already has answers here: How to create a dictionary of two pandas DataFrame columns (5 answers) Convert a Pandas DataFrame to a dictionary (11 answers) Closed 3 years ago. I have the dataframe ID A B C 0 p 1 3 2 1 q 4 3 2 2 r 4 0 9 And … Read more