Pandas
From charlesreid1
Installing
This was a royal pain in the ass.
I had some problems with my easy_install and pip, probably b/c they were not associated with the right versions of Python. I was able to use Virtualenv to debug some of the problems, and basically ended up having to remove my custom-installed version of numpy in favor of the pip-installed version of numpy. This broke my custom-installed version of scipy and ipython as well, so I had to install those using pip. These ended up not working, and I had to go back to my old, by-hand installations. So, in the end, I don't actually know what the right procedure is; I just have a vague sense that there were some problems, that got resolved, by something I did, at some point.
Like I said, it was a big damn mess.
First, I downloaded and installed easy_install from source.
Then, I ran the following commands:
$ sudo easy_install pip $ sudo pip install numpy $ sudo pip install pandas
and installing tables stuff:
$ sudo pip install numexpr $ sudo pip install tables
Data
Creating a Table of Arbitrary Data Types
Let's say you're trying to create a data table where you store the result of a simulation. This simulation has a set of inputs and outputs, each with a different data type. For example, the following inputs are scalars:
- Flowrate_in (float)
- Temperature_in (float)
- Pressure_in (float)
But temperature and species profiles are vectors, not scalars:
- Temperature_profile (numpy array)
- Oxygen_profile (numpy array)
Two ways of populating a Pandas data object (a DataFrame, in this case) are:
- Create arbitrary, concrete data with the type you are interested in storing
- Grab the types of the data you are interested in storing
Initializing with Data
A simple illustration of the first technique:
In[99]: reactors = [ { "flowrate_in" : 0.0, "temperature_in" : 0.0, "pressure_in" : 0.0, "temperature_profile" : zeros(100,), "oxygen_profile" : zeros(100,) } for i in arange(10) ]
This creates a list of 10 dicts containing the same initial values, which can then be used to initialize a DataFrame object:
In[100]: pandas.DataFrame(reactors)
Out[100]:
flowrate_in oxygen_profile pressure_in temperature_in \
0 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
1 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
2 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
3 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
4 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
5 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
6 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
7 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
8 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
9 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
temperature_profile
0 [0.0, 0.0, 0.0, 0.0, 0.0]
1 [0.0, 0.0, 0.0, 0.0, 0.0]
2 [0.0, 0.0, 0.0, 0.0, 0.0]
3 [0.0, 0.0, 0.0, 0.0, 0.0]
4 [0.0, 0.0, 0.0, 0.0, 0.0]
5 [0.0, 0.0, 0.0, 0.0, 0.0]
6 [0.0, 0.0, 0.0, 0.0, 0.0]
7 [0.0, 0.0, 0.0, 0.0, 0.0]
8 [0.0, 0.0, 0.0, 0.0, 0.0]
9 [0.0, 0.0, 0.0, 0.0, 0.0]
Initializing with Types
A simple illustration of the second technique:
In[101]: df = reactors = [ { "flowrate_in" : numpy.float32, "temperature_in" : numpy.float32, "pressure_in" : numpy.float32, "temperature_profile" : numpy.ndarray, "oxygen_profile" : numpy.ndarray } for i in range(10) ]
This creates a list of 10 dicts that are all empty:
In[102]: df = pandas.DataFrame(reactors)
Out[102]:
flowrate_in oxygen_profile pressure_in \
0 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
1 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
2 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
3 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
4 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
5 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
6 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
7 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
8 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
9 <type 'numpy.float32'> <type 'numpy.ndarray'> <type 'numpy.float32'>
temperature_in temperature_profile
0 <type 'numpy.float32'> <type 'numpy.ndarray'>
1 <type 'numpy.float32'> <type 'numpy.ndarray'>
2 <type 'numpy.float32'> <type 'numpy.ndarray'>
3 <type 'numpy.float32'> <type 'numpy.ndarray'>
4 <type 'numpy.float32'> <type 'numpy.ndarray'>
5 <type 'numpy.float32'> <type 'numpy.ndarray'>
6 <type 'numpy.float32'> <type 'numpy.ndarray'>
7 <type 'numpy.float32'> <type 'numpy.ndarray'>
8 <type 'numpy.float32'> <type 'numpy.ndarray'>
9 <type 'numpy.float32'> <type 'numpy.ndarray'>
Modifying a Table with Data
When you treat data as a 2D array of arbitrary data types, each of those numpy.ndarray objects can be whatever size it wants - all that Pandas cares about is the fact that it is a numpy array. Beyond that, Pandas doesn't care about the shape or size of the array.
This means that, in practice, you could have temperature or oxygen profiles of entirely different sizes:
In [117]: df
Out[117]:
flowrate_in oxygen_profile pressure_in temperature_in \
0 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
1 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
2 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
3 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
4 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
5 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
6 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
7 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
8 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
9 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
temperature_profile
0 [0.0, 0.0, 0.0, 0.0, 0.0]
1 [0.0, 0.0, 0.0, 0.0, 0.0]
2 [0.0, 0.0, 0.0, 0.0, 0.0]
3 [0.0, 0.0, 0.0, 0.0, 0.0]
4 [0.0, 0.0, 0.0, 0.0, 0.0]
5 [0.0, 0.0, 0.0, 0.0, 0.0]
6 [0.0, 0.0, 0.0, 0.0, 0.0]
7 [0.0, 0.0, 0.0, 0.0, 0.0]
8 [0.0, 0.0, 0.0, 0.0, 0.0]
9 [0.0, 0.0, 0.0, 0.0, 0.0]
Now set the temperature profiles to be profiles of different lengths:
In [122]: df['temperature_profile'][0] = 25*ones(3,)
In [123]: df['temperature_profile'][1] = 28*ones(5,)
In [124]: df['temperature_profile'][2] = 30*ones(8,)
In [125]: df
Out[125]:
flowrate_in oxygen_profile pressure_in temperature_in \
0 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
1 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
2 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
3 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
4 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
5 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
6 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
7 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
8 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
9 0 [0.0, 0.0, 0.0, 0.0, 0.0] 0 0
temperature_profile
0 [25.0, 25.0, 25.0]
1 [28.0, 28.0, 28.0, 28.0, 28.0]
2 [30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0]
3 [0.0, 0.0, 0.0, 0.0, 0.0]
4 [0.0, 0.0, 0.0, 0.0, 0.0]
5 [0.0, 0.0, 0.0, 0.0, 0.0]
6 [0.0, 0.0, 0.0, 0.0, 0.0]
7 [0.0, 0.0, 0.0, 0.0, 0.0]
8 [0.0, 0.0, 0.0, 0.0, 0.0]
9 [0.0, 0.0, 0.0, 0.0, 0.0]