3 All in one chapter

Reading documentation is no fun. Moreover the AstroAsciiData module promised to be convenient for users (see Sect. 1.2). This Section gives a fast introduction on all features of the AstroAsciiData module and how these features are used to work with ASCII tables. On the basis of some sample session the most important classes and methods are introduced without explicitly listing all their names and modes of usage. A complete and detailed overview on all AstroAsciiData classes and their methods is given in Section 4.

3.1 Working with existing data

This chapter shows how to load and work with the ASCII table 'example.txt'. This tables looks like:

#
# Some objects in the GOODS field
#
unknown 189.2207323  62.2357983 26.87 0.32
galaxy  189.1408929  62.2376331 24.97 0.15
star    189.1409453  62.1696844 25.30 0.12
galaxy  188.9014716  62.2037839 25.95 0.20

$\Rightarrow$

Before actually loading the table, the AstroAsciiData module must be imported with:

>>> import asciidata

$\Rightarrow$

The ASCII table is loaded with:

>>> example = asciidata.open('example.txt')

$\Rightarrow$

Just to check whether the table was loaded correctly you do:

>>> print str(example)
#
# Some objects in the GOODS field
#
unknown  189.2207323  62.2357983  26.87  0.32
 galaxy  189.1408929  62.2376331  24.97  0.15
   star  189.1409453  62.1696844  25.30  0.12
 galaxy  188.9014716  62.2037839  25.95  0.20

$\Rightarrow$

As a first application, you want to compute the average from the numbers in the second and third row:

>>> sum1=0.0
>>> sum2=0.0
>>> for index in range(example.nrows):
...     sum1 += example[1][index]
...     sum2 += example[2][index]
... 
>>> ave1 = sum1/example.nrows
>>> ave2 = sum2/example.nrows
>>> print ave1, ave2
189.101010525 62.211724925

Please note that indices start with 0, so the first row in the first column is example[0][0].

$\Rightarrow$

You want to change the table values, but before that perhaps it would be wise to keep a copy of the original ASCII table :

>>> example.writeto('example_orig.txt')

This gives you a file 'example_orig.txt', which is identical to the original 'example.txt'.

$\Rightarrow$

Now you may want to compute and save the differences between the average and the individual values:

>>> for index in range(example.nrows):
...     example['diff1'][index] = example[1][index] - ave1
...     example['diff2'][index] = example[2][index] - ave2
... 
>>> print str(example)
#
# Some objects in the GOODS field
#
unknown  189.2207323  62.2357983  26.87  0.32  1.197218e-01  2.407338e-02
 galaxy  189.1408929  62.2376331  24.97  0.15  3.988237e-02  2.590817e-02
   star  189.1409453  62.1696844  25.30  0.12  3.993477e-02 -4.204052e-02
 galaxy  188.9014716  62.2037839  25.95  0.20 -1.995389e-01 -7.941025e-03

There are two new columns, which were created by addressing elements in an unknown column with the name 'diff1' and 'diff2'.

$\Rightarrow$

To remember the new columns and their meaning, you would like to put a note into the table header:

>>> example.header.append('Nov 16 2005: computed and stored differences!')
>>> print str(example)
#
# Some objects in the GOODS field
#
# Nov 16 2005: computed and stored differences!
unknown  189.2207323  62.2357983  26.87  0.32  1.197218e-01  2.407338e-02
 galaxy  189.1408929  62.2376331  24.97  0.15  3.988237e-02  2.590817e-02
   star  189.1409453  62.1696844  25.30  0.12  3.993477e-02 -4.204052e-02
 galaxy  188.9014716  62.2037839  25.95  0.20 -1.995389e-01 -7.941025e-03

There is a new commented line at the beginning of the table with your note.

$\Rightarrow$

That was enough for now, and the best is to save the modified ASCII table:

>>> example.flush()

Now the file 'example.txt' also has the two new columns.

$\Rightarrow$

OK, there is a column with the name 'diff1' and another named 'diff2', but what are the names of the original columns? To get all information, just type:

>>> print example.info()
File:       example.txt
Ncols:      7
Nrows:      4
Delimiter:  None
Null value: ['Null', 'NULL', 'None', '*']
Comment:    #
Column name:        column1
Column type:        <type 'str'>
Column format:      ['% 7s', '%7s']
Column null value : ['Null']
Column name:        column2
Column type:        <type 'float'>
Column format:      ['% 11.7f', '%12s']
Column null value : ['Null']
Column name:        column3
Column type:        <type 'float'>
Column format:      ['% 10.7f', '%11s']
Column null value : ['Null']
Column name:        column4
Column type:        <type 'float'>
Column format:      ['% 5.2f', '%6s']
Column null value : ['Null']
Column name:        column5
Column type:        <type 'float'>
Column format:      ['% 4.2f', '%5s']
Column null value : ['Null']
Column name:        diff1
Column type:        <type 'float'>
Column format:      ['% 12.6e', '%13s']
Column null value : ['Null']
Column name:        diff2
Column type:        <type 'float'>
Column format:      ['% 12.6e', '%13s']
Column null value : ['Null']

So the original columns had default names such as 'column1', 'column2', ... Moreover the method info() returns the column type and format for every column .

3.2 Creating an ASCII table from scratch

In this Section an ASCII table is created from scratch using functions classes and methods in the AstroAsciiData module.

$\Rightarrow$

To create an empty AsciiData object, import the AstroAsciiData module and type:

>>> import asciidata
>>> example2 = asciidata.create(4,10)
>>> print example2
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null
      Null       Null       Null       Null

The object has four columns with ten rows, all empty so far.

$\Rightarrow$

The first column should contain an index:

>>> for index in range(example2.nrows):
...     example2[0][index] = index+1
... 
>>> print example2
    1       Null       Null       Null
    2       Null       Null       Null
    3       Null       Null       Null
    4       Null       Null       Null
    5       Null       Null       Null
    6       Null       Null       Null
    7       Null       Null       Null
    8       Null       Null       Null
    9       Null       Null       Null
   10       Null       Null       Null

For convenience the index starts with 1.

$\Rightarrow$

Now the rest is filled using a functional form:

>>> for cindex in range(1, example2.ncols):
...   for rindex in range(example2.nrows):
...     example2[cindex][rindex] = 0.3*float(example2[0][rindex])**cindex
... 
>>> print example2
    1  3.000000e-01  3.000000e-01  3.000000e-01
    2  6.000000e-01  1.200000e+00  2.400000e+00
    3  9.000000e-01  2.700000e+00  8.100000e+00
    4  1.200000e+00  4.800000e+00  1.920000e+01
    5  1.500000e+00  7.500000e+00  3.750000e+01
    6  1.800000e+00  1.080000e+01  6.480000e+01
    7  2.100000e+00  1.470000e+01  1.029000e+02
    8  2.400000e+00  1.920000e+01  1.536000e+02
    9  2.700000e+00  2.430000e+01  2.187000e+02
   10  3.000000e+00  3.000000e+01  3.000000e+02

Please note that the column type of the first column is int since only integer type data was entered. In all other columns numbers of type float were entered, hence their column type is also float.

$\Rightarrow$

Inserting one element with a more generic data type changes the type of the whole column:

>>> example2[0][1] = 2.0
>>> print example2
 1.000000e+00  3.000000e-01  3.000000e-01  3.000000e-01
 2.000000e+00  6.000000e-01  1.200000e+00  2.400000e+00
 3.000000e+00  9.000000e-01  2.700000e+00  8.100000e+00
 4.000000e+00  1.200000e+00  4.800000e+00  1.920000e+01
 5.000000e+00  1.500000e+00  7.500000e+00  3.750000e+01
 6.000000e+00  1.800000e+00  1.080000e+01  6.480000e+01
 7.000000e+00  2.100000e+00  1.470000e+01  1.029000e+02
 8.000000e+00  2.400000e+00  1.920000e+01  1.536000e+02
 9.000000e+00  2.700000e+00  2.430000e+01  2.187000e+02
 1.000000e+01  3.000000e+00  3.000000e+01  3.000000e+02

Now the first column is of type float as well.

$\Rightarrow$

Eventually, some comments are added for specific rows:

>>> newcol = example2.append('comment')
>>> example2[newcol][0] =  'small!'            
>>> example2[newcol][2] =  'bigger!'
>>> example2[newcol][7] =  'Huge!'
>>> print example2
 1.000000e+00  3.000000e-01  3.000000e-01  3.000000e-01 small!
 2.000000e+00  6.000000e-01  1.200000e+00  2.400000e+00   Null
 3.000000e+00  9.000000e-01  2.700000e+00  8.100000e+00 bigger!
 4.000000e+00  1.200000e+00  4.800000e+00  1.920000e+01   Null
 5.000000e+00  1.500000e+00  7.500000e+00  3.750000e+01   Null
 6.000000e+00  1.800000e+00  1.080000e+01  6.480000e+01   Null
 7.000000e+00  2.100000e+00  1.470000e+01  1.029000e+02   Null
 8.000000e+00  2.400000e+00  1.920000e+01  1.536000e+02  Huge!
 9.000000e+00  2.700000e+00  2.430000e+01  2.187000e+02   Null
 1.000000e+01  3.000000e+00  3.000000e+01  3.000000e+02   Null

Obviously it also is possible to create a new column using the AsciiData append() method. This method returns the column number, which then can be used to fill the new column.

$\Rightarrow$

Now its time to safe the AsciiData object:

>>> example2.flush()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File ".../site-packages/asciidata/asciidata.py", line 279, in flush
    raise "No filename given. Use 'writeto()' instead."
No filename given. Use 'writeto()' instead.
>>> example2.writeto('example2.txt')

Since the object was created from scratch, there is no filename associated with and the method flush() can not be used. The method writeto() must be used instead!

3.3 Woorking with SExtractor formatted data

This chapter shows how to load and work with the table 'SExample.txt', which was produced by SExtractor. This object catalogue looks like:

#   1 NUMBER          Running object number
#   2 MAG_APER        Fixed aperture magnitude vector                 [mag]
#   5 MAGERR_APER     RMS error vector for fixed aperture mag.        [mag]
#   8 FLUX_AUTO       Flux within a Kron-like elliptical aperture     [count]
#   9 FLUXERR_AUTO    RMS error for AUTO flux                         [count]
#  10 X_IMAGE         Object position along x                         [pixel]
#  11 Y_IMAGE         Object position along y                         [pixel]
#  12 FLAGS           Extraction flags
 1  -7.1135  -9.9589 -11.4873 0.1151 0.0168 0.0082 52533.1 580.708 379.715  72.461 3
 2  -7.8412  -9.5452 -10.8191 0.0591 0.0246 0.0152  171543 2014.45 341.365 320.621 9
 3  -8.1548  -9.6216 -11.0307 0.0444 0.0229 0.0125  267764 1844.97 379.148 196.397 3
 4  -9.4534 -11.0534 -11.9600 0.0134 0.0061 0.0053  178541 1290.41 367.213 123.803 3
 5  -7.8104  -9.1967 -10.5169 0.0609 0.0339 0.0201  131343 1648.34 305.545 307.027 3
 6 -12.1666 -13.2193 -14.1293 0.0011 0.0008 0.0007 2.22738 1938.01 258.692 260.341 3
 7  -8.8319 -10.3340 -11.3343 0.0238 0.0119 0.0095  111597 1525.78 336.462  97.060 3
 8  -8.9203 -10.3532 -11.7252 0.0219 0.0117 0.0066  129934 917.641 177.377 199.843 3
 9  -7.5366  -9.0374 -10.3321 0.0784 0.0393 0.0238 72761.7 1603.15  94.196 131.380 3
10  -6.7963  -8.4304  -9.5685 0.1552 0.0687 0.0482   14072 895.465 265.404  46.241 3

Please note the the jump in the column numbers from #2 MAG_APER to #5 MAGERR_APER and then #8 FLUX_AUTO. MAG_APER and MAGERR_APER are both vector data with three items each. There are three colmns with MAG_APER-values and three columns with MAGERR_APER-values in the table data, however the header contains only one explicit entry for MAG_APER and MAGERR_APER.

In AstroAsciiData an individual column name is given to each of these multiple columns by adding a number to the basic name given in the header.

$\Rightarrow$

First you load the AstroAsciiData module and the table:

>>> import asciidata
>>> SExample = asciidata.open('SExample.txt')

$\Rightarrow$

Now you check the columns in the AsciiData object

>>> print SExample.info()
File:       SExample.cat
Ncols:      12
Nrows:      10
Delimiter:  None
Null value: ['Null', 'NULL', 'None', '*']
comment_char:    #
Column name:        NUMBER
Column type:        <type 'int'>
Column format:      ['%5i', '%5s']
Column null value : ['Null']
Column comment : Running object number
Column name:        MAG_APER
Column type:        <type 'float'>
Column format:      ['% 6.4f', '%7s']
Column null value : ['Null']
Column unit : mag
Column comment : Fixed aperture magnitude vector
Column name:        MAG_APER1
Column type:        <type 'float'>
Column format:      ['% 6.4f', '%7s']
Column null value : ['Null']
Column name:        MAG_APER2
Column type:        <type 'float'>
Column format:      ['% 7.4f', '%8s']
Column null value : ['Null']
Column name:        MAGERR_APER
Column type:        <type 'float'>
Column format:      ['% 6.4f', '%7s']
Column null value : ['Null']
Column unit : mag
Column comment : RMS error vector for fixed aperture mag.
Column name:        MAGERR_APER1
Column type:        <type 'float'>
Column format:      ['% 6.4f', '%7s']
Column null value : ['Null']
Column name:        MAGERR_APER2
Column type:        <type 'float'>
Column format:      ['% 6.4f', '%7s']
Column null value : ['Null']
Column name:        FLUX_AUTO
Column type:        <type 'float'>
Column format:      ['% 7.1f', '%8s']
Column null value : ['Null']
Column unit : count
Column comment : Flux within a Kron-like elliptical aperture
Column name:        FLUXERR_AUTO
Column type:        <type 'float'>
Column format:      ['% 7.3f', '%8s']
Column null value : ['Null']
Column unit : count
Column comment : RMS error for AUTO flux
Column name:        X_IMAGE
Column type:        <type 'float'>
Column format:      ['% 7.3f', '%8s']
Column null value : ['Null']
Column unit : pixel
Column comment : Object position along x
Column name:        Y_IMAGE
Column type:        <type 'float'>
Column format:      ['% 6.3f', '%7s']
Column null value : ['Null']
Column unit : pixel
Column comment : Object position along y
Column name:        FLAGS
Column type:        <type 'int'>
Column format:      ['%5i', '%5s']
Column null value : ['Null']
Column comment : Extraction flags

In this list there are the expanded column names MAG_APER, MAG_APER1 and MAG_APER2, and now every data column has a proper name.

$\Rightarrow$

You compute the signa-to-noise-ratio, set the column comment and check it:

>>> for ii in range(SExample.nrows):
...     SExample['SNR'][ii]=SExample['FLUX_AUTO'][ii]/SExample['FLUXERR_AUTO'][ii]
...
>>> SExample['SNR'].set_colcomment('Singal-to-Noise-Ratio')
>>> print SExample['SNR'].info()
Column name:        SNR
Column type:        <type 'float'>
Column format:      ['% 12.6e', '%13s']
Column null value : ['Null']
Column comment : Singal-to-Noise-Ratio

$\Rightarrow$

The object is sorted according to the signal-to-noise-ratio, the result is checked and the AsciiData object re-written to the disk:

>>> SExample.sort('SNR', descending=1)
>>> print SExample['SNR']
Column: SNR
 1.451319e+02
 1.415957e+02
 1.383599e+02
 9.046388e+01
 8.515625e+01
 7.968198e+01
 7.314095e+01
 4.538671e+01
 1.571474e+01
 1.149313e-03
>>> SExample.flush()

$\Rightarrow$

Your favoured plotting program can not deal with any kind of header. You tranfer the AsciiData object to plain format and write it to a special plotting file 'SExample.plot':

>>> SExample.toplain()
>>> SExample.writeto('SExample.plot')
>>>

$\Rightarrow$

You are finshed now, leave python and, since you do not trust the AstroAsciiData module, check both files:

~>more SExample.cat
# 1  NUMBER  Running object number
# 2  MAG_APER  Fixed aperture magnitude vector  [mag]
# 3  MAG_APER1
# 4  MAG_APER2
# 5  MAGERR_APER  RMS error vector for fixed aperture mag.  [mag]
# 6  MAGERR_APER1
# 7  MAGERR_APER2
# 8  FLUX_AUTO  Flux within a Kron-like elliptical aperture  [count]
# 9  FLUXERR_AUTO  RMS error for AUTO flux  [count]
# 10 X_IMAGE  Object position along x  [pixel]
# 11 Y_IMAGE  Object position along y  [pixel]
# 12 FLAGS  Extraction flags
# 13 SNR  Singal-to-Noise-Ratio
    3 -8.1548 -9.6216 -11.0307  0.0444  0.0229  0.0125  267764.0  1844.970  379.148  196.397     3  1.451319e+02
    8 -8.9203 -10.3532 -11.7252  0.0219  0.0117  0.0066  129934.0  917.641  177.377  199.843     3  1.415957e+02
    4 -9.4534 -11.0534 -11.9600  0.0134  0.0061  0.0053  178541.0  1290.410  367.213  123.803     3  1.383599e+02
    1 -7.1135 -9.9589 -11.4873  0.1151  0.0168  0.0082  52533.1  580.708  379.715  72.461     3  9.046388e+01
    2 -7.8412 -9.5452 -10.8191  0.0591  0.0246  0.0152  171543.0  2014.450  341.365  320.621     9  8.515625e+01
    5 -7.8104 -9.1967 -10.5169  0.0609  0.0339  0.0201  131343.0  1648.340  305.545  307.027     3  7.968198e+01
    7 -8.8319 -10.3340 -11.3343  0.0238  0.0119  0.0095  111597.0  1525.780  336.462  97.060     3  7.314095e+01
    9 -7.5366 -9.0374 -10.3321  0.0784  0.0393  0.0238  72761.7  1603.150  94.196  131.380     3  4.538671e+01
   10 -6.7963 -8.4304 -9.5685  0.1552  0.0687  0.0482  14072.0  895.465  265.404  46.241     3  1.571474e+01
    6 -12.1666 -13.2193 -14.1293  0.0011  0.0008  0.0007     2.2  1938.010  258.692  260.341     3  1.149313e-03
~>
~>more SExample.plot
    3 -8.1548 -9.6216 -11.0307  0.0444  0.0229  0.0125  267764.0  1844.970  379.148  196.397     3  1.451319e+02
    8 -8.9203 -10.3532 -11.7252  0.0219  0.0117  0.0066  129934.0  917.641  177.377  199.843     3  1.415957e+02
    4 -9.4534 -11.0534 -11.9600  0.0134  0.0061  0.0053  178541.0  1290.410  367.213  123.803     3  1.383599e+02
    1 -7.1135 -9.9589 -11.4873  0.1151  0.0168  0.0082  52533.1  580.708  379.715  72.461     3  9.046388e+01
    2 -7.8412 -9.5452 -10.8191  0.0591  0.0246  0.0152  171543.0  2014.450  341.365  320.621     9  8.515625e+01
    5 -7.8104 -9.1967 -10.5169  0.0609  0.0339  0.0201  131343.0  1648.340  305.545  307.027     3  7.968198e+01
    7 -8.8319 -10.3340 -11.3343  0.0238  0.0119  0.0095  111597.0  1525.780  336.462  97.060     3  7.314095e+01
    9 -7.5366 -9.0374 -10.3321  0.0784  0.0393  0.0238  72761.7  1603.150  94.196  131.380     3  4.538671e+01
   10 -6.7963 -8.4304 -9.5685  0.1552  0.0687  0.0482  14072.0  895.465  265.404  46.241     3  1.571474e+01
    6 -12.1666 -13.2193 -14.1293  0.0011  0.0008  0.0007     2.2  1938.010  258.692  260.341     3  1.149313e-03
~>

The two files contain the same data. In the SExtractor version the column names are still present in the header.