Tutorial 05
Working with Data:  Creating, Formatting, Plotting, and Fitting

Copyright © 2003 by Bill Titus, Carleton College,
Department of Physics and Astronomy, Northfield, MN  55057-4025
March 1, 2003

Part 1 - Getting Started

Step 1.  Start up Mathematica and open the notebook  Tutorial05.nb.  Remember, this notebook is located in the workshop folder.

Step 2.  OPTIONAL.  Since you're going to edit this notebook, you need to make a copy of this notebook and work with that copy.  To do so, select  File / SaveAs  from the Mathematica menu and save the notebook in the workshop folder under the name  myTutorial05.nb.

Part 2 - The Problem

Jane, the geologist, has been given the following experimentally determined accumulation rates  p  for the formation of carbonate sediments off a reef of a small island near Australia as a function of the depth  z  below sea level.

z (m)	       p (m (ky^(-1)))

						00.0		3.05
						02.0		2.75
						04.0		2.55
						06.0		2.10
						08.0		1.95
						10.0		1.85
						12.0		1.60
						14.0		1.50
						16.0		1.35
						18.0		1.15
						20.0		1.10

Jane knows that the percent uncertainties in the measurement for  z  are small in comparison to those for  p.  Also each  p  measurement has roughly the same uncertainty, although the geologist who gave Jane the data neglected to tell her the magnitude of that uncertainty.   Jane knows that the data should approximately fit the equation  p = (Subscript[p, 0])exp(-z/(Subscript[z, 0])),  where  (Subscript[p, 0]) and  (Subscript[z, 0])  are constants.  She'd like to use this equation and her data to estimate values for  (Subscript[p, 0])  and  (Subscript[z, 0]) as well as the uncertainties in those quantities.

You're going to help Jane out by doing the following tasks:

(i)     Create a list of the data so you can work with it within Mathematica.
(ii)    Plot the data so you have some sense for how the data behaves.
(iii)   Do a fit to the data with the prescribed equation and estimate numerical values for  (Subscript[p, 0])  and  (Subscript[z, 0 ])  and their uncertainties.

Part 3 - Creating a List of Data

Topics and Skills

1.  Producing 2D data lists in Mathematica

Step 1.  Produce a list, whose elements are of the form  {z, p}  from the data above and give it the name   janeData.  To get you started, the cell below contains the first 5 elements of that list.   Complete the list and then enter the results.

Clear[janeData]
janeData = {{0.0, 3.05}, {2.0, 2.75}, {4.0, 2.55}, {6.0, 2.10}, {8.0, 
          1.95},  }

Comment 1

1.  Data in Mathematica is always handled by lists, lists of lists, etc.  In your case,  janeData  is a list of elements, each of which is itself a list of two numbers, the first number corresponding to  z  and the second number corresponding to  p.
2.  One potential visual problem with data lists in Mathematica is that the results are not in tabular form but instead are displayed linearly.  The next section will show you how to format your data in table form so that the data is easier to read.

Part 4 - Commands that Format Data

Topics and Skills

1.  The Mathematica command  TableForm[]
2.  Formatting with  NumberForm[]  and its option  NumberPadding 
3.  The nesting properties of functions

Step 1.  You'll first display the data in tabular form.  To do so, get extended help on the Mathematica command   TableForm[]  and then apply the command to the list  janeData.

Comment 1

1.  Note the title of this last output cell,  Out[3]//TableForm= .  The title indicts that the text in this cell has special formatting characteristics.  Usually you cannot perform mathematical operation on formatted data, but only on the original list from which the formatted data came.
2.  While  TableForm[]  did create a table of values, the numbers were not formatted well, at least to compulsive Bill's standards.  You'll see one way to do the formatting in the next several steps.

Step 2.  There are a number of Mathematica formatting commands that allow flexibility in how numbers are displayed; you have ready used  ScientificForm[]  and  PaddedForm[]  in previous tutorials.  Here you'll use the command   NumberForm[].  Get extended help on this function by typing  ??NumberForm.

Comment 2

1.  This command has several options.  You'll use the  NumberPadding  option in this tutorial.

Step 3.  Start by defining   myNum  to equal  17.0  and look at the output for the function  NumberForm[myNum,n],  with  n  going from  1  to  4,  by activating the cell below and then changing the value of  n  appropriately.

Clear[myNum, n]
myNum = 17.0; n = 2;
NumberForm[myNum, n]

Comment 3

1.  Note the title,  Out[7]//NumberForm==,   of the output cell after you invoke  NumberForm[].  Again, it tells you that this cell contains special text formatting characteristics.
2.  Even though you typed in  17.0 for  myNum,  Mathematica echoed back  17. and thus didn't include the zero after the decimal point.
3.  The symbol  n  controls the number of significant figures, provided you don't ask for more significant figures than occur.
4.  When  n  equals one, a warning message appears that actually makes sense considering the output given.

Step 4.  Try out the  NumberPadding  option, with  0  as the padding symbol by activating the code below.  See what happens as you change the value of   n.  Then repeat with an empty space   " "    replacing the first  0  in the number padding list.

Clear[myNum, n]
myNum = 17.0; n = 2;
NumberForm[myNum, n, 
      NumberPadding -> {"0", "0"}]

Comment 4

1.  For  n = 1,  you should find that a  0  appears in front of  myNum.  It's there to hold the place for the plus sign that doesn't appear; the plus sign can be added by invoking another option, if you want.  For other values of  n,  the number of  0's  that appear in front of   myNum   corresponds   n - 1.
2.  The symbol  0  isn't particularly useful in this case, but the empty space is since it allows you to specify, for formatting purposes, how many blank spaces occur before your number.

Step 5.  Suppose you want to "pad" an extra  0  after the decimal point to signify that there are really three significant figures in  myNum.  Even though it doesn't seem to be documented in Mathematica help, the command  NumberForm[]  has a list argument that replaces  n  with  {n,f},  where  n  is now the total number of places to be printed (including the padded symbols) and  f  is the number of places to the right of the decimal point that appears.  Use this information to print  myNum  to three significant figures.  Be sure to set  NumberPadding  to  {" ", "0"}.

Step 6.  Activate the cell below which formats the list   janeData   so that the data is displayed in tabular form, with two places after the decimal point and with all the numbers lined up with respect to the decimal points.  Note that Mathematica requires that you use  TableForm[]  before you apply formatting commands like  NumberForm[].

NumberForm[TableForm[janeData], {4, 2}, 
      NumberPadding -> {" ", "0"}]

Comment 6

1.  If you want to add headings to your table, you can use the option  TableHeading.  See Mathematica help for more information on this option if you ever have the occasion to use it.
2.  The above code is nice example of the nesting properties of function, i.e., one function can take another function as its argument.

Part 5 - Plotting Data

Topics and Skills

1.  The Mathematica command  ListPlot[],  its option  PlotJoined,  and the graphics directive  PointSize[].
2.  Using Mathematica help in conjunction with the "wild card"  *.

Step 1.  You now want to plot this list of  2D data, but you're not quite certain of the Mathematica command that will do this.  One way to find out the name of the appropriate Mathematica command is to use an  *  in conjunction with the help symbol ? .  The *  serves as a wild card.  For example,  ?*List  will display all the Mathematica commands that end with  List  while  ?*List*  will display all the commands that have  List  somewhere embedded inside its name.  Try out both versions of this command in the cell below.

Step 2.  Just from the name alone, it looks like  ListPlot[]  is the function to use.  Get extended help on this function and find out what it does and what are its associated options.

Comment 2

1.  Most of the options for  ListPlot[]  should look familiar from the options associated with  Plot[].
2.  You'll find the new option  PlotJoined   useful at times.  If you set  PlotJoined -> True,  straight line segments connect successive points.  However, points are then no longer plotted.  If you want to display both the lines joining the points and the points themselves, you must create two plots and then superimpose the plots using  Show[].
3.  The graphics directive  PointSize[] for the option  PlotStyle  allows one to control the size of the points to be plotted.

Step 3.  Use  ListPlot[]  to create a framed plot for  janeData  by activating the code below.  The output has been named  plot1  so it can be referred to latter.

Clear[plot1]
\(\(plot1 = 
ListPlot[
          janeData, 
PlotStyle -> {RGBColor[1, 0, 0], 
              PointSize[0.02]}, 
Frame -> 
            True, 
FrameLabel -> {"z(m)", "p(m/ky)", "Reef Accumation Rate as a Function of Depth", ""}, 
Axes -> None, 
GridLines -> 
            Automatic];\)\)

Comment 3

1.  To make the plot more readable, you may want to expand it size.   Remember, to do so, just click anywhere on the plot and then drag on one of the bars that appear on the plot outline.
2.  Note that the options in  ListPlot[]  have been put on separate lines to enhance readability.  Mathematica could care less whether the options are on separate lines as long as there are commas between the options.

Part 6 - Fitting Data to a Decreasing Exponential:  Attempt 1

Topics and Skills

1.  The Mathematica command  Fit[]

Step 1.  You'd now like to fit the data to the theoretical function  (Subscript[p, 0])(e^(-z/(Subscript[z, 0])))  and find the best values for  (Subscript[p, 0])  and  (Subscript[z, 0]).  You'll first (sort of) do this by using the Mathematica command  Fit[].  Get extended help on  Fit[].

Comment 1

1.  With the data in its present form,  Fit[]  can't be used to determine (Subscript[p, 0])  and  (Subscript[z, 0]) since fit functions must be independent of the fitting parameters.  
2.  But you could, for example, fit the data to a set of powers in  z  of the form  {1, z, (z^2), (z^3), ...}   and, in fact, that's what you're going to do for the rest of this section.  Expanding in term of this set of functions may be reasonable since the exponential can, itself, be expanded in terms of a power series in  z.  This task will give you some good practice in using  Fit[].

Step 2.  Determine the third order polynomial fit to  janeData  by activating the code below.

Clear[t]
Fit[janeData, {1, t, (t^2), (t^3)}, t]

Comment 2

1.  Here the  1  in  the list of functions {1, t, (t^2),(t^3)}  indicates that the constant term is to be included in the fit.

Step 3.  It would be nice to know how well this third order polynomial fit Jane's data.  One way would be to plot the equation and superimpose it on a plot of Jane's data.   The first step in this procedure is to define a function corresponding to the fit given in Step 2.  Call that function  h[t]  and define it through the code below by activating the code's cell.

Clear[h, t]
h[t_] = Fit[janeData, {1, t, (t^2), (t^3)}, t]

Step 4.  Create a "bare" plot of   h[t]   for  t  going from  0.0  to  20.0  and called the output  plot2.  Then superimpose  plot1  (the plot of Jane's data found in Step 3, Part 4) and  plot2  on the same graph using  Show[].  Add an option to  Show[]  to give the plot a title.

Part 7 - Fitting Data to a Decreasing Exponential - Attempt 2

Topics and Skills

1.  The Mathematica command  Log[]
2.  The purpose of Mathematica packages.
3.  The Mathematica command  Regress[]  in the package  Statistics`LinearRegression`

Step 1.  Because of the exponential nature of the assumed behavior of  p as a function of  z,  a better way to analyze this problem would be to take the natural log of  p  and plot it against  z.  Theoretically,  the curve should then be the straight line

				ln[p]  =  ln[(Subscript[p, 0])] - (z/(Subscript[z, 0]))

The intercept of the line will give  ln[(Subscript[p, 0])]  while the slope yields the value of  1/(Subscript[z, 0]).    First get help on the Mathematica function  Log[].

Step 2.  You're now going to transform the list  janeData,  whose elements are of the form  {z,p}  to the list  janeLog,  whose elements will be of the form {z,Log[z]}.  You'll do this by using the Mathematica symbol   /.  Get help on this symbol.

Step 3.  OK, that wasn't a big help.  Instead, just activate the cell below and see if the output makes sense in light of the usage statement for  /.  that you generated above.

janeData
janeLog  =  janeData /. {x_, y_} -> {x, Log[y]}

Comment 3

1.  I read  /.  as  "such that"  and  ->  as  "goes to".  So in this case,  each element of  janeData  of the form  {x, y}  goes to the element  {x, Log[y]}.  This technique is a very useful and flexible way in which to transform data.
2.  The underscores, as in  {x_, y_},  just tells Mathematica that anything that appear as the order pair  {x, y}  will be appropriately transformed.  If instead, you had  {x_, 2}  in place of   {x_, y_},  Mathematica would only transform elements of the original list that had a  2  in the second entry of the list.

Step 4.  Now apply  Fit[]  to the list   janeLog  and determine the best values for  (Subscript[p, 0])  and  (Subscript[z, 0 ]).   The latter task will require a little thought and bit more calculation on your part since the fit doesn't give  (Subscript[p, 0]) and  (Subscript[z, 0])  directly.

Comment 4

1.  I found that  (Subscript[z, 0])= 19.3573  and  (Subscript[p, 0]) = 3.02639.  Clearly the results for  (Subscript[z, 0]) and  (Subscript[p, 0])  are not good to six significant figures.  If you want an idea of the uncertainty in these coefficients, you can apply the function  Regress[]  that includes lots more information about the least square regression fit.  This function is available in the Mathematica package  Statistics`LinearRegression`.
2.  Whoa!  What's a Mathematica package.  There's a large number of functions written in Mathematica's own programming language that extend and add to Mathematica's default library of functions.  These functions are in packages, where each package is associated with a context.  Remember, you saw that your symbols and function definitions are stored in the context   Global`,  while Mathematica's definitions are stored in the context   System`.  Likewise, supplementary functions are associated with packages that have their own individual contexts.  For example, the function  Regress[]  is associated with the package  Statistics`LinearRegression`.
3.  The functions in packages aren't loaded initially by Mathematica since they're specialized and they can take up lots of memory and many people will never use them.
4.  There's a list of the Mathematica packages and a description of their contents in the book  Mathematics 4:  Standard Add-On Packages.  Also, information on the packages and their corresponding functions is available in the help browser.
5.  You can write your own packages in Mathematica's programming language.
6.  You access functions in a package by first loading the package with the Mathematica command  Needs[],  the argument of which is the context for the package.  
7.  For  Regress[],  the context is  Statistics`LinearRegression`.   Note that the single quote mark needed is the one that's located above the tab key on the upper left hand side of the keyboard.  Double quote marks surround the name of the context when it's placed inside the argument of  Needs[].

Step 5.  Get help on the Mathematica function  Needs[].  Then use it to load the package  Statistics`LinearRegression` by activating the cell below.  Finally get help on the function  Regress[].

Needs["Statistics`LinearRegression`"]

Comment 5

1.  No output occurs when you activate the  Needs[]  command unless you've made an error.
2.  As far as arguments are concerned,  Regress[]  and  Fit[]  have the same requirements.

Step 6.  Apply  Regress[]  to  janeLog,  and this time fit the data to the functions {1,t} (these functions are used since the fit is to be linear).

Comment 6

1.  The command  Regress[]  yields lots of output information, enough to make a statistician jump with joy.  To find the uncertainties in the coefficients, the information you're looking for is the standard error (SE) corresponding to each coefficient.  The standard error  cSE  for an estimated coefficient  c  can be interpreted as saving that the real value of the coefficient lies in the interval  {c - cSE, c + cSE}  about two-thirds of the time.  So if you are happy with those confidence intervals, you would quote  1/(Subscript[z, 0])  to have a value of  0.051 ± 0.002  and  ln[(Subscript[p, 0])]  to be  1.11 ± 0.02.
2.  How you go from  1/(Subscript[z, 0]) to the best value and uncertainty for  (Subscript[z, 0])  and from   ln[(Subscript[p, 0])]  to the best value and uncertainty for  (Subscript[p, 0])  is another story for which physicists, for example, would use propagation of uncertainty.

Step 7.  Finally activate the code below to produce a plot that shows that data for  janeLog   and the corresponding log fit.

Clear[plot1, plot2]
plot1 = 
ListPlot[janeLog, 
          PlotStyle -> {RGBColor[1, 0, 0], PointSize[0.02]}, 
          Frame -> True, 
          FrameLabel -> {"z(m)", "Log[p(m/ky)]", "Reef \
Accumation Rate as a Function of Depth", ""}, Axes -> None, 
          GridLines -> Automatic, 
          DisplayFunction -> Identity];
plot2 = 
        Plot[h[t], {t, 0, 20}, Frame -> True, 
          DisplayFunction -> Identity];
Show[{plot1, plot2}, 
        FrameLabel -> {"z", "p", "Jane's Data and Linear Log \
Fit", ""}, DisplayFunction -> $DisplayFunction];

Part 8 - Save Your Notebook (OPTIONAL)

Step 1.  Remove all output cells from this notebook, and to do one final save of your notebook before you quit Mathematica.