\chapter{Data file details}
\label{app-datafile}

\section{Basic native format}
\label{native}

In \app{gretl}'s native data format, a data set is stored in XML
(extensible mark-up language). Data files correspond to the simple DTD
(document type definition) given in \verb+gretldata.dtd+, which is
supplied with the \app{gretl} distribution and is installed in the
system data directory (e.g.\ \url{/usr/share/gretl/data} on Linux.)
Data files may be plain text or gzipped.  They contain the actual data
values plus additional information such as the names and descriptions
of variables, the frequency of the data, and so on.

Most users will probably not have need to read or write such files
other than via \app{gretl} itself, but if you want to manipulate them
using other software tools you should examine the DTD and also take a
look at a few of the supplied practice data files: \verb+data4-1.gdt+
gives a simple example; \verb+data4-10.gdt+ is an example where
observation labels are included.

\section{Traditional ESL format}
\label{traddata}

For backward compatibility, \app{gretl} can also handle data files in
the ``traditional'' format inherited from Ramu Ramanathan's \app{ESL}
program.  In this format a data set is represented by two text files;
one contains the actual data and the other information on how the data
should be read.  To be more specific:

\begin{enumerate}
\item \emph{Actual data}: A rectangular matrix of white-space
  separated numbers. By default, each column represents a variable,
  each row an observation. The data columns can be separated by spaces
  or tabs. Traditionally the data filename has no extension/suffix.
\item \emph{Header}: The data file must be accompanied by a header
  file which has the same basename as the data file plus the suffix
  \verb+.hdr+.  This file contains, in order:
  \begin{itemize}
  \item (Optional) \emph{comments} on the data, set off by the opening
    string \verb+(*+ and the closing string \verb+*)+, each of these
    strings to occur on lines by themselves.
  \item (Required) a list of the \emph{names of the variables} in the
    data file, separated by white space. Names are limited to 8
    characters, must start with a letter, and are limited to
    alphanumeric characters plus the underscore.  The list may
    continue over more than one line; it should be terminated with a
    semicolon.
  \item (Required) an \emph{observations} line of the form
    \verb+1 1 85+.  The first element gives the data frequency (1 for
    undated or annual data, 4 for quarterly, 12 for monthly).  The
    second and third elements give the starting and ending
    observations.  These should be 1 and the number of observations,
    respectively, for undated data.  For time-series data one can use
    dates of the form \cmd{1959.1} (quarterly, one digit after the
    point) or \cmd{1967.03} (monthly, two digits after the point).
  \item The keyword \verb+BYOBS+ (but see below).
  \end{itemize}
\end{enumerate}

Here is an example of a well-formed data header file; the
corresponding data file contains three columns of data, each having 90
entries.

\begin{code} 
(* 
  DATA9-6: 
  Data on log(money), log(income) and interest rate from US. 
  Source: Stock and Watson (1993) Econometrica 
  (unsmoothed data) Period is 1900-1989 (annual data). 
  Data compiled by Graham Elliott. 
*) 
lmoney lincome intrate ; 
1 1900 1989 BYOBS
\end{code}

Three further features of the traditional ESL data format may be
noted.
    
\begin{enumerate}
\item If the \verb+BYOBS+ keyword is replaced by \verb+BYVAR+ this
  indicates that in the corresponding data file the data are written
  out by variable rather than by observation.
\item If \verb+BYOBS+ is followed by the keyword \verb+MARKERS+,
  \app{gretl} expects a data file in which the \emph{first column}
  contains strings (8 characters maximum) used to identify the
  observations.  This may be useful in the case of cross-sectional data
  where the units of observation are identifiable: countries, states,
  cities or whatever.  It can also be useful for irregular time series
  data, such as daily stock price data where some days are not trading
  days --- in this case the observations can be marked with a date
  string such as \cmd{10/01/98}.  (Remember the 8-character maximum.)
  Note that \cmd{BYVAR} and \cmd{MARKERS} are mutually exclusive
  flags.  Also note that the ``markers'' are not considered to be a
  variable: this column does not have a corresponding entry in the
  list of variable names in the header file.
\item If a file with the same base name as the data file and header
  files, but with the suffix \verb+.lbl+, is found, it is read to fill
  out the descriptive labels for the data series. The format of the
  label file is simple: each line contains the name of one variable
  (as found in the header file), followed by one or more spaces,
  followed by the descriptive label. Here is an example, giving
  a label for a variable named ``price'':
  \verb+price New car price index, 1982 base year+
\end{enumerate}

If you want to save data in traditional format, use the
\verb|--traditional| flag with the \cmd{store} command, either in the
command-line program or in the console window of the GUI program.


\section{Binary database details}
\label{dbdetails}

A \app{gretl} database consists of two parts: an ASCII index file
(with filename suffix \verb+.idx+) containing information on the
series, and a binary file (suffix \verb+.bin+) containing the actual
data.  Two examples of the format for an entry in the \verb+idx+ file
are shown below:

\begin{code}
G0M910  Composite index of 11 leading indicators (1987=100) 
M 1948.01 - 1995.11  n = 575
currbal Balance of Payments: Balance on Current Account; SA 
Q 1960.1 - 1999.4 n = 160
\end{code}

The first field is the series name.  The second is a description of
the series (maximum 128 characters).  On the second line the first
field is a frequency code: \verb+M+ for monthly, \verb+Q+ for
quarterly, \verb+A+ for annual, \verb+B+ for business-daily (daily
with five days per week) and \verb+D+ for daily (seven days per week).
No other frequencies are accepted at present.  Then comes the starting
date (N.B. with two digits following the point for monthly data, one
for quarterly data, none for annual), a space, a hyphen, another
space, the ending date, the string ``\verb+n = +'' and the integer
number of observations. In the case of daily data the starting and
ending dates should be given in the form \verb+YYYY/MM/DD+. This
format must be respected exactly.

Optionally, the first line of the index file may contain a short
comment (up to 64 characters) on the source and nature of the data,
following a hash mark.  For example:

\begin{code}
# Federal Reserve Board (interest rates)
\end{code}

The corresponding binary database file holds the data values,
represented as ``floats'', that is, single-precision floating-point
numbers, typically taking four bytes apiece.  The numbers are packed
``by variable'', so that the first \emph{n} numbers are the
observations of variable 1, the next \emph{m} the observations on
variable 2, and so on.

\input{odbc}

\chapter{Building \app{gretl}}
\label{app-build}

\section{Requirements}
\label{sec:build-req}

\app{Gretl} is written in the C programming language, abiding as far
as possible by the ISO/ANSI C Standard (C90) although the graphical
user interface and some other components necessarily make use of
platform-specific extensions.
  
The program was developed under Linux. The shared library and
command-line client should compile and run on any platform that
supports ISO/ANSI C and has the libraries listed in
Table~\ref{tab:depend}.  If the GNU readline library is found on the
host system this will be used for \app{gretcli}, providing a much
enhanced editable command line.  See the
\href{http://cnswww.cns.cwru.edu/~chet/readline/rltop.html}{readline
  homepage}.

\begin{table}[htbp]
  \centering
  \begin{tabular}{lll}
\textit{Library} & \textit{purpose} & \textit{website} \\ [4pt]
zlib & data compression &  
   \href{http://www.info-zip.org/pub/infozip/zlib/}{info-zip.org} \\
libxml2 & XML manipulation &
   \href{http://xmlsoft.org/}{xmlsoft.org} \\
LAPACK & linear algebra & 
   \href{http://www.netlib.org/lapack/}{netlib.org} \\
FFTW3 & Fast Fourier Transform & 
   \href{http://www.fftw.org/}{fftw.org} \\
glib-2.0 & Numerous utilities & 
  \href{http://www.gtk.org/}{gtk.org}
  \end{tabular}
  \caption{Libraries required for building gretl}
  \label{tab:depend}
\end{table}

The graphical client program should compile and run on any system
that, in addition to the above requirements, offers GTK version 2.4.0
or higher (see \href{http://www.gtk.org/}{gtk.org}).\footnote{Up till
  version 1.5.1, \app{gretl} could also be built using GTK 1.2.
  Support for this was dropped at version 1.6.0 of \app{gretl}.}
  
\app{Gretl} calls gnuplot for graphing. You can find gnuplot at
\href{http://www.gnuplot.info/}{gnuplot.info}.  As of this writing the
most recent official release is 4.2.6 (of September, 2009).  The 
\app{gretl} packages for MS Windows and Mac OS X come with 
current CVS gnuplot (version 4.5), and the \app{gretl} website
offers information on building or installing gnuplot 4.5 on
Linux.
  
Some features of \app{gretl} make use of portions of Adrian Feguin's
\app{gtkextra} library.  The relevant parts of this package are
included (in slightly modified form) with the \app{gretl} source
distribution.
  
A binary version of the program is available for the Microsoft Windows
platform (Windows 2000 or higher). This version was cross-compiled
under Linux using mingw (the GNU C compiler, \app{gcc}, ported for use
with win32) and linked against the Microsoft C library,
\verb+msvcrt.dll+.  The (free, open-source) Windows installer program
is courtesy of Jordan Russell
(\href{http://www.jrsoftware.org/}{jrsoftware.org}).

\section{Build instructions: a step-by-step guide}
\label{sec:build-inst}

In this section we give instructions detailed enough to allow a user
with only a basic knowledge of a Unix-type system to build \app{gretl}.
These steps were tested on a fresh installation of Debian Etch. For
other Linux distributions (especially Debian-based ones, like Ubuntu
and its derivatives) little should change. Other Unix-like operating
systems such as MacOSX and BSD would probably require more substantial
adjustments.

In this guided example, we will build \app{gretl} complete with
documentation.  This introduces a few more requirements, but gives you
the ability to modify the documentation files as well, like the help
files or the manuals.

\subsection{Installing the prerequisites}

We assume that the basic GNU utilities are already installed on the
system, together with these other programs:
\begin{itemize}
\item some \TeX/\LaTeX system (\texttt{texlive} will do beautifully)
\item Gnuplot
\item ImageMagick
\end{itemize}
We also assume that the user has administrative privileges and knows
how to install packages.  The examples below are carried out using the
\texttt{apt-get} shell command, but they can be performed with
menu-based utilities like \texttt{aptitude}, \texttt{dselect} or the
GUI-based program \texttt{synaptic}. Users of Linux distributions
which employ rpm packages (e.g.\ Red Hat/Fedora, Mandriva, SuSE) may
want to refer to the
\href{http://gretl.sourceforge.net/depend.html}{dependencies} page on
the \app{gretl} website.

The first step is installing the C compiler and related basic
utilities, if these are not already in place. On a Debian system,
these are contained in a bunch of packages that can be installed via
the command
\begin{code}
apt-get install gcc autoconf automake1.9 libtool flex bison gcc-doc \
libc6-dev libc-dev gfortran gettext pkgconfig
\end{code}

Then it is necessary to install the ``development'' (\texttt{dev})
packages for the libraries that \app{gretl} uses:
\begin{center}
  \begin{tabular}{ll}
    \textit{Library} & \textit{command} \\ [4pt]
    GLIB     & \texttt{apt-get install libglib2.0-dev} \\
    GTK 2.0  & \texttt{apt-get install libgtk2.0-dev} \\
    PNG      & \texttt{apt-get install libpng12-dev} \\
    XSLT     & \texttt{apt-get install libxslt1-dev} \\
    LAPACK   & \texttt{apt-get install liblapack-dev} \\
    FFTW     & \texttt{apt-get install libfftw3-dev} \\
    READLINE & \texttt{apt-get install libreadline-dev} \\
    ZLIB     & \texttt{apt-get install zlib1g-dev} \\
    XML      & \texttt{apt-get install libxml2-dev} \\
    GMP      & \texttt{apt-get install libgmp3-dev} \\
    MPFR     & \texttt{apt-get install libmpfr-dev}
  \end{tabular}
\end{center}

(MPFR is optional, but recommended.)  The \texttt{dev} packages for
these libraries are necessary to \emph{compile} \app{gretl} --- you'll
also need the plain, non-\texttt{dev} library packages to \emph{run}
\app{gretl}, but most of these should already be part of a standard
installation.  In order to enable other optional features, like audio
support, you may need to install more libraries.

\tip{The above steps can be much simplified on Linux systems
that provide deb-based package managers, such as Debian and its
derivatives (Ubuntu, Knoppix and other distributions). The command

\texttt{apt-get build-dep gretl}

will download and install all the necessary packages for building the
version of \app{gretl} that is currently present in your APT
sources. Techincally, this does not guarantee that all the software
necessary to build the CVS version is included, because the version of
\app{gretl} on your repository may be quite old and build requirements
may have changed in the meantime. However, the chances of a mismatch
are rather remote for a reasonably up-to-date system, so in most cases
the above command should take care of everything correctly.}

\subsection{Getting the source: release or CVS}

At this point, it is possible to build from the source.  You have two
options here: obtain the latest released source package, or retrieve
the current CVS version of \app{gretl} (CVS = Concurrent Versions
System).  The usual caveat applies to the CVS version, namely, that it
may not build correctly and may contain ``experimental'' code; on the
other hand, CVS often contains bug-fixes relative to the released
version.  If you want to help with testing and to contribute bug
reports, we recommend using CVS \app{gretl}.

To work with the released source:
\begin{enumerate}
\item Download the \app{gretl} source package from
  \href{http://gretl.sourceforge.net/}{gretl.sourceforge.net}.
\item Unzip and untar the package.  On a system with the GNU utilities
  available, the command would be \cmd{tar xvfz gretl-N.tar.gz}
  (replace \cmd{N} with the specific version number of the file you
  downloaded at step 1).
\item Change directory to the gretl source directory created at step 2
  (e.g.\ \verb+gretl-1.6.6+).
\item Proceed to the next section, ``Configure and make''.
\end{enumerate}

To work with CVS you'll first need to install the \app{cvs} client
program if it's not already on your system.  Relevant resources
you may wish to consult include the CVS website at
\href{http://www.nongnu.org/cvs/}{www.nongnu.org/cvs},
general information on sourceforge CVS on the 
  \href{http://sourceforge.net/docman/display_doc.php?docid=14035&group_id=1}{SourceForge
    CVS page}, and instructions specific to \app{gretl} at the
\href{http://sourceforge.net/cvs/?group_id=36234}{SF gretl CVS page}.

When grabbing the CVS sources \textit{for the first time}, you should
first decide where you want to store the code.  For example, you might
create a directory called \texttt{cvs} under your home directory.
Open a terminal window, \texttt{cd} into this directory, and type
the following commands:
%
\begin{code}
cvs -d:pserver:anonymous@gretl.cvs.sourceforge.net:/cvsroot/gretl login
cvs -z3 -d:pserver:anonymous@gretl.cvs.sourceforge.net:/cvsroot/gretl co -P gretl
\end{code}
%
After the first command you will be prompted for a password: just hit
the Enter key.  After the second command, \app{cvs} should create a
subdirectory named \texttt{gretl} and fill it with the current
sources.

When you want to \textit{update the source}, this is very simple: just move into
the \texttt{gretl} directory and type
\begin{code}
cvs update -d -P
\end{code}

Assuming you're now in the CVS \texttt{gretl} directory, you can
proceed in the same manner as with the released source package.


\subsection{Configure the source}
          
The next command you need is \texttt{./configure}; this is a complex
script that detects which tools you have on your system and sets
things up. The \texttt{configure} command accepts many
options; you may want to run 
\begin{code}
./configure --help
\end{code}
first to see what options are available. One option you way wish to
tweak is \cmd{--prefix}.  By default the installation goes under
\verb+/usr/local+ but you can change this.  For example
\begin{code}
./configure --prefix=/usr
\end{code}
will put everything under the \verb+/usr+ tree.  Another useful option
refers to the fact that, by default, \app{gretl} offers support for
the \app{gnome} desktop.  If you want to suppress the
\app{gnome}-specific features you can pass the option
\option{without-gnome} to \cmd{configure}.

In order to have the documentation built, we need to pass the relevant
option to \texttt{configure}, as in
\begin{code}
./configure --enable-build-doc
\end{code}
But please note that this option will work only if you are using
the CVS source.

You will see a number of checks being run, and if everything goes
according to plan, you should see a summary similar to that displayed
in Example~\ref{configure-output}.

\begin{script}[htbp]
  \caption{Output from \texttt{./configure --enable-build-doc}}
  \label{configure-output}
\begin{scode}
Configuration:

  Installation path:                      /usr/local
  Use readline library:                   yes
  Use gnuplot for graphs:                 yes
  Use PNG for gnuplot graphs:             yes
  Use LaTeX for typesetting output:       yes
  Gnu Multiple Precision support:         yes
  MPFR support:                           no
  LAPACK support:                         yes
  FFTW3 support:                          yes
  Build with GTK version:                 2.0
  Script syntax highlighting:             yes
  Use installed gtksourceview:            yes
  Build with gnome support:               no
  Build gretl documentation:              yes
  Build message catalogs:                 yes
  Gnome installation prefix:              NA
  X-12-ARIMA support:                     yes
  TRAMO/SEATS support:                    yes
  Experimental audio support:             no

Now type 'make' to build gretl.
\end{scode}
\end{script}

\tip{If you're using CVS, it's a good idea to re-run the
  \texttt{configure} script after doing an update.  This is not always
  necessary, but sometimes it is, and it never does any harm.  For
  this purpose, you may want to write a little shell script that calls
  \texttt{configure} with any options you want to use.}


\subsection{Build and install}

We are now ready to undertake the compilation proper: this is done by
running the \texttt{make} command, which takes care of compiling all
the necessary source files in the correct order. All you need to do is
type
\begin{code}
make 
\end{code}

This step will likely take several minutes to complete; a lot of
output will be produced on screen. Once this is done, you can install
your freshly baked copy of \app{gretl} on your system via
\begin{code}
make install
\end{code}

On most systems, the \texttt{make install} command requires you to
have administrative privileges.  Hence, either you log in as
\texttt{root} before launching \texttt{make install} or you may want
to use the \texttt{sudo} utility:
\begin{code}
sudo make install
\end{code}


\chapter{Numerical accuracy}
\label{app-accuracy}

\app{Gretl} uses double-precision arithmetic throughout --- except for
the multiple-precision plugin invoked by the menu item ``Model, Other
linear models, High precision OLS'' which represents floating-point
values using a number of bits given by the environment variable
\verb+GRETL_MP_BITS+ (default value 256).  

The normal equations of Least Squares are by default solved via
Cholesky decomposition, which is highly accurate provided the matrix
of cross-products of the regressors, $X'X$, is not very ill
conditioned.  If this problem is detected, \app{gretl} automatically
switches to use QR decomposition.

The program has been tested rather thoroughly on the statistical
reference datasets provided by NIST (the U.S.  National Institute of
Standards and Technology) and a full account of the results may be
found on the gretl website (follow the link ``Numerical accuracy'').

To date, two published reviews have discussed \app{gretl}'s accuracy:
Giovanni Baiocchi and Walter Distaso \citeyearpar{baiocchi03}, and
Talha Yalta and Yasemin Yalta \citeyearpar{yalta07}.  We are grateful
to these authors for their careful examination of the program.  Their
comments have prompted several modifications including the use of
Stephen Moshier's \app{cephes} code for computing p-values and other
quantities relating to probability distributions (see
\href{http://www.netlib.org/cephes/}{netlib.org}), changes to the
formatting of regression output to ensure that the program displays a
consistent number of significant digits, and attention to compiler
issues in producing the MS Windows version of \app{gretl} (which at
one time was slighly less accurate than the Linux version).

\app{Gretl} now includes a ``plugin'' that runs the NIST linear
regression test suite.  You can find this under the ``Tools'' menu in
the main window.  When you run this test, the introductory text
explains the expected result.  If you run this test and see anything
other than the expected result, please send a bug report to
\verb+cottrell@wfu.edu+.

All regression statistics are printed to 6 significant figures in the
current version of \app{gretl} (except when the multiple-precision
plugin is used, in which case results are given to 12 figures).  If
you want to examine a particular value more closely, first save it
(for example, using the \cmd{genr} command) then print it using
\cmd{printf}, to as many digits as you like (see the \GCR).  

\chapter{Related free software}
\label{app-advanced}

\app{Gretl}'s capabilities are substantial, and are expanding.
Nonetheless you may find there are some things you can't do in
\app{gretl}, or you may wish to compare results with other programs.
If you are looking for complementary functionality in the realm of
free, open-source software we recommend the following programs.  The
self-description of each program is taken from its website.

\begin{itemize}

\item \textbf{GNU R} \href{http://www.r-project.org/}{r-project.org}:
  ``R is a system for statistical computation and graphics. It
  consists of a language plus a run-time environment with graphics, a
  debugger, access to certain system functions, and the ability to run
  programs stored in script files\dots\ It compiles and runs on a wide
  variety of UNIX platforms, Windows and MacOS.''  Comment: There are
  numerous add-on packages for R covering most areas of statistical
  work.

\item \textbf{GNU Octave}
  \href{http://www.octave.org/}{www.octave.org}:
  ``GNU Octave is a high-level language, primarily intended for
  numerical computations. It provides a convenient command line
  interface for solving linear and nonlinear problems numerically, and
  for performing other numerical experiments using a language that is
  mostly compatible with Matlab. It may also be used as a
  batch-oriented language.''

\item \textbf{JMulTi} \href{http://www.jmulti.de/}{www.jmulti.de}:
  ``JMulTi was originally designed as a tool for certain econometric
  procedures in time series analysis that are especially difficult to
  use and that are not available in other packages, like Impulse
  Response Analysis with bootstrapped confidence intervals for VAR/VEC
  modelling. Now many other features have been integrated as well to
  make it possible to convey a comprehensive analysis.''  Comment:
  JMulTi is a java GUI program: you need a java run-time environment to
  make use of it.

\end{itemize}

As mentioned above, \app{gretl} offers the facility of exporting
data in the formats of both Octave and R.  In the case of Octave, the
\app{gretl} data set is saved as a single matrix, \verb+X+. You can
pull the \verb+X+ matrix apart if you wish, once the data are loaded
in Octave; see the Octave manual for details.  As for R, the exported
data file preserves any time series structure that is apparent to
\app{gretl}.  The series are saved as individual structures. The data
should be brought into R using the \cmd{source()} command.
  
In addition, \app{gretl} has a convenience function for moving data
quickly into R.  Under \app{gretl}'s ``Tools'' menu, you will find the
entry ``Start GNU R''.  This writes out an R version of the current
\app{gretl} data set (in the user's gretl directory), and sources it
into a new R session.  The particular way R is invoked depends on the
internal \app{gretl} variable \verb+Rcommand+, whose value may be set
under the ``Tools, Preferences'' menu.  The default command is
\cmd{RGui.exe} under MS Windows. Under X it is \cmd{xterm -e R}.
Please note that at most three space-separated elements in this
command string will be processed; any extra elements are ignored.

\chapter{Listing of URLs}
\label{app-urls}

Below is a listing of the full URLs of websites mentioned in the text.

\begin{description}

\item[Estima (RATS)] \url{http://www.estima.com/}
\item[FFTW3] \url{http://www.fftw.org/}
\item[Gnome desktop homepage] \url{http://www.gnome.org/}
\item[GNU Multiple Precision (GMP) library]
  \url{http://gmplib.org/}
\item[GNU Octave homepage] \url{http://www.octave.org/}
\item[GNU R homepage] \url{http://www.r-project.org/}
\item[GNU R manual]
  \url{http://cran.r-project.org/doc/manuals/R-intro.pdf}
\item[Gnuplot homepage] \url{http://www.gnuplot.info/}
\item[Gnuplot manual] \url{http://ricardo.ecn.wfu.edu/gnuplot.html}
\item[Gretl data page]
  \url{http://gretl.sourceforge.net/gretl_data.html}
\item[Gretl homepage] \url{http://gretl.sourceforge.net/}
\item[GTK+ homepage] \url{http://www.gtk.org/}
\item[GTK+ port for win32]
  \url{http://www.gimp.org/~tml/gimp/win32/}
\item[Gtkextra homepage] \url{http://gtkextra.sourceforge.net/}
\item[InfoZip homepage]
  \url{http://www.info-zip.org/pub/infozip/zlib/}
\item[JMulTi homepage] \url{http://www.jmulti.de/}
\item[JRSoftware] \url{http://www.jrsoftware.org/}
\item[Mingw (gcc for win32) homepage] \url{http://www.mingw.org/}
\item[Minpack] \url{http://www.netlib.org/minpack/}
\item[Penn World Table] \url{http://pwt.econ.upenn.edu/}
\item[Readline homepage]
  \url{http://cnswww.cns.cwru.edu/~chet/readline/rltop.html}
\item[Readline manual]
  \url{http://cnswww.cns.cwru.edu/~chet/readline/readline.html}
\item[Xmlsoft homepage] \url{http://xmlsoft.org/}

\end{description}


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

