CH 2 Systems and Modeling

Research Methodology and Presentation ECEg 4002
Chapter 2: Systems and modeling

2.1. Systems: The subject of research as a system concept
The subject on which research is to be done may be conceptualized as a system in its simplest form. A system is generally an individual entity (usually non
isolated) or an assemblage of interacting or interdependent entities (real or abstract) forming an integrated whole and that may have one or more inputs and one
or more recognizable outputs. The input (independent variable) and outputs (dependent variable) associated to the system may be related with known or unknown
(yet to be determined) rules.
In addition to the input and output variables, a system may be subjected to variables of external nature (exogenous) generally called undesired disturbances. The
system itself may be described using system variables (endogenous) or as usually called system parameters.
Fig: Simplified system

Confounding variables are variables with a significant effect on the dependent variable that the researcher failed to control or eliminate - sometimes because the
researcher is not aware of the effect of the confounding variable. The key is to identify possible confounding variables and somehow try to eliminate or control
them.
Most naturally occurring phenomenon and studied in physics, chemistry, biology, nature ; physical occurrences/ processes studied in engineering; phenomenon
in business and society studied in socio-economics; behavioral and social sciences etc all may be visualized and described using the system concept.
A system approach to research can give insights to the researcher on how to make an appropriate experimental setup as well as the modeling and/or simulation of
the problem at hand which helps him intuitively guide what to explore, test, observe and interpret.
Description of the system variables
Depending on the problem at hand, an input may also be called as an excitation, predictor, stimulus, or cause. Correspondingly an output may also be called a
response or an effect.
The input, output, endogenous, exogenous variables as well as the system parameters may be continuous or discrete in time and/or position; time varying
(dynamic) or non-time varying (static/steady); deterministic or stochastic. Further each can be represented either quantitatively and/or qualitatively.
Characterization/types of systems (in the context as a subject of study/reserch )
A system may be of the following character:
Continuous: a continuous system reacts/responds to inputs continuously in time, position and/or other continuous variables.
Discrete: a discrete system reacts/responds to inputs only discretely in time, position and/or other discrete variable. That is reaction occurs only at discrete
instants of time, position and/or other variables.
Time invariant: a time invariant system has parameters or characteristics that do not change with time.
Time variant: a time variant system has at least one parameter or characteristics that changes with time.
Homogenous: the properties or characteristics of a homogenous system components are identical in all locations, times and conditions. As regards to population
study of any type, taking sample from such homogeneity becomes easier. (Stationarity)
Non homogenous: the properties or characteristics of a non-homogenous system components are not identical in all locations, time and conditions. As regards to
population study of any type, taking sample from such non-homogeneity requires a thoughtful strategy. (Non-stationarity)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Extractions: by Tezazu Bireda, AAU, Addis Ababa Institute of Technology, ECE Dept May 2012

Dynamic: In a dynamic system, the present output is dependent not only on the present input values but also on the past values represented as initial conditions.
In other words, a dynamic system is said to have memory and thus has associated with it initial conditions. The initial conditions are due to inherent inertial
components which are characteristic of a dynamic system. If describable mathematically, a dynamic system relates its inputs and outputs in the time domain
either by differential equations or difference equations of one or higher order or a state-space method. Frequency domain method can also be used. The
inputs and outputs may vary with time.
Static: In a static system, the present output is dependent only on the present input values and not on the past values. This means that there are no initial
conditions associated to a static system. In other words, a static system is memory-less and has no inertial components. If describable mathematically, a
static system relates its inputs and outputs by algebraic equations that may be linear, quadratic, cubic or any order of polynomials; logarithmic equations,
exponential equations, hyperbolic or trigonometric equations. The inputs and outputs may vary with time. A static system may also be called a zero order
system
Linear: a system is said to be linear if it satisfies the superposition principle. That is, if for input x1 it generates an output y1, and for input x2 it generates an
output y2, etc, then for input x1 + x2 + the output becomes y1 + y2 + ..
Non-linear: a system is said to be nonlinear if it doesnt satisfy the superposition principle.
Deterministic: for an identical input/s at different instants of time (but with identical initial conditions for dynamic systems) the system generates identical
output/s at the respective time instants.
A deterministic model is one in which every set of variable states is uniquely determined by parameters in the model and by sets of previous states of these
variables. Therefore, deterministic models perform the same way for a given set of initial conditions.
Non-deterministic: for an identical input/s at different instants of time (but with identical initial conditions for dynamic systems) the system generates nonidentical (non-unique) output/s [or more specifically random outputs] at the respective time instants. The input and output (as well as the degree of variation
of the output) can only be expressed and determined probabilistically and/or statistically governed by a certain probability distribution.
Causal: a system is said to be causal if it will have out put if and only if after application of input.
Non-causal: If with no input a system has an output then it is said to be non-causal.
Stable: a system is said to be stable if for a finite input it will have a finite out put.
Unstable: a system is said to be unstable if for a finite input it may have an infinite out put.
We may note that a given system may be characterized by a number of (if not all) the above list.
System representation: A system may be mathematically, graphically, and/or qualitatively describable
2.2. Modeling
Scientific model
Scientific modeling is the process of generating abstract conceptual, graphical and/or mathematical model/ description that is supposed to sufficiently (accurately
but simplistically) represent some or most realms of a physical/abstract phenomenon/system.
The Free Dictionary in the Internet defines a model as a description of a system, theory, or phenomenon (in the form of schematics or mathematics) that
accounts for its known or inferred properties and may be used for further study of its characteristics. Examples are the model of an atom, an economic model or a
business model.
Modelling is an essential and inseparable part of all scientific activity, and many scientific disciplines have their own ideas about specific types of modelling.
There is an increasing attention for scientific modelling in fields such as of systems theory, knowledge visualization and philosophy of science.
It is recalled that the system concept may be used to represent a physical, a biological, a socioeconomic as well as abstract entities. Thus, the entity to be studied
in scientific method may be considered as a system.
A scientific model seeks to represent empirical objects, phenomena, and physical processes or generally a system in a logical and objective way.
All models are simplified reflections of reality, but, despite their inherent falsity, they are nevertheless extremely useful. Building and disputing [to question or
doubt the truth or validity of something] models is fundamental to the scientific enterprise. Complete and true representation may be impossible, but scientific
debate often concerns which is the better model for a given task, e.g., which is the more accurate climate model for seasonal forecasting.

Attempts to formalize the principles of the empirical sciences use an interpretation to model reality in the same way logicians axiomatize the principles of logic.
The aim of these attempts is to construct a formal system that will not produce theoretical consequences that are contrary to what is found in reality. Predictions
or other statements drawn from such a formal system mirror or map the real world only insofar as these scientific models are true.
Modelling as a substitute for direct measurement and experimentation
Direct measurement of outcomes under controlled conditions in scientific method will always be more reliable than modelled estimates of outcomes.
Models are typically used when it is either impossible or impractical to create experimental conditions in which scientists can directly measure outcomes.
Models can also be used in companion with empirical experiments in order to facilitate and accelerate a study. By feeding findings of an experiment to a model
under development or an outcome of a model as an insightful manipulation of the experiment, studies can be accelerated towards convergence to solutions and
conclusions faster.
Reasons we do modeling and simulation:
We are constrained by linear thinking: We cannot understand how all the various parts of the system interact and add up to the whole
We cannot imagine all the possibilities that the real system could exhibit
We cannot foresee the full effects of cascading and/or layering upon of events with our limited mental models
We cannot foresee novel events that our mental models cannot even imagine
We model for insights, not numbers
As an exercise in thought space to gain insights into key variables and their causes and effects
To construct reasonable arguments as to why events can or cannot occur based on the model
We model to make qualitative or quantitative predictions about the future
The process of generating a model
Modelling refers to the process of generating a model as a conceptual representation of some phenomenon. Modelling requires selecting and identifying relevant
aspects of a situation in the real world. Typically a model will refer only to some aspects of the phenomenon in question. Two models of the same phenomenon
may be essentially different, that is to say that the differences between them are more than just a simple renaming of components.
Such differences may be due to differing requirements of the model'
s end users, or to conceptual or aesthetic differences among the modellers and to contingent
decisions made during the modelling process.
s preference for a reduced ontology (nature of being), preferences
Aesthetic considerations that may influence the structure of a model might be the modeller'
regarding probabilistic models vis-a-vis deterministic ones, discrete vs continuous time, etc. For this reason, users of a model need to understand the model'
s
original purpose and the assumptions made that are pertinent to its validity.
The process of evaluating a model
A model is evaluated first and foremost by its consistency to empirical data. Any model inconsistent with reproducible observations must be modified or rejected.
One way to modify the model is by restricting the domain over which it is credited with having high validity.
A case in point is Newtonian physics, which is highly useful except for the very small, the very fast, and the very massive phenomena of the universe. However,
a fit to empirical data alone is not sufficient for a model to be accepted as valid. Other factors important in evaluating a model include:
Ability to explain past observations

Ability to predict future observations
Cost of use, especially in combination with other models
Refutability, enabling estimation of the degree of confidence in the model
Simplicity, or even aesthetic appeal
One may attempt to quantify the evaluation of a model using a utility function.
Simulation and visualization
For the scientist, a model is also a way in which the human thought processes can be amplified. For instance, models that are rendered in software allow scientists
to leverage computational power to simulate, visualize, manipulate and gain intuition about the entity, phenomenon, or process being represented. Such computer
models are called in silico (an expression used to mean "performed on computer or via computer simulation.").
Other types of scientific model may be in vivo (living models, such as laboratory rats), in vitro (in glassware, such as tissue culture) and a lot more types in other
disciplines such as pilot plants.
Simulation
A simulation is the implementation of a model. A steady state simulation provides information about the system at a specific instant in time usually at
equilibrium. A dynamic simulation provides information over time. A simulation brings a model to life and shows how a particular object or phenomenon will
behave. Such a simulation can be useful for testing, analysis, or training in those cases where real-world systems or concepts can be represented by models.
Simulation approach involves the construction of an artificial environment within which relevant information and data can be generated. This permits an
observation of the dynamic behaviour of a system (or its sub-system) under controlled conditions.
The term simulation in the context of business and social sciences applications refers to the operation of a numerical model that represents the structure of a
dynamic process. Given the values of initial conditions, parameters and exogenous variables, a simulation is run to represent the behaviour of the process over
time.
Simulation approach can also be useful in building models for understanding future conditions.
Visualization is any technique for creating images, diagrams, or animations to communicate a message. Visualization through visual imagery has been an
effective way to communicate both abstract and concrete ideas since the dawn of man. Examples from history include cave paintings, Egyptian hieroglyphs,
Greek geometry, and Leonardo da Vinci'
s revolutionary methods of technical drawing for engineering and scientific purposes.

One application of scientific modelling is in the field of "Modelling and Simulation", generally referred to as "M&S". M&S has a spectrum of applications which
range from concept development and analysis, through experimentation, measurement and verification, to disposal analysis.
Examples of some computational, simulation and visualization tools in science and engineering include:
MATLAB,
MathCAD,
Mathematica,
LabView of NI
MultiSim,
Proteus
NS or Packet Tracer
etc
Mathematical model
A mathematical model is a description of a system using mathematical language. That is, a mathematical model relates the input variables and output variables of
a system representing a real world phenomenon in order to describe it in the form of one or a set of equations. Variables are abstractions of quantities of interest
in the described systems.
The process of developing a mathematical model is termed mathematical modelling.
Mathematical models are used not only in the natural sciences (such as physics, biology, earth science, meteorology) and engineering disciplines, but also in the
social sciences (such as economics, psychology, sociology and political science).
Physicists, engineers, statisticians, operations research analysts and economists use mathematical models most extensively.
Mathematical models can take many forms, including but not limited to dynamical systems, statistical models, differential equations, or game theoretic models.
These and other types of models can overlap, with a given model involving a variety of abstract structures.
The purpose of the mathematical model of the system may be at least either of the following:
i) Description: i.e. to sufficiently (accurately but simply) describe the characteristics of the real world physical phenomenon described as the model
when subjected to input as well as exogenous variables
ii) (Inferential) to explore how the system parameters can be varied/manipulated in order to obtain an optimal performance (response) for a given set
of input variables and a known set of external factors. Studying predictive (interpolating and extrapolating) accuracy is also another purpose of this
category.
In an optimization problem, the function to be optimized is called an objective function. The conditions that should be satisfied during optimization are called
constraints of the system. Objective functions will depend on the perspective of the model'
s user. Depending on the context, an objective function is also known
as an index of performance, as it is some measure of interest to the user. Although there is no limit to the number of objective functions and constraints a model
can have, using or optimizing the model becomes more involved (computationally) as the number increases.
Description and performance optimization are, thus, two important purposes of a mathematical model of a real world physical or abstract system.
Thus, often when engineers are interested to study a system to be controlled or optimized, they use a mathematical model. In analysis, engineers can build a
descriptive model of the system as a hypothesis of how the system could work, or try to estimate how an unforeseeable event could affect the system.
Similarly, in control of a system, engineers can try out different control approaches in simulations.
The mathematical model of a system may be classified in accordance to the type of system or operator. Thus we may have linear or non-linear models;
deterministic or non-deterministic (stochastic/probabilistic) model; static or dynamic model etc.

The system manifests itself as a mathematical operator on the input variables to deliver the output variables as a result. The type of operator depends on the type
of system. Thus we may have an algebraic operator, a differential operator etc. The system itself has parameters called endogenous variables. The system may
also be subjected to external undesired but inevitable factors such as noise manifested and are termed as exogenous variables.
If all the operators in a mathematical model exhibit linearity, the resulting mathematical model is defined as linear. A model is considered to be nonlinear
otherwise.
However, the question of linearity and nonlinearity is dependent on context, and linear models may have nonlinear expressions in them. For example, in a
statistical linear model, it is assumed that a relationship is linear in the parameters, but it may be nonlinear in the predictor variables.
Similarly, a differential equation is said to be linear if it can be written with linear differential operators, but it can still have nonlinear expressions in it.
In a mathematical programming model, if the objective functions and constraints are represented entirely by linear equations, then the model is regarded as a
linear model. If one or more of the objective functions or constraints are represented with a nonlinear equation, then the model is known as a nonlinear model.
Nonlinearity, even in fairly simple systems, is often associated with phenomena such as chaos and irreversibility. Although there are exceptions, nonlinear
systems and models tend to be more difficult to study than linear ones. A common approach to nonlinear problems is linearization, but this can be problematic if
one is trying to study aspects such as irreversibility, which are strongly tied to nonlinearity.
Mathematical models can be divided into either analytical models or numerical models.
An analytical model is a mathematical model that has a closed form expression, i.e. the output and input variables can be related in such a way that the two sets
of variables are disassociated from themselves or expressed as a mathematical analytic function.
On the other hand, a numerical model is a mathematical model that uses some sort of numerical time-stepping procedure to obtain the models behavior over time.
Eg recursive methods
A priori information
Mathematical modelling problems are often classified into black box or white box models, according to how much a priori information is available of the system.
A black-box model is a system of which there is no a priori information available. A white-box model (also called glass box or clear box) is a system where all
necessary information is available. Practically all systems are somewhere between the black-box and white-box models, so this concept is useful only as an
intuitive guide for deciding which approach to take.
Usually it is preferable to use as much a priori information as possible to make the model more accurate. Therefore the white-box models are usually considered
easier, because if you have used the information correctly, then the model will behave correctly. Often the a priori information comes in forms of knowing the
type of functions relating different variables.
For example, suppose we want to make a model of how a medicine works in a human system knowing the fact that (i.e. a priori information) usually the amount
of medicine in the blood is an exponentially decaying function. But we are still left with several unknown parameters; how rapidly does the medicine amount
decay, and what is the initial amount of medicine in blood? This example is therefore not a completely white-box model. These parameters have to be estimated
through some means (say experiment) before one can use the model.
In black-box models one tries to estimate both the functional form of relations between variables (input and output) and the numerical parameters (of the system)
in those functions. Using a priori information we could end up, for example, with a set of functions that probably could describe the system adequately. If there is
no a priori information we would try to use functions as general as possible to cover all different models.
An often used approach for black-box models are neural networks which usually do not make assumptions about incoming data. The problem with using a large
set of functions to describe a system is that estimating the parameters becomes increasingly difficult when the amount of parameters (and different types of
functions) increases.
Any model which is not pure white-box contains some parameters that can be used to fit the model to the system it is intended to describe. If the modelling is
done by a neural network, the optimization of parameters is called training. In more conventional modelling through explicitly given mathematical functions,
parameters are determined by curve fitting.

Model complexity vs accuracy
In general, model complexity involves a trade-off between simplicity and accuracy of the model. Occam's Razor is a principle particularly relevant to modelling;
the essential idea being that among models with roughly equal predictive power, the simplest one is the most desirable.
While added complexity usually improves the realism of a model, it can make the model difficult to understand and analyze, and can also pose computational
problems, including numerical instability. Thomas Kuhn argues that as science progresses, explanations tend to become more complex before a Paradigm shift
offers radical simplification.
That is, when a proposed solution hypothesis of a problem is not sufficient in its explanation of essential scenarios of the problem at hand, then the hypothesis
needs to be either modified or altered by a new one (paradigm shift).
For example, when modelling the flight of an aircraft, we could embed each mechanical part of the aircraft into our model and would thus acquire an almost
white-box model of the system. However, the computational cost of adding such a huge amount of detail would effectively inhibit the usage of such a model.
Additionally, the uncertainty would increase due to an overly complex system, because each separate part induces some amount of variance into the model. It is
therefore usually appropriate to make some approximations to reduce the model to a sensible size. Engineers often can accept some approximations in order to
get a more robust and simple model.
For example Newton'
s classical mechanics is an approximated model of the real world. Still, Newton'
s model is quite sufficient for most ordinary-life situations,
that is, as long as particle speeds are well below the speed of light, and we study macro-particles only.
Model evaluation
A crucial part of the modelling process is the evaluation of whether or not a given mathematical model describes a system accurately. This question can be
difficult to answer as it involves several different types of evaluation.
Fit to empirical data
Usually the easiest part of model evaluation is checking whether a model fits experimental measurements or other empirical data. In models with parameters, a
common approach to test this fit is to split the data into two disjoint subsets: training data and verification data. The training data are used to estimate the model
parameters. An accurate model will closely match the verification data even though this data was not used to set the model'
s parameters. This practice is referred
to as cross-validation in statistics.
Defining a metric to measure distances between observed (measured) and predicted data is a useful tool of assessing model fit. In statistics, decision theory, and
some economic models, a loss function plays a similar role.
While it is rather straightforward to test the appropriateness of parameters, it can be more difficult to test the validity of the general mathematical form of a
model. In general, more mathematical tools have been developed to test the fit of statistical models than models involving Differential equations. Tools from
nonparametric statistics can sometimes be used to evaluate how well data fits a known distribution or to come up with a general model that makes only minimal
assumptions about the model'
s mathematical form.
Scope of the model
Assessing the scope of a model, that is, determining what situations the model is applicable to, can be less straightforward. If the model was constructed based on
a set of data, one must determine for which systems or situations the known data is a "typical" set of data.
The question of whether the model describes well the properties of the system between data points is called interpolation, and the same question for events or
data points outside the observed data is called extrapolation.
As an example of the typical limitations of the scope of a model, in evaluating Newtonian classical mechanics, we can note that Newton made his measurements
without advanced equipment, so he could not measure properties of particles traveling at speeds close to the speed of light. Likewise, he did not measure the
movements of molecules and other small particles, but macro particles only. It is then not surprising that his model does not extrapolate well into these domains,
even though his model is quite sufficient for ordinary life physics.

Philosophical considerations
Many types of modelling implicitly involve claims about causality. This is usually (but not always) true of models involving differential equations. As the
purpose of modelling is to increase our understanding of the world, the validity of a model rests not only on its fit to empirical observations, but also on its
ability to extrapolate to situations or data beyond those originally described in the model.
One can argue that a model is worthless unless it provides some insight which goes beyond what is already known from direct investigation of the phenomenon
being studied.
Examples of mathematical models in engineering, science, demography, economics etc
Population Growth. A simple (though approximate) model of population growth is the Malthusian growth model, sometimes called the simple
exponential growth model, is essentially exponential growth based on a constant rate of compound interest. The model is named after the Reverend
Thomas Malthus, who authored An Essay on the Principle of Population, one of the earliest and most influential books on population and exponential
growth.
where
P0 = initial population, r = growth rate, sometimes also called Malthusian parameter, t = time.
Model of a particle in a potential-field. In this model we consider a particle as being a point of mass which describes a trajectory in space which is
modeled by a function giving its coordinates in space as a function of time. The potential field is given by a function V : R3 R and the trajectory is
a solution of the differential equation
Note this model assumes the particle is a point mass, which is certainly known to be false in many cases in which we use this model; for example, as a
model of planetary motion.
Model of rational behavior of a consumer. In this model we assume a consumer faces a choice of n commodities labeled 1,2,...,n each with a market
price p1, p2,..., pn. The consumer is assumed to have a cardinal utility function U (cardinal in the sense that it assigns numerical values to utilities),
depending on the amounts of commodities x1, x2,..., xn consumed. The model further assumes that the consumer has a budget M which is used to
purchase a vector x1, x2,..., xn in such a way as to maximize U(x1, x2,..., xn). The problem of rational behavior in this model then becomes an
optimization problem, that is:
subject to:
This model has been used in general equilibrium theory, particularly to show existence and Pareto efficiency of economic equilibria. However, the
fact that this particular formulation assigns numerical values to levels of satisfaction is the source of criticism (and even ridicule). However, it is not
an essential ingredient of the theory and again this is an idealization.

Statistics and statistical modelling
... it is only the manipulation of uncertainty that interests us. We are not concerned with the matter that is uncertain. Thus we do not study the mechanism of
rain; only whether it will rain. Dennis Lindley, "The Philosophy of Statistics", The Statistician (2000).
Overview of statistics
Statistics is the science of the collection, organization, and interpretation of data. It deals with all aspects of this, including the planning of data collection
(generation) in terms of the design of experiments and surveys.
The mathematical foundations of statistics were laid in the 17th century with the development of probability theory by Blaise Pascal and Pierre de Fermat.
Probability theory arose from the study of games of chance. The method of least squares was first described by Carl Friedrich Gauss around 1794. The use of
modern computers has expedited large-scale statistical computation, and has also made possible new methods that are impractical to perform manually.
Because of its empirical roots and its focus on applications, statistics is usually considered to be a distinct mathematical science rather than a branch of
mathematics.
The data and variables mostly dealt with statistics are random or stochastic in their nature. This means that a random or a stochastic data or variable is one whose
magnitude and/or occurrence is uncertain and estimated only probabilistically. Random variables are generated from a random process or equivalently from a
non-deterministic system.
Statistics is thus closely related to probability theory, with which it is often grouped. Randomness is studied using probability theory, which is also a
mathematical discipline. Probability is used to study the sampling distributions of sample statistics and, more generally, the properties of statistical procedures.
The use of any statistical method is valid when the system or population under consideration satisfies the assumptions of the method.
The quality of (collected or generated) data is improved by carefully employing two important techniques: design of experiments and survey sampling. Statistics
is not only used to describe about an entity but also provides tools for prediction and forecasting using data and statistical models.
Statistics is applicable to a wide variety of academic disciplines, including natural and social sciences, government, and business. Statistics also form a key basis
tool in business and manufacturing. It is used to understand measurement systems variability, control processes (as in statistical process control or SPC), for
summarizing data, and to make data-driven decisions. In these roles, it is a key tool, and perhaps the only reliable tool.
Population and sample
In applying statistics to a scientific, industrial, or societal problem, it is necessary to begin with a population or process to be studied. Populations can be diverse
topics such as "all persons living in a country" or "every atom composing a crystal". A population can also be composed of observations of a process at various
times, with the data from each observation serving as a different member of the overall group. Data collected about this kind of "population" constitutes what is
called a time series.
For practical reasons, a chosen subset of the population called a sample is studied as opposed to compiling data about the entire group (an operation called
census). Once a sample that is representative of the population is determined, data is collected for the sample members in an experimental or observational
setting. This data can then be subjected to statistical analysis, serving two related purposes: description and inference.
For a sample to be used as a guide to an entire population, it is important that it is truly a representative of that overall population. Representative sampling
assures that the inferences and conclusions can be safely extended from the sample to the population as a whole. A major problem lies in determining the extent
to which the sample chosen is actually representative. Statistics offers methods to estimate and correct for any random trending within the sample and data
collection procedures. There are also methods for designing experiments that can lessen these issues at the outset of a study, strengthening its capability to
discern truths about the population. Statisticians describe stronger methods as more "robust".
Two main statistical methods (purposes) [for analysis and synthesis]
Descriptive statistics and inferential statistics (a.k.a., predictive statistics) form the two main methods and together comprise applied statistics to which sample
data from experiment or observation are subjected.
Descriptive statistics: summarizes the population data by describing what was observed in the sample numerically or graphically. Numerical
descriptors include mean, variance and standard deviation for continuous data types (like heights or weights), while frequency and percentage are
more useful in terms of describing categorical data (like race). This method is useful in research, when communicating the results of experiments.
Inferential statistics: uses patterns in the sample data to draw inferences (predictions) about the population represented, accounting for randomness.
These inferences may take the form of:
modeling relationships within the data (regression), extrapolation, interpolation, or other modeling techniques like ANOVA (Analysis of
variance), time series, and data mining. This means patterns in the data may be modeled in a way that accounts for randomness and
uncertainty in the observations, and are then used to draw inferences about the process (system) or population being studied;
answering yes/no questions about the data (hypothesis testing),
estimating numerical characteristics of the data (estimation),
describing associations within the data (correlation),
Inference is a vital element of scientific advance, since it provides a prediction (based on data) for where a theory logically leads. To further prove the guiding
theory, these predictions are tested (i.e hypothesis testing) as well, as part of the scientific method. If the inference holds true, then the descriptive statistics of the
new data increase the soundness of that hypothesis.
The concept of correlation is particularly noteworthy for the potential confusion it can cause. Statistical analysis of a data set often reveals that two variables
(properties) of the population under consideration tend to vary together, as if they were connected. For example, a study of annual income that also looks at age
of death might find that poor people tend to have shorter lives than affluent people. The two variables are said to be correlated; however, they may or may not be
the cause of one another. The correlation phenomena could be caused by a third, previously unconsidered phenomenon, called a lurking variable or confounding
variable. For this reason, there is no way to immediately infer the existence of a causal relationship between the two variables. Correlation does not imply
causation.
Experimental and observational studies
A common goal for a statistical research project is to investigate causality, and in particular to draw a conclusion on the effect of changes in the values of
predictors or independent variables on dependent variables or response.
There are two major types of causal statistical studies: experimental studies and observational studies. In both types of studies, the effect of differences of an
independent variable (or variables) on the behavior of the dependent variable are observed. The difference between the two types lies in how the study is actually
conducted. Each can be very effective.
An experimental study involves taking measurements of the system under study, manipulating the system, and then taking additional measurements using the
same procedure to determine if the manipulation has modified the values of the measurements.
In contrast, an observational study does not involve experimental manipulation. Instead, data are gathered and correlations between predictors and response are
investigated.
Experiments
The basic steps of a statistical experiment are:
1.
Planning the research, including finding the number of replicates of the study, using the following information:
Preliminary estimates regarding the size of treatment effects, alternative hypotheses, and the estimated experimental variability.
Consideration of the selection of experimental subjects and the ethics of research is necessary.
Statisticians recommend that experiments compare (at least) one new treatment with a standard treatment or control, to allow an unbiased estimate of
the difference in treatment effects.
2.
Design of experiments, using blocking to reduce the influence of confounding variables, and randomized assignment of treatments to subjects to
allow unbiased estimates of treatment effects and experimental error.
At this stage, the experimenters and statisticians write the experimental protocol that shall guide the performance of the experiment and that specifies
the primary analysis of the experimental data.

3.
Performing (execute) the experiment and analyzing the data following the experimental protocol.
4.
Further examining the data set in secondary analyses, to suggest new hypotheses for future study.
5.
Documenting and presenting the results of the study.
Experiments on human behavior have special concerns. The famous Hawthorne study examined changes to the working environment at the Hawthorne plant of
the Western Electric Company.
The researchers were interested in determining whether increased illumination would increase the productivity of the assembly line workers. The researchers first
measured the productivity in the plant, then modified the illumination in an area of the plant and checked if the changes in illumination affected productivity.
It turned out that productivity indeed improved (under the experimental conditions). However, the study is heavily criticized today for errors in experimental
procedures, specifically for the lack of a control group and blindness. The Hawthorne effect refers to finding that an outcome (in this case, worker productivity)
changed due to observation itself. Those in the Hawthorne study became more productive not because the lighting was changed but because they were being
observed.
Observational study
An example of an observational study is one that explores the correlation between smoking and lung cancer. This type of study typically uses a survey to collect
observations about the area of interest and then performs statistical analysis. In this case, the researchers would collect observations of both smokers and nonsmokers, perhaps through a case-control study, and then look for the number of cases of lung cancer in each group.
Levels of measurement
There are four main levels of measurement used in statistics:
ratio
interval,
ordinal and
nominal,
They have different degrees of usefulness in statistical research.

Ratio measurements have both a meaningful zero value and the distances between different measurements defined; they provide the greatest flexibility in
statistical methods that can be used for analyzing the data.
Interval measurements have meaningful distances between measurements defined, but the zero value is arbitrary (as in the case with longitude and temperature
measurements in Celsius or Fahrenheit).
Ordinal measurements have imprecise differences between consecutive values, but have a meaningful order to those values.
Nominal measurements have no meaningful rank order among values.
Because variables conforming only to nominal or ordinal measurements cannot be reasonably measured numerically, sometimes they are grouped together as
categorical variables.
On the other hand ratio and interval measurements are grouped together as quantitative or continuous variables due to their numerical nature.
Key terms used in statistics: Null hypothesis, alternative hypothesis and hypothesis testing
A hypothesis is a suggested explanation of a phenomenon.
Interpretation of statistical information can often involve the development of a null hypothesis in that the assumption is that whatever is proposed as a cause has
no effect on the variable being measured.

A null hypothesis is a hypothesis which a researcher tries to disprove. Normally, the null hypothesis represents the current view/explanation of an aspect of the
world that the researcher wants to challenge.
Research methodology involves the researcher providing an alternative hypothesis, a research hypothesis, as an alternate way to explain the phenomenon.
The researcher tests the hypothesis to disprove the null hypothesis, not because he/she loves the research hypothesis, but because it would mean coming closer to
finding an answer to a specific problem. The research hypothesis is often based on observations that evoke suspicion that the null hypothesis is not always
correct.
The best illustration for a novice is the predicament encountered by a jury trial. The null hypothesis, H0, asserts that the defendant is innocent, whereas the
alternative hypothesis, H1, asserts that the defendant is guilty.
The indictment comes because of suspicion of the guilt. The H0 (status quo) stands in opposition to H1 and is maintained unless H1 is supported by
evidence beyond a reasonable doubt.
However, failure to reject H0 in this case does not imply innocence, but merely that the evidence was insufficient to convict.
So the jury does not necessarily accept H0 but fails to reject H0. While to the casual observer the difference appears moot, misunderstanding the difference is one
of the most common and arguably most serious errors made by non-statisticians.
Failure to reject the H0 does NOT prove that the H0 is true, as any crook with a good lawyer who gets off because of insufficient evidence can attest to.
While one can not prove a null hypothesis one can test how close it is to being true with a power test, which tests for type II errors.
Error (i.e. errors in research)
Working from a null hypothesis, logically there are two basic forms of errors that are recognized when drawing conclusions in research:
Type I errors:
where the null hypothesis is falsely rejected giving a "false positive".
That is, it is when we accept the alternative hypothesis when the null hypothesis is in fact correct.
Type II errors:
where the null hypothesis fails to be rejected and an actual difference between populations is missed.
That is, when we reject the alternative hypothesis even if the null hypothesis is wrong.
Error also refers to the extent to which individual observations in a sample differ from a central value, such as the sample or population mean. Many statistical
methods seek to minimize the mean-squared error, and these are called "methods of least squares."
Measurement processes that generate statistical data are also subject to error. Many of these errors are classified as random (noise) or systematic (bias), but other
important types of errors (e.g., blunder, such as when an analyst reports incorrect units) can also be important.
Confidence intervals
Most studies will only sample part of a population and then the result is used to interpret the null hypothesis in the context of the whole population. Any
estimates obtained from the sample only approximate the population value.
Confidence intervals allow statisticians to express how closely the sample estimate matches the true value in the whole population. Often they are expressed as
95% confidence intervals. Formally, a 95% confidence interval of a procedure is a range where, if the sampling an analysis were repeated under the same
conditions, the interval would include the true (population) value 95% of the time. This does not imply that the probability that the true value is in the confidence
interval is 95%. (From the frequentist perspective, such a claim does not even make sense, as the true value is not a random variable. Either the true value is or is
not within the given interval.)

One quantity that is in fact a probability for an estimated value is the credible interval from Bayesian statistics.
Statistical Significance
Statistics rarely give a simple Yes/No type answer to the question asked of them. Interpretation often comes down to the level of statistical significance applied to
the numbers and often refer to the probability of a value accurately rejecting the null hypothesis (sometimes referred to as the p-value).
Referring to statistical significance does not necessarily mean that the overall result is significant in real world terms. For example, in a large study of a drug it
may be shown that the drug has a statistically significant but very small beneficial effect, such that the drug will be unlikely to help the patient in a noticeable
way.
Some well-known statistical tests and procedures are:
Analysis of variance (ANOVA)

Chi-square test
Correlation
Factor analysis
Student'
s t-test
Time series analysis
Statistical computing
The rapid and sustained increases in computing power starting from the second half of the 20th century have had a substantial impact on the practice of statistical
science.
Early statistical models were almost always from the class of linear models. However, nowadays, powerful computers, coupled with suitable numerical
algorithms, caused an increased interest in nonlinear models (such as neural networks) as well as the creation of new types, such as generalized linear models
and multilevel models.
Increased computing power has also led to the growing popularity of computationally intensive methods based on resampling, such as permutation tests and the
bootstrap, while techniques such as Gibbs sampling have made use of Bayesian models more feasible.
The computer revolution has implications for the future of statistics with new emphasis on "experimental" and "empirical" statistics. A large number of both
general and special purpose statistical software are now available such as gretl, which is an example of an open source statistical package.
Misuse of statistics
Misuse of statistics can produce subtle, but serious errors in description and interpretation subtle in the sense that even experienced professionals make such
errors, and serious in the sense that they can lead to devastating decision errors. For instance, social policy, medical practice, and the reliability of structures like
bridges all rely on the proper use of statistics. Even when statistics are correctly applied, the results can be difficult to interpret for those lacking expertise.
The statistical significance of a trend in the data which measures the extent to which a trend could be caused by random variation in the sample may or
may not agree with an intuitive sense of its significance. The set of basic statistical skills (and skepticism) that people need to deal with information in their
everyday lives properly is referred to as statistical literacy.
There is a general perception that statistical knowledge is all-too-frequently intentionally misused by finding ways to interpret only the data that are favorable to
the presenter.
The famous saying, "There are three kinds of lies: lies, damned lies, and statistics". which was popularized in the USA by Samuel Clemens, has come to
represent the general mistrust [and misunderstanding] of statistical science.
Harvard President Lawrence Lowell wrote in 1909 that statistics, "...like veal pies, are good if you know the person that made them, and are sure of the
ingredients."
If various studies appear to contradict one another, then the public may come to distrust such studies. For example, one study may suggest that a given diet or
activity raises blood pressure, while another may suggest that it lowers blood pressure. The discrepancy can arise from subtle variations in experimental design,

such as differences in the patient groups or research protocols, which are not easily understood by the non-expert. (Media reports usually omit this vital
contextual information entirely, because of its complexity.)
By choosing (or rejecting, or modifying) a certain sample, results can be manipulated. Such manipulations need not be malicious or devious; they can arise from
unintentional biases of the researcher. The graphs used to summarize data can also be misleading.
Deeper criticisms come from the fact that the hypothesis testing approach, widely used and in many cases required by law or regulation, forces one hypothesis
(the null hypothesis) to be "favored," and can also seem to exaggerate the importance of minor differences in large studies. A difference that is highly statistically
significant can still be of no practical significance. (See criticism of hypothesis testing and controversy over the null hypothesis.)
One response is by giving a greater emphasis on the p-value than simply reporting whether a hypothesis is rejected at the given level of significance. The p-value,
however, does not indicate the size of the effect. Another increasingly common approach is to report confidence intervals. Although these are produced from the
same calculations as those of hypothesis tests or p-values, they describe both the size of the effect and the uncertainty surrounding it.
Statistical modelling
A statistical model is a formalization of relationships between variables in the form of mathematical equations. A statistical model describes how one or more
random variables are stochastically related to one or more random variables.
Statistical models include issues such as statistical characterization of numerical data, estimating the probabilistic future behavior of a system based on past
behavior, extrapolation or interpolation of data based on some best-fit, error estimates of observations, or spectral analysis of data or model generated output.
In mathematical terms, a statistical model is frequently thought of as a pair (Y, P) where Y is the set of possible observations and P is the set of possible
probability distributions on Y. It is assumed that there is a distinct element of P which generates the observed data. Statistical inference enables us to make
statements about which element(s) of this set are likely to be the true one.
Most statistical tests can be described in the form of a statistical model. For example, the Student'
s t-test for comparing the means of two groups can be
formulated as seeing if an estimated parameter in the model is different from 0. Another similarity between tests and models is that there are assumptions
involved. Error is assumed to be normally distributed in most models.[1]
More formally a statistical model, P, is a collection of probability distribution or density functions. A parametric model is a collection of distributions, each of
which is indexed by a unique finite-dimensional parameter:
of parameters, which is a subset of d-dimensional Euclidean space.
, where
is a parameter and
is the feasible region
A statistical model may be used to describe the set of distributions from which one assumes that a particular data set is sampled. For example, if one assumes that
data arise from a univariate Gaussian distribution, then one has assumed a Gaussian model:
.
Performance of a student class, for instance, can best be modelled using normal distribution.
Arrival of a customer in a bank queue or call making to a telephone network is modelled using Poison (exponentially distributed) distribution.
Other models include Rayleigh distribution, etc.
A non-parametric model is a set of probability distributions with infinite dimensional parameters, and might be written as
A semi-parametric model also has infinite dimensional parameters, but is not dense in the space of distributions. For example, a mixture of Gaussians with one
Gaussian at each data point is dense is the space of distributions. Formally, if d is the dimension of the parameter, and n is the number of samples, if
as
and
as
, then the model is semi-parametric.

Comparison of statistical models
Models can be compared to each other. This can either be done when you have done an exploratory data analysis or a confirmatory data analysis.
In an exploratory analysis, you formulate all models you can think of, and see which describes your data best.
In a confirmatory analysis you test which of your models you have described before the data was collected fits the data best, or test if your only model fits the
data.
In linear regression analysis you can compare the amount of variance explained by the independent variables, R2, across the different models.
In general, you can compare models that are nested by using a Likelihood-ratio test. Nested models are models that can be obtained by restricting(setting) a
parameter in a more complex model to be zero.
Example
Length and age are probabilistically distributed over humans. They are stochastically related, when you know that a person is of age 7, this influences the chance
of this person being 6 feet tall. You could formalize this relationship in a linear regression model of the following form: lengthi = b0 + b1agei + i, where b0 is the
intercept, b1 is a parameter that age is multiplied by to get a prediction of length, is the error term, and i is the subject.
This means that length starts at some value, there is a minimum length when someone is born, and it is predicted by age to some amount. This prediction is not
perfect as error is included in the model. This error contains variance that stems from sex and other variables.
When sex is included in the model, the error term will become smaller, as you will have a better idea of the chance that a particular 16-year-old is 6 feet tall when
you know this 16-year-old is a girl.
The model would become lengthi = b0 + b1agei + b2sexi + i, where the variable sex is dichotomous. This model would presumably have a higher R2. The first
model is nested in the second model: the first model is obtained from the second when b2 is restricted to zero.
Classification of statistical models
According to the number of the endogenous variables and the number of equations, models can be classified as complete models (the number of equations equals
to the number of endogenous variables) and incomplete models.
Some other statistical models are the general linear model (restricted to continuous dependent variables), the generalized linear model (for example, logistic
regression), the multilevel model, and the structural equation model.

Empirical modeling
Empirical models are those (mathematical or graphical expressions) that are based entirely on data. The important distinction between empirical models and
analytical models is that the empirical models are not derived from assumptions concerning the relationship between variables and they are not based on
physical principles.
The first step in deriving an empirical models is thus to get the scatter-plot of the data.
Interaction between data and models occurs in a number of ways:
Data are needed to suggest a right model. The models called empirical are based entirely on data. [training data]
Data are needed to estimate the values of the parameters appearing in a model. This is sometimes called calibrating a model.
Data are needed to test a model. [verification/testing data]
It happens very often that the data given at the beginning is not sufficient for making a good model. In these cases further data collection is needed. Considering
the following questions might be useful:
What is the relevant data? Exactly what kind of data is needed?
How can the relevant data be obtained?
In what form do you need the data?
Once the data is collected, you need to decide on the techniques you want to use in order to find an appropriate model. There are two major groups of techniques
based on two different ideas
Interpolation- finding a function that contains all the data points.
Model fitting - finding a function that is as close as possible to containing all the data points. Such function is also called a regression curve.
Sometimes you would need to combine these methods since the interpolation curve might be too complex and the best fit model might not be sufficiently
accurate.
Model Fitting: Modeling using Regressions
Most software applications such as Excel, Matlab and also graphing calculators can be used to find regression curves and a variable monitoring the validity of the
model, the coefficient of determination usually denoted by R2. This coefficient takes values in interval [0,1] and indicates how close the data points are to be
exactly on the regression curve. If R2 is close to 1, the model is reliable. If R2 is close to 0, other model should be considered.
Some possible types models for curve fitting
i. Linear: y = ax+b: Easiest, simplest, used very frequently. A simple test can be performed in order to determine if data is linear: if independent variable
values are equally spaced, simply check if difference of consecutive y-values is the same. If b is sufficiently small, y is said to
be proportional to x i.e. (y x)
ii.
Quadratic: y = ax2 + bx + c: Appropriate for fitting data with one minimum or one maximum. To find out if equally spaced data is quadratic, check if the
differences of the successive differences of consecutive y-values are constant. If a > 0, this function is concave up; if a < 0, it is
concave down.
iii.
Cubic: y = ax3 + bx2 + cx + d: Appropriate for fitting data with one minimum and one maximum.
iv.
Quartic: y = ax4 + bx3 + cx2 + dx + e: Convenient for fitting data with two minima and one maximum or two maxima and one minimum. When working
with polynomial models a thing to keep in mind: balance between complexity and precision. For examples, see section Testing
the effectiveness of a model. Also, monitor the long term behavior and check how realistic it is. For example, see sections
Testing the validity and Choosing the right model.
v.
Exponential: y = abx or y = aekx: While a linear function has constant average rate of change (a constant difference of two consecutive y-values), an
exponential function has constant percent (relative) rate of change (a constant quotient of two consecutive y-values). Thus, an
easy test to check if data is linear: if independent variable values are equally spaced, simply check if quotient of consecutive yvalues is the same. If k > 0, then the function is increasing and concave up. If k < 0, then the function is decreasing and
concave up. This model is appropriate if the increase is slow at first but then it speeds up (or, if k < 0 if the decrease is fast at
first but then slows down).
vi.
Logarithmic: y = a + b ln x: If b > 0, then the function is increasing and concave down. If b < 0, then the function is decreasing and concave up. If the data
indicates an increase, this model is appropriate if the increase is fast at first but then it slows down.
vii.
Logistic: y = c/(1+ae-bx) : Increasing for b > 0: In this case, the increase is slow at first, then it speeds up and then it slows down again and approaches the yvalue c for x
.

viii.
Power y = axb: If a > 0, y is increasing for b > 0 and decreasing for b < 0. It is called a power model since an increase of x by factor of t causes an increase of
y by the power tb of t (for b > 0). Increasing power function will not increase as rapidly as an increasing exponential function.
Connection with linear model: If y = axb, then ln y = ln a + b ln x. So, if y is a power
function, ln y is a linear function of ln x. Note: If y = abx, then ln y = ln a + x ln b. So, if y is
an exponential function, ln y is a linear function of x.
In conclusion:
ix. Sine y = a sin(bx+c)+d: Appropriate for periodic data. In the formula, a is the amplitude, b/360 the period, c/b the horizontal shift, and d the vertical shift.
Modeling with piecewise defined functions
A piecewise defined function is a function defined by different formulas for different values of the independent variable. Sometimes, it is better to model a given
data using one piecewise defined function than with a single function.
Analytic Methods of Model fitting
In this section, we look into the mathematics behind obtaining regression models and curve fitting.
Least-Squares Criterion
This criterion is the most frequently used curve- fitting criterion. It is based on an idea that in order to fit a set of data points (xi , yi), i = 1, , m onto a curve y =
f(x), we again want the differences between yi and f(xi) to be as small as possible.
Recall that the square of distance of the vector between the observed value yi and the predicted value f(xi) is given by
Thus, a way to make the differences yi - f(xi) small is to minimize the sum
Least-Squares Line Fit.

Suppose that we would like to fit the points (xi, yi); i = 1, . . ., m onto a line y = ax+b. So, we need to minimize the sum
This will be satisfied when
and
We can view the equations above as two linear equations in the unknowns a and b. Solving them for a and b, we obtain

The two equations above are called the normalizing equations. It is not hard to write a computer code that will compute these values.
Least-Squares Power Fit
Suppose that a positive integer n is fixed and that we would like to fit the points (xi, yi), i = 1, . . ., m onto a curve of the form y = axn. So, we need to minimize
the sum
This will be satisfied when
Solving this equation for a, we obtain
Transformed Least-Squares Fit.

The formula computing the coefficients of the linear and power least-squares fits is not too complex because the partial derivatives considered were linear
functions of the unknown coefficients.
With exponential, logarithmic or power model (with n not fixed in y = axn), the equations determined by the partial derivatives are not linear equations in
unknown coefficients. In cases like this, one may modify the data (considering ln of one or both of the variables instead of the original data) in order to reduce a
non-linear model to a linear one.
Thus
i) Finding an exponential model y = beax using the transformed data
Taking ln of y = beax, obtain that ln y = ax + ln b = ax + B for ln b = B. Thus, you can find a linear model for (xi, ln yi) instead of (xi, yi), i = 1, , m. Note that
in this case
ii) Finding a logarithmic model y = a ln x + b using the transformed data

Simply consider (xi, yi) = (ln xi, yi) instead of (xi, yi); i = 1, . . ., m and find the linear model for the transformed set of data. The linear model for y = aX + b is the
logarithmic model that we are looking for. Note that in this case
iii) Finding a power model y = bxa using the transformed data.

Taking ln of y = bxa, gives you ln y = a ln x + ln b. Consider (ln xi, ln yi) instead of (xi, yi), i = 1, , m and find the linear model for this transformed set of data.
Denote ln b = B. The linear model for Y = ln y = a ln x + ln b = aX + B can be found as:
An important fact to note is that if the partial derivative equations are used to find the unknown coefficients a and B in y = Beax, y = a ln x + b or y = Bxa without
transforming the original data, we would end up with different models than the models that we get when we transform the data and find a linear regression for the
new data. It is important to keep in mind how different software applications handle the data.
Transforming the original data often simplifies the formulas even for models with less parameters than exponential, power or logarithmic. The next example
illustrates this.
Example 1: Find the formula computing the coefficient a for the quadratic fit y = ax2 and use it to find a transformed quadratic fit for the data below:
Take ln of y = ax2. Obtain then ln y = ln a + 2 ln x. Let us denote ln a by A. Thus, we need to consider ln y = A + 2 ln x and to minimize
The partial derivative is
Setting this derivative to 0 and solving for A gives us
which can be easily solved for a.

Using the data from Example 1, we obtain that A = 1.1432 and so a = 3.1368. Compare this model y = 3.1368x2 with the model y = 3.1869x2 obtained using the
direct power curve fit.
Measuring the validity of a model
Let us first see the Chebyshev Approximation Criterion.
Let us assume that m points (xi, yi), i = 1, , m are given and that we need to fit them on a curve y = f(x). As we would like the differences between yi and f(xi)
to be as small as possible, the idea is to minimize the largest absolute value |yi - f(xi)|.
Thus, the only difference between the least square fit and Chebyshevs method is that instead of the sum of squares of yi - f(xi), we are minimizing the maximum
of the absolute values of yi - f(xi).
Following steps achieve this goal.
1. Let ri = yi - f(xi) for i = 1, , m. The variables ri are called residuals.
2. Find the maximum of |ri| for i = 1, ,m. Let us call this quantity r.
3. Solve the following optimization problem: minimize r subject to constraints -r
r - ri 0, r + ri 0 for i = 1, ..., m.
ri
r for i = 1, , m i.e.
Example 2. Suppose that a point B is somewhere inside a line segment AC. Assume that measuring the distances AB, BC and AC, we obtain that AB = 13, BC =
7 and AC = 19. As 13 + 7 = 20 19, we need to resolve the discrepancy.
We can do that using the Chebyshev approximation criterion. Let x1, x2 and x3 stand for exact lengths of AB, BC and AC respectively. Then r1 = x1 13, r2 = x2
7, and r3 = x1 + x2 19.
Let r stands for the maximum of the absolute values of r1, r2 and r3 (note that we don'
t know which one is r). We would like to minimize r subject to
r - x1 + 13
0, r + x1 - 13
0,
r - x2 + 7
0, r + x2 - 7

r - x1 - x2 + 19
0, r + x1 + x2 - 19
Using the methods of linear programming, we obtain that r = 1/3, x1 = 12 &2/3 and x2 = 6 & 2/3.
Although in this example linear programming is used, note that in some cases the constraints will not be linear equations and other optimization methods should
be used.
Now let us come back to Measuring the Validity of a Model
In the previously seen examples, two different quadratic power models y = 3.1368x2 and y = 3.1869x2 are obtained for the same set of data. If Chebyshev
criterion is to be used for the same set of data, it would yield yet another model y = 3.17073x2 for the same set of data.
A natural question is: how can we choose the best model?
Before we attempt to answer, let us introduce some notation. If the data (xi, yi), i = 1, . . ., m is to be fit on the curve y = f1(x) obtained using the Chebyshev
criterion, let ci denote the absolute deviations | yi - f(xi)|. If the same data is to be fit on the curve y = f2(x) obtained using the least-squares criterion, let di denote
the absolute deviations |yi - f2(xi)|. Also, let cmax denote the largest of ci and dmax denote the largest of di. Since the least-squares criterion is such that the squares
of the deviations are minimal, we have that
Thus,
Also, since the Chebyshev criterion is such that the maximum of the absolute deviations is minimal, we have that cmax dmax. Thus,
D cmax dmax
This last equation can help us determine which model to use: the least-squares criterion is convenient to use but there is always to concern that the difference
between D and dmax might be too large. Balancing these two conditions might help us decide which model to use.
Let us go back to the example with three different quadratic power models. Computing the deviations for all three models we conclude
1. For the model y = 3.1869x2 obtained using the least-squares fit. The largest absolute deviation is 0.347. The smallest is 0.0976.
2. For the model y = 3.1368x2 obtained using transformed least-squares fit. The largest absolute deviation is 0.495. The smallest is 0.0842. This model was the
easiest to compute but has the largest absolute deviation.
3. For the model y = 3.17073x2 obtained using Chebyshev criterion. The largest absolute deviation is 0.282. The smallest is 0.0659. Computationally, this model
was the hardest to get but has the smallest absolute deviation.
However, considering the sum of squares of deviations instead of the maximum of the absolute deviations, the least-squares fit model is better than the
Chebyshev (.2095 for the least squares versus .2256 for Chebyshev).
The conclusion that we can draw from this example is that there is not a single right answer when trying to decide which model is the best. Thus, in each of the
following cases, our choice of models would be different.
1. If it is more important to minimize the sum of squares of deviations than the maximal absolute deviation, the least-squares criterion should be used.
2. If it is more important that the model is computationally simple than to have small maximum or sum of squares of absolute deviations, the
transformed least-squares criterion should be used.
3. If it is more important to minimize the maximal absolute deviation than the sum of squares of deviation, the Chebyshev criterion should be used.
So, we need to decide which model is the best on case-by-case basis, taking all the specifics into account (what is the purpose of the model, how precise should it
be, how accurate is the data, etc).
Linear Regression
Note: regression is a measure of the relation between the mean value of one variable and corresponding values of other variables.)

We can gain further insight in the basic linear model when we consider some further parameters of the statistical analysis. For the data (xi, yi), i = 1, , m,
consider the following values.
1. The error (or explained) sum of squares ESS
ESS reflects the variation about the regression line.

2. The total (or corrected) sum of squares TSS
where
is the average of the y-values of the data points. TSS reflects the variation in the y values about the line y =
3. The regression sum of squares is RSS = TSS ESS.
4. The coefficient of determination R2
Thus defined R2 is independent on the units of x and y and is also independent on which of the two variables is labeled as independent and which as dependent
(in other words, data (yi, xi), i = 1, , m, would produce the same value of R2).
If R2 = 0.9 for example, then 90% of the total variation of y-values from the line y =
is accounted for by a linear relationship with the values of x.
Verification and Validation of Simulation Models

Model verification and validation (V&V) are essential parts of the model development process if models to be accepted and used to support decision making
One of the very first questions that a person who is promoting a model is likely to encounter is has your model been validated?
If the answer to this critical question is No, experience has shown that the model is unlikely to be adopted or even tried out in a real-world setting. Often the
model is sent back to the drawing board. The challenge then becomes one of being able to say yes to this critical question
Verification: is concerned with building the model right. It is utilized in the comparison of the conceptual model to the computer representation that implements
that conception. It asks the questions: Is the model implemented correctly in the computer? Are the input parameters and logical structure of the model correctly
represented?
Verification is done to ensure that:
The model is programmed correctly
The algorithms have been implemented properly
The model does not contain errors, oversights, or bugs
Verification ensures that the specification is complete and that mistakes have not been made in implementing the model
Verification does not ensure the model:
Solves an important problem
Meets a specified set of model requirements
Correctly reflects the workings of a real world process
No computational model will ever be fully verified, guaranteeing 100% error-free implementation
A high degree of statistical certainty is all that can be realized for any model as more cases are tested
Statistical certainty is increased as important cases are tested
In principle, a properly structured testing program increases the level of certainty for a verified model to acceptable levels
Exercise model for all possible cases
Automated testing process
Model verification proceeds as more tests are performed, errors are identified, and corrections are made to the underlying model, often resulting in
retesting requirements to ensure code integrity
The end result of verification is technically not a verified model, but rather a model that has passed all the verification tests!

Validation: is concerned with building the right model. It is utilized to determine that a model is an accurate representation of the real world system. Validation
is usually achieved through the calibration of the model, an iterative process of comparing the model to actual system behavior and using the discrepancies
between the two, and the insights gained, to improve the model. This process is repeated until model accuracy is judged to be acceptable.
The Model Validation Gold Standard
In Physics, The Standard Model is the name given to the current theory of fundamental particles and their interactions
The Standard Model is a good theory because it has been validated
Its predictions have matched experimental data, decimal place
for decimal place, with amazing precision
All the particles predicted by this theory have been found
Can such a theoretical model be found for social systems?
Important Questions about Model Validation
How can the model be validated if
Controlled experiments cannot be performed on the system, for example, if only a single historical
data set exists?
The real-world system being modeled does not exist?
The model is not deterministic (has random elements)?
How can agent-based models be validated?
Agent behaviors and interaction mechanisms
Adaptive agent behaviors of emergent organizations
Model Validation:
Does the model represent and correctly reproduce the behaviors of the real world system?
Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained
The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate
information about the system being modeled, and to makes the model actually used
Validation Depends on the Purpose of the Model and Its Intended Use
Reasons we do modeling and simulation:
We are constrained by linear thinking: We cannot understand how all the various parts of the system interact and add up to the whole
We cannot imagine all the possibilities that the real system could exhibit
We cannot foresee the full effects of cascading events with our limited mental models
We cannot foresee novel events that our mental models cannot even imagine
We model for insights, not numbers
As an exercise in thought space to gain insights into key variables and their causes and effects
To construct reasonable arguments as to why events can or cannot occur based on the model
We model to make qualitative or quantitative predictions about the future
Practical Validation
Validation exercises amount to a series of attempts to invalidate a model
One recently proposed V&V technique, Active Nonlinear Tests (ANTs), explicitly formulates a series of mathematical tests designed to
break the model
Presumably, once a model is shown to be invalid, the model is salvageable with further work and results in a
model having a higher degree of credibility and confidence
The end result of validation
Technically not a validated model, but rather a model that has passed all the validation tests
A better understanding of the models capabilities, limitations, and appropriateness for addressing a range of important questions
Establishing Credibility
Unlike physical systems, for which there are well established procedures for model validation, no such guidelines exist for social modeling
In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model
Verification and validation work together by removing barriers and objections to model use
The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that stand in
for comparing model results to data from the real system
The process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use
Pathways to Validation
Cases

Exploration of critical cases
Exhaustive exploration of cases
Using models as exploratory e-laboratories
Rapid prototyping
Multiple models
Maximally diverse model ensembles
Using subject matter experts
Evaluation
Role playing, participatory simulation
Computational simulations as a special cases of analytical modeling
Validity and Reliability
Validity refers to what degree the research reflects the given research problem, while reliability refers to how consistent a set of measurements are. Consider the
center point in the set of concentric circles is the real world fact The more clustered the repeated trials, the more reliable the model and the more nearer the
cluster is to the center the more valid is the model.
Some types of validity:
External Validity
Internal Validity
Convergent and Discriminant Validity
Test Validity
Criterion Validity
Concurrent Validity
Predictive Validity
A definition of reliability may be "Yielding the same or compatible results in different clinical experiments or statistical trials" (the free dictionary). Research
methodology lacking reliability cannot be trusted. Replication studies are a way to test reliability. Consider the center point in the set of concentric circles is the
real world fact
Types of Reliability:
Test-Retest Reliability
Internal Consistency Reliability
Instrument Reliability
Statistical Reliability
Reproducibility
Both validity and reliability are important aspects of the research methodology to get better explanations of the world.

CH 2 Systems and Modeling

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CH 2 Systems and Modeling

Uploaded by

Copyright:

Available Formats

Research Methodology and Presentation ECEg 4002

Chapter 2: Systems and modeling

Fig: Simplified system

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Ability to explain past observations

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

answering yes/no questions about the data (hypothesis testing),

estimating numerical characteristics of the data (estimation),

describing associations within the data (correlation),

Research Methodology and Presentation ECEg 4002

Documenting and presenting the results of the study.

They have different degrees of usefulness in statistical research.

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Analysis of variance (ANOVA)

Research Methodology and Presentation ECEg 4002

is the feasible region

, then the model is semi-parametric.

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Least-Squares Line Fit.

This will be satisfied when

Research Methodology and Presentation ECEg 4002

This will be satisfied when

Solving this equation for a, we obtain

Transformed Least-Squares Fit.

ii) Finding a logarithmic model y = a ln x + b using the transformed data

iii) Finding a power model y = bxa using the transformed data.

Research Methodology and Presentation ECEg 4002

Setting this derivative to 0 and solving for A gives us

which can be easily solved for a.

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

ESS reflects the variation about the regression line.

4. The coefficient of determination R2

is accounted for by a linear relationship with the values of x.

Verification and Validation of Simulation Models

Research Methodology and Presentation ECEg 4002

Research Methodology and Presentation ECEg 4002

Some types of validity:

You might also like