Timhowardriley

User:TimHowardRiley

Contributing editor, database programmer, and musician

BScThis user has a Bachelor of Science degree in Computer Science. This user graduated magna cum laude from Florida International University.

This user has a Business Administration (Accountancy) degree from California State University, Sacramento. This user is writing a book.

This user is a software engineer. C-5This user is a professional C programmer.

This user programmed the research databases at the Everglades National Park. This user used to commute daily to here.

This user's first computer was a TRS-80 Model I. This user contributes using Vim.

mysqlThis user writes programs that access MySQL. This user contributes using Linux.

This user misses his 1966 Ford Mustang Convertible. This user just LOVES to cycle here.

This user plays the trombone. This user served in the Army Band at Fort Benning.

This user has a website, which can be found here. This user believes Be Afraid is an unfortunate reality. "You should also learn that Wikipedia users often display ownership of articles they've edited."

R programming language

R is a programming language for statistical computing, data visualization, and data analysis.

Examples

Mean -- a measure of center

A numeric data set may have a central tendency — where some of the most typical data points reside. The arithmetic mean (average) is the most commonly used measure of central tendency. The mean of a numeric data set is the sum of the data points divided by the number of data points.

Let $x$ = a list of data points.

Let $n$ = the number of data points.

Let ${\bar {x}}$ = the mean of a data set.

${\bar {x}}={\frac {x_{1}+x_{2}+\cdots +x_{n}}{n}}$

Suppose a sample of four observations of Celsius temperature measurements were taken 12 hours apart.

Let $x$ = a list of degrees Celsius data points of 30, 27, 31, 28.

This R computer program will output the mean of $x$:

# The c() function "combines" a list into a single object.
x <- c( 30, 27, 31, 28 )

sum <- sum( x )
length <- length( x )
mean <- sum / length

message( "Mean:" )
print( mean )

Note: R can have the same identifier represent both a function name and its result. For more information, visit scope.

Output:

Mean:
[1] 29

This R program will execute the native mean() function to output the mean of ${\displaystyle }$x:

x <- c( 30, 27, 31, 28 )

message( "Mean:" )
print( mean( x ) )

Output:

Mean:
[1] 29

Standard Deviation -- a measure of dispersion

A standard deviation of a numeric data set is an indication of the average distance all the data points are from the mean. For a data set with a small amount of variation, then each data point will be close to the mean, so the standard deviation will be small.

Let $x$ = a list of data points.

Let $n$ = the number of data points.

Let $s$ = the standard deviation of a data set.

$s={\sqrt {\frac {\sum \left(x_{i}-{\bar {x}}\right)^{2}}{n-1}}}$

Suppose a sample of four observations of Celsius temperature measurements were taken 12 hours apart.

Let $x$ = a list of degrees Celsius data points of 30, 27, 31, 28.

This R program will output the standard deviation of $x$:

x <- c( 30, 27, 31, 28 )
distanceFromMean <- x - mean( x )
distanceFromMeanSquared <- distanceFromMean ** 2
distanceFromMeanSquaredSum <- sum( distanceFromMeanSquared )
variance <- distanceFromMeanSquaredSum / ( length( x ) - 1 )
standardDeviation <- sqrt( variance )

message( "Standard deviation:" )
print( standardDeviation )

Output:

Standard deviation:
[1] 1.825742

This R program will execute the native sd() function to output the standard deviation of $x$:

x <- c( 30, 27, 31, 28 )

message( "Standard deviation:" )
print( sd( x ) )

Output:

Standard deviation:
[1] 1.825742

Linear regression -- a measure of relation

A scatter plot resembling a linear relationship has infinitely many straight lines that will pass close to all the data points (depicted in red). The blue regression line (generally called curve fit) is the one straight line that has the minimum average distance (depicted in green) from all the points to the line.

A phenomenon may be the result of one or more observable events. For example, the phenomenon of skiing accidents may be the result of having snow in the mountains. A method to measure whether or not a numeric data set is related to another data set is linear regression.

Let $x$ = a data set of independent data points, in which each point occurred at a specific time.

Let $y$ = a data set of dependent data points, in which each point occurred at the same time of an independent data point.

If a linear relationship exists, then a scatter plot of the two data sets will show a pattern that resembles a straight line. If a straight line is embedded into the scatter plot such that the average distance from all the points to the line is minimal, then the line is called a regression line. The equation of the regression line is called the regression equation.

The regression equation is a linear equation; therefore, it has a slope and y-intercept. The format of the regression equation is ${\hat {y}}=b_{0}+b_{1}x$.

Let $b_{1}$ = the slope of the regression equation.

$b_{1}={\frac {\sum \left(x-{\bar {x}}\right)\left(y-{\bar {y}}\right)}{\sum \left(x-{\bar {x}}\right)^{2}}}$

Let $b_{0}$ = the y-intercept of the regression equation.

$b_{0}={\bar {y}}-b_{1}{\bar {x}}$

Suppose a sample of four observations of Celsius temperature measurements were taken 12 hours apart. At the same time, the thermometer was switched to Fahrenheit temperature and another measurement was taken.

Let $x$ = a list of degrees Celsius data points of 30, 27, 31, 28.

Let $y$ = a list of degrees Fahrenheit data points of 86.0, 80.6, 87.8, 82.4.

This R program will output the slope and y-intercept of a linear relationship in which $y$ depends upon $x$:

x <- c( 30, 27, 31, 28 )
y <- c( 86.0, 80.6, 87.8, 82.4 )

# Build the numerator
independentDistanceFromMean <- x - mean( x )
sampledDependentDistanceFromMean <- y - mean( y )

independentDistanceTimesSampledDistance <-
    independentDistanceFromMean *
    sampledDependentDistanceFromMean

independentDistanceTimesSampledDistanceSum <-
    sum( independentDistanceTimesSampledDistance )

# Build the denominator
independentDistanceFromMeanSquared <-
    independentDistanceFromMean ** 2

independentDistanceFromMeanSquaredSum <-
    sum( independentDistanceFromMeanSquared )

# Slope is rise over run
slope <-
    independentDistanceTimesSampledDistanceSum /
    independentDistanceFromMeanSquaredSum

yIntercept <- mean( y ) - slope * ( mean( x ) )

message( "Slope:" )
print( slope )

message( "Y-intercept:" )
print( yIntercept )

Output:

Slope:
[1] 1.8
Y-intercept:
[1] 32

This R program will execute the native functions to output the slope and y-intercept:

x <- c( 30, 27, 31, 28 )
y <- c( 86.0, 80.6, 87.8, 82.4 )

# Execute lm() with Fahrenheit depends upon Celsius
linearModel <- lm( y ~ x )

# coefficients() returns a structure containing the slope and y intercept
coefficients <- coefficients( linearModel )

# Extract the slope from the structure
slope <- coefficients[["x"]]

# Extract the y intercept from the structure
yIntercept <- coefficients[["(Intercept)"]]

message( "Slope:" )
print( slope )

message( "Y-intercept:" )
print( yIntercept )

Output:

Slope:
[1] 1.8
Y-intercept:
[1] 32

Coefficient of determination -- a percentage of variation

The coefficient of determination determines the percentage of variation explained by the independent variable. It always lies between 0 and 1. A value of 0 indicates no relationship between the two data sets, and a value near 1 indicates the regression equation is extremely useful for making predictions.

Let ${\hat {y}}$ = the data set of predicted response data points when the independent data points are passed through the regression equation.

Let $r^{2}$ = the coefficient of determination in a relationship between an independent variable and a dependent variable.

$r^{2}={\frac {\sum \left({\hat {y}}-{\bar {y}}\right)^{2}}{\sum \left(y-{\bar {y}}\right)^{2}}}$

This R program will output the coefficient of determination of the linear relationship between $x$ and $y$:

x <- c( 30, 27, 31, 28 )
y <- c( 86.0, 80.6, 87.8, 82.4 )

# Build the numerator
linearModel <- lm( y ~ x )
coefficients <- coefficients( linearModel )
slope <- coefficients[["x"]]
yIntercept <- coefficients[["(Intercept)"]]
predictedResponse <- yIntercept + ( slope * x )

predictedResponseDistanceFromMean <-
    predictedResponse - mean( y )

predictedResponseDistanceFromMeanSquared <-
    predictedResponseDistanceFromMean ** 2

predictedResponseDistanceFromMeanSquaredSum <-
    sum( predictedResponseDistanceFromMeanSquared )

# Build the denominator
sampledResponseDistanceFromMean <- y - mean( y )

sampledResponseDistanceFromMeanSquared <-
    sampledResponseDistanceFromMean ** 2

sampledResponseDistanceFromMeanSquaredSum <-
    sum( sampledResponseDistanceFromMeanSquared )

coefficientOfDetermination <-
    predictedResponseDistanceFromMeanSquaredSum /
    sampledResponseDistanceFromMeanSquaredSum

message( "Coefficient of determination:" )
print( coefficientOfDetermination )

Output:

Coefficient of determination:
[1] 1

This R program will execute the native functions to output the coefficient of determination:

x <- c( 30, 27, 31, 28 )
y <- c( 86.0, 80.6, 87.8, 82.4 )

linearModel <- lm( y ~ x )
summary <- summary( linearModel )
coefficientOfDetermination <- summary[["r.squared"]]

message( "Coefficient of determination:" )
print( coefficientOfDetermination )

Output:

Coefficient of determination:
[1] 1

Scatter plot

This R program will display a scatter plot with an embedded regression line and regression equation illustrating the relationship between $x$ and $y$:

x <- c( 30, 27, 31, 28 )
y <- c( 86.0, 80.6, 87.8, 82.4 )

linearModel <- lm( y ~ x )
coefficients <- coefficients( linearModel )
slope <- coefficients[["x"]]
intercept <- coefficients[["(Intercept)"]]

# Execute paste() to build the regression equation string
regressionEquation <- paste( "y =", intercept, "+", slope, "x" )

# Display a scatter plot with the regression line and equation embedded
plot(
    x,
	y,
	main = "Fahrenheit Depends Upon Celsius",
	sub = regressionEquation,
	xlab = "Degress Celsius",
	ylab = "Degress Fahrenheit",
	abline( linearModel ) )

Output:

Programming

R is an interpreted language, so programmers typically access it through a command-line interpreter. If a programmer types 1+1 at the R command prompt and presses enter, the computer replies with 2. Programmers also save R programs to a file then execute the batch interpreter Rscript.

Object

R stores data inside an object. An object is assigned a name which the computer program uses to set and retrieve a value. An object is created by placing its name to the left of the symbol-pair <-. The symbol-pair <- is called the assignment operator.

To create an object named x and assign it the integer value 82:

x <- 82L
print( x )

Output:

[1] 82

The [1] displayed before the number is a subscript. It shows the container for this integer is index one of an array.

Vector

The most primitive R object is the vector. A vector is a one dimensional array of data. To assign multiple elements to the array, use the c() function to "combine" the elements. The elements must be the same data type. R lacks scalar data types, which are placeholders for a single word — usually an integer. Instead, a single integer is stored into the first element of an array. The single integer is retrieved using the index subscript of [1].

R program to store and retrieve a single integer:

store <- 82L
retrieve <- store[1]
print( retrieve[1] )

Output:

[1] 82

Element-wise operation

When an operation is applied to a vector, R will apply the operation to each element in the array. This is called an element-wise operation.

This example creates the object named x and assigns it integers 1 through 3. The object is displayed and then again with one added to each element:

x <- 1:3
print( x )
print( x + 1 )

Output:

[1] 1 2 3
[1] 2 3 4

To achieve the many additions, R implements vector recycling. The numeral one following the plus sign (+) is converted into an internal array of three ones. The + operation simultaneously loops through both arrays and performs the addition on each element pair. The results are stored into another internal array of three elements which is returned to the print() function.

Numeric vector

A numeric vector is used to store integers and floating point numbers. The primary characteristic of a numeric vector is the ability to perform arithmetic on the elements.

Integer vector

By default, integers (numbers without a decimal point) are stored as floating point. To force integer memory allocation, append an L to the number. As an exception, the sequence operator : will, by default, allocate integer memory.

R program:

x <- 82L
print( x[1] )
message( "Data type:" )
typeof( x )

Output:

[1] 82
Data type:
[1] "integer"

R program:

x <- c( 1L, 2L, 3L )
print( x )
message( "Data type:" )
typeof( x )

Output:

[1] 1 2 3
Data type:
[1] "integer"

R program:

x <- 1:3
print( x )
message( "Data type:" )
typeof( x )

Output:

[1] 1 2 3
Data type:
[1] "integer"

Double vector

A double vector stores real numbers, which are also known as floating point numbers. The memory allocation for a floating point number is double precision. Double precision is the default memory allocation for numbers with or without a decimal point.

R program:

x <- 82
print( x[1] )
message( "Data type:" )
typeof( x )

Output:

[1] 82
Data type:
[1] "double"

R program:

x <- c( 1, 2, 3 )
print( x )
message( "Data type:" )
typeof( x )

Output:

[1] 1 2 3
Data type:
[1] "double"

Logical vector

A logical vector stores binary data — either TRUE or FALSE. The purpose of this vector is to store the result of a comparison. A logical datum is expressed as either TRUE, T, FALSE, or F. The capital letters are required, and no quotes surround the constants.

R program:

x <- 3 < 4
print( x[1] )
message( "Data type:" )
typeof( x )

Output:

[1] TRUE
Data type:
[1] "logical"

Two vectors may be compared using the following logical operators:

Operator	Syntax	Tests
>	a > b	Is a greater than b?
>=	a >= b	Is a greater than or equal to b?
<	a < b	Is a less than b?
<=	a <= b	Is a less than or equal to b?
==	a == b	Is a equal to b?
!=	a != b	Is a not equal to b?

Character vector

A character vector stores character strings. Strings are created by surrounding text in double quotation marks.

R program:

x <- "hello world"
print( x[1] )
message( "Data type:" )
typeof( x )

Output:

[1] "hello world"
Data type:
[1] "character"

R program:

x <- c( "hello", "world" )
print( x )
message( "Data type:" )
typeof( x )

Output:

[1] "hello" "world"
Data type:
[1] "character"

Factor

A Factor is a vector that stores a categorical variable. The factor() function converts a text string into an enumerated type, which is stored as an integer.

In experimental design, a factor is an independent variable to test (an input) in a controlled experiment. A controlled experiment is used to establish causation, not just association. For example, one could notice that an increase in hot chocolate sales is associated with an increase in skiing accidents.

An experimental unit is an item that an experiment is being performed upon. If the experimental unit is a person, then it is known as a subject. A response variable (also known as a dependent variable) is a possible outcome from an experiment. A factor level is a characteristic of a factor. A treatment is an environment consisting of a combination of one level (characteristic) from each of the input factors. A replicate is the execution of a treatment on an experimental unit and yields response variables.

This example builds two R programs to model an experiment to increase the growth of a species of cactus. Two factors are tested:

1. water levels of none, light, or medium
2. superabsorbent polymer levels of not used or used

R program to setup the design:

# Step 1 is to establish the levels of a factor.
# Vector of the water levels:
waterLevel <-
    c(
        "none",
        "light",
        "medium" )

# Step 2 is to create the factor.
# Vector of the water factor:
waterFactor <-
    factor(
        # Although a subset is possible, use all of the levels.
        waterLevel,
        levels = waterLevel )

# Vector of the polymer levels:
polymerLevel <-
    c(
        "notUsed",
        "used" )

# Vector of the polymer factor:
polymerFactor <-
    factor(
        polymerLevel,
        levels = polymerLevel )

# The treatments are the Cartesian product of both factors.
treatmentCartesianProduct <-
    expand.grid(
        waterFactor,
        polymerFactor )

message( "Water factor:" )
print( waterFactor )

message( "\nPolymer factor:" )
print( polymerFactor )

message( "\nTreatment Cartesian product:" )
print( treatmentCartesianProduct )

Output:

Water factor:
[1] none   light  medium
Levels: none light medium

Polymer factor:
[1] notUsed used   
Levels: notUsed used

Treatment Cartesian product:
    Var1    Var2
1   none notUsed
2  light notUsed
3 medium notUsed
4   none    used
5  light    used
6 medium    used

R program to store and display the results:

experimentalUnit <- c( "cactus1", "cactus2", "cactus3" )

replicateWater <- c( "none", "light", "medium" )
replicatePolymer <- c( "notUsed", "used", "notUsed" )
replicateInches <- c( 82L, 83L, 84L )

response <-
    data.frame(
        experimentalUnit,
        replicateWater,
        replicatePolymer,
        replicateInches )

print( response )

Output:

  experimentalUnit replicateWater replicatePolymer replicateInches
1          cactus1           none          notUsed              82
2          cactus2          light             used              83
3          cactus3         medium          notUsed              84

Data frame

A data frame stores a two-dimensional array. The horizontal dimension is a list of vectors. The vertical dimension is a list of rows. It is the most useful structure for data analysis. Data frames are created using the data.frame() function. The input is a list of vectors (of any data type). Each vector becomes a column in a table. The elements in each vector are aligned to form the rows in the table.

R program:

integer <- c( 82L, 83L )
string <- c( "hello", "world" )
data.frame <- data.frame( integer, string )
print( data.frame )
message( "Data type:" )
class( data.frame )

Output:

  integer string
1      82  hello
2      83  world
Data type:
[1] "data.frame"

Data frames can be deconstructed by providing a vector's name between double brackets. This returns the original vector. Each element in the returned vector can be accessed by its index number.

R program to extract the word "world". It is stored in the second element of the "string" vector:

integer <- c( 82L, 83L )
string <- c( "hello", "world" )
data.frame <- data.frame( integer, string )
vector <- data.frame[["string"]]
print( vector[2] )
message( "Data type:" )
typeof( vector )

Output:

[1] "world"
Data type:
[1] "character"

Vectorized coding

Vectorized coding is a method to produce quality R computer programs that take advantage of R's strengths. The R language is designed to be fast at logical testing, subsetting, and element-wise execution. On the other hand, R does not have a fast for loop. For example, R can search-and-replace faster using logical vectors than by using a for loop.

For loop

A for loop repeats a block of code for a specific amount of iterations.

Example to search-and-replace using a for loop:

vector <- c( "one", "two", "three" )

for ( i in 1:length( vector ) )
{
    if ( vector[ i ] == "one" )
    {
        vector[ i ] <- "1"
    }
}

message( "Replaced vector:" )
print( vector )

Output:

Replaced vector:
[1] "1"  "two" "three"

Subsetting

R's syntax allows for a logical vector to be used as an index to a vector. This method is called subsetting.

R example:

vector <- c( "one", "two", "three" )

print( vector[ c( TRUE, FALSE, TRUE ) ] )

Output:

[1] "one"    "three"

Change a value using an index number

R allows for the assignment operator <- to overwrite an existing value in a vector by using an index number.

R example:

vector <- c( "one", "two", "three" )
vector[ 1 ] <- "1"

print( vector )

Output:

[1] "1"   "two"   "three"

Change a value using subsetting

R also allows for the assignment operator <- to overwrite an existing value in a vector by using a logical vector.

R example:

vector <- c( "one", "two", "three" )
vector[ c( TRUE, FALSE, FALSE ) ] <- "1"

print( vector )

Output:

[1] "1"     "two"   "three"

Vectorized code to search-and-replace

Because a logical vector may be used as an index, and because the logical operator returns a vector, a search-and-replace can take place without a for loop.

R example:

vector <- c( "one", "two", "three" )
vector[ vector == "one" ] <- "1"

print( vector )

Output:

[1] "1"     "two"   "three"

Functions

A function is an object that stores computer code instead of data. The purpose of storing code inside a function is to be able to reuse it in another context.

Native functions

R comes with over 1,000 native functions to perform common tasks. To execute a function:

1. type in the function's name
2. type in an open parenthesis (
3. type in the data to be processed
4. type in a close parenthesis )

This example rolls a die one time. The native function's name is sample(). The data to be processed are:

1. a numeric integer vector from one to six
2. the size parameter instructs sample() to execute the roll one time

sample( 1:6, size=1 )

Possible output:

[1] 6

The R interpreter provides a help screen for each native function. The help screen is displayed after typing in a question mark followed by the function's name:

?sample

Partial output:

Description:
     ‘sample’ takes a sample of the specified size from the elements of
     ‘x’ using either with or without replacement.

Usage:
     sample(x, size, replace = FALSE, prob = NULL)

Function parameters

The sample() function has available four input parameters. Input parameters are pieces of information that control the function's behavior. Input parameters may be communicated to the function in a combination of three ways:

1. by position separated with commas
2. by name separated with commas and the equal sign
3. left empty

For example, each of these calls to sample() will roll a die one time:

sample( 1:6, 1, F, NULL )
sample( 1:6, 1 )
sample( 1:6, size=1 )
sample( size=1, x=1:6 )

Every input parameter has a name. If a function has many parameters, setting name = data will make the source code more readable. If the parameter's name is omitted, R will match the data in the position order. Usually, parameters that are rarely used will have a default value and may be omitted.

Data coupling

The output from a function may become the input to another function. This is the basis for data coupling.

This example executes the function sample() and sends the result to the function sum(). It simulates the roll of two dice and adds them up.

sum( sample( 1:6, size=2, replace=TRUE ) )

Possible output:

[1] 7

Functions as parameters

A function has parameters typically to input data. Alternatively, a function (A) can use a parameter to input another function (B). Function (A) will assume responsibility to execute function (B).

For example, the function replicate() has an input parameter that is a placeholder for another function. This example will execute replicate() once, and replicate() will execute sample() five times. It will simulate rolling a die five times:

replicate( 5, sample( 1:6, size=1 ) )

Possible output:

[1] 2 4 1 4 5

Uniform distribution

Because each face of a die is equally likely to appear on top, rolling a die many times generates the uniform distribution. This example displays a histogram of a die rolled 10,000 times:

hist( replicate( 10000, sample( 1:6, size=1 ) ) )

The output is likely to have a flat top:

Central limit theorem

Whereas a numeric data set may have a central tendency, it also may not have a central tendency. Nonetheless, a data set of the arithmetic mean of many samples will have a central tendency to converge to the population's mean. The arithmetic mean of a sample is called the sample mean. The central limit theorem states for a sample size of 30 or more, the distribution of the sample mean (${\bar {x}}$) is approximately normally distributed, regardless of the distribution of the variable under consideration ($x$). A histogram displaying a frequency of data point averages will show the distribution of the sample mean resembles a bell-shaped curve.

For example, rolling one die many times generates the uniform distribution. Nonetheless, rolling 30 dice and calculating each average (${\bar {x}}$) over and over again generates a normal distribution.

R program to roll 30 dice 10,000 times and plot the frequency of averages:

hist(
    replicate(
        10000,
        mean(
            sample(
                1:6,
                size=30,
                replace=T ) ) ) )

The output is likely to have a bell shape:

Programmer-created functions

To create a function object, execute the function() statement and assign the result to a name. A function receives input both from global variables and input parameters (often called arguments). Objects created within the function body remain local to the function.

R program to create a function:

# The input parameters are x and y.
# The return value is a numeric double vector.
f <- function(x, y)
{
    first_expression <- x * 2
    second_expression <- y * 3
    first_expression + second_expression

    # The return statement may be omitted
    # if the last expression is unassigned.
    # This will save a few clock cycles.
}

Usage output:

> f(1, 2)
[1] 8

Function arguments are passed in by value.

Generic functions

R supports generic functions, which is also known as polymorphism. Generic functions act differently depending on the class of the argument passed in. The process is to dispatch the method specific to the class. A common implementation is R's print() function. It can print almost every class of object. For example, print(objectName).

If statements

R program illustrating if statements:

minimum <- function( a, b )
{
	if ( a < b )
		minimum <- a
	else
		minimum <- b

	return( minimum )
}

maximum <- function( a, b )
{
	if ( a > b )
		maximum <- a
	else
		maximum <- b

	return( maximum )
}

range <- function( a, b, c )
{
	range <-
		maximum( a, maximum( b, c ) ) -
		minimum( a, minimum( b, c ) )

	return( range )
}

range( 10, 4, 7 )

Output:

[1] 6

Programming shortcuts

R provides three notable shortcuts available to programmers.

Omit the print() function

If an object is present on a line by itself, then the interpreter will send the object to the print() function.

R example:

integer <- 82L
integer

Output:

[1] 82

Omit the return() statement

If a programmer-created function omits the return() statement, then the interpreter will return the last unassigned expression.

R example:

f <- function()
{
    # Don't assign the expression to an object.
    82L + 1L
}

Usage output:

> f()
[1] 83

Alternate assignment operator

The symbol-pair <- assigns a value to an object. Alternatively, = may be used as the assignment operator. However, care must be taken because = closely resembles the logical operator for equality, which is ==.

R example:

integer = 82L
print( integer )

Output:

[1] 82

Normal distribution

If a numeric data set has a central tendency, it also may have a symmetric looking histogram — a shape that resembles a bell. If a data set has an approximately bell-shaped histogram, it is said to have a normal distribution.

Chest size of Scottish militiamen data set

In 1817, a Scottish army contractor measured the chest sizes of 5,732 members of a militia unit. The frequency of each size was:

Chest size (inches)	Frequency
33	3
34	19
35	81
36	189
37	409
38	753
39	1062
40	1082
41	935
42	646
43	313
44	168
45	50
46	18
47	3
48	1

Create a comma-separated values file

R has the write.csv() function to convert a data frame into a CSV file.

R program to create chestsize.csv:

chestsize <-
c( 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48 )

frequency <-
c( 3, 19, 81, 189, 409, 753, 1062, 1082, 935, 646, 313, 168, 50, 18, 3, 1 )

dataFrame <- data.frame( chestsize, frequency )

write.csv(
    dataFrame,
    file="chestsize.csv",
    # By default, write.csv() creates the first column as the row number.
    row.names = FALSE )

Import a data set

The first step in data science is to import a data set.

R program to import chestsize.csv into a data frame:

dataFrame <- read.csv( "chestsize.csv" )
print( dataFrame )

Output:

   chestsize frequency
1         33         3
2         34        19
3         35        81
4         36       189
5         37       409
6         38       753
7         39      1062
8         40      1082
9         41       935
10        42       646
11        43       313
12        44       168
13        45        50
14        46        18
15        47         3
16        48         1

Transform a data set

The second step in data science is to transform the data into a format that the functions expect. The chest-size data set is summarized to frequency; however, R's normal distribution functions require a numeric double vector.

R function to convert a summarized to frequency data frame into a vector:

# Filename: frequencyDataFrameToVector.R

frequencyDataFrameToVector <-
    function(
        dataFrame,
        dataColumnName,
        frequencyColumnName = "frequency" )
{
    dataVector <- dataFrame[[ dataColumnName ]]
    frequencyVector <- dataFrame[[ frequencyColumnName ]]

    vectorIndex <- 1
    frequencyIndex <- 1
    vector <- NA

    for ( datum in dataVector )
    {
        frequency <- frequencyVector[ frequencyIndex ]

        for ( i in 1:frequency )
        {
            vector[ vectorIndex ] <- datum
            vectorIndex <- vectorIndex + 1
        }

        frequencyIndex <- frequencyIndex + 1
    }

    return ( vector )
}

R has the source() function to include another R source file into the current program.

R program to load and display a summary of the 5,732 member data set:

source( "frequencyDataFrameToVector.R" )

dataFrame <- read.csv( "chestsize.csv" )

chestSizeVector <-
    frequencyDataFrameToVector(
        dataFrame,
        "chestsize" )

message( "Head:" )
head( chestSizeVector )

message( "\nTail:" )
tail( chestSizeVector )

message( "\nCount:" )
length( chestSizeVector )

message( "\nMean:" )
mean( chestSizeVector )

message( "\nStandard deviation:" )
sd( chestSizeVector )

Output:

Head:
[1] 33 33 33 34 34 34

Tail:
[1] 46 46 47 47 47 48

Count:
[1] 5732

Mean:
[1] 39.84892

Standard deviation:
[1] 2.073386

Visualize a data set

The third step in data science is to visualize the data set. If a histogram of a data set resembles a bell shape, then it is normally distributed.

R program to display a histogram of the data set:

source( "frequencyDataFrameToVector.R" )

dataFrame <- read.csv( "chestsize.csv" )

chestSizeVector <-
    frequencyDataFrameToVector(
        dataFrame,
        "chestsize" )

hist( chestSizeVector )

Output:

Standardized variable

Any variable ($x_{i}$) in a data set can be converted into a standardized variable ($z_{i}$). The standardized variable is also known as a z-score. To calculate the z-score, subtract the mean and divide by the standard deviation.

Let $x$ = a set of data points.

Let ${\bar {x}}$ = the mean of the data set.

Let $\sigma$ = the standard deviation of the data set.

Let $x_{i}$ = the $i^{th}$ element in the set.

Let $z_{i}$ = the z-score of the $i^{th}$ element in the set.

$z_{i}={\frac {x_{i}-{\bar {x}}}{\sigma }}$

R function to convert a measurement to a z-score:

# Filename: zScore.R

zScore <- function( measurement, mean, standardDeviation )
{
    ( measurement - mean ) / standardDeviation
}

R program to convert a chest size measurement of 38 to a z-score:

source( "zScore.R" )

print( zScore( 38, 39.84892, 2.073386 ) )

Output:

[1] -0.8917394

R program to convert a chest size measurement of 42 to a z-score:

source( "zScore.R" )

print( zScore( 42, 39.84892, 2.073386 ) )

Output:

[1] 1.037472

Standardized data set

A standardized data set is a data set in which each member of an input data set was run through the zScore function.

R function to convert a numeric vector into a z-score vector:

# Filename: zScoreVector.R

source( "zScore.R" )

zScoreVector <- function( vector )
{
    zScoreVector = NA

    for ( i in 1:length( vector ) )
    {
        zScoreVector[ i ] <-
            zScore(
                vector[ i ],
                mean( vector ),
                sd( vector ) )
    }

    return( zScoreVector )
}

Standardized chest size data set

R program to standardize the chest size data set:

source( "frequencyDataFrameToVector.R" )
source( "zScoreVector.R" )

dataFrame <- read.csv( "chestsize.csv" )

chestSizeVector <-
    frequencyDataFrameToVector(
        dataFrame,
        dataColumnName = "chestsize" )

zScoreVector <-
    zScoreVector(
        chestSizeVector )

message( "Head:" )
head( zScoreVector )

message( "\nTail:" )
tail( zScoreVector )

message( "\nCount:" )
length( zScoreVector )

message( "\nMean:" )
round( mean( zScoreVector ) )

message( "\nStandard deviation:" )
sd( zScoreVector )

hist( zScoreVector )

Output:

Head:
[1] -3.303253 -3.303253 -3.303253 -2.820950 -2.820950 -2.820950

Tail:
[1] 2.966684 2.966684 3.448987 3.448987 3.448987 3.931290

Count:
[1] 5732

Mean:
[1] 0

Standard deviation:
[1] 1

Standard normal curve

The standard normal curve is centered at 0 with most of the area between -3 and 3.

A histogram of a normally distributed data set that is converted to its standardized data set also resembles a bell-shaped curve. The curve is called the standard normal curve or the z-curve. The four basic properties of the z-curve are:

1. The total area under the curve is 1.
2. The curve extends indefinitely to the left and right. It never touches the horizontal axis.
3. The curve is symmetric and centered at 0.
4. Almost all of the area under the curve lies between -3 and 3.

Area under the standard normal curve

The probability that a future measurement will be a value between a designated range is equal to the area under the standard normal curve of the designated range's two z-scores.

For example, suppose the Scottish militia's quartermaster wanted to stock up on uniforms. What is the probability that the next recruit will need a size between 38 and 42?

R program:

library( tigerstats )
source( "frequencyDataFrameToVector.R" )
source( "zScore.R" )

dataFrame <- read.csv( "chestsize.csv" )

chestSizeVector <-
    frequencyDataFrameToVector(
        dataFrame,
        dataColumnName = "chestsize" )

zScore38 <-
    zScore( 38, mean( chestSizeVector ), sd( chestSizeVector ) )

zScore42 <-
    zScore( 42, mean( chestSizeVector ), sd( chestSizeVector ) )

areaLeft38 <- tigerstats::pnormGC( zScore38 )
areaLeft42 <- tigerstats::pnormGC( zScore42 )

areaBetween <- areaLeft42 - areaLeft38

message( "Probability:" )
print( areaBetween )

Output:

Probability:
[1] 0.6639757

The pnormGC() function can compute the probability between a range without first calculating the z-score.

R program:

library( tigerstats )
source( "frequencyDataFrameToVector.R" )

dataFrame <- read.csv( "chestsize.csv" )

chestSizeVector <-
    frequencyDataFrameToVector(
        dataFrame,
        dataColumnName = "chestsize" )

areaBetween <-
    tigerstats::pnormGC(
        c( 38, 42 ),
        mean = mean( chestSizeVector ),
        sd = sd( chestSizeVector ),
        region = "between",
        graph = TRUE )

message( "Probability:" )
print( areaBetween )

Output:

Probability:
[1] 0.6639757

XMLHttpRequest

XMLHttpRequest is a JavaScript class containing methods to asynchronously transmit HTTP requests from a web browser to a web server. The methods allow a browser-based application to make a fine-grained server call and store the result in the XMLHttpRequest responseText attribute. The XMLHttpRequest class is a component of Ajax programming. Without Ajax, the "Submit" button will send to the server an entire HTML form. The server will respond by returning an entire HTML page to the browser.

Constructor

Generating an asynchronous request to the web server requires first to instantiate (allocate the memory of) the XMLHttpRequest object. The allocated memory is assigned to a variable. The programming statement in JavaScript to instantiate a new object is new. The new statement is followed by the constructor function of the object. The custom for object-oriented language developers is to invoke the constructor function using same name as the class name. In this case, the class name is XMLHttpRequest. To instantiate a new XMLHttpRequest and assign it to the variable named request:

var request = new XMLHttpRequest();

The open method

The open method prepares the XMLHttpRequest. It can accept up to five parameters, but requires only the first two.

var request = new XMLHttpRequest();

request.open( RequestMethod, SubmitURL, AsynchronousBoolean, UserName, Password );

• RequestMethod: The HTTP request method may be GET for smaller quantities of data. Among the other request methods available, POST will handle substantial quantities of data. After the return string is received, then send the DELETE request method to .open() to free the XMLHttpRequest memory. If DELETE is sent, then the SubmitURL parameter may be null.

* request.open( "DELETE", null );

• SubmitURL: The SubmitURL is a URL containing the execution filename and any parameters that get submitted to the web server. If the URL contains the host name, it must be the web server that sent the HTML document. Ajax supports the same-origin policy.
• AsynchronousBoolean: If supplied, it should be set to true. If set to false, then the browser will wait until the return string is received. Programmers are discouraged to set AsynchronousBoolean to false, and browsers may experience an exception error.
• UserName: If supplied, it will help authenticate the user.
• Password: If supplied, it will help authenticate the user.

The setRequestHeader method

If the request method of POST is invoked, then the additional step of sending the media type of Content-Type: application/x-www-form-urlencoded is required. The setRequestHeader method allows the program to send this or other HTTP headers to the web server. Its usage is setRequestHeader( HeaderField, HeaderValue ). To enable the POST request method:

* request.setRequestHeader( "Content-Type", "application/x-www-form-urlencoded" );

The send method

If the request method of POST is invoked, then the web server expects the form data to be read from the standard input stream. To send the form data to the web server, execute request.send( FormData ), where FormData is a text string. If the request method of GET is invoked, then the web server expects only the default headers. To send the default headers, execute request.send( null ).

The onreadystatechange event listener

onreadystatechange is a callback method that is periodically executed throughout the Ajax lifecycle. To set a callback method named listenMethod(), the syntax is request.onreadystatechange = listenMethod. For convenience, the syntax allows for an anonymous method to be defined. To define an anonymous callback method:

var request = new XMLHttpRequest();

request.onreadystatechange = function()
{
// code omitted
}

The XMLHttpRequest lifecycle progresses through several stages – from 0 to 4. Stage 0 is before the open() method is invoked, and stage 4 is when the text string has arrived. To monitor the lifecycle, XMLHttpRequest has available the readyState attribute. Stages 1-3 are ambiguous and interpretations vary across browsers. Nonetheless, one interpretation is:

• Stage 0: Uninitialized
• Stage 1: Loading
• Stage 2: Loaded
• Stage 3: Interactive
• Stage 4: Completed

When readyState reaches 4, then the text string has arrived and is set in the responseText attribute.

var request = new XMLHttpRequest();

request.onreadystatechange = function()
{
    if ( request.readyState == 4 )
    {
        // request.responseText is set
    }
}

Linux examples

Upon request, the browser will execute a JavaScript function to transmit a request for the web server to execute a computer program. The computer program may be the PHP interpreter, another interpreter, or a compiled executable. In any case, the JavaScript function expects a text string to be transmitted back and stored in the responseText attribute.

To create an example JavaScript function:

• cd /var/www/html
• Edit a file named ajax_submit.js:

function ajax_submit( destination_division, submit_url, person_name )
{
    var request = new XMLHttpRequest();
    var completed_state = 4;

    submit_url = submit_url + "?person_name=" + person_name;
    request.open( "GET", submit_url, true );
    request.send( null );

    request.onreadystatechange = function()
    {
        if ( request.readyState == completed_state )
        {
            document.
                getElementById( destination_division ).
                innerHTML =
                    request.responseText;

            request.open( "DELETE", null );
        }
    }
}

PHP example

PHP is a scripting language designed specifically to interface with HTML. Because the PHP engine is an interpreter – interpreting program statements as they are read – there are programming limitations and performance costs. Nonetheless, its simplicity may place the XMLHttpRequest set of files in the same working directory – probably /var/www/html.

PHP server component

The server component of a PHP XMLHttpRequest is a file located on the server that does not get transmitted to the browser. Instead, the PHP interpreter will open this file and read in its PHP instructions. The XMLHttpRequest protocol requires an instruction to output a text string.

• cd /var/www/html
• Edit a file named ajax_server.php:

<?php
    $person_name = $_GET[ 'person_name' ];
    echo "<p>Hello $person_name";
?>

PHP browser component

The browser component of a PHP XMLHttpRequest is a file that gets transmitted to the browser. The browser will open this file and read in its HTML instructions.

• cd /var/www/html
• Edit a file named ajax_php.html:

<!doctype html>
<html>
<head>
    <title>Hello World</title>
    <script type=text/javascript src=ajax_submit.js></script>
</head>
<body>
    <p>What is your name?
    <input type=text id="person_name" size=10>

    <div id=destination_division></div>

    <button
        onclick="ajax_submit(
            'destination_division',
            'ajax_server.php',
            document.getElementById( 'person_name' ).value )">
        Submit
    </button>
</body>
</html>

1. Point your browser to http://localhost/ajax_php.html
2. Type in your name.
3. Press Submit

CGI example

The Common Gateway Interface (CGI) process allows a browser to request the web server to execute a compiled computer program.

CGI server component

The server component of a CGI XMLHttpRequest is an executable file located on the server. The operating system will open this file and read in its machine instructions. The XMLHttpRequest protocol requires an instruction to output a text string.

Compiled programs have two files: the source code and a corresponding executable.

• cd /usr/lib/cgi-bin
• Edit a file named ajax_server.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

void main( void )
{
    char *query_string;
    char *person_name;

    query_string = getenv( "QUERY_STRING" );

    /* Skip "person_name=" */
    person_name = query_string + strlen( "person_name=" );

    /* CGI requires the first line to output: */
    printf( "Content-type: text/html\n" );

    /* CGI requires the second line to output: */
    printf( "\n" );

    printf( "<p>Hello %s\n", person_name );
}

• Compile the source code to create the executable:

cc ajax_server.c -o ajax_server

CGI browser component

The CGI browser component is the same as the PHP browser component, except for a slight change in the submit_url. The syntax to tell the web server to execute an executable is /cgi-bin/ followed by the filename. For security, the executable must reside in a chroot jail. In this case, the jail is the directory /usr/lib/cgi-bin/.

• cd /var/www/html
• Edit a file named ajax_cgi.html:

<!doctype html>
<html>
<head>
    <title>Hello World</title>
    <script type=text/javascript src=ajax_submit.js></script>
</head>
<body>
    <p>What is your name?
    <input type=text id="person_name" size=10>

    <div id=destination_division></div>

    <button
        onclick="ajax_submit(
            'destination_division',
            '/cgi-bin/ajax_server',
            document.getElementById( 'person_name' ).value )">
        Submit
    </button>
</body>
</html>

1. Point your browser to http://localhost/ajax_cgi.html
2. Type in your name.
3. Press Submit

See also

• Computer program – Instructions a computer can execute
• Field (computer science) – Variable in a record
• Unix domain socket – Mechanism for data exchange between processes executing on the same host