Programming

# Accidental Art   I’m working on phased entry with spatial price discrimination. This got spat out, and I’m really enjoying the subtle patterns.

Programming

# Using “cbind” and “as.vector”: Computationally intensive commands

As perhaps a mere interesting note, cbind when combined with as.vector can be a particularly RAM-intensive set of commands. I noted the following script excerpt caused my computer to quickly consume  11GB of RAM on a 300k entry dataset. :

for(j in c(1:200)){
mod.out\$coefficients\$count[1:80]<-lim.my.df[1:80,j]
mod.out\$coefficients\$zero[1:80]<-lim.my.df[81:160,j]
a<-predict(mod.out,clip.1)
b<-predict(mod.out,clip.mean)
diff.j<-mean(a-b)
# diff[,paste(i,”.”,j)]<-diff.j
diff<-as.vector(cbind(diff,diff.j))
}

The purpose of this script is to use bootstrapped coefficients generate an average partial effect between clip.1 and clip.mean. We will later use this to get a estimate of the standard errors of the APE. As it stands, it eats all my RAM quite promptly and causes the computer to crash.  The following script, nearly identical, does not have this problem:

for(j in c(1:200)){
mod.out\$coefficients\$count[1:80]<-lim.my.df[1:80,j]
mod.out\$coefficients\$zero[1:80]<-lim.my.df[81:160,j]
a<-predict(mod.out,clip.1)
b<-predict(mod.out,clip.mean)
diff.j<-mean(a-b)
# diff[,paste(i,”.”,j)]<-diff.j
diff<-cbind(diff,diff.j)
}
diff<-as.vector(diff)

And this works just fine! In fact, it barely  consumes 25% of my RAM.

Programming

# Basic programming in Linux

I’ve just written my first program in Linux(Ubuntu) using bash commands. I’d love to give more details about what’s happening and how I’m doing it, but basically I’m following instructions and then playing with the little that I know about it.  I’ll be writing a program in two parts here, basically just as a demonstration of what they do and how they work.

Here’s the first program. Put it into a text file and save it as “testprogram”.

#!/usr/bin/env bash
# A simple command in a shell script
name=”COMPUTER”
printf “Hello, world!\n”
printf “My name is \n”
printf “\${name}! \n”
printf “What’s your name? \n”
printf “I didn’t get that, is your name \$name? \n”
printf “What’s your name again? \n”
while [ \$user_name = \$user_name2 ]
do
printf “I heard your name is \$user_name! No need to SHOUT!\n”
done

After doing that, navigate using “cd” and “ls” to the appropriate directory that you saved “testprogram” to.
Now, enter the command, this is the second part:

chmod a+rx testprogram

This will make the program “testprogram”, for All users (a), Readable and eXecutable (+rx).
You may now run the file by entering:

./testprogram

The program will produce the following output:

Hello World!
My name is COMPUTER!
-prompt1-
I didn’t get that, is your name <returns prompt>?
-prompt2, irrelevant what you say here, yes, your name again, etc.-
What’s your name again?
-prompt3-
If your name in the first prompt is exactly the same as that of the third, it will scream:
I heard your name is <return prompt 1>! No need to SHOUT!
It will repeat this until the program is terminated by the user.

I’m really pumped about this. This is part of my ongoing work at completing my dissertation (I need to make a standalone script to start the programs I want and pass arguments to it), as well as getting me to be a more useful part of the universe, so I’m always excited to get things done.

Statistics

# Generating correlated variables in R, and an uninvertability hypotheises.

This has been a problem I’ve been suffering with as of late.  How do you generate variables with an arbitrary covariance in R?  There’s no “covariance=3” button!  This creates some issues when I’m trying to simulate various behaviours in my econometrics studying (such as omitted variable bias, forthcoming). So I attempted to come up with a solution– only to discover the problem is much more difficult than I initially thought!

Say x is an arbitrary NxK matrix composed of N draws from K different random variables, and we would like to generate a matrix y (NxK) such that it has  an arbitrary covariance matrix c (KxK) and arbitrary mean m=E(y)=1xK. Define 1 as a 1xN vector of 1’s, which allows 1*x=E(x)=1xK, and N remains a scalar. $c=E(y'x)-E(y)'E(x)$ $c+E(y)'E(x)=E(y'x)$ $c+E(y)'(1*x/N)=y'x/N$ Convert into sample analogs, E(y) is a selected parameter, so it has no sample analog at the moment. $cN+E(y)'(1*x)=y'x$ Keep in mind N is a scalar $cNx'+E(y)'(1*x)x'=y'xx'$ $N*cx'inv(xx')+E(y)'1=y$

But unfortunately, xx‘ needs to be invertible! xx‘ rarely is invertable as an NxN when N is large and K is small-> only x’x, a KxK is likely invertable!  I think we can show that as N approaches infinity, xx’ is singular with probability approaching 1 for a finite K. At a minimum, we can show that this isn’t going to work in R! Work below.

N<-50
K<-round(runif(1,1,10))
c<-matrix(runif(K^2,-1,1), ncol=K, nrow=K)
c[upper.tri(c)]<-c[lower.tri(c)] #correlation matrixes are symmetric by definition.
#diag(c)<-1
ey<-matrix(rnorm(K,0,1),1,K)
eine<-matrix(1,1,N)

ex<-rnorm(1,0,10)
x<-matrix(rnorm(N,ex,10)) #one row is alcready created.
tick<-0 #cheap trick to make sure your loop repeated successfully. You’ll see it.
for(i in 2:K){
ex<-rnorm(1,0,10)
holding<-as.matrix(rnorm(N,ex,1)) #it’s an Nx1, k=1
x<-cbind(x,holding)
tick<-tick+1
}
print(tick)

solve(t(x)%*%x/N) #kxk is invertable

solve(x%*%t(x)/N) #nxn is not!

My thought is that k squared terms: $(x_{n1}^2+x_{n2}^2+...x_{nk}^2)/N$ for K different terms. But if K is small, then the sum of $x_k^2$‘s is an unincreasing finite number, divided by N, which approaches infinity. As such, all the diagonal elements approach 0.  The same is true of the cross diagonal elements. An NxN matrix with all 0’s in all elements has rank less than its dimension and cannot be inverted.

http://www.r-bloggers.com/simulating-random-multivariate-correlated-data-continuous-variables/

# Basic bootstrapping in R

I’ve been having some trouble determining how strata works in the boot() command.   My intuition says it should select from within each type of strata. But no guarantees!  I can always just type “boot” and read the whole command line for line… but this isn’t always easy to interpret without comments.
So here’s a quick test to make sure it does what I think it does.

library(boot) #load the package
x<-rep(c(0,100,0,1), each=20) #make a long list of 20 zeros,20 one hundreds, 20 zeros, 20 ones.pool<-matrix(x, ncol=2) #make that list into a 20 by 2 matrix, filling it downward.

pool #let’s look at it.
f<-function(pool,i){
mean(pool[i,1]) #mean of the first column of pool, using individuals i
} #create a function that uses (data, individuals)
boot(pool,f, R=500) #resample and perform that operation 500 times.  Has variation in output.
boot(pool,f, R=500, strata=pool[,2]) #resample and perform that operation -reselecting from within the strata on the rhs. Since all observations within each strata are identical, the standard deviation should be zero.

Here’s an interesting mistake that I made while creating this command.

f<-function(pool,i){
mean(pool[,1])
} #create a function that uses (data, individuals)
boot(pool,f, R=500) #Has no variation in bootstrapping.  Why? What’s up with that?

Answer: There’s no individuals. It just resamples the entire population of the first column of pooled 500 times, getting the final result as shown.

# Ordering and structuring text in STATA

Like most researchers, I have difficulty finding data. On occasion, when I manage to find data, it comes in a useless or meaningless format. For example, a local police department offered to print out all of its police reports for me for the last 15 years at \$0.25 a page… which I would have to enter by hand.  I politely declined.  Here’s an example of sorting out city titles from a poorly formatted document- a short version of that document can be found here.

The first and most important thing is to have your data loaded.

```clear all
use "location", clear```

This loads the data and clears any extra junk STATA may have floating around.  The next thing to do is identify any sort of regular pattern.  The first identifiable patterns were that titles for cities began every 64 lines, and sometimes went for between 3 or 4 lines, before having another 64 lines of regular income reports (useless at this point) are posted. The second pattern is that they all are either villages, cities, or towns.  Below are some attempts at accomplishing this goal via making some sort of indicator every 64 lines (didn’t work), or marking every area that starts with CITY (inefficient). Don’t forget to put strings in quotation marks!

```*Marks every line with a list that counts 0...63. I thought the 63rd would be useful, but because of the irregular length of the header, it's worthless.
*gen city_titles=mod(_n-1,64)
*Is the line saying city? If so, give it a mark, literally.
*gen mark=cond(var1=="CITY","mark",".",".")```

Notice that using (*) comments out the line in STATA, and (.) is STATA’s version of NA.  Here is a more successful attempt:

```*Identify if it's probably a title
gen mark=1 if var1=="CITY"
replace mark=1 if var1=="TOWN"
replace mark=1 if var1=="VILLAGE"```

Don’t forget the double equals!  In STATA, all if statements I have observed are followed by (==) instead of (=), and I still manage to forget them often!  Next, I will create 3 more indicators, aptly labeled mark_1 to mark_3, all showing if it follows the indicator above by 1, 2, or 3 variables. In programming, it is better to give clear, boring, brief names rather than unclear mess. I would rather have a long object name than an unclear short one, since you never know who is going to have to read your code:

```*Is it following a title?
gen mark_1=mark[_n-1]
gen mark_2=mark[_n-2]
gen mark_3=mark[_n-3]```

In retrospect, I could have perhaps consolidated those more efficiently. Does anyone else see how I could have done that in a single variable (mark) instead of having multiple indicators (mark_1…mark_3)?  Either way, I have some extra chaff I have to eliminate. having (.)’s in my data set is agitating. Let’s set those all as 0’s, and sum the remaining columns together in something called “total”, so I know if they have an indicator (1 or more) or not (0).  Then we can drop everything we don’t want!

```*Eliminate non-numerical entries as numeric.
replace mark=0 if mark==.
replace mark_1=0 if mark_1==.
replace mark_2=0 if mark_2==.
replace mark_3=0 if mark_3==.

*Is it marked?
gen total=mark+mark_1+mark_2+mark_3

drop if total!=1```

Bam!  Much better. We dropped somewhere around 80% of our unwanted junk right there! We still could never compile this all by hand, though.  Here, we want it to concatenate all our variables 1…11 in a downward direction.  So we make a local marco list called to_combine, since that’s what we’ll be doing with them.

`local to_combine var1 var2 var3 var4 var5 var6 var7 var8 var9 var10 var11`

Below is a STATA style do-loop, a common element in meaningful programming everywhere.  We want to repeat for each and every variable in the “to_combine” list a series of operations designed to deal with a few regular problems we observed. Firstly, we want to fix an issue with names. You may notice “FOND DU LAC” takes up 3 columns rather than the intended 1, being as that is the name of a city/town/village/county in Wisconsin.  This is the first thing we fix, by dropping the spaces with “DU” and “LAC” and replacing all “FOND” entries with “FOND DU LAC”.  Such a procedure is repeated for other problems we noted. There are 300+ problems we found, so this is not a perfect fix yet, (as noted by entry v9/2 in the “after” image below), but it’s much better. The second thing we fix is putting “COUNTY” horizontally under “TOTAL”, since in the original document (pdf), we know that after an entry says “TOTAL”, it always says the total for the county, not for any other grouping.  Lastly, we want it to concatenate horizontally but stop before it hits any part of the income sheet.  Stata uses & for “and” and | for “or”. You may use both in long chains for a single if statement. This last was done by a bit of trial and error to make sure I got it right. That’s fine, since computers are cheap and thinking hurts.  It stops at the line containing every end of line indicator I could think of, like “TAXES”, “POPULATION”, or “REVENUES”, which are clearly not city titles.

```foreach var in `to_combine'{

*We have a space problem (Part 1)
replace `var'="" if `var'=="DU"
...snipped...
replace `var'="LA CROSSE" if `var'=="LA"

*Puts county under total.  It's a mandatory identifier visible in the pdf. (Part 2)
replace `var'="" if `var'=="COUNTY"
replace `var'="COUNTY" if `var'[_n-2]=="TOTAL"

*Concatenates appropriately, stopping if it hits anything
replace `var'=`var'[_n-1]+"."+`var' if var2[_n-1]!="POPULATION" & var1[_n-1]!="REVENUES" & var1[_n-1]!="TAXES" & mark_3[_n-1]!=1 & var2!="POPULATION" & var1!="REVENUES" & var1!="TAXES"

}```

Now, let’s just drop the stuff we don’t need, and we’re looking at a much much cleaner worksheet, giving us the before/after image shown below:

```*Drops excess rows or columns.
drop if mark==1
drop if mark_1==1
drop if mark_2==1
drop if var2=="POPULATION"
drop if var1=="REVENUES"
drop if var1=="TAXES"``` 