Indirect Utility

Suppose the indirect utility of choosing alternative is specified as

where is a vector of features of alternative and follows the GEV.

Derivation

Alternative is selected if for any . Hence, the probability of choosing is

Note that

where is a constant.

The probability of choosing is then written as

where , , and .

Logit Regression

Reference Link: Logistic regression - Wikipedia

Let one of the alternatives be an opt-out and normalize the opt-out be zero. The logit form is given by

where is a set of mutually exclusive alternatives included in the analysis.

Example

Data (5 Transactions and 5 Items)

Support and Confidence

if Antecedent then Consequent	supp	conf	supp X conf
if buy milk, then buy bread
if buy milk, then buy eggs
if buy bread, then buy fruit
if buy fruit, then buy eggs
if buy milk and bread, then buy fruit

• Itemset has a support of 0.4 since it occurs in 40% of all transactions.

• The rule has a confidence value of , suggesting butter is bought 50% of the times when milk and bread are bought.

Encouragement Design

A: Smoking during Pregnancy (yes/no)

Y: Birth Weight

X: Parity, Mother’s Age, Weight, etc.

Z: Randomize to either receive encouragement to stop smoking (Z=1) or receive usual care (Z=0)

The encouragement will not affect the baby’s birth weight through any path other than stopping smoking.

• Intention-to-treat Analysis: Focus on the causal effect of encouragement

  

  Note it is NOT a causal effect of smoking.

Causal Effects

A motivation for using IV methods is concern about possible unmeasured confounding.

• If there is unmeasured confounding, we cannot marginalize over alll confounders via matching, IPTW, etc.

  Both matching and IPTW create a balanced sample or pseudo-population to measure the causal effect averaged over the whole population.

IV methods do not focus on the average causal effect for the population. They focus on a local average treatment effect (LATE) evaluated from a subpopulation of compliers.

• It is also known as the complier average causal effect (CACE).

Observed Data

				Class
0	0	0	?	Never-takers or Compliers
0	1	1	?	Always-takers or Defiers
1	0	?	0	Never-takers or Defiers
1	1	?	1	Always-takers or Compliers

• Compliance classes are also known as principal strata, which are latent.

• Without additional assumptions, we cannot identify each subject into one of four categories. We can only narrow it down to two categories, however.

Summary

• IV requires 2 key assumptions, the strongest of which is the exclusion restriction.

• If one also makes the monotonicity assumption, then the complier average causal effect is identified.

Strength of IVs

Estimate the proportion of compliers:

• If this number is close to 1, it has a strong instrument;

• If close to 0, it has a weak instrument.

This proportion implies an effective sample size.

Suppose only 1% of the population are compliers. This leads to very large variance estimates, resulting in unstable estimates of causal effects. It would likely produce confidence intervals that are too wide to be useful.

Strengthen Instrument: Near/far Matching

• Match so that covariates are similar but the instrument is very different

• Reference: Baiocchi et al. (2012) “Near/far matching: a study design approach to instrumental variables.”

Trimming the Tails

• Large weights occur at observations in the tails of the propensity score distribution

• Trimming the tails can eliminate some of the extreme weights

  • A common strategy:

    • Remove treated subjects whose propensity scores are above the 98th percentile from the distribution among controls

    • Remove control subjects whose propensity scores are below the 2nd percentile from the distribution among treated subjects

Weight Truncation

Truncation Steps:

1. Determine a maximum allowable weight

   • Could be a specific value (e.g., 100)

   • Could be based on a percentile (e.g., 99th)

2. If one weight is greater than the maximum allowable, set it to the maximum allowable value

   • For example, if your upper limit is 100, someone who weights 1000 will be set to 100 instead.

Truncation changes the data and hence introduces bias. So there involves a bias-variance trade-off:

• Truncation: Bias, smaller variance

• No Truncation: Unbiased, larger variance

Judge if the trade-off is worth based on the mean squared error. If truncating a small number of observations (not too many) that have extreme weights, you’ll probably be better off in terms of mean squared error.

Motivation

If there is hidden bias, how severe it would have to be to change conclusions.

• Let and be the probability that person and receive treatment

• Suppose person and are perfectly matched so that their observed covariates and are the same.

• Inequality

  

  where is odds ratio.

  • If , then , suggesting no hidden bias

  • implies hidden bias

  • is the odds of treatment for person

Balancing Score

Formally,

where represents the distributions of covariates at every value and the propensity score function generates the value of .

Implication: If we match according to the propensity score, we should achieve balance.

Estimated Propensity Score

Estimate to obtain

• The outcome is A, which is binary

• Estimate the propensity score using logistic regression

  1. Fit a logistic regression model: outcome A and covariates X

  2. From that model, get the predicted probability (fitted value) for each subject, which is the estimated propensity score,

In the randomized control trials, the propensity score is 0.5 by design, since given X subjects are randomly assigned to the treatment or the control group.

Trimming Tails

Remove subjects who have extreme values of the propensity scores

• Control subjects whose propensity score is less than the minimum in the treatment group

• Treated subjects whose propensity score is greater than the maximum in the control group

Trimming the tails makes the positivity assumption more reasonable and prevents extrapolation.

Matching

Compute a distance between the propensity score for each treated subject with every control. Then use the nearest neighbor (i.e., greedy mathcing) or optimal matching as before.

In practice, logit (log-odds) of the propensity score is often used, rather than the propensity score itself.

• The propensity score is bounded between 0 and 1, making many values similar

• Logit of the propensity score is unbounded – this transformation essentially stretches the distribution, while preserving ranks.

• Match on rather than

Caliper

In practice, a common choice for a caliper (i.e., the maximum distance we are willing to tolerate in matching) is 0.2 times the standard deviation of the logit of the propensity score.

1. Estimate the propensity score using logistic regression

2. logit-transform the propensity score

3. Take the standard deviation of this transformed variable

4. Set the caliper to 0.2 times the value from step 3

A smaller caliper suggests less bias but more variance.

Post-matching

The outcome analysis methods can be the same as those that would be used if matching directly on covariates. For example,

• Randomization tests

• Conditional logistic regression, GEE, Stratified Cox Model

Matching

Matching is a method that attempts to make an observational study more like a randomized trial.

• Controlling for confounders is achieved at the design phase without looking at the outcome. In other words, matching is similar to randomized control trials (RCT) that we are blind to the outcomes.

• Matching reveals the lack of overlap in covariate distribution.

• Once matched, data are treated as if from a randomized trial.

Stochastic Balance

In a randomized trial, treated and control subjects are not perfectly matched. The distribution of covariates, however, is balanced between groups, which we call stochastic balance.

• Matching on covariates can achieve stochastic balance.

• We start with the treated group and make the distribution of covariates in the control population look like that in the treated population.

  • The estimate from this procedure is a causal effect of treatment on the treated.

• In matching, we typically focus on the causal effect of treatment on the treated, because we find matches for each treated person from the control population. So what we are making inferences about is the treated population.

  • We can do matching where we make the treated and the control populations not only look like each other but also look like the population as a whole. It involves more complicated techniques but is feasible.

Fine Balance

Sometimes we might be willing to accept some non-ideal matches if treated and control groups have the same distribution of covariates.

[Example]

Match 1: treated, male, age 40 and control, female, age 45

Match 2: treated, female, age 45 and control, male, age 40

It is not an ideal match because the treated male and female are matched with the control female and male of different ages. We don’t achieve the stochastic balance. However, the overall average age and percentage of males are the same in both groups, even though neither match is great. This is considered a fine match.

It’s always going to look for the best pairwise kind of matches, but we will tolerate some bad matches as long as we get a fine balance.

Number of Matches

• One-to-one (pair matching)

• Many-to-one (k controls to every treated subject)

• Mixed

  • Sometimes match 1, sometimes more than 1, control to treated subjects

  • If multiple good matches are available, use them.

记忆中的清明

往年清明节当天，都是父亲领着我们兄妹几个，上午去上坟，逐个祭拜爷爷奶奶，母亲，及更早的祖辈，挂些纸钱，烧些纸金银包、冥币。晚上在祖屋厅堂供上米饭鱼肉，点香祭拜列祖列宗，每次重复默念的祭词总少不了“父亲不要睡着了，要保佑父亲一身之下的人，保佑大家在外上车下车平平安安、工作顺顺利利，小孩读书年年升级、步步升高，好人相逢，恶人躲避……”。

从1982年到南昌读书，再到84年毕业分配至上饶工作起，因路途遥远、交通不便，记忆中就没有回官坑过清明了。但唇齿间总回味着清明果、蕨菜、小竹笋、猪头花[①]等家乡食物的味道，脑海里放映着石板路旁、小河边、房前屋后、稻田里、漫山遍野的各种油菜花、野花盛开的景象。到1998年，我考取了机动车驾驶证，从此就能自己开车往返工作地和官坑了。再后来，交通路况也逐步好起来了，也买了私家车。虽然因工作变动，我搬到了南昌，但感觉回官坑的路程也不那么遥远了。往后，官坑就成了清明节必须去的地方，这个习惯也就慢慢变成了责任。

模范农民

小学期间的事，因为年纪小，我大多已经记不得了。只是模糊记着，读小学的时候，我不爱读书，家里给每个人都分配了任务。我的主要任务是早晚放牛，牵牛喝水，投放牛料，打猪草。没有事的时候，我会经常跑到六队队屋里玩。那时父亲是生产队会计，每到生产队分玉米、分红薯的时候，我们小孩就特别高兴，而父亲就要算账，特别忙。每人或每户应分多少，按什么标准（权重）分，而且那时只能用算盘来加减乘除。因此，父亲也成了乡间算盘高手。有一次，县茶厂来招统计员，办事的专员发现父亲是乡间秀才，希望调父亲去县茶厂，可后来大队不同意，就此作罢。那时父亲也常常跟我们讲，经手财会类的事，一分钱都不能少、不能错。后来父亲也不止一次跟我们讲起一个事。有次生产队的社员质疑有一笔上级补助款大家没有分到，而且是好多年前的事。父亲就把账本、单据找出来跟大家仔细讲，多少用于计划生育妇女补助，多少替大家抵交农业税等等，讲得大家心服口服。平常父亲也跟我们几个兄妹讲“命里有时终须有，命里无时莫强求”，我们也时刻把父亲的话记在心里。

去年这个时候我们在父亲房间为他守灵时，看到父亲房间里两把用得破旧的算盘和一些记着生产队账务的落满灰尘的账本、发黄的单据依然挂在昏暗的木板墙上。

也许因为父亲读了点书，且特别尊尚传统文化、古人警言，常常在跟没有读过书的社员们讲道理时，挂着口头禅，经常说“过去老几（辈）人，如何如何,我们应该怎样怎样”。因此，村里人给父亲取了个绰号，叫“清朝”。

当年父亲不仅队长、会计做得称职，田地里的农活、挑驮力气活样样不落下。父亲常常跟我们讲，那时到秋口挑米，到屯溪挑茶是家常便饭。还说有一次清明前后挑担路过段莘溪边一户亲戚家，一口气吃了十八个清明果，可见长途挑担体力消耗之大。

还有一次，不记得我和兄弟谁，进七里源砍柴回家，路过上村对面父亲做砖瓦的草棚时，肚子饿得难受。父亲赶紧跑到棚边上，摘了一个自己种的嫩南瓜，炒起来让我们当饭吃，我们非常高兴，父亲也经常当笑话说起这事。

自力更生建房

记得1976年，我11岁，家里人多，祖上分下来的老屋住不下了。经过大队批准，父亲开始在太平灶北边的一个山坳里伐木，从段莘水库移民老屋基捡挑旧砖，在老屋后边大约80平米的菜园地里做了两直传统木屋。父亲常在我们面前自豪地说，当时猪肉价0.74元一斤，做这栋房屋请工匠等总共花费现金大约两千元，其余基本都是自己完成。我和华锋哥虽然也常常帮忙去山上驮一些小方料，放学后也到官坑口挑几块砖，但那时我们力气很小，只能帮一点点忙。想来父亲到老年，右肩驼得厉害，比左肩凸出好多，可能是中年时用力过度所致。何况当时驮木头，从山坳里上来要经过500米左右的羊肠上坡路。官坑也没通公路，从官坑口挑砖也要经过一段600米左右的上岭。一座房子的物料需要多少次日日夜夜的挑运啊！

终于在1978年，一家人住进了新屋，小弟华斌就是在新屋里出生的。也就是这段日子，父亲自力更生、勤劳节俭、精打细算的特质影响了我们兄妹几个一辈子。

送我读书

1979年，我在官坑附中读完两年制初中，毕业后前往河坑源段莘乡中学读高中。第一次要离开父母，离开官坑，住校读书。可能是年龄小不懂事，不习惯陌生环境，读书兴趣也不高。记得读了一个多月，便借故跑回家里不想读，拿起镰刀就要上山砍柴去，硬是被父亲打回学校的。印象中那是被打得最厉害的一次，我也感觉当时是我犟得最厉害的一次，还回嘴对骂，言语也很难听。如果要说一辈子我有什么对不住父母的话，那是唯一的一次，也是最后悔的一次。

两年高中，我很快就混到了毕业，1981年第一次参加高考。也许是段莘中学英语教学差，大家基本上都报考中专，结果只有一个女生去了上饶卫校。记得恢复高考前几年，段莘中学考上的人很少。官坑村只有两个人，一个是同一个生产队的汪新华，读了上饶师范，另一个是堂叔汪天祥，读了上饶卫校。后来不知是因为官坑有个慈祥的长辈洪应天老师在江湾中学当校长之故，还是华锋哥学了一年手艺又回到学校想读书的事影响了我，或是受父亲经常在我们面前讲“万般皆下品，唯有读书高”的影响，1981年下半年我又到江湾中学复读去了。

记得当时16岁的我，也许是农村人那时普遍营养不良，体重大约70斤左右，很是瘦小。第一天去江湾中学报到是父亲挑着被子和木箱子送我走路去的。快到江湾中学附近，路过一片雪梨地时，记得父亲买了两个让我们尝了尝。

从那天开始，我感觉才真正懂事了，才开始刻苦读书了。一方面，想想父母送两兄弟复读，在农村的确不易，也不多见。另一方面，或许是暑假期间跟着母亲田地里除草，热得受不了。或许是上山摘箬皮卖钱时，头怕山蜂，脚怕毒蛇的情景刺激着我。功夫没有白费，一年后终于以江湾中学第一名的成绩被录取到江西银行学校农村金融专业，更高兴的是华锋哥也同时考进了上饶农校。

家书抵万金

在通讯落后、交通不便的日子里，“家书抵万金”是现在很多年轻人无法体会的。就是一元钱的路费常常也舍不得花，乡愁就只能浓缩在一枚八分钱的邮票里了。从到了南昌读书后，直到上世纪九十年代末，与家里的联系就靠这枚小小的邮票了。

我和华锋哥进了中专学校读书后，乡村里许多一辈子跟泥土打交道的亲戚朋友都很羡慕。可是命运却开了个不小的玩笑，好像上帝故意要考验父亲一样，坏消息一个接一个：1982年下半年母亲生病，后来被诊断出是鼻咽癌；1983年大弟华树又生了一次大病，送外地就医；再后来父亲在田里干活时手又被蛇咬了……这些都靠书信来往得知。那时很希望一下课就能收到家里的来信，但信捏在自己手里的时候，又不敢拆开，怕看到坏消息。最后，最不幸的事还是发生了，1983年5月母亲还是离世了。

再后来，父亲家事、农活一个人更忙了，还有弟妹要读书要照顾抚养，有时一两个月都没有时间回我们的信。时间长了，心里就格外担心害怕。直到我和华锋哥毕业参加工作，一切才慢慢好起来。这时我们不仅常常写信向父亲报告一些好消息，也会通过邮局寄点钱回家补贴家用。

1988年，我写信回家，告诉父亲我找对象了，1989年春节带回官坑过年。父亲虽然言语不多，但内心的高兴还是看得出来的。当年4月28日，父亲和华锋哥来上饶参加我的结婚典礼，记得那天晚上，父亲们很晚到上饶。一方面是客车在路上耽误了，另一方面是两个人都没有来过上饶，下车后不认识路。那时小城市即买不到地图，路牌也很不规范，更没有电话手机。记得到我单位上饶农业银行都晚上十点钟了。第二天，新娘下婚车放鞭炮时，父亲也许是激动紧张，用一根火柴，划了好几次都未成功。这时我的一位退休留用的阿姨同事，一把抓过父亲手中的火柴，几根火柴一起划，着了，鞭炮响了。后来，我那位阿姨同事常常在我面前提起这个小插曲。

1990年2月小岛出生，我也是第一时间写信向父亲报喜。转眼到了1994年，华树毕业，分配到婺源邮电局[②]工作，一切都好起来了，交通也改善了，写信的次数就越来越少了。又过了几年，到了本世纪初，官坑外边溪关娣家安装了一部无线电话[③]，我们有急事、好事就通过关娣家电话联系了。好像又过了一年多，华树就帮我们自己家也装了一部无线电话，从那以后就再也没有写过信了，一家人相聚也容易多了。

幸福的晚年

2005到2006年间，我们四兄妹在双河汇合处合资建新房。那时父亲虽已年近七十，但头脑清晰，很会算账，又有建房经验，仍担任家里的总指挥。记忆中，这是父亲最后做的一件大事。2007年春节，我们兄妹五家大大小小都住到了新房里过年。从那时算起，父亲才开始真正享受自己的晚年生活，吃喝、洗衣有旺生和兴珠伺候。

日子好了，天天都好像过年过节一样，想吃什么都有。电话方便了，不仅座机有了，华树还帮父亲办了手机。有什么事或想谁了，一个电话就解决问题了。交通也方便多了，水泥路、高速路陆续通了。我们兄弟几个也都买了私家车，逢年过节大家总会不约而同地跑去官坑看望父亲。与此同时，我们也了却了心中的乡愁，品尝了小时候的味道。碰上不错的时机，我们也会带父亲出门走走，看看外面的世界，体验一下电视新闻里才看得到的地方。

记得带父亲到南昌过年的时候，父亲说城市里怎么到处是人山人海；记得开车带父亲去赣州玩，父亲还说车子没有坐够；记得路过吉安吃饭时，上了一份驴肉，回来后常常说好吃；带父亲去杭州玩时，父亲总回想小时候书上讲“上有天堂，下有苏杭”，一看西湖怎么这么大；带父亲去上海时，父亲感觉酒店里的自助早餐真实在，东方明珠塔怎么能建得那么高，同时也去看了我爷爷在上海生活期间置办的小屋，了结了多年的心愿。再从上海坐飞机回南昌时，父亲说怎么就只坐了一下下子。

我还记得带父亲去惠州华斌家过年的时候，父亲在街上看到人家过年打赤膊很是惊奇。我还记得父亲，常常讲华锋带着去婺源乡下吃米饭蒸狗肉，低头一口气吃了三小碗。也常讲华斌带父亲去北京时，北京的饺子如何如何实惠。还常讲县里华树家声琴做的饺子、包子是如何如何好吃。

也因为父亲走得多、看得多，在官坑，父亲跟亲朋好友谈天时，或是跟兴珠家游客吹牛时，话题自然丰富了很多。近年来，我问父亲想去哪，他说想去的都去过了，不想了，也许这就是知足吧。

晚年期间，每当我们有好友去官坑玩，或是兴珠家游客高兴起哄时，父亲偶尔也在小院里唱唱婺源小调。我们听得最多的是《劝善记》，虽然调子有些凄凉，但歌中劝大家和谐相处、与人为善的词句让人深受启发，我们兄妹几个也都受益匪浅。

因父亲略知中国民间风水历学，特别是熟悉婺源本地民间风俗礼仪，所以闲空之时官坑本地亲朋好友举办红白喜事、建屋盖房都喜欢请父亲挑日子、看风水、写对联、做司仪。亦或是分家，也愿意请父亲当公证人或代理书写契约。对于这些事，父亲从不开价收费。相反，他总是乐于相助，在忙碌中实现个人价值。

最后的时光

近几年来，父亲的听力减退了，我们也不常通电话。每逢节假日，我几乎都要开车去官坑看父亲。每次只要是讲了大概什么时间会到，父亲一定会站在停车场等候。同样，每次开车离开，父亲也一定会去停车场送别。虽然煽情的话不多，但彼此的内心一定是温暖的，现在这种感觉突然就没有了。

近几年，每次大年三十晚上团圆饭后，父亲都会提着小蓝子给我们大家发红包。无论男女老少，每人200元，很是热闹。小岛在美求学，因此也缺席了多年。但每次父亲见到我，总会提起，“小岛什么时候毕业，让他回来工作”。这份惦记，也许就是父亲心底里对孙辈的挂念吧。

近几年清明节，或是大年初二，父亲带我们去爷爷奶奶坟前祭拜结束后，他都会指着爷爷坟墓右边30米左右的一处空地，跟我们小声说上一句，“那个地方可以，离这也近，我以后可以放那”。去年清明，我问父亲寒食节是怎么来的，为什么婺源规矩不能挂纸钱，父亲跟我讲过后又拿出一本古书让我查对。最后还补上一句，“这本书人家出一万元想收购我都不肯”，又说，“以后这本书就传给你好了”。感觉是父亲自己知道时日不多，在妥善安排后事一样。没想到几天后，他就真要离开我们了。那时也没有机会说上一句话，也许这就是天意。

俗话说，“父母在，家就在。父母离去，人生只剩下归途”。现在终于体会到了其中的深意。在父亲离开的这一年里，我时常想起他过往的点点滴滴。各种场景涌上心头，感觉挥之不去。于是，我打算找个时间做个记录，传于后人。恰好这个清明假期，哪里也去不了，便打开电脑写下这些文字，当作对父亲的周年纪念吧。愿父亲在另一个世界里还是那么乐观、知足、快乐！

华亮

二零二二年农历三月初五 清明节

①: 猪头花是官坑一种可以蒸起来吃的野花。

②: 1994年，邮电局负责邮政、电信、移动三项业务。

③: 官坑的无线电话依靠段莘乡山顶一个无线差转台传送信号。

Example 1

• i.e., C is independent of all variables

• i.e.,$ BD$,

• , because B and D are marginally dependent

• 

Example 2

• i.e.,
• i.e.,

Example 3

• i.e.,
• i.e.,

Decomposition of Joint Distribution

We can decompose the join distribution by sequential conditioning only on sets of parents.

• Start with roots (nodes with no parents)

• Proceed down the descendant line, always conditional on parents

Type of Paths

• Fork

  

• Chain

  

• Inverted Fork

  • E is a collider

  

Blocking

• Paths and associations can be blocked by conditioning on nodes in the path.

• However, the opposite situation occurs in inverted paths if a collider is conditioned on.

Rules of d-separation

A path is d-separated by a set of nodes C if:

• It contains a chain (e.g., ) and the middle part is in C (i.e., )

  Example D is temperature, E is sidewalks are icy, and F is someone falls. If we restrict to situation where sidewalks are icy (condition on G), then temperature and falling are not associated via this path.

• It contains a fork (e.g., ) and the middle part is in C (i.e., )

  In this path, D and F are dependent because of E. If E is given or fixed, E no longer affects D and F. Hence, they are independent (i.e., the path is blocked).

• It contains an inverted fork (e.g., ) and the middle part is NOT in C, nor are any descendants of it. In other words, no need to control anything in this path where is collider and the relation between and is independent.

  Example is the state of an ON/OFF switch, while is the state of another switch. Both switches control the same light. We decide the state of and by flipping a fair coin, respectively. is the event that the light is ON, where the light is ON if only if both switches are ON.

  Hence, and are independent if the information on is not known. In contrast, and are dependent if is given/controlled.

Confounders

A confounder is a variable that affects both the treatment and the outcome.

Frontdoor Paths

A frontdoor path from A to Y is one that begins with an arrow emanating out of A.

Backdoor Paths

A backdoor path from treatment A to outcome Y is a path from A to Y that travels through arrows going into A.

Note: A path is a link between two variables regardless of arrow directions

Example

• There is one backdoor path from A to Y, that is,

  • It is blocked by a collider M

  • No confounding (Information from V and W does not flow to Y and A because of M)

• Set of variables that are sufficient to control for confounding:

  • Solutions: {},{V},{W},{M, W}, {M, V}, {M, V, W}

  • Never only control {M} since it will open a path between A and Y that is not causal

• There are two paths

  • 

    • No colliders

    • Solutions: {Z}, {V}, or {Z, V}

  • 

    • Z is a collider

    • Solutions: {V}, or {Z, V}, {Z, W}, and {Z, V, W}

      Once we control for Z, we need to block the backdoor path by including V, or W, or V and W

Summary

To use the backdoor path criterion, we are required to know the whole DAG.

Example

• Observed pre-treatment variables: {M, W, V}

• Unobserved pre-treatment variables: {U1, U2}

• Suppose W and V are causes of A, Y, or both and M is not a cause of either A or Y

Solution 1: Use all pre-treatment covariates, i.e., {M, W, V}

Solution 2: Use variables based on disjunctive cause criterion, i.e., {W, V}

Let’s check if these solutions fit in the DAG for

Hypothetical DAG 1

Hypothetical DAG 2

Hypothetical DAG 3

Hypothetical DAG 4

Summary

The disjunctive cause criterion:

• does not always select the smallest set of variables to control for (e.g., if we only have one collider in the whole DAG, the smallest set is , that is, doing nothing)

• but is conceptually simpler (i.e., list variables that are causes of treatment or outcome or both)

• is guaranteed to select a set of variables that are sufficient to control for confounding if

  • such a set exists

  • all of the observed causes of A and Y are correctly identified (no need to know the whole DAG but do have to know about the relationship between observed variables)

Robustness Check

1. We can include extra demographics and use a polynomial regression including , , or to fit the data as close as possible.

2. We care most about the observations right around the threshold rather than observations far away from the center. Therefore, we should only include the students who had scores just barely under and over 70. Accordingly, We can fit the same model and restrict it to students within a smaller window, or bandwidth, like , or .

tutoring[, entrance_centered := entrance_exam - 70]

model_simple <- lm(exit_exam ~ entrance_centered + tutoring,
                   data = tutoring)
# round(summary(model_simple)$coef, 4)

model_poly <- lm(exit_exam ~ entrance_centered + I((entrance_centered/10)^2) + 
                   I((entrance_centered/10)^3) + I((entrance_centered/10)^4) + 
                   tutoring, data = tutoring)
# round(summary(model_ploy)$coef, 4)

model_bw_10 <- lm(exit_exam ~ entrance_centered + tutoring,
                  data = filter(tutoring,
                                entrance_centered >= -10 &
                                  entrance_centered <= 10))
# round(summary(model_bw_10)$coef, 4)

model_bw_5 <- lm(exit_exam ~ entrance_centered + tutoring,
                 data = filter(tutoring,
                               entrance_centered >= -5 &
                                 entrance_centered <= 5))
# round(summary(model_bw_5)$coef, 4)

model_bw_5_poly <- lm(exit_exam ~ entrance_centered + I((entrance_centered/10)^2) +
                        I((entrance_centered/10)^3) + I((entrance_centered/10)^4) + 
                        tutoring,
                      data = filter(tutoring,
                                    entrance_centered >= -5 &
                                      entrance_centered <= 5))
# round(summary(model_bw_5_poly)$coef, 4)

The effect of tutoring differs a lot across these different models, from 9.1 to 10.8. The comparison between RMSEs from model_bw_5 and model_bw_5_poly might suggest the overfitting issue resulting from the over-specification with several polynomial terms and the small size of sample.

Robustness Check

1. Try different Kernel functions

   By default rdrobust() uses a triangular kernel (linearly decreasing weights). We can also use Epanechnikov (non-linearly decreasing weights) or uniform (equal weights, i.e., unweighted).

2. Try different the bandwidth algorithms

rdbwselect(y = tutoring$exit_exam, x = tutoring$entrance_exam, c = 70, all = TRUE) %>%
  summary()

## [1] "Mass points detected in the running variable."
## Call: rdbwselect
## 
## Number of Obs.                 1000
## BW type                         All
## Kernel                   Triangular
## VCE method                       NN
## 
## Number of Obs.                  237          763
## Order est. (p)                    1            1
## Order bias  (q)                   2            2
## Unique Obs.                     155          262
## 
## =======================================================
##                   BW est. (h)    BW bias (b)
##             Left of c Right of c  Left of c Right of c
## =======================================================
##      mserd     9.969      9.969     14.661     14.661
##     msetwo    11.521     10.054     17.067     14.907
##     msesum    12.044     12.044     17.631     17.631
##   msecomb1     9.969      9.969     14.661     14.661
##   msecomb2    11.521     10.054     17.067     14.907
##      cerrd     7.058      7.058     14.661     14.661
##     certwo     8.156      7.118     17.067     14.907
##     cersum     8.526      8.526     17.631     17.631
##   cercomb1     7.058      7.058     14.661     14.661
##   cercomb2     8.156      7.118     17.067     14.907
## =======================================================

1. Compare the results using the ideal bandwidth, twice the ideal, and half the ideal

# Ideal bandwidth 
rdbwselect(y = tutoring$exit_exam, x = tutoring$entrance_exam, c = 70) %>%
  summary()

## [1] "Mass points detected in the running variable."
## Call: rdbwselect
## 
## Number of Obs.                 1000
## BW type                       mserd
## Kernel                   Triangular
## VCE method                       NN
## 
## Number of Obs.                  237          763
## Order est. (p)                    1            1
## Order bias  (q)                   2            2
## Unique Obs.                     155          262
## 
## =======================================================
##                   BW est. (h)    BW bias (b)
##             Left of c Right of c  Left of c Right of c
## =======================================================
##      mserd     9.969      9.969     14.661     14.661
## =======================================================

rdrobust(y = tutoring$exit_exam, x = tutoring$entrance_exam, c = 70, h = 9.969) %>%
  summary()

## [1] "Mass points detected in the running variable."
## Call: rdrobust
## 
## Number of Obs.                 1000
## BW type                      Manual
## Kernel                   Triangular
## VCE method                       NN
## 
## Number of Obs.                  237          763
## Eff. Number of Obs.             144          256
## Order est. (p)                    1            1
## Order bias  (q)                   2            2
## BW est. (h)                   9.969        9.969
## BW bias (b)                   9.969        9.969
## rho (h/b)                     1.000        1.000
## Unique Obs.                     155          262
## 
## =============================================================================
##         Method     Coef. Std. Err.         z     P>|z|      [ 95% C.I. ]       
## =============================================================================
##   Conventional    -8.578     1.601    -5.359     0.000   [-11.715 , -5.441]    
##         Robust         -         -    -3.276     0.001   [-12.483 , -3.138]    
## =============================================================================

rdrobust(y = tutoring$exit_exam, x = tutoring$entrance_exam, c = 70, h = 9.969 * 2) %>%
  summary()

## [1] "Mass points detected in the running variable."
## Call: rdrobust
## 
## Number of Obs.                 1000
## BW type                      Manual
## Kernel                   Triangular
## VCE method                       NN
## 
## Number of Obs.                  237          763
## Eff. Number of Obs.             206          577
## Order est. (p)                    1            1
## Order bias  (q)                   2            2
## BW est. (h)                  19.938       19.938
## BW bias (b)                  19.938       19.938
## rho (h/b)                     1.000        1.000
## Unique Obs.                     155          262
## 
## =============================================================================
##         Method     Coef. Std. Err.         z     P>|z|      [ 95% C.I. ]       
## =============================================================================
##   Conventional    -9.151     1.130    -8.100     0.000   [-11.365 , -6.937]    
##         Robust         -         -    -4.980     0.000   [-11.670 , -5.078]    
## =============================================================================

rdrobust(y = tutoring$exit_exam, x = tutoring$entrance_exam, c = 70, h = 9.969 / 2) %>%
  summary()

## [1] "Mass points detected in the running variable."
## Call: rdrobust
## 
## Number of Obs.                 1000
## BW type                      Manual
## Kernel                   Triangular
## VCE method                       NN
## 
## Number of Obs.                  237          763
## Eff. Number of Obs.              82          109
## Order est. (p)                    1            1
## Order bias  (q)                   2            2
## BW est. (h)                   4.984        4.984
## BW bias (b)                   4.984        4.984
## rho (h/b)                     1.000        1.000
## Unique Obs.                     155          262
## 
## =============================================================================
##         Method     Coef. Std. Err.         z     P>|z|      [ 95% C.I. ]       
## =============================================================================
##   Conventional    -8.201     2.348    -3.493     0.000   [-12.803 , -3.600]    
##         Robust         -         -    -2.032     0.042   [-13.618 , -0.246]    
## =============================================================================

Fuzzy Gap

First, let’s look at a histogram that shows the probability of being in the tutoring program at different entrance exam scores.

tutoring_fuzzy[, exam_bin := cut(entrance_exam, breaks = seq(0, 100, 5))]

tutoring_bins <- tutoring_fuzzy[, .(count = .N, count_t = sum(tutoring), 
                                    per = 100*sum(tutoring)/.N)
                                , by = 'exam_bin'][order(exam_bin)]

ggplot(data = tutoring_bins, aes(x = exam_bin, y = per)) + 
  geom_col() + geom_vline(xintercept = 8.5) +
  labs(x = "Entrance exam score", y = "Proportion of students participating in program")

If this were a sharp design, every single bar to the left of the cut point would be 100% and every single bar to the right would be 0%, but that’s not the case in this fuzzy design.

• 100% of students who scored between 25 and 50 on the entrance exam used tutoring
• This rate drops to 80ish% up until the cut point at 70
• There’s about 15% chance of using tutoring if students are above the threshold

Discontinuity in Outcome

We can visualize the gap by making scatter plots of running variable against outcome. Moreover, we can fit data with linear and local polynomial regressions on both sides of the threshold.

Fit data with linear regressions

Fit data with local polynomial regressions

Parametric and Non-parametric Estimation

Given the compliance issues, we need to isolate causal effects for compliers from others, i.e., the complier average causal effect (CACE).

As a result, we need an instrument which implies what should have happened rather than what actually happened. In a fuzzy design, the variable indicating if someone is above or below the threshold is a valid instrument.

Particularly, in this example, let “entrance score less than 70”, “participation in tutoring”, and “exit score” be denoted as , , and . The indication variable of “entrance score less than 70” satisfies all requirements for a valid instrument:

• Relevance: The cutoff affects the access to the tutoring program; and .
• Exclusion: The cutoff affects exit exam scores only through the tutoring program; and .
• Exogeneity: Unobserved confounders between the tutoring program and exit exam scores are unrelated to the cutoff in local. For example, IQ is an unobserved variable in the outcome equation. The implicit assumption is that students who scored barely below or above the cutoff have the same level of IQ.

tutoring_fuzzy[, entrance_centered := entrance_exam - 70]

# Run a regression in a sharp RD design
model_sharp <- lm(exit_exam ~ entrance_centered + tutoring,
                  data = filter(tutoring_fuzzy,
                                entrance_centered >= -10 &
                                  entrance_centered <= 10))

# Run a regression in a fuzzy RD design
model_fuzzy <- estimatr::iv_robust(exit_exam ~ entrance_centered + tutoring |
                                     entrance_centered + leq70,
                                   data = filter(tutoring_fuzzy,
                                                 entrance_centered >= -10 & 
                                                   entrance_centered <= 10))

modelsummary(list("Sharp RD (wrong)" = model_sharp,
                  "Fuzzy RD (bw = 10)" = model_fuzzy),
             gof_omit = "IC|Log|Adj|p\\.value|statistic|se_type|RMSE|Std.Errors",
             stars = TRUE, output = "table2.tex") 

Note:

• We’re estimating a CACE or a local average treatment effect (LATE) for people in the bandwidth, because we’re working with regression discontinuity.
• We’re estimating the CACE/LATE for compliers only, because we’re using instruments.
• We should check the robustness of estimates by modifying the bandwidth, adding polynomial terms, and others that we discussed in Practice 6.

Also, we can estimate the CACE in non-parametric methods. We use the fuzzy argument in rdrobust() to specify the treatment column (or tutoring in our case). Importantly, we do not need to specify an instrument (or even create one!). All you need to specify is the column that indicates treatment status. rdrobust() will do all the above/below-the-cutoff instrument stuff automatically for us.

rdrobust(y = tutoring_fuzzy$exit_exam, x = tutoring_fuzzy$entrance_exam,
         c = 70, fuzzy = tutoring_fuzzy$tutoring) %>%
  summary()

## Call: rdrobust
## 
## Number of Obs.                 1000
## BW type                       mserd
## Kernel                   Triangular
## VCE method                       NN
## 
## Number of Obs.                  238          762
## Eff. Number of Obs.             170          347
## Order est. (p)                    1            1
## Order bias  (q)                   2            2
## BW est. (h)                  12.985       12.985
## BW bias (b)                  19.733       19.733
## rho (h/b)                     0.658        0.658
## Unique Obs.                     238          762
## 
## =============================================================================
##         Method     Coef. Std. Err.         z     P>|z|      [ 95% C.I. ]       
## =============================================================================
##   Conventional     9.683     1.893     5.116     0.000     [5.973 , 13.393]    
##         Robust         -         -     4.258     0.000     [5.210 , 14.095]    
## =============================================================================

Note:

• We should check the robustness of estimates by modifying the bandwidth (ideal, half, double) and using different kernels (e.g., uniform, triangular, Epanechnikov).

Recall:

There is an alternative way to evaluate an causal effect with ONE instrument in regressions:

where the first stage and the reduced form are specified respectively as

model_FirstStage <- lm(tutoring ~ entrance_centered + leq70, 
                       data = filter(tutoring_fuzzy, 
                                     entrance_centered >= -10 & 
                                       entrance_centered <= 10))

model_Reduced <- lm(exit_exam ~ entrance_centered + leq70, 
                    data = filter(tutoring_fuzzy, 
                                  entrance_centered >= -10 & 
                                    entrance_centered <= 10))

model_Reduced$coef['leq70TRUE']/model_FirstStage$coef['leq70TRUE']

## leq70TRUE 
##  9.741044

Intent to Treat (ITT)

We can calculate ITT by assuming the proportion of compliers (c), never takers (n), and always takers (a) are equally spread across treatment and control. This assumption is plausible in a randomized control trial.

Suppose treatment doesn’t make someone more likely to be an always taker or a never taker, we have and . Hence,

Finding ITT Using regression

itt_model <- lm(health ~ treatment, data = bed_nets_time_machine)
summary(itt_model)$coeff %>%
  (function(x) round(x, 4))

##                    Estimate Std. Error t value Pr(>|t|)
## (Intercept)         40.9381     0.4444 92.1143        0
## treatmentTreatment   5.9880     0.6298  9.5081        0

CACE/LATE

Given$ $, we can calculate by obtaining .

table_net <- bed_nets_time_machine[, .N, by=c('treatment', 'bed_net')][
  order(treatment, bed_net)]
table_net[, percent:=round(N/sum(N), 4), by='treatment']
table_net

##    treatment    bed_net   N percent
## 1:   Control    Bed net 196  0.1952
## 2:   Control No bed net 808  0.8048
## 3: Treatment    Bed net 608  0.6104
## 4: Treatment No bed net 388  0.3896

cACE <- itt_model$coef["treatmentTreatment"] / 
  (table_net[treatment == 'Treatment' & bed_net == 'Bed net']$percent - 
     table_net[treatment == 'Control' & bed_net == 'Bed net']$percent)
names(cACE) <- "CACE"
round(cACE, 2)

##  CACE 
## 14.42

model_2sls <- estimatr::iv_robust(health ~ as.factor(bed_net) | as.factor(treatment),
                                  data = bed_nets_time_machine)
round(summary(model_2sls)$coef, 2)

##                              Estimate Std. Error t value Pr(>|t|) CI Lower
## (Intercept)                     52.54       0.84    62.2        0    50.89
## as.factor(bed_net)No bed net   -14.42       1.25   -11.5        0   -16.88
##                              CI Upper   DF
## (Intercept)                     54.20 1998
## as.factor(bed_net)No bed net   -11.96 1998

Directed Acyclic Graph (DAG)

• Identify mediator, confounder, and collider

  • Example

  • DAG for nets

    

    • Control variables?
    • Do we need to control eligible and household

cor(nets$health, nets$household)

## [1] 9.785337e-05

cor(nets$income, nets$resistance)

## [1] 0.01371297

lm(malaria_risk ~ household + net + health + income + temperature,
   data = nets) %>%
  (function(x) summary(x)$coeff) %>%
  (function(x) round(x, 4))

##             Estimate Std. Error  t value Pr(>|t|)
## (Intercept)  76.2067     0.9658  78.9062   0.0000
## household    -0.0155     0.0893  -0.1730   0.8626
## netTRUE     -10.4370     0.2665 -39.1633   0.0000
## health        0.1483     0.0107  13.8997   0.0000
## income       -0.0751     0.0010 -72.5635   0.0000
## temperature   1.0058     0.0310  32.4829   0.0000

Naive Comparison

model_naive <- lm(malaria_risk ~ net, data = nets) 
summary(model_naive)

## 
## Call:
## lm(formula = malaria_risk ~ net, data = nets)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -26.937  -9.605  -1.937   7.063  55.395 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  41.9365     0.4049  103.57   <2e-16 ***
## netTRUE     -16.3315     0.6495  -25.15   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 13.25 on 1750 degrees of freedom
## Multiple R-squared:  0.2654, Adjusted R-squared:  0.265 
## F-statistic: 632.3 on 1 and 1750 DF,  p-value: < 2.2e-16

for (v in c('income', 'temperature', 'health')) {
  plot_command <- paste0('ggplot(data = nets, aes(x =', v,
                         ', fill=net)) +geom_density(alpha = 0.7)')
  print(eval(parse(text = plot_command)))
  test_result <- t.test(as.formula(paste0(v, '~net')), data=nets)
  print(test_result)
}

## 
##  Welch Two Sample t-test
## 
## data:  income by net
## t = -8.8113, df = 1284.7, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -100.79660  -64.08593
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            872.7526            955.1938

## 
##  Welch Two Sample t-test
## 
## data:  temperature by net
## t = 3.492, df = 1404.3, p-value = 0.0004943
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  0.3098586 1.1042309
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            24.08796            23.38091

## 
##  Welch Two Sample t-test
## 
## data:  health by net
## t = -7.6447, df = 1346.9, p-value = 3.961e-14
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -8.610309 -5.093693
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            48.05696            54.90896

Regression

model_regression <- lm(malaria_risk ~ net + income + temperature + health, data = nets)
summary(model_regression)

## 
## Call:
## lm(formula = malaria_risk ~ net + income + temperature + health, 
##     data = nets)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -13.143  -3.915  -0.561   3.333  16.461 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  76.159901   0.926835   82.17   <2e-16 ***
## netTRUE     -10.441932   0.264906  -39.42   <2e-16 ***
## income       -0.075144   0.001035  -72.58   <2e-16 ***
## temperature   1.005855   0.030953   32.50   <2e-16 ***
## health        0.148362   0.010668   13.91   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 5.244 on 1747 degrees of freedom
## Multiple R-squared:  0.8852, Adjusted R-squared:  0.8849 
## F-statistic:  3366 on 4 and 1747 DF,  p-value: < 2.2e-16

Matching (Mahalanobis Distance)

• Mahalanobis distance between and

  

  where is the covariance matrix.

library(MatchIt)

matched <- matchit(net ~ income + temperature + health, data = nets,
                   method = "nearest", 
                   distance = "mahalanobis", 
                   replace = TRUE)
matched

## A matchit object
##  - method: 1:1 nearest neighbor matching with replacement
##  - distance: Mahalanobis
##  - number of obs.: 1752 (original), 1120 (matched)
##  - target estimand: ATT
##  - covariates: income, temperature, health

# Dimension of the data used for matching
dim(matched$X)

## [1] 1752    3

# Number of the treated
sum(matched$treat)

## [1] 681

# Number of the matched control
sum(matched$weights>0)

## [1] 1120

# Number of the unmatched control
sum(matched$weights==0)

## [1] 632

• The matchit() function determines the pair weights by measuring how close the matched pair are. The weights can be used in the regression to account for the variation in distance.

nets_matched <- match.data(matched)
model_matched <- lm(malaria_risk ~ net, data = nets_matched, weights = weights)
summary(model_matched)

## 
## Call:
## lm(formula = malaria_risk ~ net, data = nets_matched, weights = weights)
## 
## Weighted Residuals:
##     Min      1Q  Median      3Q     Max 
## -36.312  -7.605  -1.681   5.395  55.395 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  36.0940     0.5951   60.65   <2e-16 ***
## netTRUE     -10.4890     0.7632  -13.74   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 12.47 on 1118 degrees of freedom
## Multiple R-squared:  0.1445, Adjusted R-squared:  0.1438 
## F-statistic: 188.9 on 1 and 1118 DF,  p-value: < 2.2e-16

for (v in c('income', 'temperature', 'health')) {
  test_result <- t.test(as.formula(paste0(v, '~net')), data=nets_matched)
  print(test_result)
}

## 
##  Welch Two Sample t-test
## 
## data:  income by net
## t = -3.5017, df = 990.87, p-value = 0.0004829
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -64.12955 -18.06677
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            914.0957            955.1938 
## 
## 
##  Welch Two Sample t-test
## 
## data:  temperature by net
## t = 0.79239, df = 952.76, p-value = 0.4283
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -0.2955968  0.6959627
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            23.58109            23.38091 
## 
## 
##  Welch Two Sample t-test
## 
## data:  health by net
## t = -3.0287, df = 977.39, p-value = 0.002521
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -5.567087 -1.189325
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            51.53075            54.90896

Matching (Propensity Score)

Other Source：Simon Ejdemyr’s notes on propensity score matching

matched_psm <- matchit(net ~ income + temperature + health, data = nets,
                   method = "nearest",
                   replace = TRUE)
matched_psm

## A matchit object
##  - method: 1:1 nearest neighbor matching with replacement
##  - distance: Propensity score
##              - estimated with logistic regression
##  - number of obs.: 1752 (original), 1098 (matched)
##  - target estimand: ATT
##  - covariates: income, temperature, health

nets_matched_psm <- match.data(matched_psm)
model_matched_psm <- lm(malaria_risk ~ net, data = nets_matched_psm, weights = weights)
summary(model_matched_psm)

## 
## Call:
## lm(formula = malaria_risk ~ net, data = nets_matched_psm, weights = weights)
## 
## Weighted Residuals:
##     Min      1Q  Median      3Q     Max 
## -57.745  -7.605  -1.605   5.395  59.226 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  36.3025     0.6187   58.67   <2e-16 ***
## netTRUE     -10.6975     0.7857  -13.62   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 12.63 on 1096 degrees of freedom
## Multiple R-squared:  0.1447, Adjusted R-squared:  0.1439 
## F-statistic: 185.4 on 1 and 1096 DF,  p-value: < 2.2e-16

model_treatment <- glm(net ~ income + temperature + health, data = nets,
                       family = binomial(link = "logit"))
nets_pred <- copy(nets)
nets_pred$pred <- model_treatment$fitted.values

ggplot(data = nets_pred, aes(x = pred, fill=net)) + 
  geom_histogram(position = "identity", alpha = 0.7, bins = 50) + xlim(c(0, 1)) + 
  labs(title=paste0('Full Sample (N = ', nrow(nets_pred), ')'))

ggplot(data = nets_pred[id %in% nets_matched_psm$id], aes(x = pred, fill=net)) + 
  geom_histogram(position = "identity", alpha = 0.7, bins = 50) + xlim(c(0, 1)) + 
  labs(title=paste0('Matched Sample (N = ', nrow(nets_matched_psm), ')'))

for (v in c('income', 'temperature', 'health')) {
  plot_command <- paste0('ggplot(data = nets_matched_psm, aes(x =', v,
                         ', fill=net)) +geom_density(alpha = 0.7)')
  print(eval(parse(text = plot_command)))
  test_result <- t.test(as.formula(paste0(v, '~net')), data=nets_matched_psm)
  print(test_result)
}

## 
##  Welch Two Sample t-test
## 
## data:  income by net
## t = -3.548, df = 954.79, p-value = 0.0004071
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -64.64167 -18.59971
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            913.5731            955.1938

## 
##  Welch Two Sample t-test
## 
## data:  temperature by net
## t = 1.4522, df = 910.52, p-value = 0.1468
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -0.1294926  0.8664727
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            23.74940            23.38091

## 
##  Welch Two Sample t-test
## 
## data:  health by net
## t = -2.1015, df = 920.12, p-value = 0.03587
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -4.6145611 -0.1577902
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            52.52278            54.90896

Inverse Probability Weighting

# Predict the probability of a household using bed nets
model_treatment <- glm(net ~ income + temperature + health, data = nets,
                       family = binomial(link = "logit"))
nets_ipw <- copy(nets)
nets_ipw$pred <- model_treatment$fitted.values
nets_ipw[, ipw := (net_num / pred) + (1 - net_num) / (1 - pred)]

# Evaluate the effect with inverse probability weights
model_ipw <- lm(malaria_risk ~ net, data = nets_ipw, weights = ipw)
summary(model_ipw)

## 
## Call:
## lm(formula = malaria_risk ~ net, data = nets_ipw, weights = ipw)
## 
## Weighted Residuals:
##     Min      1Q  Median      3Q     Max 
## -45.705 -14.622  -4.924  10.791 162.642 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  39.6788     0.4684   84.71   <2e-16 ***
## netTRUE     -10.1312     0.6583  -15.39   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 19.54 on 1750 degrees of freedom
## Multiple R-squared:  0.1192, Adjusted R-squared:  0.1187 
## F-statistic: 236.8 on 1 and 1750 DF,  p-value: < 2.2e-16

• Inverse Probability Weights

  

  Think of this formula in a controlled experiment scenario:

  • Individuals are more likely to be never-takers (in a control group) if their likelihood values of receiving treatment are lower than the average and they do not receive treatment

  • Individuals are more likely to be always-takers (in a treatment group) if their likelihood values of receiving treatment are higher than the average and they do receive treatment

  • Individuals are more likely to be compliers (in both the control and the treatment group) if their likelihood values of receiving treatment are inconsistent with their actual treatment

To the end, we evaluate the change in the compliers’ outcomes due to their different exposures to the treatment.

Results

## 
## ===============================================================================
##              Naive        Regression   Matching (M)  Matching (PS)  IPW        
## -------------------------------------------------------------------------------
## (Intercept)    41.94 ***    76.16 ***    36.09 ***     36.30 ***      39.68 ***
##                (0.40)       (0.93)       (0.60)        (0.62)         (0.47)   
## netTRUE       -16.33 ***   -10.44 ***   -10.49 ***    -10.70 ***     -10.13 ***
##                (0.65)       (0.26)       (0.76)        (0.79)         (0.66)   
## income                      -0.08 ***                                          
##                             (0.00)                                             
## temperature                  1.01 ***                                          
##                             (0.03)                                             
## health                       0.15 ***                                          
##                             (0.01)                                             
## -------------------------------------------------------------------------------
## R^2             0.27         0.89         0.14          0.14           0.12    
## Adj. R^2        0.27         0.88         0.14          0.14           0.12    
## Num. obs.    1752         1752         1120          1098           1752       
## ===============================================================================
## *** p < 0.001; ** p < 0.01; * p < 0.05

• Overestimate or underestimate? Why?

  Suppose where is an omitting variable.

  

  where if or .

Change after 1980

Regression

model_simple <- lm(log_duration ~ highearn + after_1980 + highearn * after_1980,
                   data = injury)
summary(model_simple)

## 
## Call:
## lm(formula = log_duration ~ highearn + after_1980 + highearn * 
##     after_1980, data = injury)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.9666 -0.8872  0.0042  0.8126  4.0784 
## 
## Coefficients:
##                         Estimate Std. Error t value Pr(>|t|)    
## (Intercept)             1.125615   0.030737  36.621  < 2e-16 ***
## highearnTRUE            0.256479   0.047446   5.406 6.72e-08 ***
## after_1980              0.007657   0.044717   0.171  0.86404    
## highearnTRUE:after_1980 0.190601   0.068509   2.782  0.00542 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.269 on 5622 degrees of freedom
## Multiple R-squared:  0.02066,    Adjusted R-squared:  0.02014 
## F-statistic: 39.54 on 3 and 5622 DF,  p-value: < 2.2e-16

Add more controls: male, married, age, hosp (1 = hospitalized), indust (1 = manuf, 2 = construc, 3 = other), injtype (1-8; categories for different types of injury), lprewage (log of wage prior to filing a claim)

model_full <- lm(log_duration ~ highearn + after_1980 + highearn * after_1980 + male + 
                   married + age + hosp + as.factor(indust) + as.factor(injtype) + lprewage,
                 data = injury)
summary(model_full)

## 
## Call:
## lm(formula = log_duration ~ highearn + after_1980 + highearn * 
##     after_1980 + male + married + age + hosp + as.factor(indust) + 
##     as.factor(injtype) + lprewage, data = injury)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.0606 -0.7726  0.0931  0.7314  4.3409 
## 
## Coefficients:
##                          Estimate Std. Error t value Pr(>|t|)    
## (Intercept)             -1.528202   0.422212  -3.620 0.000298 ***
## highearnTRUE            -0.151781   0.089116  -1.703 0.088591 .  
## after_1980               0.049540   0.041321   1.199 0.230624    
## male                    -0.084289   0.042315  -1.992 0.046427 *  
## married                  0.056662   0.037305   1.519 0.128845    
## age                      0.006507   0.001338   4.863 1.19e-06 ***
## hosp                     1.130493   0.037006  30.549  < 2e-16 ***
## as.factor(indust)2       0.183864   0.054131   3.397 0.000687 ***
## as.factor(indust)3       0.163485   0.037852   4.319 1.60e-05 ***
## as.factor(injtype)2      0.935468   0.143731   6.508 8.29e-11 ***
## as.factor(injtype)3      0.635466   0.085442   7.437 1.19e-13 ***
## as.factor(injtype)4      0.554550   0.092849   5.973 2.49e-09 ***
## as.factor(injtype)5      0.641201   0.085435   7.505 7.15e-14 ***
## as.factor(injtype)6      0.615041   0.086312   7.126 1.17e-12 ***
## as.factor(injtype)7      0.991336   0.190532   5.203 2.03e-07 ***
## as.factor(injtype)8      0.434082   0.118912   3.650 0.000264 ***
## lprewage                 0.284481   0.080056   3.554 0.000383 ***
## highearnTRUE:after_1980  0.168721   0.063975   2.637 0.008381 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.15 on 5329 degrees of freedom
##   (279 observations deleted due to missingness)
## Multiple R-squared:  0.1899, Adjusted R-squared:  0.1873 
## F-statistic: 73.46 on 17 and 5329 DF,  p-value: < 2.2e-16

Summary

## 
## =================================================
##                          Simple       Full       
## -------------------------------------------------
## (Intercept)                 1.13 ***    -1.53 ***
##                            (0.03)       (0.42)   
## highearnTRUE                0.26 ***    -0.15    
##                            (0.05)       (0.09)   
## after_1980                  0.01         0.05    
##                            (0.04)       (0.04)   
## highearnTRUE:after_1980     0.19 **      0.17 ** 
##                            (0.07)       (0.06)   
## -------------------------------------------------
## Additional Controls        NO          YES       
## R^2                         0.02         0.19    
## Adj. R^2                    0.02         0.19    
## Num. obs.                5626         5347       
## =================================================
## *** p < 0.001; ** p < 0.01; * p < 0.05

Motivation

• Understand what drives your business and provide insights for business decisions
  
• Understand causal relationship

Prerequisites

• The control and testing groups can be clearly defined
• Metrics of interest can be quantified
• Data can be collected in a timely manner

Five Stages in Practice

• Define a study question
• Make hypotheses (variations) and identify metrics
• Identify variables
  • Determine the data to be collected
• Run experiments and collect data
  • Determine a detectable difference (i.e., how small of a difference you would like to detect, for example, 10% increase in your metric of interest)
  • Calculate the proper sample size using power analysis
  • Determine what fraction of traffic can be used in the treatment
  • Conduct a prior A/A test to check unfavorable impacts on business and a simultaneous A/A test to track seasonality and systematic biases/trend if any
• Measure results

Some Details in A/B Testing

# Here is a function in R
# https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/power.prop.test
# power.prop.test(n = NULL, p1 = NULL, p2 = NULL, 
#                 power = NULL, sig.level = 0.05,
#                 alternative = c("two.sided", "one.sided"),
#                 strict = FALSE, tol = .Machine$double.eps^0.25)
# Examples
power.prop.test(p1 = 0.5, p2 = 0.75, power = 0.90) ## =>     n = 76.7 in each group
power.prop.test(p1 = 0.5, p2 = 0.501, 
                power = 0.90, sig.level=.001)      ## =>     n = 10451937 in each group
power.prop.test(n = 50, p1 = 0.5, p2 = 0.75)       ## => power = 0.740
power.prop.test(n = 50, p1 = 0.5, power = 0.90)    ## =>    p2 = 0.8026
power.prop.test(n = 50, p1 = 0.5, p2 = 0.9,
                power = 0.90, sig.level=NULL)      ## => sig.l = 0.00131

Some Challenges in A/B Testing

Navigating in Command Prompt

In Command Prompt, the default cd command allows us to change the working directory within the same drive. When changing to a directory in another drive,

1. we need switch to the target drive first by typing the letter of the drive and then direct us to the target directory using cd, or
2. we can add a parameter in the cd command, i.e., cd /d "<path in another drive>".

Username and Token

1. Creating a personal access token: https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-token

2. Using a token on the command line: https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token#using-a-token-on-the-command-line

SSH

1. Checking for existing SSH keys: https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh/checking-for-existing-ssh-keys
2. Generating a new SSH key and adding it to the ssh-agent: https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
3. Adding a new SSH key to your GitHub account: https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account

With SSH keys, you can connect to GitHub without supplying your username and personal access token at each visit.

Adding an existing project to GitHub

https://docs.github.com/en/github/importing-your-projects-to-github/importing-source-code-to-github/adding-an-existing-project-to-github-using-the-command-line

Cloning a repository

https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/cloning-a-repository-from-github/cloning-a-repository

Branch: Checkout/Branch

1. Create a new branch using git branch <branch_name>.
2. Switch to the new branch using git checkout <branch_name>.

Execute two command lines in one: git checkout -b <branch_name>

Adding a file to a repository

https://docs.github.com/en/github/managing-files-in-a-repository/managing-files-using-the-command-line/adding-a-file-to-a-repository-using-the-command-line

Remove files from the working directory

1. Remove files (e.g. trash.txt) from local directory manually.
2. git rm trash.txt in Git Bash.
3. Commit the change using git commit -m "Remove trash.txt".

Pull and Push

1. Check the target remote repository on GitHub using git remote -v.
2. Pull the latest changes from GitHub with git pull.
3. Push your work to GitHub with git push origin <branch_name>" where origin is referred to as the target remote repository.

Merge

Merge code changes in two different branches (target branch and to be merged branch)

1. Checkout the target branch using git checkout <target_bread>.
2. Merge the changes from to-be-merged-branch into the target branch using git merge <to_be_merged_branch>.

Revert and Reset

git revert <commit_ID> will undo the changes in a particular commit.

• git reset

• git commit --amend

Controlled Regression

• We have Y => usage as measured by completing the 2nd week of a course

• and X => First Week NPS (net promoter score) which is a satisfaction rating on a 1-10 scale

• Univariate regression of Y (usage) on X (product quality)

• Multiple regression of Y (usage) on X (product quality) and a set of controls

If

1. the R squared in the multiple regression increases “a lot” from that in the univariate regression

2. coefficient on X is similar in the two models

then by the theory of controlled regression, we can use it as the causal impact.

Sources of Error

• Omitted Variable Bias

  Look at the R squared in the multiple regression with controls and how large it is (if R² is not close to 100% we know there are a lot of omitted variables in our regression)

• Included Variable Bias

  This is the opposite of omitted variable bias and involves the inclusion of too many controls. No direct way to identify this problem but generally can leave out controls that are not determined at time observe X. In the ML context, it is the information leakage (e.g., target encoding), that is, using future to predict the past.

Controlled regression can be a bit of art form and a bit of random. People nowadays prefer the other causal inference techniques that are considered to be more rigorous and reliable.

Regression Discontinuity

Focus on cut-off point that can be thought of as local randomized experiment.

Arbitrarily assigned cutoff point randomly splits the sample into two comparable groups around the cutoff point.

• Y: Revenue or course enrollment

• X: Advertisement (advertised, e.g., via email, if the percentage of subtitled materials in a course is greater than 80%)

• Assumption

  1. Courses below and above the 80% subtitle threshold are similar to one another, so the discontinuity point effectively randomizes the sample.

     Check:

     1. Sample size similar below and above cutoff (i.e., roughly balanced)
     2. Sample below and above cutoff are similar on observable/confounders

  2. Advertising available or not is assumed to be the only differentiator between courses at the 70% subtitle value and the 90% subtitle value.

     Check:

     Conduct placebo tests, that is, run regression discontinuity at points other than the cutoff and check for no effect (e.g., run regression discontinuity at 20%, 30%, and 40% and we shouldn’t find any significant difference at these levels)

Using regression discontinuity design (RDD), we estimate a linear regression with a term of the X variable with the discontinuity, an indicator for the discontinuity (which is a function of the X variable being above or below the cutoff point), and the interaction of the X variable and the discontinuity indicator. where is the variable with the discontinuity, is an indicator function, and is the randomly assigned cutoff point.

Note this regression fits two linear models (with different coefficients) below and above the cutoff point. When , the intercept and the coefficient of the X variable are and . In contrast, when , the intercept and the coefficient become and .

This change in the model structure is analog to the formulation of the Chow test for examining the structural break.

Comparing pre and post the discontinuity, we obtain the causal effect given by

Difference-in-Difference

• Y: Revenue

• X: Price change

• Assumption: Parallel Trends

  Check revenue in control country with no price change was similar and highly correlated with revenue in treatment country with price change (ensure control can serve as counterfactual)

  1. Graph control and treatment groups in the pre period and see if highly correlated
  2. Build a regression model to check whether trends are identical (hypotheses testing if there is no difference in slopes between two groups)

• Extension: Synthetic Control

  • Problem with regular DID: Picking a single control group that satisfies parallel trends can be arbitrary
  • Synthetic control creates a synthetic control group that is a weighted average of many control groups
    1. Choose weights to minimize tracking error with treatment group pre intervention (parallel trends)
    2. Causal estimate is difference post intervention between treatment and “synthetic control”
  • R packages: Synth; CausalImpact (Bayesian Version)

Instrumental Variables

• Assumptions

  1. Strong First Stage: Instruments are strong predictors for the X variable

     Hypothesis tests for weak instruments

  2. Exclusion Restriction: Instruments have an impact on the Y variable only through their impact on the X variable

     There is no test for this assumption. We need to use logic to strengthen the validity of this assumption. In a randomized encouragement trial (i.e., a RCT used for product promotion campaign; think of email notification) where we create the instrument as a randomly assigned nudge that prompts the X variable, we can ensure exclusion restriction through random assignment.

Machine Learning (ML) + Causal Inference

• Weakness of classic causal approaches

  • Fail with many covariates
  • Model selection unprincipled
  • Linear relationships and no interactions

• Benefits of ML

  • Handle high dimensionality
  • Principled ways to choose model
  • Nonlinear model using higher order features

• ML form controlled regression

  • Use ML models to control for many potential confounders and/or nonlinear effects.

    Standard controlled regression steps

    • Regress Y on X and a set of controls C to identify coefficient of interest on X
    • Be wary of omitted and included variable biases

  • There are two types of ML form controlled regressions:

    • Double Selection (LASSO)
    • Double Debiased (Generic ML models)

    Double Selection on AB test data:

    • Reduce the noises and increase statistical power (give smaller confidence intervals in a shorter experiment period)
    • Good for small samples and small effect sizes (i.e., small expected difference between A and B groups)

  • Steps

    • Define outcome Y, treatment indicator X, and high dimensional set of controls C

    • Split data into two sets: Training and Testing (this can be generalized to k-folds)

    • Fit two Lassos of X~C and Y~C on the training data set

      Use the one standard error rule to choose the optimal (i.e., the shrinkage parameter) for the LASSO regression

      The final set of controls is a union of controls from the LASSO regression X~C and Y~C

    • Take fitted models and apply to the testing data set

    • Get all nonzero variables in C and use as controls in controlled regression of Y on X

• Causal Trees/Forests

  • Previous models assume homogeneous treatment effects. However, causal trees/forests estimates heterogeneous treatment effects where impact differs on observed criteria
    • Use trees (or forests) to identify partition of the demographic space that maximizes observed difference of Y between treatment and control while balancing overfitting
  • Steps
    • Split data into two halves
    • Fit trees/forests on one half (i.e., the splitting subsample) and apply to second half (i.e., the estimating subsample) to estimate treatment effects
    • Heterogeneous treatment effects from difference in Y in leaf nodes (i.e., the treatment effect conditioned on C attributes in leaf nodes)
    • Optimization criteria set up to find the best fit given the splitting subsample
    • Forest is an average of a collection of trees with sampling