1. THE SURVEY DESIGN AND AN INTERVENTION-BASED PROBABILITY MODEL

We consider a simple set-up of an SRSWOR (N, n) sampling design in conjunction with equal-sized clustered population units. The problem is to estimate P, the unknown population proportion of a qualitative feature ‘A’ in terms of its incidence [Yes/No] on each individual in the population. Naturally, the sample incidence proportion p suffices for the task. However, in our study design we encounter a different situation: The investigators, and also the supervisors, are likely to ‘intervene and distort’ the responses received from the sampled units. How should we estimate P in this modified situation of randomized reporting?

Revisiting the example described in Maiti and Sinha [1], which studies a quantitative variable, in this paper we allow investigators and/or supervisors to affect the response profiles of a qualitative variable. In other words, we keep a provision for wise and thoughtful circumstantial intervention effects on the quality/category of responses given by the respondents, before the data are finally received by the data collection agency.

Let the population consist of N Clustered Respondent Units (CRUs), each of which is composed of a fixed number m of individual respondents. These clusters are already formed before the sampling operation takes place. The investigators/supervisors are supposed to provide information on each of the n sampled CRUs in the form of subtotals of the original responses from all m individuals within each sampled CRU. The i^th sampled CRU will be denoted by C_i, for i=1,2,…,n. Let Li denote the number of schedule-based subtotals collected on C_i by various selected combinations of investigators j=1,2,…,J and supervisors k=1,2,…,K, specified by the study design. That is,

where Ii;j,k=1 if Investigator j and Supervisor k have both worked on a schedule assigned to CRU C_i; otherwise, Ii;j,k=0. Of course, Li≥1 for each responding C_i. Whenever Ii;j,k=1, we will denote by Fi;j,k the underlying response (the subtotal) on the qualitative feature ‘A’ for C_i, reported by the (j, k)-combination of investigator and supervisor.

Note that the same CRU may be surveyed by multiple investigators and/or multiple supervisors; that is, Li may exceed 1. This happens, for example, when several investigators visit the same CRU to collect information on different parts of a large-scale survey, but there is smaller set of questions (including the question on the presence or absence of the qualitative feature ‘A’) that are included on each investigator's schedule. In case there is no intervention effect of the investigators/supervisors, all Li observations on CRU C_i will be identical for each sampled CRU. On the other hand, if such interventions are anticipated, these observations can potentially differ. Also, the supervisor must be blinded from the CRU label, so that he/she can genuinely assess the subtotals provided by different investigators on the same CRU and possibly provide independent thoughtful modifications on each of them.

As in Maiti and Sinha [1], let us consider the following simple example of a study design: The population consists of N=70 CRUs, each composed of m=10 individuals. A sample of n=7 CRUs is taken using a fixed size sampling design [say, for example, SRSWOR(70, 7)]. Let there be J=7 investigators and K=2 supervisors engaged in the process of data collection. We name the selected CRUs as C₁, …, C₇. Next, we assign investigators (considered as treatments) to gather information from someselected CRUs (considered as blocks) by utilizing a symmetric BIBD(7, 7, 3, 3, 1), developed from the initial set (1, 2, 4) following Bose's technique. See Raghavarao [2]. Additionally, Investigators 1–3 are supervised by Supervisor 1, Investigators 5–7 are supervised by Supervisor 2, and Investigator 4 is supervised by both Supervisors 1 and 2. Table 1 gives the complete details of the study design.

In essence, the allocation design of Table 1 specifies that the first CRU C₁ will be approached by Investigators 1 and 2, and their response profiles will be checked by Supervisor 1. Further, the same CRU C₁ will also be approached by Investigator 4 and the response profile will be checked by both Supervisors 1 and 2. Similar explanations apply to the other selected CRUs as well. Since the CRU sizes are all equal (m=10), we will ignore the CRU size effect and treat each one as a singleton.

Let us denote by F1,F2,…,F7 the ‘data’ on the selected CRUs accrued from the field. Without any intervention effect from the investigators/supervisors, we would have regarded such data as ‘error-free;’ and so standard estimation techniques could be routinely used. Note that in such an ‘error-free’ scenario, there is no difference among F1;1,1,F1;2,1,F1;4,1 and F1;4,2, for example. However, we want to examine the possibility of intervention effect(s) from one or the other group or possibly from both groups. So we adopt an intervention probability distribution which is fully described in Section 2.

2. RANDOMIZATION DISTRIBUTIONS

To fix ideas, as mentioned earlier, let each CRU be composed of m=10 households (HHs). The response on each selected CRU is taken to be the sum of the responses of the 10 HHs within the CRU. Since we are dealing with a binary response to the qualitative feature ‘A’, each such subtotal F[i;j,k will take on a count value in the range 0,1,…,10. Therefore, an intervention would amount to either keeping the response count intact or ‘twisting’ the response count. This will apply at the investigator level as well as at the supervisor level. More specifically, an investigator takes the true score T from the field and after intervention produces a revised score R; and a supervisor takes this revised score R and after intervention produces a final declared score D. Also, we assume that for each supervisor, the modification (through intervention, if any) will be independent of whichever investigator forwards the response to him/her; that is, there is no interaction effect between investigator and supervisor.

For simplicity, for each of Investigators 1–4, we assume the same randomization probability distribution given by the stochastic diagram in Fig. 1(a), where the label of each node represents the true response T from the field, and the randomized revised score R is the label of either that node itself (with probability written inside that node), or of an adjacent node (with probability written on the outgoing arc). For Investigators 5–7, we assume a different randomization probability distribution asdepicted in Fig. 1(b). For Supervisor 1, the randomization distribution is shown in Fig. 1(c), where the node label represents the revised score R given to her by the investigator, and the final declared score D is the label of either that node (withprobability written inside that node), or of an adjacent node (with probability written on the outgoing arc); while for Supervisor 2, the randomization distribution is the same as in Fig. 1(a).

Figure 1

Stochastic diagrams of randomization probability distributions for (a) Investigators 1–4, and Supervisor 2; (b) Investigators 5–7; and (c) Supervisor 1.

Instead of Fig. 1, equivalently the randomization probability distribution is also given by an 11 × 11 stochastic matrix Q(q), whose rows and columns are labeled as 0, 1, …, 10, and where q denotes the retention probability at each node. That is,

In particular, Fig. 1(a)–(c) are equivalent to matrices Q.9,Q.92,Q.88, respectively.

3. DATA GATHERING MECHANISM

The statistics software R has been used throughout this paper for data generation, computations, data analysis and simulation.

We have denoted by P the proportion of HHs in the whole population possessing the qualitative feature ‘A’. Our purpose is to estimate this parameter based on the data declared by the supervisors. In case of no-intervention effect, we would have gathered ‘true scores’ from the n=7 selected CRUs, each comprising of m=10 HHs. Assuming independent behavior of the HHs with respect to possession of the attribute ‘A’, each true score obtained from a selected CRU behaves like an observation from a binomial (10, P) distribution. Therefore, the sample mean of 7 scores divided by 10 (or equivalently, the sum of 7 scores divided by 70) would provide a valid estimate for P. How should estimation proceed under the intervention effect(s)?

Let us explain the entire data generating process: We set P = 0.2, and generate 70 Bernoulli observations, in sets of 10, shown in Table 2. These are the true responses of the HHs. The i^th row sum is the true subtotal Fi from the field corresponding to CRU C_i.

Under a direct response scenario without any distortion from any external source, the investigator-cum-supervisor allocation design has no role whatsoever on the estimation of P. From the raw data of Table 2, we obtain:

Also, routine computation yields a standard error (SE) of the estimate as

On the other hand, if there is/are intervention(s) by investigator/supervisor, in order to explain the effect of the distortion mechanism on the estimation of P, let us carefully review the generation of ‘declared’ scores according to the allocation design (of investigators and supervisors to CRUs) given in Section 1, and the randomization probability distributions adopted by them described in Section 2.

For the CRU C₁, the investigator–supervisor combinations are: j,k=1,1,2,1,4,1,4,2. Let us explain how the declared score distribution arises out of the j,k=1,1 combination, given that the true score is T=3: First, construct a matrix for which the rows represent the randomized revised values 2,3,4 by Investigator 1, and the columns represent the randomized declared values 1,2,3,4,5 by Supervisor 1. The (u, v)-th element of the matrix is the joint probability obtained by multiplying the randomization probability that Investigator 1 reports a value u given T=3, and the conditional randomization probability that Supervisor 1 reports a value v given that Investigator 1 reported u. Then the randomization distribution of ‘declared’ score is simply the column sum of the matrix of joint distribution. The details of the calculations are shown in Table 3.

The calculations in Table 3 show that the declared score can take on values 1,2,3,4,5 with probabilities .003,.098,.798,.098,.003.

Alternatively, a more efficient algorithm to construct this randomization distribution of the declared score (ranging from 0 to 10) is to compute the product matrix Q(.90) Q(.88), and then simply take the row labeled 3 (since T=3), which is

We generate a random number according to this randomization distribution, and obtain a declared score of ‘3’.

For the CRU C₁, with true score T=3, we must construct the declared score distributions corresponding to three more investigator–supervisor combinations. For j,k=2,1,4,1 combinations, the same conditional distribution holds as for the j,k=1,1 combination just shown above; and we generate declared scores ‘3’ and ‘3’ in these two cases. Lastly, for the j,k=4,2 combination, we have a different declared score distribution given by the row labeled 3 of the product matrix Q.90 Q.90, which is

We generate a random number according to this randomization distribution and obtain a declared score of ‘2’. Thus, for CRU C₁, while the true score is 3, the intervention-based scores are (3, 3, 3, 2) corresponding to the four investigator–supervisor combinations reporting on CRU C₁.

Next, for the CRU C₂, the true score is T=2. According to the design plan, the possible intervention pairs are j,k=2,1,3,1,5,2. The declared score distributions arising out of j,k=2,1,3,1 pairs are exactly the same; namely,

We draw random numbers from this randomization distribution and derive a score of ‘3’ for (j, k) = (2, 1), and ‘2’ for j,k=3,1. Lastly, for j,k=5,2, the declared score distribution is given by

We generate a random number according to this randomization distribution, and obtain a declared score of ‘2’. Thus, for CRU C₂, while the actual score is 2, the three intervention-based scores are (3, 2, 2).

In this manner we obtain the randomization distributions and obtain declared scores for the remaining CRUs for each investigator–supervisor combination mentioned in the study design. These are summarized in Table 4.

4. DATA ANALYSIS

We have explained the generation of declared scores in all 24 cases of investigator–supervisor interventions based on the adopted study design. The frequency (f)(x)[x, f] = 0, 2, 1, 5, 2, 10, 3, 7. Computation yields a mean score of ∑fx/∑f = 46/24 = 1.9167, which provides an estimate of P as P^2=0.19167, which is very close to the true value of P = 0.20. The associated SE, if computed by the usual formula assuming that all the 24 estimates are independent, becomes SE2=0.0189. One may be tempted to conclude that estimator P^2 is quite sound, and we are able to provide an estimate of P with utmost accuracy based on this intervention-based study design. However, there is a sharp criticism regarding the validity of this approach. The ‘scores’ arising out of ‘within CRU intervention sessions’ are heavily dependent; and hence simple averaging of 24 scores is not a valid estimate.

We may, however, adopt a two-stage estimation procedure. In the first stage, we estimate the ‘true’ scores for the CRUs in the presence of intervention mechanism. These are essentially simple means of all declared scores within each CRU. In the second stage, we compute the simple average of these first stage estimates, and hence obtain estimator P^3 of P. That is,

Alternatively, the first-stage estimates of true scores for the CRUs may be calculated as the mean declared score across the investigators—that is, first we replace the two declared scores reported by Supervisors 1 and 2 based on the revised score produced by Investigator 4, by their mean, and we consider this mean as the single score produced by Investigator 4; then we take the mean of the scores across the three investigators reporting on each CRU. Then in the second stage we compute the simple average of these revised first-stage means to obtain estimator P^4 of P. That is,

Applying these two two-stage approachs, we compute the first-stage means of all declared scores and the first-stage means of investigator scores as shown in Table 5.

In the second stage, we compute the simple average of these first stage estimates of true CRU scores to obtain P^3=0.1928 and also P^4=0.1928. Since the CRUs are selected according to SRSWOR sampling, the simple averaging of CRU-based estimates in the second stage produces a valid point estimate of P. In fact, for our particular data set, the two estimates happen to be exactly the same. The sampling distributions of P^3 and P^4 are comparable, though the former seem to have a smaller SE while the later seem to have a narrower central 95% coverage region, as we shall see shortly.

Actually, the computation of the associated SE for P^3 or P^4 does not seem to be easy. We may argue that since only the calculation of CRU subtotals have changed due to interventions, as a first approximation it may suffice to simply use the same SE formula for P^1, given in Eq. (2), which we use in case of no-intervention. That is, a first approximation of SE of P^3 or P^4 is about 1/n−1/NP^31−P^3/m=.04473. This crude computation of SE indicates that there is not much loss of precision due to investigator/supervisor intervention(s). A more refined SE formula can be derived through probability theory by incorporating the effects of all possible randomization distributions on thetrue score of a typical CRU. We leave the details to the reader.

In the absence of any analytical expressions for the SE of P^3 or P^4, we will take recourse to a (parametric) bootstrap computation. See Efron and Tibshirani [3] or Davison and Hinkley [4], for a justification of the bootstrap method. The bootstrap set up is described as follows: Assume the true proportion of HHs exhibiting the qualitative feature ‘A’ is P0=P^. In each bootstrap iteration, we generate the true responses (the number of HHS exhibiting feature ‘A’ out of the 10 HHs in the cluster) from the 7 sampled CRUs using a binomial (10, P₀) distribution. Then for each investigator–supervisor combination (j, k) assigned to a CRU receiving true score T, we obtain the declared score according to the randomization distribution given by the row labeled T of the product matrix Q(q_j) Q(q_k). Using these declared scores for all 24 investigator–supervisor combinations, we compute P^3 as the simple average of first-stage means of declared scores, and we compute P^4 as the simple average of three investigators' scores (assuming that Investigator 4's score is given by the average of declared scores by the two supervisors), within each sampled CRU. We repeat the process M=104 times (to ensure that the mean of the sampling distribution is close to P₀, correct to at least two decimal places). Then the SE of P^3 or P^4 is obtained as the standard deviation of the bootstrap distribution of P^3 or P^4. Also, we construct a 95% confidence interval for P based on the 2.5-th and the 97.5-th percentiles of the bootstrap distributions. The results are summarized below

We only exhibit the bootstrap distribution of P^4 in Fig. 2. The bootstrap distribution of P^3 is very similar. These bootstrap distributions suggest that these estimators are slightly positively skewed. We can justify it as follows: A revised score R due to intervention by investigator j behaves as R = T + Y, where T is the true score from the binomial (10, P₀) distribution, and Y is a trinomial variable taking on values −, 0, 1 with probabilities 1−q/2,q,1−q/2 respectively when 1≤T≤9, a binomial taking values 0, 1 with probabilities q, 1 − q when T=0, and a binomial taking values −1, 0 with probabilities 1 − q, q when T = 10. Hence, a revised score has a mean of

Figure 2

Bootstrap distribution of estimator iterations.

Carrying on a similar reasoning, a declared score D due to interventions by investigator j and supervisor k has a mean of

which exceeds mP₀, resulting in a positive skewness. If we evaluate Eq. (5) for all investigator–supervisor combinations and then average them according to the two-stage procedure of calculating P^3 and P^4, then the means of the declared scores are seen to be 1.9493 and 1.9517 respectively. Thus, the bias of estimator P^3 in estimating P is about .0021, and that of estimator P^4 is about .0023. Indeed, the bootstrap distributions of P^3 and P^4 have the same mean 0.1947=.1928+.0019. Also, from the bootstrap distributions we estimate that the investigator–supervisor interventions result in only about 5.2% increase in the SE of P^3 or about 5.3% increase in the SE of P^4, compared against the SE of 0.04473 under the no-intervention scenario. While P^3 seem to have a slightly smaller SE than P^4, a 95% confidence interval for P based on the bootstrap distribution of P^4 is slightly narrower than that based on P^3. For all practical purposes estimators P^3 and P^4 have comparable behaviors.

5. POSTERIOR ANALYSIS

We are advocating for allowing wise and thoughtful interventions by investigators and/or supervisors. Even then, in this section we are interested in trying to recover the original scores received in the field, given the declared scores. This posterior analysis merits attention in its own right. Moreover, it demonstrates that the estimators P^3 and P^4 are closer to the true proportion P than the estimator P^1 computed from ‘predicted’ true scores based on posterior analysis.

Once we have the scores declared by different investigator–supervisor combinations, we can work out the posterior probability distribution of the true score, using Bayes' formula. We obtain the posterior probability distribution of the true score T corresponding to a declared score D, reported by investigator–supervisor combination (j, k), by implementing the following three steps: (1) We choose the column labeled D of the product matrix QqjQqk. Call this vector l. (2) We multiply this column vector l coordinate-wise by the prior probability vector π0, given by the binomial 10,P0 distribution. (3) We standardize the product vector by dividing each element by the sum of all the elements, to obtain the posterior probability vector π1. For example, for CRU C₁, based on interventions by Investigator 1 and Supervisor 1, the declared score is ‘3’. Following the above three steps (and writing Q.90Q.88δ4=l4) we obtain

The same posterior distribution also applies to interventions by (Investigator 2, Supervisor 1) and (Investigator 4, Supervisor 1) combinations. For intervention by (Investigator 4, Supervisor 2), with a declared score of ‘2’, (and writing Q.90Q.90δ3=l3) we have

In this manner we obtain the posterior distribution of the true score T for each declared score D after investigator–supervisor interventions, and tabulate them in Table 6. We also compute the within CRU average posterior distribution by taking a simple average of the posterior distributions corresponding to the three or four investigator–supervisor combinations assigned to each CRU. Thereafter, we take the mode of the average posterior distribution as the ‘predicted’ true score of each CRU.

One might wonder about the strong resemblance (more specifically, exact identity) between the ‘predicted’ value given by the posterior mode and the ‘true score’ for each CRU. A natural justification lies in the fact that the ‘randomization probability distributions’ are highly peaked (around the respective true scores) for each investigator and each supervisor. It may be remarked that the randomization theory works perfectly well with any other choice of the randomized distributions. For example, we couldlower the retention probability q (from .9 down to .5) for the investigator and/or the supervisor. This would lead to an increased variation in reporting by the investigator and/or supervisor. Consequently, the strong resemblance between the posterior mode and the true score could be reduced to some extent.

Having obtained the predicted true score of each CRU given by the mode of the average posterior distribution within each CRU, we can go back to estimator P^1, but this time calculated based on the predicted true scores. Let us call it P^5. Since the predicted true scores are exactly the same as the original true score, we have P^5=P^1=.1714, and its SE is .04273. Note that the reduction in SE of P^5, compared to that of P^3 or P^4, is not much. On the other hand, the point estimate P^5 is further away from the true value P=.2. Thus, this posterior analysis strongly supports the proposal for allowing interventions by investigator/supervisor.

6. CONCLUDING REMARKS

We presented an illustrative example which exhibits the benefits of allowing wise and thoughtful interventions by investigators and/or supervisors. It is advisable that the investigators, who are directly in conversation/contact with the respondents, should apply their strong feelings and wise thoughts, to modify the responses received in the field. They should not merely write down the responses, for that may not be conducive to the case being studied. With their educated interventions, the quality of estimation will improve (measured in terms of low bias), while the precision of estimation (measured by the reciprocal of the SE of the point estimate) may decline only by a small amount. The example we presented here extends in the following two directionsto cover increasingly more realistic situations:

1. We can change the sampling design for selecting the CRUs from SRSWOR scheme to some other unequal probability sampling scheme. For example, using any fixed sample size sampling design N=70,n=7 with positive first and second order inclusion probabilities of the CRUs, we could apply Horvitz-Thompson estimator based on one of the first-stage estimates of individual CRU-based proportions. See Table 5 of Section 4.

2. We can change the transition matrices used to model the intervention of the interviewers and supervisors from having the very simple form of Qq into being arbitrary stochastic matrices. We can even allow a unique stochastic matrix to model the intervention of each investigator and each supervisor. The computations extend in a straight-forward manner under such generalized scheme of randomization.

Analytical derivations of the sampling distributions of estimators P^3 and P^4 in general, and their SEs in particular, remain open problems.

ACKNOWLEDGEMENTS

The authors sincerely thank a referee for pointing out one computational error in an earlier version of the manuscript and for indicating several places where more detailed explanations were needed. The second author thanks the authorities at the Indian Statistical Institute, Kolkata, for hosting his visit during which time the research was concluded. The editor's encouragement for carrying out the needed revision is also highly appreciated.