Pre-analysis plan: provincial responsiveness

Written 2026-10-17, before any of the analyses in sections 4–9 were run. The git commit that adds this file is its timestamp. Changes after that commit go in section 12 (Deviations), dated, with the reason; the text above it is not edited.

0. Frozen inputs

Data collected after the freeze (section 5) is added only through the procedures fixed here, logged with its own hashes, and applied to every cohort alike.

1. The claim

Provinces answered 28% of State Council directives (国发/国办发) within a year for 2013–15 directives, and 12% for 2022–26 (Cox hazard ratio 0.51, 95% CI 0.41–0.65).

This is the headline on the Insights page and point 4 of reports/briefing.md. This plan asks whether it is true, how large the change really is, and which of the rival explanations in section 6 account for it.

2. What has already been seen

The plan is written after exploration, not before. Everything below has been computed and looked at. Nothing else about the outcome has.

• Pooled shares answered within a year (pairs with a full year of observation): 2013–15 28.4% (303/1,068), 2016–18 22.3%, 2019–21 16.6%, 2022–26 12.7% (76/599). On province-years with ≥90% readable text: 28.2% → 12.5%.
• Cox model, all six provinces, stratified by province, clustered by directive:
• 2022–26 vs 2013–15: HR 0.51 (0.41–0.65); 2019–21: 0.74 (0.60–0.92).
• Mandate HR 1.55, deadline 1.27, log length 1.23, 国发 vs 国办发 0.96.
• The proportional-hazards test fails: minimum Schoenfeld p = 0.032.
• Concordance 0.62.
• The same model on Fujian, Guangdong and Anhui only: HR 0.46 (0.36–0.59). With title echoes counted as responses: 0.51 (0.41–0.64).
• Stated basis of provincial documents:
• provincial government documents citing a 国发/国办发 number in their opening: odds ×0.79 a year from 2016;
• provincial departments naming the State Council as their basis: 22% (2017–19) → 3.2% (2023–26), odds ×0.80 a year;
• Xi and Party-Centre invocations in openings: seen as a time series on the Insights page.
• Precision: 40 of 40 sampled responses are genuine (Wilson 91–100%). The analyst (Claude) coded them during development; this is not an independent check.
• Recall proxy: 73% of pairs found by title echo are also found by the basis-citation measure.
• Exposure counts: seen while writing this plan, outcome-free. Directives whose text the current MANDATE regex says orders localities to issue implementing measures:

Either the State Council stopped asking provinces to legislate, or the regex misses newer phrasing. Section 5.2 resolves which.

3. Problems with the current estimate this plan has to fix

1. The panel changes under the cohorts. Hunan is observed 2017–2019-09 and Hainan 2016–2018-10. Beijing starts in 2016. The 2022–26 share rests on Fujian, Guangdong, Beijing and Anhui. The pooled 28% → 12% mixes provinces. 2. The hazard ratio is not a safe primary measure. Proportional hazards fails; an HR averages a changing ratio over time. 3. Directive mix is not controlled. The mandate regex may undercount late directives, and the 2022–26 cohort may hold more directives that need no local response. 4. Coverage is assumed, not measured. A province enters when it has ≥15 government documents a year. We do not know what share of its output that is, and the share may change over time. 5. Response means citing. Responses that copy the directive without citing it, that cite it through a ministry's follow-up, that come from a department, or that name the Party Centre as their basis are missed. If citation habits changed, the measure confounds habit with response.

4. Estimands

Unit: a directive–province pair.

• Directive: a 国发 or 国办发 issued from 2013-01-01 whose text we hold.
• Pair rule: a pair enters only if the province is observed for at least 365 days after the directive (its window end in coverage.csv, as now). "2022–25" means the late cohort under this rule.
• Province: one of those in the panel (4.1).

Clock: days from the directive's publication to the response's own date (成文日期), at least 0. Responses dated more than 30 days before the directive are discarded, as now.

4.1 Panel

• Primary panel: Fujian (35), Guangdong (44), Anhui (34) for 2013–2025. These are the only provinces observed in every cohort.
• Extended panel (secondary): adds Beijing (2016–) and any province-year that passes the coverage rule in 5.1. Candidates are Jiangsu, Zhejiang, Shandong and Hubei, from the Common Crawl import. Coverage is computed and committed for every candidate before any response outcome is computed for it.

4.2 Primary estimand (P1)

The ratio of the one-year cumulative incidence of a basis response (5.3), 2022–25 directives against 2013–15 directives, in the primary panel. It is:

• directly standardised to the 2013–15 distribution of directive mandate level (5.2) and directive kind (国发/国办发);
• corrected for collection coverage (5.1).

RR = CI₃₆₅(2022–25) / CI₃₆₅(2013–15), with incidence estimated by Kaplan–Meier within each province × mandate × kind cell.

Inference: bootstrap resampling directives with replacement (pairs from one directive move together), 2,000 replicates, seed 20261017, percentile 95% interval. Province-level inference is impossible with three provinces. Results describe these provinces, not China.

4.3 Secondary estimands

• S1: P1 with each broadened response definition (5.3, B1–B4) in turn and all together.
• S2: P1 on the extended panel.
• S3: two-year cumulative incidence, for 2013–15 against 2019–23 directives (which have two years of observation).
• S4: the Cox hazard ratio, as now, reported for continuity only. It also comes with:
• a time-varying cohort effect (cohort × log time);
• restricted mean time to response over 365 days, in place of the HR.
• S5: per-cohort shares of:
• silent adoption: text reuse ≥ θ with no citation;
• citation without adoption: citation with text reuse < θ.
• S6 (exploratory): peer effect. A time-varying count of other panel provinces that have already responded, and whether its coefficient changes across cohorts.

S1–S5 are confirmatory for the rival explanations in section 6. S6 and anything not listed here is exploratory and labelled so.

5. Measurement

5.1 Coverage

There are two quantities, kept apart:

• Collection coverage = documents held ÷ documents published. This is our gap.
• Disclosure rate = documents published ÷ documents issued. This is the government's choice, and it is what the 2019 rule change (R1) acts on.

Methods, all computed per province × year on the provincial government's own series (粤府, 粤府办, 闽政, 闽政办, 皖政, 皖政办 … each a separate serial series):

1. Serial-number estimate of documents issued.

• For each series and year, with k distinct serials observed and maximum m: N̂ = m + m/k − 1.
• Report N̂, held ÷ N̂, and the serials missing.
• Series where m exceeds 3k are flagged: they number internal letters too.

2. Capture–recapture wherever two collection routes are independent: portal enumeration, Common Crawl, and the provincial gazette (政府公报) contents pages.

• Lincoln–Petersen with the Chapman correction, and Chao's lower bound.
• Report the overlap counts behind each estimate.

3. Self-reported counts: each government's 政府信息公开工作年度报告 (2019 onward) gives 行政规范性文件 issued that year. Normative documents are a subset of the series, so this bounds coverage of that subset.

Collection rules, fixed in advance:

• New documents may be collected only to fill serial gaps and gazette listings.
• Collection runs over all cohorts alike, in the same pass.
• Coverage is recomputed after collection, and only then are outcomes computed.

Coverage rule:

• A province-year enters a panel only if its collection coverage is ≥ 0.8 by method 2, or by method 1 where method 2 is unavailable.
• Primary-panel province-years that fail the rule stay in P1. They are corrected (below) and flagged; the sensitivity analysis drops them.

Correction:

• CI_true ≈ 1 − (1 − CI_obs) ^ (1/ĉ), where ĉ is the pair-weighted mean collection coverage over the response window.
• This treats each response document as held independently with probability ĉ, which errs toward over-correction when a province issues several responses.
• Sensitivity: uncorrected, on province-years with ĉ ≥ 0.8 only.

5.2 Mandate level of each directive

Each directive gets one of three levels:

• M2: orders localities to issue their own implementing measures (制定实施方案/实施细则/具体办法, with or without a deadline).
• M1: directs localities to implement, with no instrument required (各地区各部门要结合实际认真贯彻落实).
• M0: no local role. Examples: approvals, holiday arrangements, personnel, internal State Council matters, regulations addressed to central bodies only.

Procedure: 1. A coder manual (reports/coding_manual.md, written in step 1 of section 7) gives the rubric with examples. 2. Two human coders independently label a random sample of 150 directives (seed 20261017): 40 each from 2013–15, 2016–18 and 2019–21, and 30 from 2022–25. They do not see dates or cohorts: the 文号 year, the date, and phrases that fix the year are redacted from the text they see. 3. A classifier labels all directives: the regex extended from the coder sample's phrasing, or a model prompted with the manual, whichever has the higher α against the adjudicated labels on a held-out half. 4. The classifier is accepted if Krippendorff's α ≥ 0.667 against the adjudicated labels in every cohort separately. If it fails in any cohort, the 150 human labels are used and P1 is computed on those directives only.

5.3 Responses

• Basis response (primary; the current definition, unchanged):
• a provincial-government document (the government or its general office, including joint party–government documents) whose first 400 characters cite the directive by number or by resolved title;
• next to a basis word: 贯彻/落实/根据/按照/精神/要求/部署/依据.

Broadened responses:

• B1 Silent adoption. A provincial-government document within the window that shares at least θ of its text with the directive, with or without citing it.
• Candidates: MinHash LSH on character 5-gram shingles (datasketch, 128 permutations, Jaccard threshold 0.10).
• Alignment: character-level local alignment. Reuse = the share of the provincial document's characters in aligned blocks of ≥ 20 characters.
• Boilerplate: passages occurring in 20 or more documents are removed first.
• Setting θ: coders label 200 candidate pairs drawn across all cohorts (adopts / does not adopt), blind to cohort. θ is fixed from those labels before any cohort contrast is computed.
• B2 Ministry route. A provincial-government document whose stated basis is a central ministry document that itself states the directive as its basis, within the window.
• B3 Department response. A basis response by a provincial department (厅/委/局) rather than the government.
• B4 Party-basis response. A provincial-government document within the window that:
• echoes the directive (title-core bigram Jaccard ≥ 0.5, as now, or reuse ≥ θ);
• states as its basis the Party Centre, a 中发/中办发/厅字 document, or Xi Jinping's instructions, rather than the directive.

6. Rival explanations, with their predictions and tests

A rival accounts for the decline when applying its test moves log RR at least halfway to zero: |log RR_adjusted| ≤ ½ |log RR_P1|. It accounts for part of the decline when it moves log RR by a quarter to a half.

7. Analysis order

1. Build the coder manual and samples (5.2, 5.3 B1, 9.2). Send them to the coders. Nothing below runs until the labels are back. 2. Collect coverage sources (5.1) and compute coverage. Commit the coverage tables. 3. Fix θ and the mandate classifier from the labels. Commit them. 4. Compute P1, then S1–S5 and the R-tests. Commit outputs to reports/data/pap/ with a run log. 5. Write the results note (section 10).

Steps 1–3 involve no response outcomes for new provinces. Step 4 is the first time any new estimate is computed.

8. Decision rules for P1

The verdict is reported once for P1, and again with every rival that accounts for the decline applied together. If the decline is held under P1 but accounted for by R4, R6 or R7, it is reported as "provinces cite the State Council less, and respond through other channels", not as "provinces respond less".

9. Validation by someone other than the analyst

9.1 Known adoption

• A random sample of 30 directives coded M2 (seed 20261017): 8 per cohort, plus 6 drawn from all cohorts. Any shortfall in 2022–25 is filled from 2019–21 and reported.
• For each directive and each of Fujian, Guangdong and Anhui, a coder searches outside our corpus for the province's implementing document within two years. Search places:
• the provincial portal's search;
• gov.cn;
• Baidu limited to gov.cn;
• the provincial gazette.
• The searches need a China-side session, or Common Crawl.
• Recall = the share of documents found by the coders that our pipeline also finds, by cohort.

9.2 Blind response coding

Two coders label 200 items, blind to cohort and to our label:

• 100 pipeline-positive pairs, 25 per cohort, for precision;
• 100 pipeline-negative pairs where the province issued a document in the window with title-core Jaccard ≥ 0.3 or reuse ≥ 0.05, for missed responses.

Labels: basis response / implements without citing / cites without implementing / unrelated.

Reporting:

• Krippendorff's α per label, target ≥ 0.667;
• disagreements adjudicated by a third coder;
• precision and recall by cohort, with Horvitz–Thompson weights for the stratified design.

The existing 40-pair check is superseded.

9.3 Calibration

The mandate classifier and the reuse threshold are reported with confusion matrices against the human labels, by cohort. Any score the pipeline publishes as a probability gets a reliability diagram and a Brier score.

9.4 A dated forecast

Model: P1's model, fitted on the frozen data and committed with this plan's results.

Predictions:

• For every 国发/国办发 published from the results commit until 2027-06-30, predict each primary-panel province's probability of a basis response within 365 days.
• Commit each prediction within 14 days of the directive's publication, before any response.

Scoring on 2028-07-31:

• Brier score against two reference forecasts: the 2013–15 rate and the 2022–25 rate.
• If the decline is real, the 2022–25 rate should beat the 2013–15 rate, and the model should beat both.

10. Reporting

• reports/pap_responsiveness_results.md and a "Responsiveness: pre-registered test" section on the Data page.
• Each estimand, rival and validation is reported with its n, its interval and its verdict under section 8, including those that come out against the claim.
• The Insights headline and the briefing change to whatever P1 supports.

11. Known limits this plan does not fix

• Three provinces. Fujian, Guangdong and Anhui are coastal or near-coastal and are not a sample of China. The extended panel helps only where coverage passes.
• Only public documents. A province can respond through unpublished documents, meetings or budgets. R1 bounds the published share; it cannot see the rest.
• The directives are 国发/国办发 only. Central guidance increasingly arrives as Party–state documents, which are mostly unpublished. R6 measures provinces' stated basis, not that guidance itself.
• The coders are colleagues. Their α measures agreement, not correctness. The known-adoption search (9.1) is the closest thing to ground truth.

12. Deviations

2026-10-17 — 5.3 B1, how reuse is computed.

• What the plan said: MinHash LSH candidates (5-grams, 128 permutations, Jaccard ≥ 0.10), then character-level local alignment.
• What is used instead: reuse is computed exactly, as specified (the share of the provincial document's characters lying in blocks of ≥ 20 characters shared with the directive), by hashing every 20-character window and matching through an inverted index (analysis/pap_reuse.py).
• Boilerplate: windows in ≥ 20 documents are removed, as specified.
• Window: −30 to +730 days.
• Why: the corpus is small enough (1,145 directives × 6,353 documents) for an exact computation, which removes the candidate step's approximation and is deterministic.
• Difference: blocks are exact matches, so a one-character edit (各地 → 本省) splits a block. The measure is stricter than gapped alignment. θ is calibrated on this same measure (5.3), so the rule stays internally consistent.
• Timing: recorded before any reuse value was compared across cohorts.

2026-10-17 — 5.3 B1, the θ calibration sample.

• What the plan said: "200 candidate pairs drawn across all cohorts".
• What is used instead: the pairs are stratified by reuse level, six bins from 0.005 to 1 (30–35 each), with sampling weights in the key. A threshold can only be calibrated if coders see the whole range, and a simple random draw would be about 70% near-zero pairs.
• Seed: 20261018.
• Cohorts: not used in sampling.

2026-10-17 — blinding. Besides dates, years and document-number years, the coder pages mask plan and congress numbers, 习近平, 新冠/疫情 and period phrases (新时代, 新发展格局, 高质量发展, 中国式现代化). This is a clarification of 5.2's "phrases that fix the year", applied to all three samples.

2026-10-17 — 4.2, the P1 reference mix: 2022–25, not 2013–15.

• What the plan said: standardise to the 2013–15 distribution of cells.
• The problem: the regex counts in section 2 suggest M2 directives are rare in 2022–25 (3 of 167). A dry run of analysis/pap_estimate.py on simulated data with that pattern, where the truth is known, showed that the 2013–15 version gives a sparse late cell its full early weight. One late directive then decides the estimate.
• Evidence: 40 simulations, true RR 0.5, late M2 share 2%, coverage falling from 0.8 to 0.6.

• Decision: P1 standardises to the 2022–25 mix. It then answers: "given what the State Council now asks of provinces, do they respond less than they did?" The 2013–15 version is reported as a sensitivity.
• Timing: no real outcome had been computed.

2026-10-17 — 4.2, sparse cells (not specified in the plan).

• Level 1: in each cohort, a (province, mandate) stratum with any kind-cell under 5 pairs is pooled across kind.
• Level 2: a mandate level still under 5 pairs in some province is pooled across provinces.
• Never pooled: mandate levels are never pooled together.
• Reporting: merges are reported with the estimate. The rule is in code and tested (analysis/test_pap_estimate.py).

2026-10-17 — 8, verdict precedence. Where both the "held" and the "shrank" conditions are met (e.g. RR 0.70, upper bound 0.74), the verdict is "shrank". That is the more specific condition, and it reports that the decline is less than half the published one.

2026-10-17 — expected precision (simulation; a design fact, not a deviation). With the primary panel's size, two cohorts of about 360 and 135 directives across three provinces, 25 simulations per true value give:

• What it means: at the published size the primary panel cannot reliably reach "held". The extended panel (S2) is where the power is.
• Interval coverage: 95% bootstrap intervals cover the truth in about 90% of simulations, slightly narrow for rare events. This is noted with the result, not corrected after seeing it.
• Optimism: these figures assume perfect mandate labels and known coverage, so the real intervals will be wider.

2026-10-22 — 5.2, 5.3 B1 and 9.2: one coder's labels used as a baseline.

• What the plan said: two human coders label each sample independently, a third settles disagreements, and Krippendorff's α between the coders is reported by cohort.
• What happened: one set of labels came back, from coder XJ, for all three samples (150 + 200 + 200). The analyst has not verified how they were produced. All 550 were finalised 14 minutes after the coder pack was built (the guide estimates about 26 hours), and the CSVs were written in the export format rather than exported from the coding pages. The project lead decided to use them as a baseline because the team has no capacity for a second coder now.
• Consequences:
• There is no inter-coder α and nothing is adjudicated. Every figure from these labels is labelled "baseline (one coder)", not "validated".
• The figures are in reports/data/pap/labels_XJ.json (analysis/pap_labels.py). They are recomputed if further coders' labels arrive.
• Mandate (5.2 step 4): the regex screen against XJ's M2-or-not gives α = 0.21 (2013–15), −0.16, −0.17 and −0.26 (2022–25). The regex never calls M2 where XJ doesn't, but finds only 13 of XJ's 82 M2 directives.
• The classifier fails the α ≥ 0.667 rule, so under 5.2 step 4, P1 uses the 150 labelled directives only.
• XJ codes 13 of 30 late directives as M2. That contradicts the regex-based premise of the 2026-10-17 deviation on the reference mix (late M2 "3 of 167"). Its 2013–15 sensitivity is kept as specified.
• θ (5.3 B1): the rule was fixed before the labels were read: the reuse threshold with the highest weighted Youden J for "adopts" against related/unrelated. The result is θ = 0.013 (weighted sensitivity 0.78, specificity 0.77, precision 0.69). The labels change sharply at 0.05, so 0.05 is reported as a sensitivity.
• Responses (9.2), XJ against the pipeline:
• Basis-response precision is 0.99 overall (1.00, 1.00, 0.96, 1.00 by cohort).
• Horvitz–Thompson recall is 0.96 overall (1.00, 0.94, 1.00, 0.88).
• 14 of 200 are unsure and left out.

2026-10-22 (later) — coder instructions tightened after the first labels; a correction.

• Correction to the entry above: XJ's M2 rate is 55%. In 25 of the 69 directives where XJ says M2 and the regex does not, the text holds no instruction to issue anything (for example, "请认真组织实施" only). So XJ over-codes M2, and "13 of 30 late directives are M2" is weaker evidence against the regex's "3 of 167" than that entry implied. In 27 others the regex misses a real instruction: other instrument words (政策措施), a subject over 40 characters away, or other subjects (县级以上地方人民政府). The true late share lies between the two.
• Instructions for later coders (reports/coding_manual.md and the coder guide):
• an M2 check (an instruction sentence, addressed to local governments, for an instrument of their own), with the deciding sentence required in the notes;
• "partial adoption" narrowed to taking up a specific measure or programme;
• the "several bases" and "text problem" rules that the guide already carried, now in the manual too;
• coding must be done in the coding pages and exported from them.
• Comparability: XJ coded under the earlier wording. Agreement between XJ and any later coder is reported with that difference noted. The samples are unchanged.

2026-10-22 — decisions fixed before the first P1 computation (plan steps 2–4).

• Inputs: the panel files rebuilt on 2026-10-22, frozen in reports/data/pap/inputs_20261022/ with SHA256SUMS. Since the 17 October freeze the three primary provinces gained 3 documents (Anhui, 2014–15). Anhui gained 121 basis-response rows, from texts filled by the China-side re-read of documents held without a body (last night). That gap filling is blind to cohort, so it follows the collection rule.
• Mandate labels (5.2 step 4):
• P1 uses the 150 labelled directives, because the regex failed.
• Where XJ and DXP disagree (20 directives), DXP's label is used, since it was coded under the tightened manual. Where either coder chose "unsure", the other's label is used.
• XJ-only labels are reported as a sensitivity.
• No third coder was available.
• Collection coverage (5.1):
• Guangdong: the share of the gazette's 粤府/粤府办 documents of that year that we hold (gd-gazette contents). It is not an independent capture–recapture estimate, because part of the collection came from the gazette, but the gazette is a complete published frame.
• Fujian and Anhui (no gazette): method 1's dense-series share (series numbering internal letters excluded). Where a year has none, the province's median dense share is used.
• Method 1 counts documents issued, not published, so it understates collection coverage and the correction errs toward over-correction.
• Nearly every Fujian and Anhui province-year is below 0.8, so it is corrected and flagged. The ĉ ≥ 0.8 sensitivity is then mostly Guangdong.
• Event: a basis response (responses.csv basis = 1) with lag ≤ 365 days. Lags from −30 to 0 count as day 0; earlier ones are discarded (section 4).

2026-10-22 — first results, and one exploratory addition.

• Results: reports/pap_responsiveness_results.md. P1 RR 0.56 (0.27–0.96), verdict shrank.
• Exploratory addition (not in the plan, labelled as such): R7 recomputed with each text-only match weighted by the adoption precision of its reuse band (0.013–0.05 or ≥ 0.05) and period (2013–18 or 2019–25), from both coders' labels. It was added after the pre-fixed θ = 0.013 gave "accounts for the decline", once the labels showed that late low-band matches are not adoptions (0 of 12). The pre-fixed result is reported alongside it.

2026-10-22 — correction of an implementation error in the first run.

• The error: analysis/pap_run.py first counted an event only when the *first* document citing a directive had the basis flag. In responses.csv, the first basis document is basis_doc_id with basis_lag_days, and it can be a later document. 29 panel rows were affected.
• How it was found: checking a title-only case (Anhui's 空气质量持续改善 plan, a basis response 202 days after 国发〔2023〕24号, behind an earlier passing citation).
• The fix: the rule now follows the plan's definition, unchanged. Every result was recomputed.
• Effect: P1 moved from 0.56 (0.27–0.96, shrank) to 0.61 (0.28–1.07, failed). The results note was rewritten with the corrected figures, and the first version is in git history (574fd8e).

2026-10-22 — 9.4, the dated forecast.

• What the plan said: P1's model.
• What is used instead: a new directive has no mandate label (the regex screen failed, and no coder sees new directives in time). The forecast is the 2022–25 one-year basis-response rate by province and kind, averaged over that cohort's mandate mix and corrected for coverage as in P1. Cells under 5 pairs are pooled over kind, with half-count smoothing so no forecast is 0 or 1. The references are the 2013–15 and 2022–25 rates.
• Code: analysis/pap_forecast.py. It appends to reports/data/pap/forecasts.csv with a timestamp. Run it after each import that brings new directives, and commit within 14 days of each directive.
• Status: no directive has been published since the results commit; the corpus ends at 2026-10-14 for central documents.