The pre-registered test of the Insights headline ("28% of directives answered within a year for 2013–15, 12% for 2022–26"), run under reports/pap_responsiveness.md with the deviations logged in its section 12. Outputs are in reports/data/pap/ (p1_20261022.json, rivals_20261022.json, s3_…, r6_…, title_only_…), from analysis/pap_run.py, pap_rivals.py, pap_s3.py, pap_r6.py and pap_title_only.py on inputs frozen in reports/data/pap/inputs_20261022/.
A first version of this note was written with an event rule that missed later basis responses. It used the first citing document's flag instead of basis_doc_id, affecting 29 response rows. Everything below uses the corrected rule; the correction is logged in the plan's section 12.
Verdict
Failed, narrowly. The point estimate says provinces in the primary panel answered State Council directives about 40% less often in 2022–25 than in 2013–15. But on the 150 labelled directives the 95% interval reaches just past 1, so under the plan's rules the decline is not established.
| Estimand | RR (2022–25 ÷ 2013–15) | 95% interval | Verdict (section 8) |
|---|---|---|---|
| P1: basis response within a year, standardised to the 2022–25 mix, corrected for coverage | 0.61 | 0.28–1.07 | failed |
| Published figure (for comparison) | 0.45 | — | — |
P1 uses only the directives with coder labels (40 early, 30 late; 120 and 90 directive–province pairs), as 5.2 step 4 requires once the regex mandate screen failed. The expected-precision simulation (section 12) already showed that the primary panel cannot reliably reach "held" at this size. The extended panel (S2) is where the power is.
Sensitivities
| Version | RR | 95% interval | Verdict |
|---|---|---|---|
| 2013–15 mix (the plan as first written) | 0.58 | 0.28–1.01 | failed |
| No coverage correction | 0.54 | 0.24–0.99 | shrank |
| No correction, province-years with coverage ≥ 0.8 only (in effect Guangdong, 82 pairs) | 0.76 | 0.22–1.60 | failed |
| XJ's mandate labels instead of the tie-broken ones | 0.57 | 0.26–1.01 | failed |
| Unstandardised, uncorrected | 0.46 | — | — |
Fujian's gazette (collected 2026-10-25, 2012–2016 only). It is a complete published list for those years. We hold 19–40% of its 闽政 and 闽政办 documents (0.22, 0.19, 0.29, 0.36, 0.40), lower than the serial estimate used (0.39–0.46). With these as Fujian's 2012–16 coverage, P1 is 0.59 (0.27–1.04, failed): the early correction was a little too small, not the cause of the verdict. Late coverage for Fujian and Anhui is still serial-based (fj_gazette_sensitivity_20261025.txt). The gazette's own documents are not added to the panel, since collection for one cohort only would break the collection rule (5.1).
The verdict sits on the line: every version's upper bound is between 0.99 and 1.07, apart from the Guangdong-only subset.
Raw one-year response rates by window, all three provinces (r1_windows_20261022.txt):
| Directives | Pairs | Rate | Anhui | Guangdong | Fujian |
|---|---|---|---|---|---|
| Window ends before 2019-05-15 | 201 | 0.35 | 0.49 | 0.39 | 0.16 |
| Window straddles 2019-05-15 | 51 | 0.35 | 0.47 | 0.41 | 0.18 |
| 2019-05-15 to 2021 | 96 | 0.28 | 0.31 | 0.31 | 0.22 |
| 2022 on | 99 | 0.14 | 0.18 | 0.18 | 0.06 |
The fall is concentrated in directives from 2022 on. Guangdong's coverage is essentially complete against its gazette (0.92–1.00 every year), and it falls too, from 0.39–0.41 to 0.18.
Two-year incidence (S3)
| Estimand | RR | 95% interval | Verdict |
|---|---|---|---|
| S3: basis response within two years, 2019–23 against 2013–15 directives | 0.80 | 0.57–1.09 | failed |
Its late cohort, 2019–23, is mostly the 2019–21 directives, which are answered only somewhat less than 2013–15 ones. A one-year ratio on the same pairs is close to the two-year one, so responses are not arriving later.
Cox model, for continuity (S4)
| Estimand | Estimate | 95% interval |
|---|---|---|
| Cox HR, 2022–25 against 2013–15 (mandate and kind adjusted, stratified by province, errors clustered by directive; no coverage correction) | 0.42 | 0.20–0.87 (p = 0.02) |
| Time-varying cohort effect (late × log t) | coefficient −0.43 | p = 0.31: no evidence the gap changes over the year |
| Restricted mean time to response within 365 days | 293 days early, 337 late | +44 days |
Like P1 without the coverage correction (0.54, 0.24–0.99), the Cox model excludes 1. What pulls P1's interval past 1 is the correction for Fujian's and Anhui's low collection coverage, which raises and spreads both cohorts' corrected incidence. So the verdict depends on how coverage is measured for those two provinces. Their serial estimates count documents issued, not published, and so err toward over-correction (5.1). A published-document frame like Guangdong's gazette would settle it.
Peer effect (S6, exploratory)
| Term (Cox, time-varying) | HR | 95% interval | p |
|---|---|---|---|
| Each other panel province that has already responded | 1.59 | 0.94–2.68 | 0.09 |
| The same, × the 2022–25 cohort | 1.82 | 0.55–6.08 | 0.33 |
There is weak evidence that a province responds sooner once its peers have, and no evidence that this changed. The errors are model-based, not clustered (s6_20261022.json).
Rival explanations
A rival accounts for the decline when it moves log RR at least halfway to zero, and for part when a quarter to a half (section 6).
| Rival | Test | Adjusted RR | Result |
|---|---|---|---|
| R1 disclosure fell (2019 rule) | Timing: is the decline concentrated in windows after 2019-05-15? | — | Partly. The rate steps from 0.35 to 0.28 after the rule, and the main fall (0.14) comes with 2022 directives. The disclosure-rate test itself was not run |
| R2 we collect less of recent years | P1 against no correction | 0.54 uncorrected | Does not account: the correction moves RR toward 1 a little, from 0.54 to 0.61 |
| R3 directive mix changed | P1 within M2 directives only | 0.69 (0.26–1.32) | Accounts for part (26%). Both coders put M2 at 48–50% of 2013–15 directives and 37% of 2022–25 ones |
| R6 authority moved to the Party | S1 with B4, θ = 0.05 (θ = 0.013 gives 1.01) | 0.87 (0.48–1.57) | Accounts, as computed, but see below |
| R7 copying without citing | S1 with B1, θ = 0.013 (pre-fixed) | 0.81 (0.51–1.19) | Accounts, as computed, but see below |
| S1 with B1, θ = 0.05 | 0.60 (0.32–0.97) | Does not account | |
| R8 our extractor drifted | P1 with cohort recall correction (1.00 early, 0.88 late) | 0.69 (0.32–1.22) | Accounts for part (26%) |
| R8, title-only citations | Citations of an 印发《X》 directive by its inner title 《X》, counted | 0.63 (0.31–1.08) | Does not account (7%); 3 such pairs |
| R9 dating artefacts | Own-dated responses only | 0.61 | Does not account: 1,164 of 1,166 response documents carry their own date |
| R10 right-censoring | — | — | Excluded by design |
| R5 responses delegated to departments | S1 with B3: department basis responses counted (b3_departments_20261022.csv) | 0.64 (0.31–1.13) | Does not account (12%). Department-only responses are 0.8%, 0.9%, 2.5% and 1.1% of pairs by cohort. Only 817 department documents are held for the three provinces, so this is a weak test |
| R4 responses through ministries | B2: a provincial-government document whose basis is a ministry document that itself states the directive as its basis (b2_ministry_20261022.csv) | — | Does not account: 1 directive–province pair in the whole panel. Of 3,278 ministry documents held, 284 state a directive as their basis, and provincial governments almost never cite those as theirs |
R7 at θ = 0.013 is driven by false matches (exploratory, not in the plan).
- At θ = 0.013, silent adoption (shared text, no citation) rises from 10.8% to 15.6% of pairs.
- But 11 of the 14 late silent adoptions have reuse between 0.013 and 0.05. Pooling both coders' adoption labels, matches in that band were adoptions in 25 of 66 early pairs and 0 of 12 late pairs.
- Weighting each text-only match by its band-and-period precision gives a combined ratio of 0.45, the same as basis responses alone (0.46, both unstandardised;
r7_precision_weighted_20261022.txt). - So silent adoption does not replace citation. The pre-fixed θ's precision is not stable across cohorts.
R6 rests on a handful of documents of one kind.
- Echoes of the directive without a citation rise from 4.2% to 7.8% of pairs (θ = 0.05). In 2022–25, all 7 open with Party or Xi language, against 1 of 5 in 2013–15 (
r6_decompose_20261022.txt). - Reading them: five are provinces' annual legislative-work plans (立法工作计划) echoing the State Council's. Their openings moved to Xi's rule-of-law thought and provincial Party decisions.
- Two are provincial work rules that refer to 《国务院工作规则》 with 参照 ("with reference to"), which is not a basis word. One is a false match (a science-award notice).
- The flags are mentions of Xi or the Party Centre in a document's opening, not a stated basis. Xi mentions rise from about 3% of openings in 2013 to 25–50% from 2016 on.
- So there is a real shift in how one recurring document type states its basis. It is too narrow to say that provinces now answer the State Council through Party channels in general.
Silent adoption and citation without adoption (S5, θ = 0.05): 3.3%, 0.9%, 5.8% and 3.3% of pairs silently adopt, and 3.3%, 2.6%, 2.5% and 2.2% cite without adopting. Neither has a trend.
Validation (section 9)
| Check | Result |
|---|---|
| Basis-response precision (9.2) | 1.00 by DXP's labels, 0.99 by XJ's |
| Recall (Horvitz–Thompson, 9.2) | 1.00, 0.94, 1.00, 0.88 by cohort |
| Inter-coder α, responses | 0.93 (basis or not 0.99) |
| Inter-coder α, mandate | 0.78 overall; 0.88, 0.79, 0.59, 0.84 by cohort |
| Inter-coder α, adoption | 0.76 overall; 2013–15 0.65 |
| Regex mandate screen against the coders | α ≈ 0 (it finds 13 of 82 M2), so it fails 5.2 step 4 |
| θ (5.3 B1) | 0.013 under either coder's labels; precision by cohort 0.74–0.87 at 0.013, 0.89–1.00 at 0.05 |
About the labels:
- XJ's labels were produced in 14 minutes, and how they were made is not verified.
- DXP's labels follow the tightened manual.
- Mandate disagreements went to DXP. There was no third coder.
- The project lead decided to use both sets (section 12, 2026-10-22).
What this means for the headline
- The direction holds; the certainty does not. The point estimate is 0.61 against the published 0.45. Guangdong, the fully covered province, falls too. But on the labelled sample the interval reaches 1.07, so the pre-registered verdict is "failed". The published "28% → 12%, HR 0.51 (0.41–0.65)" overstated precision.
- Part of the gap to 0.45 is measurement and mix. The directive mix (R3) and lower extractor recall for recent documents (R8) each account for part of the decline.
- No hidden channel replaces citation, on present evidence. Silent adoption does not rise once false matches are weighed (R7). The Party-basis channel (R6) shows up only in annual legislative plans.
- The timing points to 2022, not the 2019 disclosure rule. The fall is concentrated in directives from 2022 on.
- The published model's mandate control was ineffective. The regex agrees with neither coder, and recent directives often ask for local measures (about 37%), not "3 of 167".
- A defensible wording: "on a pre-registered test, provinces answered State Council directives about 40% less often in 2022–25 than in 2013–15, but the decline is not statistically established on the labelled sample (estimate 0.61, 95% interval 0.28–1.07)". Section 10 says the Insights headline changes to what P1 supports. That page change waits for the project lead, who kept the 28% card on 2026-10-21.
Not yet done
- S2, the extended panel: the way to the power this test lacks, but not yet viable.
- Provincial-government documents held, 2012–26: Jiangsu 824 (thin after 2018), Shandong 37, Hubei 21, Zhejiang 14. Common Crawl captured those provinces' departments and cities, not the governments' own series.
- Needed: a China-side session collecting their series (苏政发/苏政办发, 浙政发/浙政办发, 鲁政发/鲁政办发/鲁政字, 鄂政发/鄂政办发) from the portals' 政府文件 columns, as serial-gap filling under 5.1.
- Then compute coverage, then outcomes.
manage.py responsiveness_panel --extended --out <dir>builds that panel without touching the site's files. - R1 disclosure: the full test needs published-versus-issued counts. The timing check points to 2022; disclosure practice then is untested.
- R5 would be firmer with more department documents.
- Coverage for Fujian and Anhui against a published frame (their gazettes): the verdict turns on it.
- 9.1: known-adoption search (needs a China-side session). 9.4: the dated forecast.