Vermont Business Magazine State Auditor Doug Hoffer released a new investigative report examining the quality and usefulness of the Vermont Annual Outcomes Report, which is designed to tell policy makers and the public whether state government is doing a good job serving all Vermonters. His report suggests it is not.
Click HERE for full auditor's investigative report.
The Outcomes Report is produced annually by the Agency of Administration’s Chief Performance Officer using “indicators” selected by the Legislature’s joint Government Accountability Committee.
“I’m a big believer in using performance measurement to inform citizens and policymakers in hopes of improving government,” said Hoffer. “Unfortunately, in this case, our analysis uncovered several challenges that limit the quality and usefulness of the Outcomes Report. One glaring example of the problems with the report is its claim that 93% of Vermont is covered by the ‘state-of-the-art telecommunications infrastructure’.”
The Auditor’s Office drew on nine principles of effective performance measurement to assess the Annual Outcomes Report.
Key findings include:
- The Outcomes Report’s use of statewide population-level data to measure State government performance is not terribly useful because the State has limited influence over most population-level outcomes. For example, the unemployment rate, one of the indicators included in the report, is primarily impacted by national trends like recessions or booms rather than any State-level activity.
- Many of the indicators included in the Outcomes Report do not capture the most significant or relevant information. For example, the indicator measuring the median income in Vermont is not adjusted for inflation. As many Vermonters know all too well right now, if inflation is rising faster than your income, you’re actually worse off.
- Most of the indicators fail to disaggregate data to help draw meaningful conclusions about how different groups of Vermonters are faring. For example, the Outcomes Report shows only 36% of Vermont households were paying more than 30% of their income on housing costs in 2017. However, if disaggregated, we’d see that 77% of households earning less than $35,000 per year faced housing burdens in excess of 30%.
Now in its eighth year, the auditor examined the quality and usefulness of the Annual Outcomes Report using nine principles of effective performance measurement. In addition to our analyses, we met with a bipartisan group of Vermont legislators to better understand the extent to which legislators themselves feel well-served by the Report.
1. INFLUENCE: Population-level analyses have limited utility as a tool for State government performance measurement because the State has limited influence over most population-level outcomes.
2. CLEAR GOALS AND OBJECTIVES: The population-level goals are broad and undefined, which leaves the door open for multiple, possibly even contradictory, interpretations of success.
3. SIGNIFICANT AND RELEVANT: Many indicators do not capture the most significant or relevant information.
4. COMPARABLE: Most indicators lack contextual information and comparative data that would enable readers to draw meaningful conclusions.
5. DISAGGREGATED: When appropriate, disaggregating data would inform policy discussions about the disparate and unintended inequitable impacts of policies and programs and/or where to target resources.
6. UNDERSTANDABLE: There are opportunities to improve how data is presented to make it easier for the reader to understand.
7. WELL-DEFINED: Some indicators have no definitions of the variables being measured, without which the reader is left guessing.
8. VALID: Many indicators lack specific data sources or methodology information, making it difficult for the reader to assess the validity of the data.
9. TIMELY: The data is generally current, though some descriptions and indicators need updating.
As one legislator we spoke with said, it’s time to “go back and scrub [the Outcomes Report]. Ask whether we are getting what we want from these indicators.”
“Developing useful and meaningful performance data requires time, resources, and expertise, but the payoff is worth it if it helps us improve government services and policies. When done poorly, however, performance measurement can be misleading and a waste of time. The magnitude of the issues that surfaced in our analysis ranged from fundamental to technical; they all pointed to the need to reassess whether the Outcomes Report is serving its purpose,” concluded Hoffer.
Source: MONTPELIER, VT – State Auditor 1.24.2022