LSC grantees in more than ten states use similar outcomes measure systems developed by Interest on Lawyers’ Trust Accounts (IOLTA) programs, bar foundations or similar state funders (or “IOLTA” funders).
These IOLTA funders require grantees to collect and report similar types of data, which include outcomes and outputs data as well other quantitative, qualitative and narrative data. These funders use these data as part of their grantmaking and grant oversight processes.
Grantees have participated in the design and implementation of these data collection and reporting systems to ensure that they were useful for grantees and not unduly burdensome.
Legal Services Corporation of Virginia (LSCV) implemented an outcomes reporting system in 1998. All of its grantees – including six LSC grantees – now use this system. See the sections below to learn about how the LSCV data reporting system and how one of these grantees, Blue Ridge Legal Services (BRLS), uses the system.
LSCV engaged BRLS and its other grantees in the development and implementation of its outcome measures system to ensure that the data collected was useful for both LSCV and the grantees.
Although LSCV does not make funding decisions based on outcomes data, those data have enabled LSCV to enhance its grantmaking decisions and grant oversight functions, better ensure grant funds are used effectively and efficiently, and successfully demonstrate the value and benefits of state funding of legal services.
For example, identifying and documenting the results of grantees’ work improves:
- Grantees’ ability to formulate more concrete and measurable objectives in their funding proposals;
- The capacity of LSCV and grantees to assess programs’ success in accomplishing these objectives;
- Assessments of the alignment of grantees’ advocacy work and their objectives and priorities; and,
- The identification of areas for program improvement.
LSCV also uses outcomes data to improve its ability to show the impact legal services programs have in the community and to demonstrate the value and cost-effectiveness of legal services programs’ work to the public and the Virginia General Assembly. This is essential since the General Assembly funds legal services filing fees and general appropriations. LSCV used outcomes data in its publications, including the 2012-2013 Report to General Assembly and its recent report, Economic Impacts of Civil Legal Aid Organizations in Virginia.
BRLS has used the LSCV outcomes data system since it was first implemented in 1998. It has found that its outcome metrics are increasingly valuable for both external and internal purposes, because outcomes data enable the program to better “tell the story” of the benefits of its work much more powerfully than outputs data alone.
BRLS reports that outcome measures help improve its client and program services by:
- Helping it better align its advocacy work with its strategic goals and priorities;
- Generating reports that show the accomplishments of entire program, each office and each case handler;
- Assess the results of advocacy strategies and identify approaches to improve the benefits the program provides clients and the client community; and
- Informing the Board’s decision-making by profiling the focus and accomplishments of the program’s work.
More detailed information about the range of reports with outcomes data that LSCV and BRLS produce and assessments of the value of these data are in Components of the outcomes reporting system and Specific examples of the ways BRLS and LSCV use outcomes measures.
It should be emphasized that outcomes data alone do not meet LSCV and BRLS’ data needs. Both rely on the combination of a wide range of measures and data sets. Both also use data such as CSR data and numbers of persons data, Census data and data from other federal and state agencies regarding the client’s population characteristics (e.g., geographic location, economic and demographic data, education and employment status)and financial and staffing data. Equally important is BRLS’ on-going engagement with client and community groups, surveys and interviews with staff and community membersand GPS or similar data to align services with the needs of the client community need.
LSCV partnered with its grantees to develop the LSCV outcomes system. LSCV initiated what ultimately became a year-long process in 1997. Throughout this process, LSCV and its grantees worked with the staff of The Resource for Great Programs, which facilitated this process and played the lead role in designing a system that LSCV and its grantees agreed would be most effective. The Resource had previously worked with New York’s Interest on Lawyers’ Accounts (IOLA) program to develop an outcomes system for its grantees. At the same time that is was working with LSCV, it was working with the Maryland Legal Services Corporation (MLSC) to develop a similar outcomes system in that state. The Resource subsequently worked with other states to develop similar systems.
The development process began with a meeting of the LSCV executive director, the executive directors of Virginia’s legal aid programs and the director of The Resource. At the meeting, the group reviewed the outcome categories used in the NY IOLA system and discussed the benefits and shortcomings of using a similar system. Over the next several months, Virginia’s program directors and LSCV staff developed suggestions for ways the NY IOLA system could be modified to best meet the needs of LSCV and the Virginia programs. LSCV then worked with The Resource staff to develop a system that incorporated these elements.
During this period, management of the individual programs engaged their staff in a process to identify how the system would be most useful and efficient.
In addition to developing the substantive content of the system (see Major outcomes categories and metrics and Components of the outcomes reporting system) the partners also had to identify the case management system (CMS) or other data systems that would be most effective in capturing the data. About two years after the implementation of the outcomes system, LSCV purchased the same CMS for all of its grantees. The CMS had all of the capacities needed for the effective and efficient collection, analysis and reporting of the outcomes system.
The LSCV outcomes data system used by Virginia’s legal aid programs has four major outcomes data categories. Outcomes data are collected only for extended service cases (LSC CSR case closing codes F-L), not for limited services cases (LSC CSR case closing codes A and B).
Outcomes Category 1: Major Benefits from Direct Legal Representation of Individuals
The system collects outcomes for cases closed in twelve areas that correspond to the LSC CSR legal problem categories (e.g. consumer/finance, education, employment, family, housing). The number of possible outcomes in these categories varies.
For example, the consumer/finance category has 20 different possible outcomes. Two examples of the outcomes in this category are: “Obtained federal bankruptcy protection,” and “Stopped or reduced debt collection activity.” The family category has 32 possible categories. Outcomes in this category include: “Obtained or maintained custody of children,” “Obtained protection from domestic violence,” and “Obtained, preserved, or increased child support.” Information is also provided for the “Number of persons directly affected” for each of these outcomes. See a full list of the possible outcomes in each of the twelve categories.
Outcomes Category 2: Direct Dollar Benefits for Clients
These are for “affirmative dollar awards” to clients, not cost savings from judgments of payments avoided, which are reported separately (see next bullet). Affirmative benefits are reported for the amounts of “lump sum awards/settlements” and “monthly benefits.” Specific financial benefits categories include, but are not limited to, “Social Security, SSI,” “Child Support,” and “Affirmative consumer judgments.” In addition, the “Other Benefits” category can include “Insurance settlements” and “Housing allowances.” See a list of the affirmative dollar award categories.
Outcomes Category 3: Dollar Savings for Clients
These outcomes are for “amount of dollar savings achieved for clients through judgments or payments avoided,” such as “Defensive Consumer Law Matters” (e.g., bankruptcy, garnishment). See a list of the affirmative dollar award categories.
Outcomes Category 4: Benefits From Direct Legal Representation of Groups
Only one outcome is captured in this area: “Obtained incorporation/tax exempt status.” All other data collected in this area are for outputs, e.g., “Obtained assistance with regulatory/licensing issues,” “Obtained assistance with other structural or governance issues.” See the list of data collected for group representation.
Blue Ridge Legal Services uses the LSCV outcomes data categories and supplements them with the results of its on-going surveys of clients served by the program. It uses three separate surveys:
- one survey to obtain targeted information from clients receiving limited services,
- another survey for clients receiving extended services,
- and a third one for clients receiving services from pro bono attorneys.
There are English and Spanish-language versions of all of these surveys. The surveys are emailed to clients who provide an email address; others receive the survey via U.S. mail (with a BRLS-addressed, postage page envelope). Each survey is coded with the case number to match the client’s responses with other case data.
The Case Management System that LSCV purchased for its grantees provides specific three key functionalities. The CMS
- enables case handlers to input required outcomes data about the case at the time of closing
- generates basic reports for the grantees’ use
- allows the grantee to seamlessly report the information the LSCV
At case closing, BRLS advocates complete a case closing memo. Once advocates enter the outcomes and other data into the CMS, the program can seamlessly transfer the necessary data into an Excel file to submit outcomes (and other data) data to LSCV as well as to “slice and dice” the data in a variety of ways to produce reports for its own purposes.
See examples of the reports that BRLS produces
BRLS’ Use of Outcomes Data
BRLS uses outcomes data – along with outputs and other data – to produce reports to assess and improve various aspects of its operations. Examples of these reports are profiled below.
- To assess and evaluate its client services, BRLS generates reports that show:
- Program-wide client services data for all substantive law areas
- Client services data for all substantive law areas for a particular office
- Client services data for a particular geographic area within the program’s service area
- Outcomes based on clients’ demographic characteristics.
- “Success rates” data: whether the goals the clients and advocates identified at the time of opening the case were achieved.
- The outcomes data are key components of the quarterly reports to the BRLS Board of Directors about the program’s activities and accomplishments.
- Reports are also generated for individual casehandlers. When BRLS first initiated the use of outcomes data, some staff were concerned that the data would be used inappropriately for performance evaluations. However, staff now embrace the use of these data. Having information which more fully shows the results of their work has increased their morale and has also led them to take on cases that provide the greatest benefits to clients. They are not “cherry picking” cases that can yield high “success rates.” Instead, they are addressing more complicated issues and increasing the number of extended service cases that have greater impact on the lives of their clients.
- BRLS considers outcomes data critical to its fundraising success in its largely rural service area. There are nine relatively small United Ways in BRLS’ service area and a limited number of small private foundations. These funders want to assess the impact their limited funding has on the community, not just outputs such as cases closed. Outcomes data more fully demonstrate to funders how supporting legal services is a cost-effective investment. The program estimates that the use of outcomes data has significantly increased the funds it receives from the United Ways. For example, in 2013, a United Way allocations panel awarded BRLS twice the amount it had requested because it was impressed by the outcomes BRLS was able to document. For some funding requests, BRLS will go beyond the requirements of the application and include a supplement that shows more fully the impact of their work.
- BRLS has also used outcomes data to help inform difficult decisions to adjust its allocation of resources to ensure the program’s advocacy provides the greatest benefits to its client community. For example, the analysis of case outcomes and information from other sources indicated that certain types of family laws cases do not provide the same level of client benefits as cases in other substantive law areas. This led the program to reduce its volume of family law cases while increasing its volume of housing and consumer cases.
LSCV’s Use of Outcomes Data
LSCV does not make funding decisions based on outcomes data, but it does use these data along with a range of other data in desk reviews and associated grant oversight activities. LSCV staff reports that outcomes data improve the ability of grantees and LSCV to assess a programs’ success in accomplishing grant goals and to identify strategies to better achieve these goals.
The outcomes data LSCV collects from grantees are integral to LSCV’s Annual Reports to the VA General Assembly and provide the foundation for a comprehensive analysis of the Economic Impacts of Civil Legal Aid Organizations in Virginia.
LSCV cannot quantify how important its use of outcomes data has been to its success in sustaining and increasing these appropriations. It believes that these data contributes greatly to LSCV’s efforts to make the “business case” for legal services – and may have increased bipartisan support for legal services. Even though it cannot affirm that outcomes have been instrumental in sustaining support in the General Assembly, LSCV has concluded that it would be remiss if it did not marshal as much data as possible to demonstrate the need for and value of civil legal services for low income people.
- You get what you measure. The use of outcomes and other performance measures can have the unintended consequence of driving program services rather than providing a means of evaluating and improving program operations. BRLS management recognized that the measures the program used could significantly shape staffers views of what is – and is not – important and influence the type of cases they took and strategies they pursued, and BRLS’ executive director John Whitfield has analyzed different aspects of this issue.
- Outcome measures are necessary, but not the only tools needed to assess effectiveness. Both BRLS and LSCV stress that programs need a wide range of data, not just outcomes data, to assess and enhance program operations and improve client services. Both use outputs data such as CSR data and numbers of persons data; Census data and data from other federal and state agencies regarding the client’s population characteristics (e.g., geographic location, economic and demographic data, education and employment status); and financial and staffing data. BRLS also considers essential the on-going engagement with client and community groups, surveys and interviews with staff and community members and GPS or similar data to align services with the needs of the client community need.
- The outcome data gap for limited services is substantial. BRLS is striving for better outcomes data for cases involving only advice and brief counsel. Conducting the follow-up necessary to obtain sufficient data for these cases could require more time and resources than those that were needed to provide the services in the first place. The program has sought to overcome this limitation by revising its client satisfaction surveys for advice and counsel clients to include:
- Whether the advice provided answered their questions;
- Whether they understood the advice provided them; and
- Whether it was helpful in resolving their legal problem.
The staff maintains these data in the CMS along with the other outcomes data, so that it can generate customized reports for a particular funding source. (See Major outcomes categories and metrics and Components of the Outcomes Reporting System.)
- On-going evaluation and improvement of the system is essential. Since the system’s establishment, LSCV, BRLS and VA’s other legal services programs have engaged in a continuous assessment of the outcomes system to identify ways it is – and is not – working. Based on its own needs and feedback from grantees, LSCV has “tweaked” its system nearly every year. And BRLS has modified it reports and generated new reports to fit its needs.
- Many important aspects of the program’s work cannot be quantified. As with other legal services programs, LSCV and BRLS recognize that critical components of a program’s work cannot be meaningfully Limitations include: the quality of an advocate’s casework; the impact of the program’s work on a client’s sense of well-being (e.g., from avoiding homelessness, escaping from an abusive partner) or empowerment; and the impact of the program’s work with community organizations.
- Every measure is imperfect. LSCV and BRLS recognize that the outcome measures they use reflect assumptions and practices that can limit their validity and value. Questions they ask when assessing what measures to use include:
- Are we measuring the right things?
- Are the definitions of success appropriate?
- Are the measures based on the appropriate indicators?
- Are the factors used to calculate the financial benefits valid and appropriate?
- Are the advocates accurately reporting the information case outcomes?
As a quality assurance mechanism, when supervising attorneys review the case file at case closing they ensure that the case results information in the case closing memo is consistent with the case outcomes data entered into the CMS.
LSCV and BRLS address these concerns through their on-going evaluation of the system and their recognition and transparency about methodologies used to calculate outcomes and the limitations of these methodologies.