Program-Owned Evaluation:
Four Recent Evaluations Show That Much Can Be Done At Low Cost

Legal aid project coordinators often ask The Resource to assist in evaluations of their projects. Usually, these are being done to fulfill the requirements of a grant; however, leaders interested in innovation and growth are embracing the idea of going beyond grant requirements to apply evaluation as a tool for developing invaluable insights about how their projects are performing.

They are finding that evaluation can help them refine their strategies by revealing what's working for clients and what's not. The story they are able to tell funders is, "Here's where the project is successful and here are the improvements we are incorporating to make it work even better in the future."

The four recent project evaluations described in this article illustrate the power of an evaluation model that is affordable even for small programs for whom expert-assisted evaluation might seem out of reach

Four recent project evaluations illustrate the power of simple evaluations that are affordable even for small programs. Each of these was done at a cost of $10,000 or less, including the data collection, analysis, and report. Considerable savings were achieved by using project staff, volunteers, or student interns to collect the data; The Resource designed the instruments, coached the staff, analyzed the data and produced the final report. The four evaluations covered a wide range, including a web- and kiosk-based self-help assistance project, a legal hotline serving non-English-speaking clients, a courthouse-based pro se assistance project, and a national support project that provides training and technical assistance to civil justice programs.

This model of evaluation focuses the expertise that The Resource brings to the table where it is most needed to ensure objectivity and reliability of results. It conserves resources by involving project staff, interns and volunteers in data collection, which is often the most expensive part of an evaluation. It thus brings the powerful insights of program-owned evaluation within reach of these programs – and, as an added benefit, increases the program's ownership of the results by involving project staff in the evaluation process.

1. Evaluating a Promising New Technology

The I-CAN! Tax Module evaluation shows the high potential of an innovative new project and explores ways to maximize its impact in the future

In 2002-2003, the Legal Aid Society of Orange County received grants from LSC and the IRS to develop a self-assisted, web- and kiosk-based system using LASOC's I-CAN! technology to enable low-income people to prepare their own income tax returns and apply for the Earned Income Credit (EIC). An evaluation by The Resource showed that in its first season, the I-CAN! tax module worked exceedingly well, producing over $1,600 in tax credits for each low-income tax filer who completed a session, and scoring over 97 percent on client satisfaction and ease-of-use measures.

A web-based technology for use by low-income people. The I-CAN! system, developed by the Legal Aid Society of Orange County (LASOC), enables low-income people to use a computer terminal, kiosk, or any internet access vehicle to perform many simple legal tasks themselves that previously would have required significant help from a legal professional. The I-CAN! tax module and its accompanying on-line video and text assistance are available in English, Spanish, and Vietnamese and target a fifth-grade reading level to maximize ease of use. Once they have completed the forms, taxpayers can then file them either in traditional paper format or electronically.

The Resource was engaged by LASOC to perform an interim evaluation of the first season's implementation of the I-CAN! EIC module. The evaluation included statistical, demographic, and financial analysis of client and outcomes data, an on-line exit survey (completed at the end of the user's session) to obtain input on client satisfaction and suggestions for improvement, and telephone interviews with project staff and agency partners. The Resource designed the data collection instruments, conducted the analysis, developed the interim conclusions, and is in the process of conducting the telephone interviews; I-CAN! staff implemented the survey instruments and collected statistical and survey data.

• Successful use by low-income clients. As of May 2003, 216 low-income clients had successfully completed their returns using the I-CAN! system.

• Exciting initial results. In its start-up season, low-income taxpayers obtained an average of $1,602 per completed session. This portends enormous payoffs for clients in the future as more I-CAN! partners come on-stream and make this technology available to their clients.

• Positive feedback from clients. More than 99 percent of respondents to the exit survey indicated they found the system either "Helpful" or "Very Helpful", and more than 97 percent found it either "Easy" or "Very Easy" to use.

• A powerful outreach network in place. A total of 60 partners (including 27 legal services programs and 33 libraries, community technology centers and other community organizations) had agreed to make the I-CAN! EIC module available to their clients. During the 2003 tax season, due to a late start, the I-CAN! tax program was fully implemented in only three partner locations; in 2004 it is anticipated that many more partners will make this tool accessible to thousands of clients.

• A clear path for improvement. The main challenges ahead consist of activating the partners and increasing public awareness of I-CAN! in the low-income community. In the first tax season, partners were slow to ramp up the use of the system by their clients. Once this issue has been overcome, the usage is expected to climb and the number of benefits of the I-CAN! system for clients and communities will be dramatically expanded.

2. Documenting Success of a Pro Se Assistance Project

The Sonoma County Self Help Access Center evaluation reveals the power of courthouse-based, pro se assistance as a tool for expanding access to our civil justice system.

In August 2000, the Self Help Access Center (SHAC) opened its doors as the culmination of a local collaborative effort to address the flood of unrepresented litigants showing up in the California court system. A "program-owned" evaluation by Sonoma County Legal Aid and The Resource showed that in its first few months of operation the project was already filling a critical gap in the justice system in Sonoma County.

An innovative strategy for helping self-represented litigants. Many in pro per (self-represented) litigants face serious legal problems such as child custody, support or eviction. They are unable to afford a private lawyer, yet face long waiting lists at Legal Aid. The Self-Help Access Center, located in the Sonoma County Courthouse, serves walk-in visitors, people referred to it by court personnel, and people sent over to it by Sonoma County Legal Aid. It provides four types of services:

• Information and materials,

• Legal advice and counseling,

• Assistance in completing legal forms, and

• Referrals to other providers of legal and non-legal assistance.

An important condition of the Equal Access Fund Partnership Grant providing SHAC's principal funding was that an evaluation be performed using both qualitative and quantitative information. With a small supplemental grant, Sonoma County Legal Aid hired The Resource to assist with the evaluation. Together SCLA and The Resource designed and carried out an evaluation based upon six components: service statistics, exit surveys, follow-up client interviews, court observation of SHAC-assisted clients (and unassisted litigants for comparison) by law student volunteers, court clerk interviews, and partner interviews.

An evaluation of the Center's first six months of operation produced six overall conclusions.

• Assisted litigants performed better. The Self Help Access Center improved the performance and prospects of people representing themselves. Assisted litigants were better prepared, presented themselves better in court, and obtained better results than they could have on their own.

• SHAC helped the court fulfill its mission. The Center helped make the justice system both more user-friendly and more efficient.

• Court users were highly satisfied. The vast majority found SHAC "very helpful," and gave the Center's overall service an average rating of 4.86 on a five-point scale.

• Partners felt the project was successful. Local agencies that partnered with SHAC felt the project produced win-win solutions to community problems.

• It was cost-effective. In the Center's first six months, its average cost per person served was $81 (far less than the cost of direct representation), providing a cost-effective way to supplement its parent legal aid program's "traditional" models of service delivery, which included pro bono and staff-model legal services projects. The Center became part of Legal Aid's intake system, intercepting people who were coming directly to the courthouse and who might otherwise never have obtained any legal assistance at all.

• The SHAC model dramatically expands access. The evaluation showed that with a modest amount of additional resources the project could expand its hours of operation and thereby help all of the people who enter the courthouse seeking to represent themselves in simple matters.

3. Evaluating a Legal Hotline:

The ALLIP evaluation shows that a legal hotline can be a viable model for meeting the special legal needs of a non-English-speaking population.

The Asian Language Legal Intake Project (ALLIP) was created in 2002 to improve access to legal services among the low-income Chinese and Vietnamese communities in the greater Los Angeles/Orange County area. A "program-owned" evaluation by ALLIP and The Resource shows that the project is already producing significant outcomes for clients and points the way to improvements that will make the project even stronger in the future.

A legal hotline model. ALLIP was created by the Asian Pacific American Legal Center (APALC) in partnership with Neighborhood Legal Services (NLS), Legal Aid Foundation of Los Angeles (LAFLA), and Legal Aid Society of Orange County (LASOC), with primary funding provided by the Open Society Institute. ALLIP's delivery model is centered around a coordinated hotline system which delivers services wholly in the native languages of the client population, namely, Mandarin, Cantonese, and Vietnamese. ALLIP staff provide a broad range of services to clients, including:

• Counsel and advice on legal matters

• Assistance in completing legal forms, documents, and letters

• Education on common legal issues and delivery of printed informational materials

• Brief intervention and advocacy on the client's behalf (e.g., calling a landlord or governmental agency)

• Referrals to other agencies for in-depth assistance, including partner legal aid programs, specialized programs, pro bono attorneys, and community based organizations

ALLIP engaged The Resource to undertake an evaluation designed to answer two fundamental questions: the extent to which the Project increases access to legal services for under-served communities, and how the Project can serve as a model for other service providers serving similarly linguistically diverse client populations. The evaluation design incorporated the perspectives of clients, Project leaders and staff, and partners and other stakeholders, in order to provide a well-rounded array of inputs from multiple viewpoints. Data collection methods included telephone interviews of ALLIP Project leaders and staff, analysis of service statistics drawn from ALLIP databases, telephone follow-up interviews of a random sample of clients, and telephone interviews of Project partners. The client surveys were conducted by multilingual ALLIP staff in the clients' native languages, using instruments designed by The Resource. Partner interviews, in addition to the data analysis and final evaluation report, were completed by principals of The Resource.

The evaluation showed that ALLIP has achieved compelling results to date:

• Significant outcomes and satisfied clients. Most clients found ALLIP's services helpful and were able to obtain at least some of the outcomes they sought; those who were least satisfied were typically those having tough problems beyond the scope of a hotline and/or those referred elsewhere for assistance.

• An effective approach for serving a hard-to-reach population. Partners felt that the Project provides a valuable service for clients, in addition to fostering collaboration among the legal aid providers in the greater Los Angeles area in addressing the challenges of serving this hard-to-reach population.

• Fulfillment of goals. The Project had met its first-year milestones and appeared to have strong momentum to carry forward successfully into its second year.

4. An Effective National Support Project:

The NTAP evaluation demonstrates the high potential of a virtual (web-based) training strategy...And provides suggestions for improving it.

The National Technology Assistance Project (NTAP) was established in 2001, with funding from LSC, to support the poverty law community's efforts to apply technology for improving efficiency and client services. A "program-owned" evaluation by NTAP and The Resource collected feedback from users of the project's training and technical assistance services. The evaluation report provides strong evidence of the project's successes to date, from a user's perspective, as well as clear indications of where future improvement efforts should be focused for maximum payoff.

A national support project fostering the effective application of technology. NTAP delivers assistance on specific technologies to LSC Technology Initiative Grant (TIG) recipients, their state justice communities, and the general poverty law community. NTAP also develops and delivers trainings on technology topics to a wide variety of audiences in the legal services community to foster coordination among legal services programs, encourage replication of working models, and build the legal aid community's collective ability to make technology solutions succeed. NTAP's role further includes acting as a clearinghouse for technologies that can be used by programs nationwide, and helping to create national networks to effectively deploy these technologies within and beyond the legal services community.

An evaluation design stressing user feedback and suggestions for improvement. The Resource coached NTAP staff on evaluation design and implementation, developing a set of outcome measures to be used going forward to evaluate program performance against critical objectives and milestones, and performed a direct analysis of outcomes and satisfaction data. The latter encompassed feedback from participants in training sessions, workshops, and other forums used by NTAP in delivering services; interviews of key personnel at programs that have received support and assistance from NTAP; and a review of satisfaction, outcome, and other data collected by NTAP as part of its ongoing quality improvement processes.

Newsletter Home

Contact Us