Another Outlook on Outcomes - Part Three

NOTE: This is Part Three of a continuing entry. See Monday 4/9 and Tuesday 4/10 for Part’s One and Two.

As it turns out, my April JTFest Online Library article, An Outlook on Outcomes, actually played a role in prompting the initial email to me. In today’s entry, I present his thoughts on outcomes and program evaluation. All the bold emphasis is his …

Outlook on Outcomes caught my eye because there is no metric in place by which to measure the desired outcomes - and very little has been done to define what the desired outcomes actually are.

Too much focus on immediate outcomes seems to be a common theme lately. Perhaps it's a side effect of all the attention deficit medication people are taking. A recent education reform act (which shall remain nameless for the sake of political neutrality) managed to transform the public school system into an institution dedicated to training children to master standardized tests irrespective of actual knowledge retention or educational value.

Because of the emphasis placed on the metric (and funding disincentives to not fail), the standardized tests are irrelevant to the root of what people really wanted to know in the first place: Is the education system preparing our children for success? Worse still, the outcome was altered by measuring it.

Just as the metric used to evaluate public schools fails to address the fundamental questions, and degrades the quality of service students receive, the measures by which youth services are judged both fail to provide answers to rudimentary questions, and erode the level of services that homeless youth receive. We are left with more questions than answers, and I don't think everyone is on the same page (or the same book) about which questions are the relevant ones.

It seems to me that the most important questions are the questions behind the questions. I believe that the questions that need answers are the long term questions. Transitional housing can get someone off of the streets, but does it help a person to succeed in the long run? What percentage of youths who have worked with street outreach programs find their way off of the streets (or conversely, into prisons) - how does this percentage compare to youths who have not had interaction with street outreach? What does the success curve look like over time; is it linear, logarithmic, or irregular? Do resource providers funnel the kids who are ready to the right resources - or just put tax dollars to work making young 'sluts' and 'junkies' comfortable (you know, the 'worthless kids')? Is there an approach which is demonstrably 'better', or does having a wide variety of resources from a wide variety of providers serving a wide variety of ethical viewpoints ensure that a broader cross section of kids is reached? Statistically speaking, how do shelters and drop in centers and street outreach programs effect the lives of homeless youths (versus homeless youth without access to these resources) 5, 10, and 20 years down the road? How do programs that heavily restrict personal freedom fare against those that strive to protect personal freedom in the long run? How many of these cases don't mutually exclude in the long run? How many do?

As far as I can tell, the most fundamental definition of success in this case is the long-term, permanent transition from homelessness to a state of self-sufficient, sustainable "un-homelessness". In other words, everyone wants results that 'stick'. If success is marked by long term cessation of the vagrant lifestyle, then the only way to measure and compare the effectiveness of different programs is to develop a metric based on long term statistics.

Of course, this would mean developing a method to track statistics in the long term. The ACLU would probably get all uppity if we were to try and radio tag these kids. Even if one could get homeless youth to voluntarily agree (without incentive or disincentive) to agencies using their government tracking numbers (id number / SSN) to monitor their long term progress 'from the shadows', the attempt to do so would likely be seen as exploitative and possibly even more evil than the radio collars.

Maybe agencies could nicely ask their clients to check in after 5 and 10 years.

I guess the hard part would be getting all of the homeless and displaced youth services around the country to remember to ask nicely for the call backs, develop a brief standardized battery of questions, secure permissions to collect criminal and employment records, perform the interviews, and not 'amplify' results to protect funding. You'd still need a central data compilation location.

For long term research, you need an entity that will still be around in 10 years. Youth services aren't well suited to the task. If you want any uniformity of data, persistence of location and the verisimilitude that comes with independent research, your best bet is finding a university to perform the study. That way the data would be handled and collected by an organization with no personal investment in the outcomes; an organization with people who know a lot more about data collection than the average bear; and an organization that will be around long enough to compile intensive long term information.

Tomorrow I’ll offer some of my thoughts on the ideas he has presented.