14 EVALUATION RESEARCH
Mike Howard is a senior program evaluator for a major private research and consulting company. Ninety percent of the company's work is connected to state and federal contracts. These contracts typically call for policy studies or program evaluation research. In policy studies the objective is to develop new or redevelop existing policies in accordance with organizational missions. These policies are then submitted as recommendations for adoption and/or revision by the contracting organization. In program evaluation the objective is to determine how well an existing program is meeting its goals. This is important for development of the program or termination of programs which are not successful. Over 300 employees of this company are involved in such work in three major offices across the country.
Mike earned B.A., M.A., and Ph.D. degrees in sociology, but his path was not exactly direct. When asked about his B.A. Mike said, "I had no specific goals, although I thought there might be some future in government. My parents were in government and I had some idea how things worked and how sociology might fit." Early on Mike knew he would pursue a Ph.D. but just when, where, and how was an open question. Initially he took a series of jobs in both teaching and social work. In one way this served to focus his attention and interests in the area of social welfare. Mike also began to see a need for concrete applications of his discipline to the problems he was studying.
After completing his doctorate Mike continued teaching but also began looking for other opportunities. He found some in contract assessment and evaluation work on a part–time basis. His first work in this area was in response to an ad in the newspaper for a person with a background in welfare policy and research. The job offered both full and part time consulting opportunities and, according to Mike, "higher salary scales than I could expect to achieve in teaching."
The contract was for a federal study of how states administer welfare and Medicaid programs. The issue was the use, by states, of data on welfare/Medicaid eligibility error rates. The potential problem is that people who are not eligible may somehow be defined as eligible. Because of errors in determining eligibility, money budgeted for these programs is not always well spent. The task is to minimize these errors. The only way to do so is research the processes by which eligibility is determined. The method of research was telephone interviews. Several different interview schedules were used so as to not artificially produce patterned results. Mike noted that, "While I was interviewing people from the same kinds of agencies it became very evident that I was dealing with different types of local organizations, different perspectives on the work being done and different ideas about the research I was conducting." Methodologically, Mike conducted content analyses, a quantitative process, to determine common denominators among similar, yet different agencies, so as to produce a uniform report. At stake was the appropriate distribution of thousands, or perhaps even millions of dollars per agency. This is important work.
After completing the study on eligibility error rates Mike decided to leave teaching and go to work full time for the same company. The following is one of the projects Mike has been working on. It is an evaluation of the implementation of substance abuse prevention programs. Funding for the study has come from the Center for Substance Abuse Prevention (CSAP). The study will take three years to complete. The CSAP has already funded 130 substance abuse prevention demonstration programs targeting youth. It is Mike's job to evaluate these programs for the purpose of developing an overall strategy for fighting substance abuse.
The tasks and issues confronting Jack are numerous and complex. The insights and skills of the sociologist are at a premium. These include, but are by no means limited to, defining the issues and problems faced by such programs, selecting data collection techniques, selecting a sample of 60 from 130 projects across the U.S., operationalizing significant variables, writing case study reports, writing cross–site reports (comparing sites), and leading site visits for data collection. In addition there are special topics such as program services to targeted populations based on gender and/or ethnicity. To get an idea of the size of the study Mike suggests looking at the final report. "There will be sixty individual program reports, each averaging forty pages. While the writing keeps you busy, it is the logistics of actually producing the program reports that really test you." An additional challenge, according to Mike, is "that the individual programs are so diverse and new that there is not much in common among them and little is known ahead of time about how they were working. This calls for a very flexible methodology so we can adjust as we go along."
To all of the research issues just mentioned you must also add, according to Mike, the "two worlds of the project manager." One of these is the research company's point of view. Here the project manager must deal with the composition of site visit teams, as well as internal project staff. Mike notes that, "The company is concerned with matters of personnel and the issues of task; what must be done and who will do it? But, you are working for a client (for example, CSAP) and they have their own interests and needs." In contracts with government there is the project officer for the government who is interested in task and a deliverable project. Then there is the government contract officer who is interested in the project budget variables and whether the work actually satisfies the contract. These people, and other related supervisors from the client perspective wield a great deal of influence. "Sometimes," according to Mike, "there is difficulty in attempting to please the client while staying within the parameters of the contract. This can be a very delicate balancing act." Then, in a multi–year contract there is always the potential for personnel turnover. This can easily upset the balance you have worked so hard to achieve. Mike adds that, "Managing a project like this is really a matter of managing all the various relationships. Because of this you must always be on top of things. If not, you can lose control quite easily."
Mike's job, as can be seen, is a varied mix of tasks, relationships and perspectives. His work is not simply researching a question, but managing the intricacies of organizational life. For example, all contracts have their beginning in a proposal. The client publishes an RFP––request for proposals––which outlines the project the client envisions. According to Mike, "It is important to anticipate RFP's, because once they are published the contracting company cannot have discussions with clients (if the client is the government). If you already have a good client relationship then you can find out what is going to happen before it happens. This way you can have all the discussions you want." This suggests the influence of structures outside what may be defined as official. "By and large," according to Mike, "a lot of what we do is informal rather than formal. Your personal network is a very valuable resource. It does matter who you know and by whom you are known."
When asked to connect his studies in sociology to his work as a researcher Mike suggested that, "There is very little that I do which is not informed by sociology." Organizational theory and management have obvious connections. "You are constantly applying your knowledge of how organizations function just to stay on top of your job. For example, the difference between formal/and informal is readily clear to the sociologist trained in organizational behavior. Such an understanding, however, is not a part of some other disciplines." General theory is also quite valuable. Mike notes that, "You are always dealing with people as they try to relate to each other. This is the essence of sociology and theory is an important tool in understanding, or framing some situation." So often people get caught up in the local or immediate issues. Sociological theory helps one see the larger and longer range picture. According to Mike, "Without this perspective there is no viable way to really evaluate a program."
In addition to theory, methodological skill is always in demand. According to Mike, "Methodological abilities are extremely important. But beyond competence, you must be creative. No two situations will be just alike." A part of methods is statistics, and in this regard Mike advises that, "You have to know how to crunch the numbers. I am always doing this to some extent." Finally according to Mike, "None of the skills will mean much if you cannot communicate. You have to be able to write all types of things from reports, to summaries, to letters, memos, etc. You will also have to make oral presentations, as well as simply speak clearly in your work." The computer is always a part of the work, so computer literacy is a must. According to Mike, "Our employees must be able to use software such as Word Perfect, Lotus, D–Base, and SPSS–PC." Without this capability it is not possible to do the job.
It is the case, however, that sociology is a discipline which does prepare the student for the above challenges. "The critical perspective of sociology sharpens the mind," according to Mike, and this promotes the kinds of competence and skills necessary. Mike does suggest that, "students actively seek internships or volunteer positions which challenge them to immediately begin applying the classroom to the real world. It's also quite good on a resume." Regardless of your preparation, Mike clearly points out that what ultimately matters most is, "Can you do the job? People don't care what your degree is as long as you are productive. It's just that sociology can give you a real advantage over other disciplines."
Finally, what about making a living? In Mike's company entry– level positions will pay $30,000 – $45,000 depending on skills, capabilities, experience and education. Senior staff, like Mike, will earn from $50,000 to $100,000. Top level vice presidents will earn from $90,000 – $120,000. Presidents will earn from $150,000 to $175,000. "But the bottom line is, can you do the job? If you can, you will make a good living."
Careers Page |