ASSESSMENT DETAILS

Summary

Interviews with workshop participants

Product analysis

Newsroom surveys

Selecting organizations to assess

Summary

Between May 2002 and July 2003, Professor William Damon – one of the original architects of the Traveling Curriculum program – and his associates interviewed 126 print journalists from seven newspapers that participated in Curriculum workshops about their experiences. What did they remember? Did they find anything particularly valuable? Were they able to implement any training lessons into their daily routines? Did they believe their newsrooms had implemented training lessons in a noticeable way? What could their newsrooms, and they as individuals, do to keep alive a spirit of striving for excellence in their journalism? These interviews became the foundation for a formal assessment of the impact of the Traveling Curriculum program.

The formal assessment began in July 2003, after a fulltime assessment associate was added to the CCJ staff. The assessment team decided to focus on three measurement tools to determine what, if any, impact the Traveling Curriculum workshops were having on the individual journalists and newsrooms that participated. These three measurement tools included: 1) One-on-one interviews with journalists who participated in workshops. 2) Analysis of staff surveys administered before and after workshop participation. And 3) analysis of samples of “product” collected before and after workshop participation.

Between July 2003 and September 2005, the assessment team traveled to 34 more news organizations that had participated in Traveling Curriculum workshops to conduct interviews and collect samples of news product. Of these 34 news organizations, 17 were newspapers, 15 were broadcast stations, one was an online organization, and one was a student newspaper. The assessment team interviewed 549 more former workshop participants and collected and analyzed almost 1,800 staff surveys. The team also collected and analyzed nearly 2,100 newspaper stories and 300 local television newscasts.

A more detailed description of the three measurement tools, the rationale for their selection, details about how they were utilized, and a description of the process for selecting organizations to assess follows:

[top]

1. Interviews with workshop participants

Interviews with journalists who’ve participated in CCJ workshops are the primary component of the assessment model. The goal was to interview at least 15-20 former workshop participants at each organization assessed. As noted above, some 675 former workshop participants have been interviewed across Damon’s preliminary assessment visits and the “formal” assessment period.

Interviews were conducted in on-on-one settings and typically lasted 30 minutes. An effort was made to interview a representative group of staff from each newsroom – a mix of participants from different job positions, hierarchical levels, ages, genders, ethnic backgrounds, etc. The top newsroom manager was interviewed in nearly every newsroom, as was the individual whose decision it was to invite CCJ in for training, if that person wasn’t the newsroom manager.

Interviews followed a protocol that was specific to the training modules to which each newsroom was exposed. For example, if a newsroom received the accuracy and verification module during its workshop, interviewees were asked what they remembered about the workshop in general, and then whether they remembered and implemented any lessons or tools from that specific module. Each battery of questions related to a particular module included open and closed-ended questions.

Interview responses were transcribed and loaded into a database for further analysis. These responses were content coded for consistent themes and ideas. Then a detailed, customized report was prepared for the staff of each newsroom that focused on what workshop participants said they learned from the CCJ training and what they felt like their newsrooms could still do to improve. These reports relied heavily on direct quotations from interviews and the assessment team’s observations of each organization’s culture, communication systems, and staff morale. In some cases, a confidential memo was prepared for the newsroom manager when it seemed important to share information from interviews deemed too inflammatory or contentious to include in the staff report.

Interviews were chosen as the primary assessment tool because they allowed us to gather rich qualitative data and, through open-ended questions, get a sense of the universe of responses and reactions to the Curriculum program. One-on-one interviews conducted “on-location” allowed CCJ assessment staff to process not only whether training lessons did or didn’t “stick,” but also how the specific circumstances of each newsroom environment impacted the saliency of training lessons for individuals and entire newsrooms.

In some cases, the opportunity to observe the culture and communication systems in a newsroom while on an assessment visit proved as valuable in terms of understanding CCJ’s impact, or lack thereof, as what interview respondents had to say about their workshop experiences. The importance of being in a newsroom while conducting interviews was reinforced at an early assessment visit to a large Midwest television station. Interviews revealed that participants had high recall of the Traveling Curriculum workshop in which they participated, but few of the suggestions put forth in the post-workshop report had been implemented. Many of the interviews were tinged with negativity toward the newsroom manager. The CCJ assessment team observed editorial meetings and the way staff interacted with each other and the manager. The manager seemed gruff and intimidating during the meetings observed. CCJ assessment staff pointed out to the manager that his management style seemed abrasive – something newsroom staff and middle managers weren’t comfortable doing themselves. We, as concerned third-parties, were able to give the manager advice in a non-threatening way, and received insight into why training suggestions hadn’t been implemented – staff were overwhelmed by the low morale caused by negative feelings toward their manager.

As the example above demonstrates, assessment interviews have been invaluable in not only providing data about which CCJ workshop tools and lessons were most memorable and useful, but also offering context and explanations for variances in impact from newsroom to newsroom.

[top]

2. Product analysis

We collected pre and post workshop “product” from 15 newspapers, 11 local broadcast stations, and one Internet newsroom.

We collected two types of “product” from the newspaper and Internet newsrooms we assessed. In order to get a sense of how a newspaper might have been impacted broadly by a CCJ workshop, we collected and analyzed four weeks of “Local/Metro” section front stories – two weeks from right before the CCJ workshop and two weeks from somewhere between six and 18 months after the workshop. The Local/Metro section of most papers we studied contained mostly original content – not a lot of wire stories contributed by reporters outside of the assessed newsrooms. We reasoned that national and international events drive much of a newspaper’s “Front” page content. But Local/Metro section stories generally are driven by reporter enterprise and the newspaper’s identification of salient local issues and interests, making the content in this section more likely to be visibly affected by the many workshop lessons aimed at making stories relevant and interesting to one’s community.

The other type of product collected from newspapers was stories from individual reporters who had participated in a CCJ workshop. One such reporter was selected from each newspaper assessed. We asked the editor of each newspaper assessed to identify a reporter who, from her perspective, seemed to get the most out of the CCJ workshop. Then we collected 20 of that reporter’s stories for analysis – ten from before the workshop and ten from six to 18 months after. For two reporters whose work we decided to use in case studies (see the “Impact on quality of news product” section), we collected an additional 20 pre and 20 post workshop stories. We applied the same coding scheme to these stories as we did to the section front stories we analyzed. The goal here was to determine whether lessons from the workshop became salient enough to measurably affect the quality of the work produced by individuals exposed CCJ training.

For the lone Internet newsroom we trained and assessed, WEB-1, the product collection process was similar. Instead of a “section front,” we collected the first five stories to appear in, next to, or below the website’s homepage “target box” on a given day. As with the newspapers we assessed, we identified an individual reporter who appeared to have been especially impacted by the CCJ workshop and collected a sample of his pre and post workshop stories.

The assessment team collected and analyzed 2,208 print and Internet stories. The pre and post workshop product samples were nearly identical in size – 1,098 stories from before the training and 1,110 from after. Of the 2,208 stories collected, 384 were from the individual reporters we identified from each newsroom. The rest were from Local/Metro and Front section stories from various authors.

Broadcast station product was a bit trickier to obtain than newspaper and Internet product. The ideal broadcast product to collect is video of newscasts, since that is what viewers actually see and experience. Unfortunately, we learned that few local television stations have the capacity to store video copies of their newscasts for long periods of time in such a way that they’re easily and affordably accessible.

We experimented with collecting newscast “director scripts” which include the dialogue that newscast anchors read and details about the visuals that accompany stories, but a single half-hour newscast script can run more than 100 pages long. It wasn’t practical to ask newsrooms to print out weeks of these scripts – the few times we asked for them it took hours to print them out, which tied up employees’ work time and depleted paper supplies and stations’ goodwill.

We settled on collecting newscast “rundowns” of each assessed station’s “broadcast of record.” A “rundown” is a condensed list of the stories that appear in a newscast with information about the order in which they ran, the manner in which they were reported – commonly referred to as their “treatment” – and how long they ran, among other things. A rundown does not contain the “text” of a story, but does include a brief phrase describing the content of each story.

We collected rundowns from 11 local broadcast stations. As with print and Internet, the sampling period we aimed for was two weeks of product from before the workshop, and two weeks from six to 18 months after the workshop. The 44 weeks of rundowns collected consist of 295 newscasts containing more than 5,000 stories. We did not collect individual reporter stories from the broadcast stations we assessed.

[top]

3. Newsroom surveys

Starting with the Traveling Curriculum’s first training visit to PR-3 in February 2001, Curriculum coordinators have administered staff surveys in advance of their workshops. The initial aim of these surveys was to provide trainers with an idea of a given staff’s morale, perceived job challenges, strengths, weaknesses, and routines. Curriculum staff used survey responses to customize workshops to the needs and aptitudes of each organization.

Importantly, these surveys also initiated contact with the staffs of each organization to get them thinking reflectively about their routines and product quality. They introduced CCJ and informally primed staff for the training they would soon receive.

Initially, staff surveys were administered in paper form and respondents could mail or fax them back to the Traveling Curriculum Coordinator for analysis. This process proved cumbersome and time-consuming, so the surveys were translated into an online format that is still used today for print workshops.

Broadcast Traveling Curriculum Director Wally Dean found that, even today, many local television staff – primarily photographers, field staff, and studio staff – have limited Internet access. So he continues to administer and collect paper surveys from the stations at which he coordinates and conducts workshops.

The wealth of information contained in the responses to these pre-workshop surveys has been invaluable to workshop trainers. Having a system already in place for surveying newsrooms also created a natural research opportunity for the assessment team. We re-administered similar versions of these surveys in advance of our assessment visits. Survey responses were then used to aid in determining how newsroom attitudes and practices may have changed in the year or so after a CCJ workshop.

We analyzed the pre and post workshop survey responses of the newsrooms we assessed. This survey sample included 695 pre-workshop surveys and 437 post-workshop surveys from 14 newspapers. It also included 382 pre-workshop surveys and 261 post-workshop surveys from 13 local television stations. In all, we analyzed 1,775 surveys.

[top]

4. Selecting organizations to assess

The earliest newsrooms assessed had participated in a workshop within the past 16 months. We tried to vary the time lapse between initial workshops and assessment visits within a six to 16-month range, hoping to anecdotally determine how much of a factor time played in the “freshness” of training lessons. The earliest we returned to a news organization for an assessment visit was six months after the initial workshop. The longest gap between an initial workshop and an assessment visit was 32 months.

The best time for an assessment visit seemed to be approximately a year after the initial workshop. By that point, enough time had passed that the initial excitement from the workshop had worn off. Interview respondents were left with the lessons, examples, and tools that were most salient for them personally. These things were, by that time, more likely to be a part of their routines and thinking. Enough time had passed that if suggested changes were being implemented on the newsroom-level, interview respondents would be aware of them and able to make a judgment about their effectiveness. We also reasoned that measurable changes in product quality that could be directly or indirectly attributed to the Traveling Curriculum workshop lessons were likely to engrain themselves in a year’s time.

Also, we found it was important to provide adequate time between administering our pre and post-workshop staff surveys. Some organizations we assessed hesitated to allow a re-surveying of their staffs within a year, citing “survey fatigue” and a drain on their workers’ time. Plus, administering an almost identical survey within too short a time period after an initial survey introduces concerns about a “Hawthorn affect” – where respondents tell surveyors what they think they want to hear because they become more aware that they’re part of an experiment or study.

Besides timing, the next most important factor in determining which newsrooms we assessed was trying to include newsrooms with a mix of staff size, audience size, geographical location, and ownership characteristics. We also hoped to include newsrooms that had been exposed to a variety of Curriculum modules.

The 41 newsrooms that ended up in the assessment sample are diverse in these areas. Of the 24 newspapers assessed, seven have staffs over 1,000, four have staffs between 500 and 1,000, eight have staffs between 200 and 500, and the remaining five have staffs under 200. Of these newspapers, seven have daily circulations over 200,000, five have circulations between 100,000 and 200,000, five have circulations between 50,000 and 100,000, and the remaining seven have circulations under 50,000.

Geographic diversity was attained to some degree in selecting newspapers to assess as well. The geographic breakdown of assessed newspapers went as follows: Eight newspapers from the Western U.S., seven from the Midwest, five from the Southeast, three from the Northeast, and one from the Southwest.

In terms of ownership, the assessed newspapers break down this way: Five newspapers are owned by local companies or trusts, four are owned by regional chains, and 15 are owned by national chains. Of the 15 owned by national chains, one is owned by Gannett, three were owned by Knight Ridder, four are owned by McClatchey, one is owned by Media News Group, one is owned by the New York Times Regional Newspaper Group, two are owned by Scripps Howard, two are owned by the Tribune Corp., and one is owned by the Washington Post Company.

Similar efforts at diversity were made in selecting local television stations to assess. Of the 15 broadcast stations in the assessment sample, five are in one of the largest 35 markets in the country, six are in markets that fall between the 36th and 75th largest markets in the country, and the remaining four are between the 95th and 120th largest markets in the country. Local television staff sizes don’t differ significantly from large to small markets. The largest station assessed had only 40 or so more newsroom staff than did the smallest station assessed.

Geographically, the local television stations assessed break down as follows: Six stations from the Midwest, six stations from the Southeast, two stations form the Southwest, and one station from the Northeast.

In terms of ownership, the assessed local television stations break down this way: Local companies own five, and national chains own the remaining ten. Of the ten chain-owned stations, one is owned by Gannett, one is owned by Granite Broadcasting, two are owned by Hearst-Argyle, two are owned by Media General, two are owned by the New York Times, one was owned by Raycom Media, and one is owned by Viacom.

[top]