During the more than a decade that Ariel Castro allegedly held kidnapped teenagers and young women captive in his home at 2207 Seymour Ave., police officers were within shouting distance of the house more than a thousand times, according to Cleveland Police Department records analyzed by NBC News.

From the time the first kidnap victim vanished in August 2002 until the three women and a 6-year-old girl emerged from the Castro house on May 6, 2013, police responded to calls on that block 1,099 times — or about once every three and a half days. Castro pleaded not guilty last week to 329 charges, including kidnapping and rape.

The records — police dispatch logs and a few follow-up reports – offer a clue to a central mystery of the case: How could the women have gone undiscovered for so long? Castro’s whitewashed home – despite its plywood-covered windows and padlocked front door — was one of the quietest on a chaotic block, one of the houses making the fewest reports to police.

Next door to the Castro house, at No. 2003 Seymour, residents called police 35 times. Two more doors down, at No. 2115, police fielded 37 calls. Across the street and a few doors down at No. 2120, police came 68 times.

---

The neighborhood emerges from the police records as a central character in the crime story, a declining neighborhood in social turmoil. Why would the quiet house on the block draw a second glance from officers who are responding to domestic abuse calls and flashers, to broken windows and prowlers, to a fight involving 20 people armed with baseball bats?

 

 

Khalid Samad, a community organizer in Cleveland who had worked with police and organized community searches for the missing women, described an incident that occurred the evening that Amanda Berry, Gina DeJesus, Michelle Knight and Berry’s 6-year-old daughter were freed. He said it painted a picture of the current state of the neighborhood.

"Not an hour after they're out, I'm standing on the street near the Castro house and a fight broke out a few doors down because a guy who was out there saw a guy who he recognized as having shot him on the street,” Samad said. “Dude took off running, and they're wrestling down in front of the church. That's the kind of thing that would go on there."

The stretch of Seymour Avenue extending from Scranton Road to West 25th Street has seven houses on the south side of the street, surrounding the whitewashed Castro house at 2207, and nine houses and two apartment buildings on the north side.

It's a tough neighborhood struggling with unemployment and poverty, with a few well-tended houses and just as many vacant lots. Several houses, including an apartment building down the block from Castro’s residence, are vacant and boarded up. While some residents described the neighborhood as close-knit, others said it has suffered from increasing drug use and violent crime.

Given the frequency with which police visited the block over the period, it’s not surprising that they were often present near milestones in the kidnapping cases.

Three days after 20-year-old Knight disappeared on Aug. 22, 2002, police were two houses away from the Castro house, responding to a call at 2221 Seymour about the theft of a cell phone. (The residents at these addresses were typically calling police to report illegal or suspicious activity -- not perpetrators but victims or witnesses.)

Nine days after 16-year-old Amanda Berry disappeared on April 21, 2003, police were at 2221 Seymour again, investigating a car with no plates that had been left on the street for months.

And the day after 14-year-old Georgina "Gina" DeJesus disappeared on April 2, 2004, police were at 2022 Seymour, investigating harassing telephone calls.

Some incidents would have placed officers right outside Castro’s front door. On April 6, 2007, for example, two cars collided almost directly in front of the home, which is set back from the street. Police were on scene for 78 minutes.

On the Fourth of July weekend in 2006, a street fight near 2115 Seymour involving 20 people with baseball bats sent a pregnant woman to the hospital and drew several police cars.

Officers came to the Castro house only twice during the 3,910 nights that Knight was missing, as police have said, and neither call had anything to do with the missing women.

The first visit was on Jan. 26, 2004, as police investigated a complaint that Castro had left a child on the public school bus that he drove. The boy said Castro kept him on the bus while Castro went to a Wendy's restaurant for lunch. He was not charged. According to his school personnel file, Castro was fired in 2012 after a traffic violation and then leaving his bus unattended in a school fire lane while he went home to rest.

The second visit came on July 3, 2009, when Castro called police to complain about a fight in the street.

A third entry for the Castro address was marked in the records as a "test" on July 3, 2009; a police department spokeswoman said that would not have been a call from the address, and the records show no officer was dispatched.

The Cleveland police spokeswoman said officials would have no comment about the records, which were provided in response to a public records request by NBC News.

There is no indication in the records that police officers were inattentive or missed clues that could have led to the discovery of the women. Some of the calls they responded to were reports of unidentified women screaming, but there is no suggestion that the screams emanated from Castro’s home.

On Jan. 20, 2003, a visually-impaired woman at 2115 Seymour, three doors down the street from the Castro house, called to say she could hear a female "screaming out front." An officer arrived within five minutes, and was on the scene for 45 minutes. There's no indication in the records of what was found.

The same woman heard, on May 6, 2008, a female voice at the apartment building across the street from her house, yelling, "Get off me!" And, the report says, the caller said the voice sounded covered or muffled. She also said she heard a baby crying. Again, no report was filed, and the dispatch log doesn't indicate police found anything. The call was cleared 19 minutes after the officer arrived.

Nor is there any support in the records for statements by a few neighbors who said — after the women were rescued — that they had alerted police to strange goings-on at the Castro house. They included some accounts suggesting that witnesses had reported seeing women chained and naked in the back yard.

Police officials have said that those calls were never made, and the women themselves told investigators  they were only allowed outside twice, and then forced to wear wigs and sunglasses and keep their heads down.

"There is no evidence to indicate that any of them were ever outside in the yard, in chains, without clothing, or any other manner," Martin Flask, Cleveland director of public safety, told reporters on May 8.

Police Chief Michael McGrath told NBC News the same day, "We have no record of anyone calling" to report anything suspicious about the Castro house.

While police came under criticism from both residents and armchair detectives after Castro’s arrest, department officials have consistently defended their investigation of the disappearances as thorough and focused.

“I can tell you personally that I busted my butt to find those girls,” Keith Sulzer, Cleveland police district commander, said at a community meeting on May 9. "Me and my guys searched every vacant lot, every vacant building, everywhere that we could legally go in and search."

Perhaps as telling as what is in the records is what isn’t: calls about the Castro house. That absence supports police accounts of the isolation in which the women were kept and their terrorized mental states after years of captivity.

After their release, the three women told police that they were chained in the basement for a time, then were allowed more freedom to spend time on the second floor of the Castro house, still behind locked doors and subject to beatings, according to a Cleveland police report. The women also said that Castro intimidated them by pretending to leave but would punish them if they tried to escape or call out for help.

Even so, Samad, the community organizer, said he believed the women might not have gone unnoticed in a different neighborhood.

"If this was an inner-ring suburban neighborhood," he said, "you'd have some nosy neighbors who would ask, ‘Why are your windows boarded up, why are you taking groceries in if you don't have family there?’"

Contact reporters Bill Dedman at bill.dedman@msnbc.com, and Matthew DeLuca at Matthew.DeLuca@nbcuni.com.

More from Open Channel:

• US intel officials: 'Dozens' of terror plots disrupted by NSA surveillance
• Victim of alleged rape at Marine base: 'I thought ... I would be safe'
• Secret court won't object to release of opinion on illegal surveillance

Follow Open Channel from NBCNews.com on Twitter and Facebook 

The FBI has dramatically increased its use of a controversial provision of the Patriot Act to secretly obtain a vast store of business records of U.S. citizens under President Barack Obama, according to recent Justice Department reports to Congress. The bureau filed 212 requests for such data to a national security court last year – a 1,000-percent increase from the number of such requests four years earlier, the reports show.

The FBI’s increased use of the Patriot Act’s “business records” provision — and the wide ranging scope of its requests -- is getting new scrutiny in light of last week’s disclosure that that the provision was used to obtain a top-secret national security order requiring telecommunications companies to turn over records of millions of telephone calls.

Taken together, experts say, those revelations show the government has broadly interpreted the Patriot Act provision as enabling it to collect data not just on specific individuals, but on millions of Americans with no suspected terrorist connections. And it shows that the Foreign Intelligence Surveillance Court  accepted that broad interpretation of the law.

---

“That they were using this (provision) to do mass collection of data is definitely the biggest surprise,” said Robert Chesney, a top national security lawyer at the University of Texas Law School. “Most people who followed this closely were not aware they were doing this.  We’ve gone from producing records for a particular investigation to the production of all records for a massive pre-collection database. It’s incredibly sweeping.”  

The Justice Department and FBI did not respond to requests for comment. But in a recent interview with NBC News, Director of National Intelligence James Clapper dismissed the idea that the records were being used to spy on innocent Americans. “The notion that we’re trolling through everyone’s emails and voyeuristically reading them, or listening to everyone’s phone calls is, on its face, absurd,” he said. “We couldn’t do that even if we wanted to.”

But little-noticed statements by FBI Director Robert Mueller in recent years – as well as interviews with former senior law enforcement officials – hint at what Chesney calls a largely unnoticed “sea change” in the way the U.S. government collects data for terrorism and other national security investigations.

The Patriot Act provision, known as Section 215, allows the FBI to require the production of business records and any other “tangible things” -- including “books, records, papers, documents and other items,” for an authorized terrorism or foreign intelligence investigation. The Patriot Act was a broad expansion of law enforcement powers enacted by Congress with overwhelming bipartisan support in the aftermath of the Sept. 11, 2001, terrorist attacks. In addition to Section 215, other provisions expanded the FBI’s power to issue so-called “national security letters,” requiring individuals and business to turn over a more limited set of records without any court order at all. 

In contrast to standard grand jury subpoenas, material obtained under both Section 215 orders and national security letters must be turned over under so-called “gag orders” that forbid the business or institution that receives the order from notifying its customers or publicly referring to the matter.

From the earliest days of the Patriot Act, Section 215 was among the most hotly disputed of its provisions. Critics charged the language – “tangible things” -- was so broad that it would even permit the FBI to obtain library and bookstore records to inspect what citizens were reading.

Ashcroft confronted criticism
Largely to tamp down those concerns, then-Attorney General John Ashcroft declassified information about the FBI’s use of the provision in September 2003, saying in a statement that “the number of times Section 215 has been used to date is zero.” Ashcroft added that he was releasing the information “to counter the troubling amount of public distortion and misinformation” about Section 215. 

But in the years since, the FBI’s use of Section 215 quietly exploded, with virtually no public notice or debate. In 2009, as part of an annual report to Congress, the Justice Department reported there had been 21 applications for business records to the Foreign Intelligence Surveillance Court (FISC) under Section 215 – all of which were granted, though nine were modified by the court. (The reports do not explain how or why the orders were modified.) 

In 2010, the number of requests jumped to 205 (all again granted, with 176 modified.) In the latest report filed on April 30, the department reported there had been 212 such requests in 2012 – all approved by the court, but 200 of them modified.

These sharp increase in the use of Section 215 has drawn little attention until now because the number of national security letters (NSLs) issued by the bureau has been so much greater -- 15,229 in 2012. But FBI Director Mueller, in little-noticed written responses to Congress two years ago, explained that the bureau was encountering resistance from telecommunications companies in turning over “electronic communication transaction” records in response to national security letters.

“Beginning in late 2009, certain electronic communications service providers no longer honored NSLs to obtain” records because of what their lawyers cited as “an ambiguity” in the law. (What Mueller didn’t say was this came at a time when all the major telecommunications companies were still facing lawsuits over their cooperation with the government on surveillance programs.) As a result, Mueller said, the FBI had switched over to demanding the same data under Section 215. “This change accounts for a significant increase in the volume of business records requests,” Mueller wrote.

What was not explained at the time, Chesney notes, is that the FBI was using the Section 215 requests to obtain a broad array of records. For example, a top-secret FISC order disclosed last week by the Guardian showed that the FBI had  used a single Section 215 request to direct Verizon  to turn over “all call detail records or telephony metadata’’ of its customers for a three month period, literally millions of records.

Saying they wanted to put an end to “secret law,  eight U.S. senators — led by Sens. Jeff Merkley, D-Ore., and Mike Lee, R-Utah — on Tuesday introduced a bill to require the Justice Department to declassify national security court decisions that have permitted the use of the “business records” provision for such purposes.

That followed a court filing Monday by the American Civil Liberties Union and allied groups asking the surveillance court to release its classified legal opinions question that have allowed the expanded use of Section 215.

The motion, filed “pursuant to the First Amendment,” the ACLU states, and under rules that, in some circumstances, permit petitions to the FISC, also cites statements by two Democratic U.S. senators, Ron Wyden of Oregon and Mark Udall of Colorado, and Obama to justify public disclosure.

Related story

On the run and out of a job: Consulting company fires professed NSA leaker 

“When the American people find out how their government has secretly interpreted the Patriot Act, they will be stunned and they will be angry,” it quoted Wyden as saying in 2011.

It also cited Obama words after last week’s disclosure of the Verizon order: “I welcome this debate.”

The motion also asks the court to consider the constitutionality of the “gag order” written into Section 215.

“There should be no room for secret law,” said Jameel Jaffer, the ACLU’s deputy legal director, adding that disclosure of the FISC rulings is essential if the debate Obama called for is to take place. “The public has a right to know what limits apply to the government’s surveillance authority, and what safeguards are in place to protect individual privacy.” 

More from Open Channel:

• Susan Komen CEO's salary draws fire as donations drop, races are canceled
• Officials: NSA mistakenly intercepted emails, phone calls of Americans
• Chinese hacked Obama, McCain campaigns, took documents, officials say

Follow Open Channel from NBCNews.com on Twitter and Facebook 

The Justice Department pledged Friday to to review its policies relating to the seizure of information from journalists after acknowledging that a controversial search warrant for a  Fox News reporter’s private emails  was approved “at the highest levels” of the Justice Department, including “discussions” with Attorney General  Eric Holder.

The statement, confirming an NBC News account of Holder’s role, defended the secret warrant to obtain reporter James Rosen’s emails as a legitimate step to obtain evidence as part of an investigation of Stephen Kim. A former intelligence analyst, Kim has since been indicted under the Espionage Act for leaking classified information to Rosen about North Korea. He has denied the charges.

In a 2010 affidavit in support of the search warrant, an FBI agent named Rosen as a possible “co-conspirator” in the case because he “asked, solicited and encouraged” Kim to give him information.

---

“After extensive deliberations, and after following all applicable laws, regulations and policies, the Department sought an appropriately tailored search warrant under the Privacy Protection Act,” said a department official, referring to a federal law that governs under what circumstances information can be subpoenaed from the news media. “And a federal magistrate judge made an independent finding that probable cause existed to approve the search warrant.”

Nevertheless, said the official, who spoke on condition of anonymity, Holder “understands the concerns that have been raised by the media and has initiated a re-evaluation of existing department policies and procedures.” The official said the department must strike “the appropriate balance” between preventing leaks of classified information and “First Amendment rights,”adding that passage of a new media shield law “and appropriate updates to the department”s internal guidelines” will help achieve that.  

 The statement comes  amid a firestorm of criticism from news media groups over the Rosen search warrant and a secret subpoena for the phone records of AP reporters. It also comes one day after President Obama addressed the issue in a major speech on counter-terrorism policy, saying "I am troubled by the possibility that leak investigations may chill the investigative journalism that holds government accountable," Obama said. "Journalists should not be at legal risk for doing their jobs."

Related stories

Obama reframes counterterrorism policy with new rules on drones

Heckler repeatedly interrupts Obama speech

Holder previously said he recused himself from the AP subpoena because he had been questioned as a witness in the underlying investigation into a leak about a foiled bomb plot in Yemen. His role in personally approving the Rosen search warrant had not been previously reported until NBC News reported it on Thursday.

A law enforcement official. who also spoke on condition of anonymity, said Holder's approval of the Rosen search, in the spring of 2010, came after senior Justice officials concluded there was "probable cause" that Rosen's communications with Kim met the legal burden for such searches. 

In an affidavit in support of a search warrant to Google for Rosen's emails, an FBI agent wrote that the Fox News journalist -- identified only as "the Reporter" -- had "asked, solicited and encouraged Mr. Kim to disclose sensitive United States internal documents and intelligence information." 

"The Reporter did so by employing flattery and playing to Mr. Kim's vanity and ego,” it continued. “Much like an intelligence officer would run a clandestine intelligence source, the Reporter instructed Mr. Kim on a covert communications plan that involved" emails from his gmail account.

The affidavit states that FBI agents had tracked Rosen’s entrances and exits of the State Department in order to show that they had coincided with Kim’s movements. Based on that and other findings, the affidavit by FBI Agent Reginald B. Reyes, stated, “There is probable cause to believe that the Reporter has committed a violation” of the Espionage Act “at the very least, either as an aider, abettor and/or co-conspirator of Mr. Kim.”

It also said that Google was specifically instructed not to notify “the subscriber” -- Rosen -- that his emails were being seized.

In new documents disclosed Thursday, the Justice Department sought and obtained approval to keep the search warrant, which was approved by a federal magistrate, under seal. It was unsealed in November 2011, but never made a part of the docket of Kim’s case and went unnoticed until this week.

Justice officials have since said they do not intend to criminally charge Rosen, but media groups have condemned the issuance of the search warrant itself.

"The Justice Department's decision to treat routine newsgathering efforts as evidence of criminality is extremely troubling and corrodes time-honored understandings between the public and the government about the role of the free press," said Bruce Brown, executive director of the Reporters Committee for Freedom of the Press.

In his speech Thursday, Obama reiterated his determination to pursue leak investigations. "We must enforce consequences for those who break the law and breach their commitment to protect classified information," he said.

  But, he said, "Our focus must be on those who break the law," not journalists. He said he was calling on Congress to pass a media shield law and had raised the issue with Holder, "who shares my concern."

As part of the Justice Department review of guidelines, the president said, Holder will convene a group of media organizations to hear their views and “report back to me by July 12th."

NBC News Chief Justice Correspondent Pete Williams contributed to this report.

More from Open Channel:

• Too much terrorism data? Connecting the dots may be getting harder
• Admission of al-Awlaki killing could affect family's lawsuit against US government
• Texas blast shows failure of emergency planning law, analysis shows

Follow Open Channel from NBCNews.com on Twitter and Facebook 

 

This story was originally published on Thu May 23, 2013 5:16 PM EDT

WASHINGTON -- The Justice Department used a secret subpoena to obtain two months of phone records for Associated Press reporters and editors without notifying the news organization, a senior department official tells NBC News, saying the step was necessary to avoid "a substantial threat to the integrity" of an ongoing leak investigation.

The seizure of the phone records, disclosed earlier Monday by AP President and CEO Gary Pruitt, is the latest move in a series of high profile and controversial investigations of leaks of classified information by the Justice Department. In a letter of protest to Attorney General Eric Holder, Pruitt said obtaining more than two months of AP phone records on 20 separate telephone lines without prior notice was a "massive and unprecedented intrusion" into news-gathering operations. 

It also drew a swift rebuke Monday from members of Congress and freedom of the press watchdogs, one of whom called the move "Nixonian."

Ronald C. Machen, Jr., the U.S. attorney in Washington, D.C., revealed in a letter to the AP on Friday that federal prosecutors obtained the records. The letter did not give a reason for obtaining the records, but Machen is conducting an investigation into the leak of classified information about a foiled terror plot in Yemen last year. An AP story last spring reported details of a CIA operation in Yemen that stopped an al Qaeda plot to detonate a bomb on an airplane bound for the United States.

---

In his letter to Holder, Pruitt said the seized phone records were from early 2012 and included phone lines for AP bureaus in New York, Washington DC, Hartford, Connecticut and the AP line at the House of Representatives. He said the records seized also included those from the home phones and cell phones of individual journalists.

"We regard this action by the Department of Justice as a serious interference with AP's constitutional rights to gather and report the news," Pruitt said.

Holder last June appointed Machen to conduct the investigation of the Yemen terror plot leak and Rod Rosenstein, the U.S. attorney in Maryland, to oversee a separate probe into the leak of U.S. government efforts to use the Stuxnet computer virus to thwart the Iranian nuclear program. In later Senate testimony, Holder said that he and FBI director Robert Mueller had both been interviewed by FBI agents as part of the investigations because they had prior knowledge of the information that was leaked. (Under Justice regulations, any subpoena for news media phone records requires the "express authorization" of the attorney general. But a Justice Department spokeswoman did not respond Monday night when asked whether the attorney general had recused himself in the investigation.)

As another sign of the sensitivity of the case, CIA Director John Brennan disclosed earlier this year that he also had been questioned by FBI agents as part of the Yemen probe, but said he was later notified that he was not a subject of the investigation.

Bill Miller, spokesman for Machen, said in an email that the subpoena for the records was done by the book.

"Consistent with DOJ regulations, the department provided notification to the Associated Press of the receipt of toll records in a letter dated May 10, 2013," He noted that Justice regulations "do not require notification to the media prior to the issuance of legal process to obtain toll records."

In a separate email, Miller wrote: "We take seriously our obligations to follow all applicable laws, federal regulations, and Department of Justice policies when issuing subpoenas for phone records of media organizations. Those regulations require us to make every reasonable effort to obtain information through alternative means before even considering a subpoena for the phone records of a member of the media. We must notify the media organization in advance unless doing so would pose a substantial threat to the integrity of the investigation.

"Because we value the freedom of the press, we are always careful and deliberative in seeking to strike the right balance between the public interest in the free flow of information and the public interest in the fair and effective administration of our criminal laws.”

The regulations cited by Miller state that subpoenas for the news media in criminal cases should be done only when there are “reasonable grounds to believe … that a crime has occurred” and that the records sought are “essential to a successful investigation.” They also state that subpoenas should, wherever possible, “be directed at material information regarding a limited subject matter and “should cover a reasonably limited period of time and … avoid requiring production of a large volume of unpublished material.”

Since President Barack Obama took office, the Justice Department has aggressively pursued leak investigation and brought more criminal prosecutions – six in five years – than any previous administration. Those cases, which also have been sharply criticized by press groups, have also targeted reporters’ phone records: James Risen, a national security reporter for the New York Times, had his phone, credit card and bank records subpoenaed as part of a Justice Department prosecution of a former CIA officer accused of leaking classified information on Iran’s nuclear program to him.

But critics say the extensive nature of the subpoena for the AP phone records goes far beyond what was seen in earlier cases.

Rep. Darrell Issa, R-Calif., and chairman of the House Oversight and Government Reform Committee, vowed to investigate.

"This is obviously disturbing," he said. Coming in the wake of other disclosures about the administration’s response to the attack on the U.S. consulate in Benghazi, Libya, and the IRS’s targeting of conservative nonprofit groups, he said it showed "top Obama administration officials increasingly see themselves as above the law and emboldened by the belief that they don't have to answer to anyone."

Sen. Patrick Leahy, D-Vt., and chairman of the Senate Judiciary Committee, said he wanted to know more about the justification for the secret subpoena.

"The burden is always on the government when they go after private information -- especially information regarding the press or its confidential sources,” he said. “… I am concerned that the government may not have met that burden. I am very troubled by these allegations and want to hear the government's explanation."

Anti-secrecy watchdogs also criticized the move.

"I've never heard of a dragnet collection effort against a media organization like this," said Stephen Aftergood, who tracks secrecy issues for the Federation of American Scientists. "This was not a targeted monitoring of an individual reporter. It's a sweeping collection of an entire bureau's communications."

"The Justice Department’s seizure of the Associated Press’ phone records is Nixonian," said Danielle Brian, executive director of the Project on Government Oversight, a group that advocates on behalf of whistleblowers. "The American public deserves a full accounting of why and how this could happen."

NBC News' Capitol Hill Correspondent Kelly O'Donnell contributed to this report.

More from Open Channel:

• IRS watchdog: Senior official knew in 2011 that Tea Party groups were targeted
• Unaware of Tsarnaev warnings, Boston counterterror unit tracked protesters
• Long before he was charged, Ariel Castro was accuser in sexual assault case

Follow Open Channel from NBCNews.com on Twitter and Facebook 

 

This story was originally published on Mon May 13, 2013 5:31 PM EDT

Who says there's no free lunch?

You may have read over the past week about the release of 1940 Census records on a new U.S. government website, a site that buckled under the huge demand from people looking up details on the lives of their friends and relatives from the Great Depression.

You may not have realized that the site was built for U.S. taxpayers for the price of — not one dime. A company from Silicon Valley built the site, and is operating it, for free. Genealogy buffs have been using the site for a week now to check millions of records. (See our earlier story for tips on searching the 1940 Census, and examples of people who have found relatives.)

Of course, the company, Inflection LLC of Redwood City, Calif., did get something in return for its effort: a free copy of those 3.8 million images of records from the 1940 Census. While other companies paid $200,000 for a set of the public records, Inflection can use those records in its for-profit business, a genealogy site called Archives.com.

It's a barter system for federal records: the public gets a free official U.S. website, and the company gets free data. It's been done before, as when the U.S. Patent and Trademark Office gave data to Google, which since 2006 has hosted the site for free as Google Patents.

---

Do you approve of the approach that the National Archives took, giving the data away in exchange for the free website? And what stories have you found in the 1940 Census? Add your story in the comments below or on our Open Channel page on Facebook.

Inflection also was hoping to get a boost to its reputation for building websites that could withstand a storm of traffic.

Performance standards in the contract
Both the company and the National Archives and Records Administration (NARA) had anticipated that the site would draw a crowd, as 72-year privacy restrictions expired and the records became available. What happened next lends credence to the boast that genealogy is the country's favorite hobby.

The contract says, "Drawing from NARA's experience in releasing the 1930 Census, and the experience of the National Archives of the United Kingdom when they released their 1901 and 1911 Censuses, NARA anticipates immense interest in the 1940 Census and a tremendous increase in traffic to its www.archives.gov web site." (Here's the contract in a PDF file.)

But how much of a crowd?

Here are the performance standards in the contract:

• "When browsing from one image to another, each image should be presented to the user in 3 seconds or less."
• "When moving from the standard rendered image to each zoom level (e.g. zoom 1x, 2x, 3x), the reformatted image should be rendered in 2 seconds or less."
• "Support up to 10 million hits per day while providing response times of less than three seconds for keyword searches of the descriptive metadata."
• "Support up to 25,000 concurrent users."

There was one more element in the contract, a somewhat vague requirement that Inflection increase service if demand was greater than anticipated.

• "Scale on demand in the event that 10 million hits and/or 25,000 concurrent users are exceeded to ensure that the performance requirements ... are still achieved."

The crowd certainly exceeded those levels, as the most old-fashioned sounding search term possible, "1940 Census," became a top "trending topic" on Google and Twitter.

Most people seemed to get little or nothing from the site on the first day, including Census leaders, who were prepared to show off how easy it was to look up their grandparents. When the site stuck on "loading image," as it did for many other users, the officials resorted to showing a PowerPoint presentation with the results from an earlier search.

A 'tsunami'
As Inflection's general manager, Joe Godfrey, told us last week, "We were expecting a flood, but we got a tsunami."

• On Day One, Monday, an estimated 100 million hits, or requests, with 22.5 million hits in just the first three hours. Though Inflection scrambled to improve service, the site was unusable for many users on the first day. The company added more servers through Amazon Simple Storage Service, its cloud data service provider, and also restricted some features on the site (such as zooming of images), until finally it was able to get on top of the traffic.
• On Day Two, Tuesday, the numbers haven't been totaled, but it's believed to be higher than on Day One, with an estimated 40.1 million hits in the three-hour peak.
• By Friday, the site was stable with about 60 million hits per day, and had served up more than 80 million images, or about 61 terabytes of data, the National Archives said. (That's more than the data contained in the first 20 years of astronomical observations by the Hubble Space Telescope.) The service quality was better than called for in the contract, with a load time of about 1.8 seconds per page, according to the Archives.

In other words, this might have been a good project for a "soft launch."

The contract called for extensive load testing before the release. We asked the National Archives for copies of those test results, but its spokeswoman said it wouldn't be able to provide them. But it said the site was tested to handle more than 70,000 simultaneous users — more than the contract called for, and fewer than the level that resulted.

A 'no-cost contract'
No-cost contracts are allowed under Federal Acquisition Regulation competitive procedures. This contract has a one-year base period and options to extend for four more one-year periods.

"NARA provided a copy of the data to Inflection at no cost, copies that were sold to others for $200K," said spokeswoman Laura Diachenko of the National Archives. "Why Inflection agreed to this is a better question for them, but we are very happy to have them as a partner. They have experience with Census data, and managing access to large data sets, the capabilities we were seeking for this project."

She added, "Even though this is called a no-cost contract, the Government did incur costs — in this case, aside from our resources, we also provided a copy of the 1940 Census to Inflection, at no cost.  In this particular case, we provided them data that they wanted in exchange for hosting access to this data.  Their interest was in getting the data (for their archives.com business), and for business development (attracting users to their site and eventually converting them to a subscriber."

Inflection's Godfrey said, "The primary value for us was in building our brand/notoriety, leveraging and expanding our technical expertise/infrastructure and helping to getting this extremely valuable record collection into the hands of as many people as possible.  Also, our engineering team (like all great engineers) are motivated by tackling challenging technical problems, and so the team was very excited to work on this."

Competition
All or most of the 1940 Census is now available free from several other companies, which had to pay for the public records. As a sort of loss leader, other genealogy sites, even the commercial ones, are making the 1940 Census records available for free, to subscribers and non-subscribers alike.

Here's how the race worked: All the commercial sites that chose to buy the data for $200,000 were handed a rack of hard drives full of 20 terabytes of images, taken from 4,745 rolls of microfilm, at 12:01 a.m. on April 2, or 72 years and a day after the Census Day in 1940.

By Thursday, a relatively new genealogy site called myHeritage, was the first to have all the images online. Also making images available for free are Ancestry.com, a commercial site, and FamilySearch.org, owned by the Church of Jesus Christ of Latter-day Saints.

Thousands of volunteers are working on the next step: indexing the records by name, just as previous Census releases have been indexed by volunteers. Until those indexes are finished, searching is done only by address or neighborhood.

Your view
Do you approve of the approach that the National Archives took, giving the data away in exchange for the free website? And what stories have you found in the 1940 Census? Add your story in the comments below or on our Open Channel page on Facebook. See our earlier story for tips on searching the 1940 Census.

Update, 5:40 p.m. ET: The firm at the center of today's census records meltdown says, "We were expecting a flood, but we got a tsunami."

"We had estimates of how much traffic was going to hit the site, and we did performance testing at several levels above that, but we were surprised by the traffic," Joe Godfrey, senior director of product and general manager for Inflection, a Silicon Valley database company."

Inflection was hired by the National Archives and Records Administration, which provided the 1940 census records. Inflection buiilt the search engine to serve up the records, and relied on Amazon Simple Storage Service (Amazon S3) as the cloud service provider. Inflection has been adding more of a pipeline to Amazon all day, adding the ability for more simultaneous connections, but so far searches for census records are running slowly or not running at all for many users.

---

The company is trying to serve up 3.8 million images of census documents, each with multiple views at different zoom levels, with each file being 10 megabytes or larger.

Godfrey said the situation has improved, and engineers are hoping by the end of today to have the situation squared away.

Earlier:

Embarrassed by a computer system that crumbled under public demand, the National Archives and Records Administration said Monday that it's working to add more servers for the release of 1940 Census records. For more users the wait to see records on family members from the Great Depression era will go on for a while longer.

The Archives had hired Inflection, a Silicon Valley database company, to run the computers, but frustrated users lit up Facebook and Twitter with complaints about images that were said to be "loading" but never arrived.

"Our testing indicated NARA and Inflection could handle the load, but 1.9 mil visitors caused issues we're working to resolve," the Archives said via Twitter. Later it added, "We'll let you know as soon as we have another update - thank you for your patience, we know it's incredibly frustrating."

Even agency officials, during the webcast to kick off the day, couldn't get images to load when they tried to look up their own relatives.

In Springfield, Ohio, Facebook user Val Lough commented on our page: "It's very sweet of them to put all of these records on line. It would be even nicer of them to make the records VISIBLE. None of them will download, I have a browser window opening that's 'loading' the documents and has been for about 20 minutes. You might want to find out what their issues are. It would be faster to mail a public records request to the National Archives." Many others are tweeting about delays.

The National Archives says it is putting more servers online to handle the crush.  At one point, the Archives said, its computers were receiving 100,000 hits per second.

Hey, you've waited 72 years to see these records, so what's another day or two.

Earlier:

A time capsule from 1940 was opened on Monday at 9 a.m. ET, and we invite readers to share what they find. If you use the new records to find information about the loved or lost in your family, please post a note in the comments below or on our Open Channel page on Facebook.

U.S. Census records for individuals from April 1, 1940, protected until now by a 72-year privacy law, are now public for the first time, revealing details about millions of Americans from that day, as the country lingered in a Great Depression, still a year away from entry into war in Europe and the Pacific.

"I'm so excited!" Gary Robert Del Carlo of Martinsburg, W.Va., posted on Facebook. "Maybe for the first time ever, I'll be able to find out something about my father. All I have is my birth certificate with his name, date of birth, state born in, and that he was in the Army stationed in Washington State. His military records burned up in St. Louis in a fire in 1973. They would have told me a lot. Wrote for his birth certificate, and there was no records of his birth. I have done nothing but hit brick walls every which way I turn. I'm praying I find something useful tomorrow, anything."

NPR describes the release as the "Super Bowl for Genealogists." Librarians around the country are ready to provide assistance. At the Family History Library in Salt Lake City, the staff will be serving cake and providing help.

 

When the 120,000 census takers counted 132,164,569 people living in the country on that day, the information collected included the address, whether the house was owned or rented, value of the home or monthly rent, is it considered a farm, names of adults and children, familiy relationships, sex, race, age, place of birth, citizenship, residence five years earlier, education. And for a small subset of people, about 5 percent, they were asked about place of birth of mother and father, language spoken in the home as a child, veteran status, wars served in, Social Security status, occupation, employment status, occupation, number of weeks worked in 1939, income and, for women, whether they had been married more than once, age at first marriage, and number of children ever born.

There is a catch. As the records go online, they can't be searched by name. For a city it's helpful to know an exact address, but often you can work with a neighborhood (near the corner of Canal and Varrick streets in New York City). Your public library may have old city directories or telephone directories from that period, allowing you to look up people by name to find an address. For a rural area, you need to know at least the county and the name of the town or township.

Genealogists, librarians and volunteers will begin the work of indexing the records, which eventually will allow searches by name. Two sites, the commercial Ancestry.com and the Mormon Church's FamilySearch.org, have announced plans to provide indexes to their customers as quickly as possible, with some images going online on Monday. FamilySearch and Ancestry.com started putting images from the Census files online early on Monday, but for now without a name index. 

For now, you must know at least an approximate address to get started. You use that address to find an "enumeration district," which in a big city might be only a few blocks, and would be a larger area in a small town.

Another approach, for those interested in a specific place, is to look at all the records for your block or street. If your area was settled in 1940, who lived there then, and what were their lives like?

Your goal: With that district number, you can look on the Census website at the online copy of the form filled out by the census taker in 1940. In 70 years, it has gone from paper to microfilm to computer.

Here are resources to help you with the search (links open in a new window), though as with most things in life, the key is: Ask a librarian.

• Most important page No. 1: Step-by-step help from private researchers with free aids to help you find the enumeration district map for a particular address
• Most important page No. 2: A Census explainer on starting your search.
• The home of the 1940 Census
• A Census page with general information on the 1940 release
• A copy of the 1940 Census form (PDF file) that you can fill in when you find information
• Census aids to finding information
• Ancestry.com, a commercial service for genealogists
• FamilySearch.org from the Church of Jesus Christ of Latter-day Saints
• Tell us what you find: Post your story on Open Channel's Facebook page

 

Ohio journalists are warning that the state is headed toward a "total eclipse" of the public's right to know what government officials are doing after Gov. John Kasich last week signed a measure that significantly limits civil penalties for destroying public documents.

The new law limits fines against a public agency to $10,000 per case and restricts attorney's fees to the same amount. There previously was no limit on damages.

The Columbus Dispatch reported that public agencies frequently reject requests for public records. It said the Kasich administration, for example, rejected its request for three e-mails related to the Ohio State University football scandal because it was "overly broad."

"Similar occurrences are increasing at all levels of Ohio government," the newspaper reported.

---

State Sen. Bill Seitz, a Republican from Cincinnati who co-sponsored the bill, said the measure was intended to stem abuse of records requests by parties who might request documents they knew didn't exist in hope of a big payday in court. He noted that criminal penalties for such destruction weren't affected.

The Blade of Toledo said Sunday there was "no evidence" of such abuse, reporting that "(s)ince the state's public-records law took effect in 1985, there have been only a few big judgments for official violations."

"It seems at least as likely that some government agencies would regard the meager financial penalties in the bill as an acceptable cost of keeping public business secret, and that some potential plaintiffs would be dissuaded from bringing valid lawsuits because they couldn't afford to pursue them," the newspaper said in an editorial.

The Dispatch agreed in an editorial Sunday that the measure is a big blow for open government:

"In the spirit of our nation celebrating its independence this July 4 weekend, maybe more public officials who believe in transparency could sponsor changes that would make more records open to taxpayers who pay the bills."

What do you think? Do public officials need more protection from records requests? Let us know in the comments.