The French called it the cabinet noir – the “black chamber”. Set behind the General Post Office in the Rue Coq Heron, its job was to take letters of interest, open them, read their contents and then seal them again without detection by their intended readers or disruption to the postal service. “It is in the secret holes and cellars that the unsealers of letters, like the coiners of false money, carry on their dark labour,” wrote a visitor from London in 1828, describing a room full of steam, staffed by 20 or 30 agents. Created under the monarchy, the cabinet noir survived the transition to the revolutionaries, and then Napoleon. Each railed against it in turn and then, on taking power, suddenly found it indispensable.

The largest black chamber ever constructed is being built now, in Utah. Called the Intelli­gence Com­munity Compre­hensive National Cyber­security Initiative Data Center, its storage capacity is estimated to exceed a yottabyte, the largest unit of data that computer scientists have. That’s around half a sextillion (500,­000,­000,­000,­000,­000,­000) pages of text, more than enough to store everything that’s ever been written, as well as every email and phone call to be made in the coming century. Australia is building its own sister system at HMAS Harman just outside Canberra, partly to accommodate the data that will flow to us from Utah. The question now is: what will these black chambers be filled with? And how?

The answer lies with the US communications intelligence organisation, the National Security Agency, which runs the Utah Data Center. Since 2001, the NSA’s approach to this question has changed. In the past weeks, debate has centred around one individual, Edward Snowden, a former NSA contractor, and one program, called PRISM. But Snowden is only one of many whistleblowers who have drawn attention to NSA activities, and PRISM is only one of dozens of database and surveillance programs. Together, they intercept, store and search vast quantities of information, raking the internet and phone systems for data and metadata. Sometimes lists of their strangely evocative names appear anonymously online: COASTLINE, CROSSBONES, MESSIAH, NIGHTSURF. Another is called PANOPTICON.

ECHELON was the most famous before PRISM, although ECHELON is no longer its name. It is a program of shared signals intelligence between Australia, the US, the UK, New Zealand and Canada (the so-called “Five Eyes”) that involves spying on material gleaned primarily from satellite communications. The Australian Defence satellite station at Geraldton in Western Australia is used for this purpose; so is Pine Gap. Every now and then ECHELON would crop up on a show like Channel Nine’s Sunday, where a reporter wearing a “this-is-all-a-bit-tinfoil-hat” expression would explain its functioning. In 1996, two New Zealand journalists with a ladder and a camera filmed the interior of an ECHELON station at Waihopai. It had no people inside it, just banks of dumb computers with blinking lights, and a screen repeating the word ENVOY.

It was in response to one of these reports that Australia became the first of the Five Eyes to acknowledge the existence of ECHELON. A Defence Signals Directorate spook walked us through the boring safeguards – there were strict controls to ensure it wasn’t used by the Directorate and ASIO to circumvent Australian domestic surveillance laws, and abuses were probably fairly rare. Australians worried meekly that the US and UK might have access to our trade and defence contracts, and then got back in the junior partner box. The oversight system seemed to work, and still governs Australia’s access to PRISM and its ilk. So what’s changed?

Oversight at the American end of the arrangement has failed. The September 11 attacks acted as a catalyst to alter not just the scale of US intelligence gathering, but its purpose and rationale. Over the last decade, battles within the NSA over privacy, oversight and legality have prompted a slew of resignations and whistleblowing. Analyst William Binney became so exasperated with aggressive domestic spying that he took the information to a Supreme Court judge, going to the press only when he was ignored. Russ Tice and Thomas Drake showed how the NSA had undertaken an extensive program of wiretaps without warrants, targeting everyone from Amnesty International to foreign journalists having phone sex with their spouses. Telecommunications whistleblowers revealed that NSA “black boxes” had been installed in some areas to directly monitor fibre-optic cables.

The revelations changed little – PRISM is the child of all these programs. Those responsible for the programs were promoted, the whistleblowers had their houses raided and legal challenges were dismissed on the grounds that the plaintiffs were unable to prove they had been spied on – partly because the necessary evidence was deemed too sensitive to be made public. This de facto legal immunity became official when the telecommunication companies involved were given retroactive immunity for any illegalities that had occurred. The most aggressive wiretaps were eventually overseen by a secret court operating under the Foreign Intelligence Surveillance Act, which in the past 33 years has rejected only 11 of the 34,000 requests put to it.

Legislative oversight has also been ineffectual. In 2006, the whistleblower Russ Tice, already fired by the NSA, offered to testify to the US Congress about surveillance program abuse, and was warned that legislators did not have the security clearance even to hear his testimony. Senator Ron Wyden has pressed the NSA again and again to provide even an approximate figure on how many US citizens are monitored under the provisions of the 2001 USA PATRIOT Act (which stands for Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism). He was eventually told that revealing the information to him would violate Americans’ privacy. Even the architect of the Act, Jim Sensenbrenner, and former counter-terrorism tsar Richard Clarke have publicly criticised the scale of surveillance, arguing it has pushed anti-terrorism laws beyond their intended use.

This overreach is bad for Americans, but it puts the citizens of Australia, the UK, Canada and New Zealand in an even more vulnerable position. US agencies are almost unfettered when it comes to monitoring foreigners. The majority of internet traffic is routed via US-based servers and, in the case of the Five Eyes, a conduit exists to provide this information to our governments for domestic use. General Michael Hayden, a former director of the NSA and CIA, has maintained that governments cannot circumvent their own surveillance laws, and that any intelligence-swapping “back door” is firmly closed. “I realise you’ve got these urban legends out there, but it does not happen,” he assured Canadians via theToronto Star.

Yet it does. Examples of precisely this abuse are occurring across the Tasman. Kim Dotcom is a wealthy internet entrepreneur and loudmouth who currently resides in New Zealand. He is the founder of Megaupload, an online service notorious for hosting a vast amount of copyright-infringing material. In 2012, indictments were prepared against Dotcom in the US, and he was arrested in a raid carried out by 76 armed New Zealand police. In the subsequent legal fallout, it was discovered that New Zealand’s foreign signals intelligence agency had illegally spied on Dotcom. More worryingly, it had also been unlawfully provided with intercepts of Dotcom’s emails that originated in an FBI surveillance program called TESSA. In other words, the intelligence and law enforcement agencies of New Zealand broke their own laws to collude with a foreign power to spy on, arrest and attempt to extradite one of their own residents, using information from a program not dissimilar to PRISM. And not over terrorism, but in a copyright case. This is not just “sharing” intelligence – it is co-operating with the investigative arm of an alternative legal system that seeks to extend its jurisdiction of interception, detention and execution almost everywhere, contaminating local law in the process.

Could this happen here? Mark Dreyfus, the Attorney-General, assures us that ministerial authority must be sought before intelligence can be collected on Australians, but it’s unclear if receiving data from an overseas agency counts as “collection”. In the UK, some lawyers argue that intelligence from a foreign source isn’t subject to domestic oversight laws. Legal problems are now being addressed simply by changing the law. The Cameron government is trying to introduce secret courts, chiefly to avoid embarrassing US military and intelligence agencies that have conducted potentially illegal activities involving UK citizens.

Last year there was an extended, often heated debate about then Attorney-General Nicola Roxon’s proposed data-retention reforms. It ended with them effectively abandoned. But in reality, the data retention regime already existed – in the US, where its scale is now being determined by the NSA’s boundless curiosity, and its use weighed up in secret by our intelligence agencies. We don’t know what has or will be decided, and our citizenry is not party to the discussion. But as billions of pieces of information enter the black chamber daily, so do we.