Archive for the ‘Transparency’ Category

What’s Up With Data Padding? (

Wednesday, March 29th, 2017

I forgot to mention in Copyright Troll Hunting – 92,398 Possibles -> 146 Possibles that while using LibreOffice, I deleted a large number of either N/A only or columns not relevant for

Except as otherwise noted, after removal of “no last name,” these fields had N/A for all records except as noted:

  1. L – Implementation Date
  2. M – Effective Date
  3. N – Related RINs
  4. O – Document SubType (Comment(s))
  5. P – Subject
  6. Q – Abstract
  7. R – Status – (Posted, except for 2)
  8. S – Source Citation
  9. T – OMB Approval Number
  10. U – FR Citation
  11. V – Federal Register Number (8 exceptions)
  12. W – Start End Page (8 exceptions)
  13. X – Special Instructions
  14. Y – Legacy ID
  15. Z – Post Mark Date
  16. AA – File Type (1 docx)
  17. AB – Number of Pages
  18. AC – Paper Width
  19. AD – Paper Length
  20. AE – Exhibit Type
  21. AF – Exhibit Location
  22. AG – Document Field_1
  23. AH – Document Field_2, not the Copyright Office, is responsible for the collection and management of comments, including the bulked up export of comments.

From the state of the records, one suspects the “bulking up” is NOT an artifact of the export but represents the storage of each record.

One way to test that theory would be a query on the noise fields via the API for

The documentation for the API is out-dated, the Field References documentation lacks the Document Detail (field AI), which contains the URL to access the comment.

The closest thing I could find was:

fileFormats Formats of the document, included as URLs to download from the API

How easy/hard it will be to download attachments isn’t clear.

BTW, the comment pages themselves are seriously puffed up. Take

Saved to disk: 148.6 KB.

Content of the comment: 2.5 KB.

The content of the comment is 1.6 % of the delivered webpage.

It must have taken serious effort to achieve a 98.4% noise to 1.6% signal ratio.

How transparent is data when you have to mine for the 1.6% that is actual content?

Transparency can have a prophylactic effect

Sunday, March 26th, 2017

Farai Chideya set out to explore:

…who reported the 2016 election, and whether political teams’ race and gender diversity had any impact on newsrooms.

That’s an important question and Chideya certainly has the qualifications and support ( fellow at Harvard’s Shorenstein Center on Media, Politics and Public Policy) to pursue it.

One problem. For reasons best known to themselves, numerous media organizations refuse to provide diversity date. (full stop)

But the most important data point for this project—numbers from newsrooms on their 2016 political team staffing—has been the hardest to collect because very few managers or business-side staff are willing to disclose their data. One company admitted off the record that they were not responding to diversity requests, period. The Wall Street Journal provided the statement that it “declined to provide specific personnel information.” An organization sent numbers for its corporate parent company, whose size is approximately a thousand times the size of the entire news team, let alone the political team. Another news manager promised verbally to cooperate with the inquiry, but upon repeated follow up completely ghosted.

Concealment wasn’t the uniform response as Chideya makes clear but useful responses were so few and far. Enough so to provoke her post.

She captures my sentiments writing:

If we journalists can’t turn as unsparing a gaze on ourselves as we do on others, it speaks poorly for us and the credibility of our profession. If the press lauds itself for demanding transparency from government but cannot achieve transparency in its newsrooms, that is cowardice. If we say we can cover all of America with representatives of only a few types of communities, we may win battles but lose the war to keep news relevant to a broad segment of Americans. This is as strong a business argument as a moral argument.

If you need additional motivation, be aware that Chideya is proceeding in the face of non-cooperation and when her study is published, there will be a list of who has been naughty and nice.

Here’s how to self-report:

Whether or not you are a news organization I’ve already contacted, please email me at

For the purposes of the reporting, I’m looking for a race/gender count of 2016-cycle political staffers—full-time or at least 25-hour-per-week contract workers (but not freelancers paid by the story). People come and go during the election season, but these should be people who spent at least six months covering the election between September 2015 and November 2016.

If you want to add to the data you disclose, you can include separate counts for freelancers; or for staff who worked on politics less than six months of the cycle, but those should be broken out separately.

Want bonus points? Produce an org chart showing how your staff diversity played out across the ranks of reporters and editors. Feel free to annotate for self-reported class background or other metrics if you want, too. But race and gender are the minimum.

We’d like on-the-record numbers and interviews from people who we can use as sources in the report: managers, corporate communications staff, anyone authorized to speak on behalf of the newsroom. Please indicate if you are speaking on the record and in what role.

Because we are not getting this information, in many cases, we also welcome interviews and information on background. That is, if you are a staffer and can provide information, please do, and tell us who you are and that you don’t want to be quoted or cited. We’ll take what you provide to us into account as we do our research, but obviously it can’t be the final word. You could also offer quotes about the topic on the record, and your assessment of staff diversity on background.

As we conclude the report, we will release information on who has provided information, and who it was requested from who did not.

Self-reporting beats being on the naughty list and/or your diversity information extracted by a ham-handed hacker who damages your systems as well.

Who knew? Transparency can have a prophylactic effect.

See Chideya’s full post at: One question that turns courageous journalists into cowards

MuckRock Needs Volunteers (9 states in particular)

Monday, November 21st, 2016

MuckRock needs your help to keep filing in all 50 states by Beryl Lipton.

From the post:

Election time excitement got you feeling a little more patriotic than usual? Looking for a way to help but not sure you have the time? Well, MuckRock is looking for a few good people to do a big service requiring little effort: serve as our resident proxies.

A few states have put up barriers at their borders, limiting required disclosure and response to requests to only residents. One more thing added to the regular rigamarole of requesting public records, it’s huge block to comparative studies and useful, outside accountability.

This is where you come in.


We’re looking for volunteers in the ten states that can whip out their residency requirements whenever they get the chance:

  • Alabama
  • Arkansas
  • Georgia
  • Missouri
  • Montana.
  • New Hampshire
  • New Jersey
  • Tennessee
  • Virginia

As a MuckRock proxy requester, you’ll serve as the in-state request representative, allowing requests to be submitted in your name and enabling others to continue to demand accountability. In exchange, you’ll get your own Professional MuckRock account – 20 requests a month and all that comes with them – and the gratitude of the transparency community.

Interested in helping the cause? Let us know at, or via the from below.

Despite my view that government disclosures are previously undisclosed government lies, I have volunteered for this project.

Depending on where you reside, you should too and/or contribute to support MuckRock.

“connecting the dots” requires dots (Support Michael Best)

Friday, November 11th, 2016

Michael Best is creating a massive archive of government documents.

From the post:

Since 2015, I’ve published millions of government documents (about 10% of the text items on the Internet Archive, with some items containing thousands of documents) and terabytes of data; but in order to keep going, I need your help. Since I’ve gotten started, no outlet has matched the number of government documents that I’ve published and made freely available. The only non-governmental publisher that rivals the size and scope of the government files I’ve uploaded is WikiLeaks. While I analyze and write about these documents, I consider publishing them to be more important because it enables and empowers an entire generation of journalists, researchers and students of history.

I’ve also pressured government agencies into making their documents more widely available. This includes the more than 13,000,000 pages of CIA documents that are being put online soon, partially in response to my Kickstarter and publishing efforts. These documents are coming from CREST, which is a special CIA database of declassified records. Currently, it can only be accessed from four computers in the world, all of them just outside of Washington D.C. These records, which represent more than 3/4 of a million CIA files, will soon be more accessible than ever – but even once that’s done, there’s a lot more work left to do.

Question: Do you want a transparent and accountable Trump presidency?

Potential Answers include:

1) Yes, but I’m going to spend time and resources hyper-ventilating with others and roaming the streets.

2) Yes, and I’m going to support Michael Best and FOIA efforts.

Governments, even Trump’s presidency, don’t spring from ocean foam.


The people chosen fill cabinet and other posts have history, in many cases government history.

For example, I heard a rumor today that Ed Meese, a former government crime lord, is on the Trump transition team. Hell, I thought he was dead.

Michael’s efforts produce the dots that connect past events, places, people, and even present administrations.

The dots Michael produces may support your expose, winning story and/or indictment.

Are you in or out?

The TPP Is Dead! Really Most Sincerely Dead! (Celebration Is In Order!)

Friday, November 11th, 2016

Obama Administration Gives Up on Pacific Trade Deal by William Mauldin.

From the post:

The Obama administration on Friday gave up all hope of enacting its sweeping Pacific trade agreement, a pact designed to preserve U.S. economic influence in fast-growing Asia that was buried by a wave of antitrade political sentiment that culminated with Tuesday’s presidential election….


I have ranted about the largely secret Trans-Pacific Partnership (TPP) trade agreement on several occasions.

Negotiated entirely in secret and even worse, designed to be kept secret from the citizens of signing countries, among the worse provisions (there were several), were those enabling investors to sue sovereign countries if laws diminished their investments.

I don’t know, like health warnings on cigarettes for example.

With the election of Donald Trump, I should say president-elect Donald Trump, the TPP is dead. (full stop)

As the proverb says:

It’s an ill wind that blows nobody any good.

Whatever your feelings about president-elect Donald Trump and any of his decisions/policies as president, the defeat of the TPP is one for the win column.

Hazards and dangers lie ahead, just as they would for any presidency, but take a moment to appreciate this win.

Weakly Weaponized Open Data

Friday, November 4th, 2016

Berners-Lee raises spectre of weaponized open data by Bill Camarda.

From the post:


Practically everybody loves open data, ie “data that anyone can access, use or share”. And nobody loves it more than Tim Berners-Lee, creator of the World Wide Web, and co-founder of the Open Data Institute (ODI).

Berners-Lee and his ODI colleagues have spent years passionately evangelizing governments and companies to publicly release their non-personal data for use to improve communities.

So when he recently told the Guardian that hackers could use open data to create societal chaos, it might have been this year’s most surprising “man bites dog” news story.

What’s going on here? The growing fear of data sabotage, that’s what.

Bill focuses on the manipulation and/or planting of false data, which could result in massive traffic jams, changes in market prices, etc.

In fact, Berners-Lee says in the original Guardian story:

“If you falsify government data then there are all kinds of ways that you could get financial gain, so yes,” he said, “it’s important that even though people think about open data as not a big security problem, it is from the point of view of being accurate.”

He added: “I suppose it’s not as exciting as personal data for hackers to get into because it’s public.”

Disruptive to some, profitable to others, but what should be called weakly weaponized open data.

Here is one instance of strongly weaponized open data.

Scenario: We Don’t Need No Water, Let The Motherfucker Burn

The United States is currently experiencing a continuing drought. From the U.S. Drought Monitor:


Keying on the solid red color around Atlanta, GA, Fire Weather, a service of the National Weather Service, estimates the potential impact of fires near Atlanta:


Impacted by a general conflagration around Atlanta:

Population: 2,783,418
Airports: 38
Miles of Interstate: 556
Miles of Rail: 2399
Parks: 4
Area: 27,707 Sq. Miles

Pipelines are missing from the list of impacts. For that, consult the National Pipeline Mapping System where even a public login reveals:


The red lines are hazardous liquid pipelines, blue lines are gas transmission pipelines, the yellow lines outline Fulton County.

We have located a likely place for a forest fire, have some details on its probable impact and a rough idea of gas and other pipelines in the prospective burn area.

Oh, we need a source of ignition. Road flares anyone?


From the WSDOT, Winter Driving Supply Checklist. Emergency kits with flares are available at box stores and online.

Bottom line:

Intentional forest fires can be planned from public data sources. Governments gratuitously suggest non-suspicious methods of transporting forest fire starting materials.

Details I have elided over, such as evacuation routes, fire watch stations, drones as fire starters, fire histories, public events, plus greater detail from the resources cited, are all available from public sources.

What are your Weaponized Open Data risks?

Attn: Secrecy Bed-Wetters! All Five Volumes of Bay of Pigs History Released!

Thursday, November 3rd, 2016

Hand-wringers and bed-wetters who use government secrecy to hide incompetence and errors will sleep less easy tonight.

All Five Volumes of Bay of Pigs History Released and Together at Last: FRINFORMSUM 11/3/2016 by Lauren Harper.

From the post:

After more than twenty years, it appears that fear of exposing the Agency’s dirty linen, rather than any significant security information, is what prompts continued denial of requests for release of these records. Although this volume may do nothing to modify that position, hopefully it does put one of the nastiest internal power struggles into proper perspective for the Agency’s own record.” This is according to Agency historian Jack Pfeiffer, author of the CIA’s long-contested Volume V of its official history of the Bay of Pigs invasion that was released after years of work by the National Security Archive to win the volume’s release. Chief CIA Historian David Robarge states in the cover letter announcing the document’s release that the agency is “releasing this draft volume today because recent 2016 changes in the Freedom of Information Act (FOIA) requires us to release some drafts that are responsive to FOIA requests if they are more than 25 years old.” This improvement – codified by the FOIA Improvement Act of 2016 – came directly from the National Security Archive’s years of litigation.

The CIA argued in court for years – backed by Department of Justice lawyers – that the release of this volume would “confuse the public.” National Security Archive Director Tom Blanton says, “Now the public gets to decide for itself how confusing the CIA can be. How many thousands of taxpayer dollars were wasted trying to hide a CIA historian’s opinion that the Bay of Pigs aftermath degenerated into a nasty internal power struggle?”

To read all five volumes of the CIA’s Official History of the Bay of Pigs Operation – together at last – visit the National Security Archive’s website.

Even the CIA’s own retelling of the story, The Bay of Pigs Invasion, ends with a chilling reminder for all “rebels” being presently supported by the United States.

Brigade 2506’s pleas for air and naval support were refused at the highest US Government levels, although several CIA contract pilots dropped munitions and supplies, resulting in the deaths of four of them: Pete Ray, Leo Baker, Riley Shamburger, and Wade Gray.

Kennedy refused to authorize any extension beyond the hour granted. To this day, there has been no resolution as to what caused this discrepancy in timing.

Without direct air support—no artillery and no weapons—and completely outnumbered by Castro’s forces, members of the Brigade either surrendered or returned to the turquoise water from which they had come.

Two American destroyers attempted to move into the Bay of Pigs to evacuate these members, but gunfire from Cuban forces made that impossible.

In the following days, US entities continued to monitor the waters surrounding the bay in search of survivors, with only a handful being rescued. A few members of the Brigade managed to escape and went into hiding, but soon surrendered due to a lack of food and water. When all was said and done, more than seventy-five percent of Brigade 2506 ended up in Cuban prisons.

100% captured or killed. There’s an example of US support.

In a less media savvy time, the US did pay $53 million (in 1962 dollars, about $424 million today) for the release of 1113 members of Brigade 2506.

Another important fact is that fifty-seven (57) years of delay enabled the participants to escape censure and/or a trip to the gallows for their misdeeds and crimes.

Let’s not let that happen with the full CIA Torture Report. Even the sanitized 6,700 page version would be useful. More so the documents upon which it was based.

All of that exists somewhere. We lack a person with access and moral courage to inform their fellow citizens of the full truth about the CIA torture program. So far.

Update: Michael Best, NatSecGeek advises CIA Histories has the most complete CIA history collection. Thanks Michael!

Inside the fight to reveal the CIA’s torture secrets [Support The Guardian]

Monday, September 12th, 2016

Inside the fight to reveal the CIA’s torture secrets by Spencer Ackerman.

Part one: Crossing the bridge

Part two: A constitutional crisis

Part three: The aftermath

Ackerman captures the drama of a failed attempt by the United States Senate to exercise oversight on the Central Intelligence Agency (CIA) in this series.

I say “failed attempt” because even if the full 6,200+ page report is ever released, the lead Senate investigator, Daniel Jones, obscured the identities of all the responsible CIA personnel and sources of information in the report.

Even if the full report is serialized in your local newspaper, the CIA contractors and staff guilty of multiple felonies, will be not one step closer to being brought to justice.

To that extent, the “full” report is itself a disservice to the American people, who elect their congressional leaders and expect them to oversee agencies such as the CIA.

From Ackerman’s account you will learn that the CIA can dictate to its overseers, the location and conditions under which it can view documents, decide which documents it is allowed to see and in cases of conflict, the CIA can spy on the Select Senate Committee on Intelligence.

Does that sound like effective oversight to you?

BTW, you will also learn that members of the “most transparent administration in history” aided and abetted the CIA in preventing an effective investigation into the CIA and its torture program. I use “aided and abetted” deliberately and in their legal sense.

I mention in my header that you should support The Guardian.

This story by Spencer Ackerman is one reason.

Another reason is that given the plethora of names and transfers recited in Ackerman’s story, we need The Guardian to cover future breaks in this story.

Despite the tales of superhuman security, nobody is that good.

I leave you with the thought that if more than one person knows a secret, then it it can be discovered.

Check Ackerman’s story for a starting list of those who know secrets about the CIA torture program.

Good hunting!

Hunters Bag > 400 Database Catalogs

Monday, August 29th, 2016

Transparency Hunters Capture More than 400 California Database Catalogs by Dave Maass.

The post in its entirety:

A team of over 40 transparency activists aimed their browsers at California this past weekend, collecting more than 400 database catalogs from local government agencies, as required under a new state law. Together, participants in the California Database Hunt shined light on thousands upon thousands of government record systems.

California S.B. 272 requires every local government body, with the exception of educational agencies, to post inventories of their “enterprise systems,” essentially every database that holds records on members of the public or is used as a primary source of information. These database catalogs were required to be posted online (at least by agencies with websites) by July 1, 2016.

EFF, the Data Foundation, the Sunlight Foundation, and Level Zero, combined forces to host volunteers in San Francisco, Washington, D.C., and remotely. More than 40 volunteers scoured as many local agency websites as we could in four hours—cities, counties, regional transportation agencies, water districts, etc. Here are the rough numbers:

680 – The number of unique agencies that supporters searched

970 – The number of searches conducted (Note: agencies found on the first pass not to have catalogs were searched a second time)

430 – Number of agencies with database catalogs online

250 – Number of agencies without database catalogs online, as verified by two people

Download a spreadsheet of the local government database catalogs we found: Excel/TSV

Download a spreadsheet of cities and counties that did not have S.B. 272 catalogs: Excel/TSV

Please note that for each of the cities and counties identified as not posting database catalogs, at least two volunteers searched for the catalogs and could not find them. It is possible that those agencies do in fact have S.B. 272-compliant catalogs posted somewhere, but not in what we would call a “prominent location,” as required by the new law. If you represent an agency that would like its database catalog listed, please send an email to

We owe a debt of gratitude to the dozens of volunteers who sacrificed their Saturday afternoons to help make local government in California a little less opaque. Check out this 360-degree photo of our San Francisco team on Facebook.

In the coming days and weeks, we plan to analyze and share the data further. Stay tuned, and if you find anything interesting perusing these database catalogs, please drop us a line at

Of course, bagging the database catalogs is like having a collection of Christmas catalogs. It’s great, but there are more riches within!

What data products would you look for first?

Updated to mirror changes (clarification) in original.

US Army committed $6.5 trillion in accounting fraud in one year (w/correction)

Saturday, August 20th, 2016

US Army committed $6.5 trillion in accounting fraud in one year by Cory Doctorow.

From the post:

In June, the Defense Department’s Inspector General released a report on the US Army’s accounting, revealing that the Army had invented $6.5 trillion in “improper adjustments” ($2.8T in one quarter!) to make its books appear balanced though it could not account for where the funds had gone.

If you are interested in transparent and trackable information systems, that’s a headline that captures your attention!

Except that when you run it back to the original story, U.S. Army fudged its accounts by trillions of dollars, auditor finds by Scot J. Paltrow, which reads in part:

The United States Army’s finances are so jumbled it had to make trillions of dollars of improper accounting adjustments to create an illusion that its books are balanced.

The Defense Department’s Inspector General, in a June report, said the Army made $2.8 trillion in wrongful adjustments to accounting entries in one quarter alone in 2015, and $6.5 trillion for the year. Yet the Army lacked receipts and invoices to support those numbers or simply made them up.

You won’t find a reference to the “June report,” as cited by Paltrow. No link, no title, no nothing.

In fact, there is no such June report.

If you look carefully enough at the Inspector General site for the DoD you will find:

Financial Management
Army General Fund Adjustments Not Adequately Documented or Supported (Project No. D2015-D000FL-0243.000)

The webpage for that July report, reads in part:


The Office of the Assistant Secretary of the Army (Financial Management & Comptroller) (OASA[FM&C]) and the Defense Finance and Accounting Service Indianapolis (DFAS Indianapolis) did not adequately support $2.8 trillion in third quarter journal voucher (JV) adjustments and $6.5 trillion in yearend JV adjustments1 made to AGF data during FY 2015 financial statement compilation.2 The unsupported JV adjustments occurred because OASA(FM&C) and DFAS Indianapolis did not prioritize correcting the system deficiencies that caused errors resulting in JV adjustments, and did not provide sufficient guidance for supporting system‑generated adjustments.

In addition, DFAS Indianapolis did not document or support why the Defense Departmental Reporting System‑Budgetary (DDRS-B), a budgetary reporting system, removed at least 16,513 of 1.3 million records during third quarter FY 2015. This occurred because DFAS Indianapolis did not have detailed documentation describing the DDRS-B import process or have accurate or complete system reports.

As a result, the data used to prepare the FY 2015 AGF third quarter and yearend financial statements were unreliable and lacked an adequate audit trail. Furthermore, DoD and Army managers could not rely on the data in their accounting systems when making management and resource decisions. Until the Army and DFAS Indianapolis correct these control deficiencies, there is considerable risk that AGF financial statements will be materially misstated and the Army will not achieve audit readiness by the congressionally mandated deadline of September 30, 2017.

Everybody makes mistakes. I’m sure I make several everyday without hardly trying.

However, if you link to original sources, readers stand some chance of discovering and correcting those errors.

If you cite a resource, link to the resource.

PS: Before you use the word “fraud” with regard to military accounting systems, realize financial accounting is not a primary or even secondary concern of a military force. There are possible solutions to military accounting issues but congressional tantrums, a/k/a mandates, aren’t among them.

Concealing the Purchase of Government Officials

Monday, June 20th, 2016

Fredreka Schouten reports in House approves Koch-backed bill to shield donors’ names the US House of Representatives, has passed a measure to conceal the purchase of government officials.

From the post:

The House approved a bill Tuesday that would bar the IRS from collecting the names of donors to tax-exempt groups, prompting warnings from campaign-finance watchdogs that it could lead to foreign interests illegally infiltrating American elections.

The measure, which has the support of House Speaker Paul Ryan, R-Wis., also pits the Obama administration against one of the most powerful figures in Republican politics, billionaire industrialist Charles Koch. Koch’s donor network channels hundreds of millions of dollars each year into groups that largely use anonymous donations to shape policies on everything from health care to tax subsidies. Its leaders have urged the Republican-controlled Congress to clamp down on the IRS, citing free-speech concerns.

The names of donors to politically active non-profit groups aren’t public information now, but the organizations still have to disclose donor information to the IRS on annual tax returns. The bill, written by Rep. Peter Roskam, R-Ill., would prohibit the tax agency from collecting names, addresses or any “identifying information” about donors.

Truth be told, however, “the House” didn’t vote in favor of H.R.5053 – Preventing IRS Abuse and Protecting Free Speech Act.

Rather, two-hundred and forty (240) identified representatives voted in favor of H.R.5053.

Two-hundred and forty representatives purchased by campaign contributions who now wish to keep their contributors secret.

Two-hundred and forty representatives who are as likely as not, guilty of criminal, financial/sexual or other forms of misconduct, that could result in their replacement.

Two-hundred and forty representatives who continue in office only so long as they are not exposed to law enforcement and the public.

Where are you going to invest your time and resources?

Showing solidarity on issues where substantive change isn’t going to happen, or taking back your government from its current purchasers?

PS: In case you think “substantive change” is possible on gun control, consider the unlikely scenario that “assault weapons” are banned from sale. So what? The ones in circulation number in the millions. Net effect of your “victory” would be exactly zero.

Breaking Californication (An Act Performed On The Public)

Monday, June 6th, 2016

Law Enforcement Lobby Succeeds In Killing California Transparency Bill by Kit O’Connell.

From the post:

A California Senate committee killed a bill to increase transparency in police misconduct investigations, hampering victims’ efforts to obtain justice.

Chauncee Smith, legislative advocate at the ACLU of California, told MintPress News that the state Legislature “caved to the tremendous influence and power of the law enforcement lobby” and “failed to listen to the demands and concerns of everyday Californian people.”

California has some of the most secretive rules in the country when it comes to investigations into police misconduct and excessive use of force. Records are kept sealed, regardless of the outcome, as the ACLU of Northern California explains on its website:

“In places like Texas, Kentucky, and Utah, peace officer records are made public when an officer is found guilty of misconduct. Other states make records public regardless of whether misconduct is found. This is not the case in California.”

“Right now, there is a tremendous cloud of secrecy that is unparalleled compared to many other states,” Smith added. “California is in the minority in which the public do not know basic information when someone is killed or potentially harmed by those are sworn to serve and protect them.”

In February, Sen. Mark Leno, a Democrat from San Francisco, introduced SB 1286, the “Enhance Community Oversight on Police Misconduct and Serious Uses of Force” bill. It would have allowed “public access to investigations, findings and discipline information on serious uses of force by police” and would have increased transparency in other cases of police misconduct, according to an ACLU fact sheet. Polling data cited by the ACLU suggests about 80 percent of Californians would support the measure.

But the bill’s progress through the legislature ended on May 27, when it failed to pass out of the Senate Appropriations committee.

“Today is a sad day for transparency, accountability, and justice in California,” said Peter Bibring, police practices director for the ACLU of California, in a May 27 press release.

Mistrust between police officers and citizens makes the job of police officers more difficult and dangerous, while denying citizens the full advantages of a trained police force, paid for by their tax dollars.

The state legislature, finding sowing and fueling mistrust between police officers and citizens has election upsides for them, fans those flames with secrecy over police misconduct investigations.

Open, not secret (read grand jury) proceedings where witnesses can be fairly examined (unlike the deliberately thrown Michael Brown investigation), can go a long way to re-establishing trust between the police and the public.

Members of the community know when someone was a danger to police officers and others. Whether their family members will admit it or not. Likewise, police officers know which officers are far to quick to escalate to deadly force. Want better community policing? What better citizen cooperation? That’s not going to happen with completely secret police misconduct investigations.

So the State of California is going to collect the evidence, statements, etc., in police misconduct investigations, but won’t share that information with the public. At least not willingly.

Official attempts to break illegitimate government secrecy failed. Even if it had succeeded you’d be paying least $0.25 per page plus a service fee.

Two observations about government networks:

  • Secret (and otherwise) government documents are usually printed on networked printers.
  • Passively capturing Ethernet traffic (network tap) captures printer traffic too.

Whistle blowers don’t have to hack heavily monitored systems, steal logins/passwords, leaking illegally withheld documents is within the reach of anyone who can plug in an Ethernet cable.

There’s a bit more to it than that, but remember all those network cables running through the ceiling, walls, closets, the next time your security consultant, assures you of your network’s security.

As a practical matter, if you start leaking party menus and football pools, someone will start looking for a network tap.

Leak when it makes a significant difference to public discussion and/or legal proceedings. Even then, look for ways to attribute the leak to factions within the government.

Remember the DoD’s amused reaction to State’s huffing and puffing over the Afghan diplomatic cables? That sort of rivalry exists at every level of government. You should use it to your advantage.

The State of California would have you believe that government information sharing is at its sufferance.

I beg to differ.

So should you.

Hidden Inspector General Report on Clinton’s Emails?

Wednesday, May 25th, 2016

If you haven’t heard about the controversy surrounding Hillary Clinton’s handling of emails during her term as Secretary of State, you are one of the lucky ones.

The rest of us have been treated to a literal circus of pettifogging over her “private” email server for years now. Truly a tempest in a teapot.

But, along comes a much awaited report by the Inspector General for the State Department on those same emails, and where can you find it?

Not on the Inspector General for the State Department homepage (as of 25 May 2016, 9:00 PM EST)!

No, you will have to find that report, the one everyone has been waiting for, Office of the Secretary: Evaluation of Email Records Management and Cybersecurity Requirements to be posted by Politico.

I have no objection to Politico having the “scoop” on this report and/or distributing a document of great public interest. All fine and good.

But why does the Inspector General choose to hide this report from the general public?

Is the Inspector General ashamed of the report?

A report that encompasses other secretaries of state, as though to argue bad and/or criminal behavior can be excused because it is customary?

I’m not familiar with the “customary therefore not criminal” defense.

Perhaps that only obtains at Cabinet level positions.

In any event, please help Steve Linick, the current Inspector General for the State Department, own this report now and forever.

FOIA – For Algorithms

Tuesday, May 24th, 2016

We need to know the algorithms the government uses to make important decisions about us by Nicholas Diakopoulos.

From the post:

In criminal justice systems, credit markets, employment arenas, higher education admissions processes and even social media networks, data-driven algorithms now drive decision-making in ways that touch our economic, social and civic lives. These software systems rank, classify, associate or filter information, using human-crafted or data-induced rules that allow for consistent treatment across large populations.

But while there may be efficiency gains from these techniques, they can also harbor biases against disadvantaged groups or reinforce structural discrimination. In terms of criminal justice, for example, is it fair to make judgments on an individual’s parole based on statistical tendencies measured across a wide group of people? Could discrimination arise from applying a statistical model developed for one state’s population to another, demographically different population?

The public needs to understand the bias and power of algorithms used in the public sphere, including by government agencies. An effort I am involved with, called algorithmic accountability, seeks to make the influences of those sorts of systems clearer and more widely understood.

Existing transparency techniques, when applied to algorithms, could enable people to monitor, audit and criticize how those systems are functioning – or not, as the case may be. Unfortunately, government agencies seem unprepared for inquiries about algorithms and their uses in decisions that significantly affect both individuals and the public at large.

Nicholas makes a great case for Freedom of Information Act (FOIA) legislation being improved to explicitly include algorithms used by government or on its behalf.

I include “on its behalf” because as Nicholas documents, some states have learned the trick of having algorithms held by vendors, thus making them “proprietary.”

If you can’t see the algorithms behind data results, there is no meaningful transparency.

Demand meaningful transparency!

SEMS 2016 (Auditable Spreadsheets – Quick Grab Your Heart Pills)

Monday, April 11th, 2016

3rd International Workshop on Software Engineering Methods in Spreadsheets

July 4, 2016 Vienna, Austria

Abstracts due: April 11th (that’s today!)

Papers due: April 22nd

From the webpage:

SEMS is the #1 venue for academic spreadsheet research since 2014 (SEMS’14, SEMS’15). This year, SEMS’16 is going to be co-located with STAF 2016 in Vienna.

Spreadsheets are heavily used in industry as they are easy to create and evolve through their intuitive visual interface. They are often initially developed as simple tools, but, over time, spreadsheets can become increasingly complex, up to the point they become too complicated to maintain. Indeed, in many ways, spreadsheets are similar to “professional” software: both concern the storage and manipulation of data, and the presentation of results to the user. But unlike in “professional” software, activities like design, implementation, and maintenance in spreadsheets have to be undertaken by end-users, not trained professionals. This makes applying methods and techniques from other software technologies a challenging task.

The role of SEMS is to explore the possibilities of adopting successful methods from other software contexts to spreadsheets. Some, like testing and modeling, have been tried before and can be built upon. For methods that have not yet been tried on spreadsheets, SEMS will serve as a platform for early feedback.

The SEMS program will include an industrial keynote, followed by a brainstorming session about the topic, a discussion panel of industrial spreadsheet usage, presentation of short and long research papers and plenty of lively discussions. The intended audience is a mixture of spreadsheet researchers and professionals.

Felienne Hermans pioneered viewing spreadsheets as programming artifacts, a view that can result in easier maintenance and even, gasp, auditing of spreadsheets.

Inspectors General, GAO and other birds of that feather should sign up for this conference.

Remember topic maps for cumulative and customized auditing data. For example, who, by name, was explaining entries that several years later appear questionable? Topic maps can capture as much or as little data as you require.

Attend, submit an abstract today and a paper in two weeks!

Nominations by the U.S. President

Tuesday, December 29th, 2015

Nominations by the U.S. President

From, a faceted listing of all nominations from 1981 to date.

Facets include Congress, Nomination Type (Civilian, Military, Select Only), Status of Nomination, Senate Committee, Nominees with US State or Territory Indicated.

I haven’t spent a lot of time with this resource but it appears to be unnecessarily difficult to use.

For example:

Let’s look up the nomination of Sonia Sotomayor to the United States Supreme Court:


Did you notice the absence of any hyperlinks to the three days of hearings, July 13-15, 2009? Or the absence of links to the Senate debate on August 4-5, 2009? Or the absence of links to any of the other documents or agreements?

I remember the WWW being around in 2009 and I am damned sure it is available now!

So, what’s with the lack of hyperlinks?

Do you think they are lurking beneath the surface, waiting to be turned on?

Afraid not. Here is a sample of the underlying content for that page:

<td class="date">07/16/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held and completed. Hearings printed: S.Hrg. 111-503.</td>
<td class="date">07/15/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held.</td>
<td class="date">07/14/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held.</td>
<td class="date">07/13/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held.</td>

I don’t see any hooks for hyperlinking later on. Do you?

Another data hook that is missing is linking historical campaign donations to nominees for offices, particularly in the State Department.

Surely you didn’t think ambassadors were appointed from the professional ranks of the Foreign Service? People who actually speak the languages of the host country and know it customs and habits. What an odd view of American (or any other) government you have.

Some of the larger ambassadorships do require some experience but out of 270 embassies around the world, there are ones that go to mega-donors.

I don’t know the going rate on ambassadorships but linking nominations to donation records could yield a target minimum for donors to shoot for.

Linking nominations to donations would be a non-trivial exercise but certainly doable.

Other suggestions for on these webpages? They respond well to suggestions. Not to say they always agree but they do respond. More than I can say for some government groups.

Why the Open Government Partnership Needs a Reboot [Governments Too]

Saturday, December 12th, 2015

Why the Open Government Partnership Needs a Reboot by Steve Adler.

From the post:

The Open Government Partnership was created in 2011 as an international forum for nations committed to implementing Open Government programs for the advancement of their societies. The idea of open government started in the 1980s after CSPAN was launched to broadcast U.S. Congressional proceedings and hearings to the American public on TV. While the galleries above the House of Representatives and Senate had been “open” to the “public” (if you got permission from your representative to attend) for decades, never before had all public democratic deliberations been broadcast on TV for the entire nation to behold at any time they wished to tune in.

I am a big fan of OGP and feel that the ideals and ambition of this partnership are noble and essential to the survival of democracy in this millennium. But OGP is a startup, and every startup business or program faces a chasm it must cross from early adopters and innovators to early majority market implementation and OGP is very much at this crossroads today. It has expanded membership at a furious pace the past three years and it’s clear to me that expansion is now far more important to OGP than the delivery of the benefits of open government to the hundreds of millions of citizens who need transparent transformation.

OGP needs a reboot.

The structure of a system produces its own behavior. OGP needs a new organizational structure with new methods for evaluating national commitments. But that reboot needs to happen within its current mission. We should see clearly that the current structure is straining due to the rapid expansion of membership. There aren’t enough support unit resources to manage the expansion. We have to rethink how we manage national commitments and how we evaluate what it means to be an open government. It’s just not right that countries can celebrate baby steps at OGP events while at the same time passing odious legislation, sidestepping OGP accomplishments, buckling to corruption, and cracking down on journalists.

Unlike Steve I didn’t and don’t have a lot of faith in governments being voluntarily transparent.

As I pointed out in Congress: More XQuery Fodder, sometime in 2016, full bill status data will be available for all legislation before the United States Congress.

A lot more data than is easy to access now but it is more smoke than fire.

With legislation status data, you can track the civics lesson progression of a bill through Congress, but that leaves you at least 3 to 4 degrees short of knowing who was behind the legislation.

Just a short list of what more would be useful:

  • Visitor/caller list for everyone who spoke to a member of Congress and their staff. With date and subject of the call.
  • All visitors and calls tied to particular legislation and/or classes of legislation
  • All fund raising calls made by members of Congress and/or their staffs, date, results, substance of call.
  • Representative conversations with reconciliation committee members or their staffers about legislation and requested “corrections.”
  • All conversations between a representative or member of their staff and agency staff, identifying all parties and the substance of the conversation
  • Notes, proposals, discussion notes for all agencies decisions

Current transparency proposals are sufficient to confuse the public with mounds of nearly useless data. None of it reflects the real decision making processes of government.

Before someone shouts “privacy,” I would point out that no citizen has a right to privacy when their request is for a government representative to favor them over other citizens of the same government.

Real government transparency will require breaking the mini-star chamber proceedings from the lowest to the highest levels of government.

What we need is a rebooting of governments.

How Computers Broke Science… [Soon To Break Businesses …]

Tuesday, November 10th, 2015

How Computers Broke Science — and What We can do to Fix It by Ben Marwick.

From the post:

Reproducibility is one of the cornerstones of science. Made popular by British scientist Robert Boyle in the 1660s, the idea is that a discovery should be reproducible before being accepted as scientific knowledge.

In essence, you should be able to produce the same results I did if you follow the method I describe when announcing my discovery in a scholarly publication. For example, if researchers can reproduce the effectiveness of a new drug at treating a disease, that’s a good sign it could work for all sufferers of the disease. If not, we’re left wondering what accident or mistake produced the original favorable result, and would doubt the drug’s usefulness.

For most of the history of science, researchers have reported their methods in a way that enabled independent reproduction of their results. But, since the introduction of the personal computer — and the point-and-click software programs that have evolved to make it more user-friendly — reproducibility of much research has become questionable, if not impossible. Too much of the research process is now shrouded by the opaque use of computers that many researchers have come to depend on. This makes it almost impossible for an outsider to recreate their results.

Recently, several groups have proposed similar solutions to this problem. Together they would break scientific data out of the black box of unrecorded computer manipulations so independent readers can again critically assess and reproduce results. Researchers, the public, and science itself would benefit.

Whether you are looking for specific proposals to make computed results capable of replication or quotes to support that idea, this is a good first stop.

FYI for business analysts, how are you going to replicate results of computer runs to establish your “due diligence” before critical business decisions?

What looked like a science or academic issue has liability implications!

Changing a few variables in a spreadsheet or more complex machine learning algorithms can make you look criminally negligent if not criminal.

The computer illiteracy/incompetence of prosecutors and litigants is only going to last so long. Prepare defensive audit trails to enable the replication of your actual* computer-based business analysis.

*I offer advice on techniques for such audit trails. The audit trails you choose to build are up to you.

Disclosing Government Contracts

Friday, August 21st, 2015

The More the Merrier? How much information on government contracts should be published and who will use it by Gavin Hayman.

From the post:

A huge bunch of flowers to Rick Messick for his excellent post asking two key questions about open contracting. And some luxury cars, expensive seafood and a vat or two of cognac.

Our lavish offerings all come from Slovakia, where in 2013 the Government Public Procurement Office launched a new portal publishing all its government contracts. All these items were part of the excessive government contracting uncovered by journalists, civil society and activists. In the case of the flowers, teachers investigating spending at the Department of Education uncovered florists’ bills for thousands of euros. Spending on all of these has subsequently declined: a small victory for fiscal probity.

The flowers, cars, and cognac help to answer the first of two important questions that Rick posed: Will anyone look at contracting information? In the case of Slovakia, it is clear that lowering the barriers to access information did stimulate some form of response and oversight.

The second question was equally important: “How much contracting information should be disclosed?”, especially in commercially sensitive circumstances.

These are two of key questions that we have been grappling with in our strategy at the Open Contracting Partnership. We thought that we would share our latest thinking below, in a post that is a bit longer than usual. So grab a cup of tea and have a read. We’ll be definitely looking forward to your continued thoughts on these issues.

Not a short read so do grab some coffee (outside of Europe) and settle in for a good read.

Disclosure: I’m financially interested in government disclosure in general and contracts in particular. With openness there comes more effort to conceal semantics and increase the need for topic maps to pierce the darkness.

I don’t think openness reduces the amount of fraud and misconduct in government, it only gives an alignment between citizens and the career interests of a prosecutor a sporting chance to catch someone out.

Disclosure should be as open as possible and what isn’t disclosed voluntarily, well, one hopes for brave souls who will leak the remainder.

Support disclosure of government contracts and leakers of the same.

If you need help “connecting the dots,” consider topic maps.

Reputation instead of obligation:…

Thursday, June 4th, 2015

Reputation instead of obligation: forging new policies to motivate academic data sharing by Sascha Friesike, Benedikt Fecher, Marcel Hebing, and Stephanie Linek.

From the post:

Despite strong support from funding agencies and policy makers academic data sharing sees hardly any adoption among researchers. Current policies that try to foster academic data sharing fail, as they try to either motivate researchers to share for the common good or force researchers to publish their data. Instead, Sascha Friesike, Benedikt Fecher, Marcel Hebing, and Stephanie Linek argue that in order to tap into the vast potential that is attributed to academic data sharing we need to forge new policies that follow the guiding principle reputation instead of obligation.

In 1996, leaders of the scientific community met in Bermuda and agreed on a set of rules and standards for the publication of human genome data. What became known as the Bermuda Principles can be considered a milestone for the decoding of our DNA. These principles have been widely acknowledged for their contribution towards an understanding of disease causation and the interplay between the sequence of the human genome. The principles shaped the practice of an entire research field as it established a culture of data sharing. Ever since, the Bermuda Principles are used to showcase how the publication of data can enable scientific progress.

Considering this vast potential, it comes as no surprise that open research data finds prominent support from policy makers, funding agencies, and researchers themselves. However, recent studies show that it is hardly ever practised. We argue that the academic system is a reputation economy in which researchers are best motivated to perform activities if those pay in the form of reputation. Therefore, the hesitant adoption of data sharing practices can mainly be explained by the absence of formal recognition. And we should change this.

(emphasis in the original)

Understanding what motivates researchers to share data is an important step towards encouraging data sharing.

But at the same time, would we say that every researcher is as good as every other researcher at preparing data for sharing? At documenting data for sharing? At doing any number of tasks that aren’t really research, but just as important in order to share data?

Rather than focusing exclusively on researchers, funders should fund projects to include data sharing specialists who have the skills and interests necessary to effectively share data as part of a project’s output. Their reputations will be more closely tied to the successful sharing of data and researchers would gain in reputation for the high quality data that is shared. A much better fit for the recommendation of the authors.

Or to put it differently, lecturing researchers on how they should spend their limited time and resources to satisfy your goals, isn’t going to motivate anyone. “Pay the man!” (Richard Prior from Silver Streak)

TPP – Just One of Many Government Secrets

Thursday, June 4th, 2015

The Trans-Pacific Partnership is just one of many government secrets.

Reading Army ELA: Weapon Of Mass Confusion? by Kevin McLaughlin, I discovered yet another.

From the post:

As DISA and VMware work on a new JELA proposal, sources familiar with the matter said the relationship between the two is coming under scrutiny from other enterprise vendors. What’s more, certain details of the JELA remain shrouded in secrecy.

DISA’s JELA document contains several large chunks of redacted text, including one entire section titled “Determination Of Fair And Reasonable Cost.”

In other parts, DISA has redacted specific figures, such as VMware’s percentage of the DOD’s virtualized environments and the total amount the DOD has invested in VMware software licenses. The redacted portions have fueled industry speculation about why these and other aspects of the contract were deemed unfit for the eyes of the public.

DISA’s rationale for awarding the Army ELA and DOD JELA to VMware without opening it up to competition is also suspect, one industry executive who’s been tracking both deals told CRN. “Typically, the justification for sole-sourcing contracts to a vendor is that they only cover maintenance, yet these contracts obviously weren’t maintenance-only,” said the source.

The situation is complex but essentially the Army signed a contract with VMware that resulted in the Army downloading suites of software when it only wanted one particular part of the suite, but the Army was billed for maintenance cost on the entire suite.

That appears to be what was specified in the VMware ELA, which should be a motivation to using topic maps in connection with government contracts.

Did that go by a little fast? The jump from the VMware ELA to topic maps?

Think about it. The “Army” didn’t really sign a contract with “VMware” anymore than “VMware” signed a contract with the “Army.”

No, someone in particular, a nameable individual or group of nameable individuals, had meetings, reviews, and ultimately decided to agree to the contract between the “Army” and “VMware.” All of those individuals has roles in the proceedings that resulted in the ELA in question.

Yet, when it comes time to discuss the VMware ELA, the best we can do is talk about it as though these large organizations acted on their own. The only named individual who might be in some way responsible for the mess is the Army’s current CIO, Lt. Gen. Robert S. Ferrell, and he got there after the original agreement but before its later extension.

Topic maps, since we don’t have to plot the domain before we start uncovering relationships and roles, could easily construct a history of contacts (email, phone, physical meetings), aligned with documents (drafts, amendments), of all the individuals on all sides of this sorry episode.

Topic maps can’t guarantee that the government, the DOD in this case, won’t make ill-advised contracts in the future. No software can do that. What topic maps can do is trace responsibility for such contracts to named individuals. Having an accountable government means having accountable government employees.

PS: How does your government, agency, enterprise track responsibility?

PPS: Topic maps could also trace, given the appropriate information, who authorized the redactions to the DISA JELA. The first person who should be on a transport as a permanent advisor in Syria. Seriously. Remember what I said about accountable government requiring accountable employees.

$100,000 reward for leaking the Trans-Pacific Partnership ‘TPP’

Tuesday, June 2nd, 2015

Wikileaks is raising a $100,000 bounty for the missing twenty-six (26) chapters of the TPP. (Hope they get the revised versions after the meeting in Lima this summer.)

Donate Today!

As of 13:23 on 2 June 2015, $23927.17 from 68 people or about 24% of the goal. No guarantees but sounds like a good plan.

Congress Can — and Should — Declassify the TPP

Saturday, May 30th, 2015

Congress Can — and Should — Declassify the TPP by Robert Naiman.

From the post:

One of the most controversial aspects of the proposed Trans-Pacific Partnership (TPP) is the fact that the Obama administration has tried to impose a public blockade on the text of the draft agreement.

When Congress votes on whether to grant the president “fast-track authority” to negotiate the TPP — which would bar Congress from making any changes to the secret pact after it’s negotiated — it will effectively be a vote to pre-approve the TPP itself.

Although the other negotiating countries and “cleared” corporate advisers to the US Trade Representative have access to the draft TPP agreement, the American people haven’t been allowed to see it before Congress votes on fast track. Members of Congress can read the draft agreement under heavy restrictions, but they can’t publicly discuss or consult on what they have read.

Correction: The Obama administration hasn’t “tried to impose a public blockade on the text of the draft agreement,” it has succeeded in imposing a public blockage on the text of TPP.

The question is: What is Congress going to do to break the current blockade on the text of the TPP?

Robert has a great writeup of all the reasons why the American public should be allowed to see the text of the TPP. Such as the other parties to the agreement already know what it says, so why not the American people? Interim texts of agreements get published all the time, so why not this one?

The United States Senate or the House of Representatives can declassify the TPP text. I would say to write to your Senators and Representatives, but not this time. Starting Monday, June 1, 2015, I am going to call both of my Senators and my Representative until I have spoken with each one of them personally to express my concern that the TPP text should be available to the American public before it is submitted to Congress for approval. Including any additional or revised versions.

I will be polite and courteous but will keep calling until contact is made. I suggest you do the same. Leave your request that the TPP be declassified (including later versions) by (appropriate body) for the American public with every message.

BTW, keep count of how many calls it takes to speak to your Senator or Representative. It may give you a better understanding of how effective democracy is in the United States.

I first saw this in a tweet by

Government and “legitimate secrets?”

Saturday, May 30th, 2015

Benjamin Wittes in An Approach to Ameliorating Press-IC Tensions Over Classified Information gives a good set of pointers to the recent dispute between the intelligence community and the New York Times:

I’ve been thinking about the exchange over the past couple of weeks—much of which took place on Lawfare—between the New York Times and the intelligence community over the naming of CIA undercover officers in a Times story. (A brief recap in links: here are Bob Litt’s original comments, the 20 former intelligence officers’ letter, Jack’s interview with Dean Baquet, my comments in response, Mark Mazzetti’s comments, and Jack’s comments.)

I want to float an idea for a mechanism that might ameliorate tensions over this sort of issue in the future. It won’t eliminate those tensions, which are inherent in the relationship between a government that has legitimate secrets to keep and a press that rightly wants to report on the activities of government, but it might give the public a lens through which to view individual disputes, and it might create a means through the which the two sides can better and more fully communicate in high-stakes situations.

The basic problem when the press has an undoubtedly newsworthy item that involves legitimately sensitive information is two-fold: the government will tend to err on the side of overstating the sensitivity of the information, because it has to protect against the biggest risk, and the government often cannot disclose to the newspaper the full reasons for its concerns.

I can’t say that I care for Wittes’ proposal because it begins with the assumption that a democratically elected government can have “legitimate secrets.” Once that point is conceded, the only argument is about the degree of ignorance of the electorate. That the electorate will never know, hopefully in eyes of some, the truth about government activities is taken as a given.

For example, did you know that the United States government supported the Pol Pot regime in Cambodia? A regime that reduced the population of Cambodia by 25%, deaths coming from executions, forced labor, starvation, etc.

Question for readers who vote in the United States:

Would United States support for a genocidal regime affect your vote in an election?

Assuming that one candidate promised continued support for such a regime and their opponent promised to discontinue support.

That seems pretty obvious, but that is exactly the sort of secrets that the government keeps from the voters.

How do I know the United States government supported Pol Pot? Good question! Sources? Would you believe diplomatic cables from the relevant time period? Recently published by Wikileaks?

The Pol Pot dilemma by Charles Parkinson, Alice Cuddy and Daniel Pye, reads in part:

A trove of more than 500,000 US diplomatic cables from 1978 released by WikiLeaks on Wednesday includes hundreds that paint a vivid picture of a US administration torn between revulsion at the brutality of Pol Pot’s government and fear of Vietnamese influence should it collapse.

“We believe a national Cambodia must exist even though we believe the Pol Pot regime is the world’s worst violator of human rights,” reads a cable sent by the State Department to six US embassies in Asia on October 11, 1978. “We cannot support [the] Pol Pot government, but an independent Kampuchea must exist.”

They are the second batch of cables to be released by the whistle-blowing website from Jimmy Carter’s presidency, which was marked by a vocal emphasis on human rights. That focus shines through in much of the correspondence, even to the point of wishing success on the Khmer Rouge in repelling Vietnamese incursions during the ongoing war between the two countries, in the hope it would, paradoxically, prevent more of the worst excesses of the government in Phnom Penh.

“While the Pol Pot government has few, if any, redeeming features, the cause of human rights is not likely to be served by the continuation of fighting between the Vietnamese and the government,” reads a cable sent by the US Embassy in Thailand to the State Department on October 17. “A negotiated settlement of [Vietnamese-Cambodian] differences might reduce the purges.”

Read also: SRV-KHMER CONFLICT PRESENTS BENEFITS AND POTENTIAL PROBLEMS FOR MOSCOW to get a feel for the proxy war status of the conflict between Cambodia and Vietnam during this period.

Although the government keeps a copy of your financial information, social security number, etc., that is your information that it holds in trust. Secrecy of that information should not be in doubt.

However, when we are talking about information that is generated in the course of government relations to other governments or in carrying out government policy, we aren’t talking about information that belongs to individuals. At least in a democracy, we are talking about information that belongs to the general public.

In your next debate about government secrecy, challenge the presumption of a need for government secrecy. History is on your side.

Why nobody knows what’s really going into your food

Tuesday, April 21st, 2015

Why nobody knows what’s really going into your food by Phillip Allen, et al.

From the webpage:

Why doesn’t the government know what’s in your food? Because industry can declare on their own that added ingredients are safe. It’s all thanks to a loophole in a 57-year-old law that allows food manufacturers to circumvent the approval process by regulators. This means companies can add substances to their food without ever consulting the Food and Drug Administration about potential health risks.

The animation is quite good and worth your time to watch.

If you think the animation is disheartening, you could spend some time at the Generally Recognized as Safe (GRAS) page over at the FDA.

From the webpage:

“GRAS” is an acronym for the phrase Generally Recognized As Safe. Under sections 201(s) and 409 of the Federal Food, Drug, and Cosmetic Act (the Act), any substance that is intentionally added to food is a food additive, that is subject to premarket review and approval by FDA, unless the substance is generally recognized, among qualified experts, as having been adequately shown to be safe under the conditions of its intended use, or unless the use of the substance is otherwise excluded from the definition of a food additive.

Links to legislation, regulations, applications, and other sources of information.

Leaving the question of regulation to one side, every product should be required to list all of its ingredients. In addition to the package, it should be required to post a full chemical analysis online.

Disclosure would not reach everyone but at least careful consumers would have a sporting chance to discover what they are eating.

UNESCO Transparency Portal

Friday, April 10th, 2015

UNESCO Transparency Portal

From the about page:

Public access to information is a key component of UNESCO’s commitment to transparency and its accountability vis-à-vis stakeholders. UNESCO recognizes that there is a positive correlation between a high level of transparency through information sharing and public participation in UNESCO-supported activities.

The UNESCO transparency portal has been designed to enable public access to information about the Organization’s activities across sectors, countries, and regions, accompanied by some detail on budgetary and donor information. We see this as a work in progress. Our objective is to enable access to as much quality data about our activities as possible. The portal will be regularly updated and improved.

The data is a bit stale, 2014 and by the site’s admission, data on “10 Category I Institutes operating as separate economic entities” and the “UNESCO Brasilia Office,” will be included “in a later phase….”

The map navigation on the default homepage works quite well and I tested it from focusing on Zimbabwe, lead by everyone’s favorite, Robert Mugabe. If you zoom in and select Zimbabwe on the map, the world map displays with a single icon over Zimbabwe. Hovering over that icon displays the number of projects, budget, so I selected projects and the screen scrolls down to show the five projects.

I then selected: UBRAF: Supporting Comprehensive Education Sector Responses to HIV, Sexual and Reproductive Health in Botswana, Malawi, Zambia and Zimbabwe and you are presented with the same summary information that was already presented.

Not a great showing of transparency. The United States Congress can do that well, most of the time. Transparency isn’t well served by totals and bulk amounts. Those are more suited to concealment than transparency.

At a minimum, transparency requires disclosure of who the funds were disbursed to (one assume some entity in Zimbabwe) and to who that entity transferred funds, and so on, until we reach consumables or direct services. Along with the identities of every actor along that trail.

I first saw this in The Research Desk: UNESCO, SIPRI, and Searching iTunes by Gary Price.

Tracking NSA/CIA/FBI Agents Just Got Easier

Saturday, March 28th, 2015

I pointed out in The DEA is Stalking You! the widespread use of automobile license reading applications by the DEA. I also suggested that citizens start using their cellphones to take photos of people coming and going from DEA, CIA, FBI offices and posting them online.

The good news is that Big Data has come to the aid of citizens to track NSA/CIA/FBI, etc. agents.

Lisa Vaas writes in Entire Oakland Police license plate reader data set handed to journalist:

Howard Matis, a physicist who works at the Lawrence Berkeley National Laboratory in California, didn’t know that his local police department had license plate readers (LPRs).

But even if they did have LPRs (they do: they have 33 automated units), he wasn’t particularly worried about police capturing his movements.

Until, that is, he gave permission for Ars Technica’s Cyrus Farivar to get data about his own car and its movements around town.

The data is, after all, accessible via public records law.

Ars obtained the entire LPR dataset of the Oakland Police Department (OPD), including more than 4.6 million reads of over 1.1 million unique plates captured in just over 3 years.

Then, to make sense out of data originally provided in 18 Excel spreadsheets, each containing hundreds of thousands of lines, Ars hired a data visualization specialist who created a simple tool that allowed the publication to search any given plate and plot its locations on a map.

How cool is that!?

Of course, your mileage may vary as the original Ars article reports:

In August 2014, the American Civil Liberties Union and the Electronic Frontier Foundation lost a lawsuit to compel the Los Angeles Police Department and the Los Angeles Sheriff’s Department to hand over a mere week’s worth of all LPR data. That case is now on appeal.

The trick being that the state doesn’t mind invading your privacy but is very concerned with you not invading its privacy or knowing enough about its activities to be an informed member of the voting electorate.

If you believe that the government wants to keep information like license reading secret to protect the privacy of other citizens, you need to move to North Korea. I understand they have a very egalitarian society.

Of course these are license reading records collected by the state. Since automobiles are in public view, anyone could start collecting license plate numbers with locations. Now there’s a startup idea. Blanket the more important parts of D.C. inside the Beltway with private license readers. That would be a big data set with commercial value.

To give you an idea of the possibilities, visit Police License Plate Readers at You will find links to a wide variety of license plate reading solutions, including:


A fixed installation device from Vigilant Solutions.

You could wire something together but if you are serious about keeping track of the government keeping track on all of us, you should go with professional grade equipment. As well as adopt an activist response to government surveillance. Being concerned, frightened, “speaking truth to power,” etc. are as effective as doing nothing at all.

Think about a citizen group based license plate data collection. Possible discoveries could include government vehicles at local motels and massage parlors, explaining the donut gaze in some police officer eyes, meetings between regulators and the regulated, a whole range of governmental wrong doing is waiting to be discovered. Think about investing in a mobile license plate reader for your car today!

If you don’t like government surveillance, invite them into the fish bowl.

They have a lot more to hide that you do.

“We live in constant fear of upsetting the WH (White House).”

Wednesday, March 18th, 2015

Administration sets record for withholding government files by Ted Bridis.

From the post:

The Obama administration set a record again for censoring government files or outright denying access to them last year under the U.S. Freedom of Information Act, according to a new analysis of federal data by The Associated Press.

The government took longer to turn over files when it provided any, said more regularly that it couldn’t find documents and refused a record number of times to turn over files quickly that might be especially newsworthy.

It also acknowledged in nearly 1 in 3 cases that its initial decisions to withhold or censor records were improper under the law — but only when it was challenged.

Its backlog of unanswered requests at year’s end grew remarkably by 55 percent to more than 200,000. It also cut by 375, or about 9 percent, the number of full-time employees across government paid to look for records. That was the fewest number of employees working on the issue in five years.

The government’s new figures, published Tuesday, covered all requests to 100 federal agencies during fiscal 2014 under the Freedom of Information law, which is heralded globally as a model for transparent government. They showed that despite disappointments and failed promises by the White House to make meaningful improvements in the way it releases records, the law was more popular than ever. Citizens, journalists, businesses and others made a record 714,231 requests for information. The U.S. spent a record $434 million trying to keep up. It also spent about $28 million on lawyers’ fees to keep records secret.

Ted does a great job detailing the secretive and paranoid Obama White House up to and including a censored document that forgot to cover up:

“We live in constant fear of upsetting the WH (White House).”

Although I must confess that I don’t know if it is worse that President Obama and company are so non-transparent or that they lie about how transparent they are with such easy smiles. No shame, no embarrassment, they lie when the truth would do just as well.

Not that I think any other member of government does any better, but that is hardly an excuse.

The only legitimate solution that I see going forward are massive leaks from all parts of government. If you aren’t leaking, you are part of the problem.

What’s all the fuss about Dark Data?…

Wednesday, March 11th, 2015

What’s all the fuss about Dark Data? Big Data’s New Best Friend by Martyn Jones.

From the post:

Dark data, what is it and why all the fuss?

First, I’ll give you the short answer. The right dark data, just like its brother right Big Data, can be monetised – honest, guv! There’s loadsa money to be made from dark data by ‘them that want to’, and as value propositions go, seriously, what could be more attractive?

Let’s take a look at the market.

Gartner defines dark data as "the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes" (IT Glossary – Gartner)

Techopedia describes dark data as being data that is "found in log files and data archives stored within large enterprise class data storage locations. It includes all data objects and types that have yet to be analyzed for any business or competitive intelligence or aid in business decision making." (Techopedia – Cory Jannsen)

Cory also wrote that "IDC, a research firm, stated that up to 90 percent of big data is dark data."

In an interesting whitepaper from C2C Systems it was noted that "PST files and ZIP files account for nearly 90% of dark data by IDC Estimates." and that dark data is "Very simply, all those bits and pieces of data floating around in your environment that aren’t fully accounted for:" (Dark Data, Dark Email – C2C Systems)

Elsewhere, Charles Fiori defined dark data as "data whose existence is either unknown to a firm, known but inaccessible, too costly to access or inaccessible because of compliance concerns." (Shedding Light on Dark Data – Michael Shashoua)

Not quite the last insight, but in a piece published by Datameer, John Nicholson wrote that "Research firm IDC estimates that 90 percent of digital data is dark." And went on to state that "This dark data may come in the form of machine or sensor logs" (Shine Light on Dark Data – Joe Nicholson via Datameer)

Finally, Lug Bergman of NGDATA wrote this in a sponsored piece in Wired: "It" – dark data – "is different for each organization, but it is essentially data that is not being used to get a 360 degree view of a customer.

Well, I would say that 90% of 2.7 Zetabytes (as of last October) of data being dark is a reason to be concerned.

But like the Wizard of Oz, Martyn knows what you are lacking, a data inventory:

You don’t need a Chief Data Officer in order to be able to catalogue all your data assets. However, it is still good idea to have a reliable inventory of all your business data, including the euphemistically termed Big Data and dark data.

If you have such an inventory, you will know:

What you have, where it is, where it came from, what it is used in, what qualitative or quantitative value it may have, and how it relates to other data (including metadata) and the business.

Really? A data inventory? Relief to know the MDM (master data management) folks have been struggling for the past two decades for no reason. All they needed was a data inventory!

You might want to recall AnHai Doan’s observation for a single enterprise mapping project:

…the manual creation of semantic mappings has long been known to be extremely laborious and error-prone. For example, a recent project at the GTE telecommunications company sought to integrate 40 databases that have a total of 27,000 elements (i.e., attributes of relational tables) [LC00]. The project planners estimated that, without the database creators, just finding and documenting the semantic mappings among the elements would take more than 12 person years.

That’s right. One enterprise, 40 databases, 12 person years.

How that works out: PersonYears x 2.7 Zetabytes = ???, no one knows.

Oh, why did I lose the 90% as “dark data?” Simple enough, the data AnHai was mapping wasn’t entirely “dark.” At least it had headers that were meaningful to someone. Unstructured data has no headers at all.

What Martyn is missing?

What is known about data is the measure of its darkness, not usage.

But supplying opaque terms (all terms are opaque to someone) for data, only puts you into the AnHai situation. Either you enlist people who know the meanings of the terms and/or you create new meanings for them from scratch. Hopefully in the latter case you approximate the original meanings assigned to the terms.

If you want to improve on opaque terms, you need to provide alternative opaque terms that may be recognized by some future user instead of the primary opaque term you would use otherwise.

Make no mistake, it isn’t possible to escape opacity but you can increase your odds that your data can be useful at some future point in time. How many alternatives = some degree of future usefulness isn’t known.

So far as I know, the question hasn’t been researched. Every new set of opaque terms (read ontology, classification, controlled vocabulary) presents itself as possessing semantics for the ages. Given the number of such efforts, I find their confidence misplaced.

MI5 accused of covering up sexual abuse at boys’ home

Saturday, February 28th, 2015

MI5 accused of covering up sexual abuse at boys’ home by Vikram Dodd and Richard Norton-Taylor.

From the post:

MI5 is facing allegations it was complicit in the sexual abuse of children, the high court in Northern Ireland will hear on Tuesday.

Victims of the abuse are taking legal action to force a full independent inquiry with the power to compel witnesses to testify and the security service to hand over documents.

The case, in Belfast, is the first in court over the alleged cover-up of British state involvement at the Kincora children’s home in Northern Ireland in the 1970s. It is also the first of the recent sex abuse cases allegedly tying in the British state directly. Victims allege that the cover-up over Kincora has lasted decades.

The victims want the claims of state collusion investigated by an inquiry with full powers, such as the one set up into other sex abuse scandals chaired by the New Zealand judge Lowell Goddard.

Amnesty International branded Kincora “one of the biggest scandals of our age” and backed the victims’ calls for an inquiry with full powers: “There are longstanding claims that MI5 blocked one or more police investigations into Kincora in the 1970s in order to protect its own intelligence-gathering operation, a terrible indictment which raises the spectre of countless vulnerable boys having faced further years of brutal abuse.

It’s too early to claim victory but, Belfast boys’ home abuse victims win legal bid by Henry McDonald:

Residents of a notorious Northern Ireland boys’ home are to be allowed to challenge a decision to exclude it from the UK-wide inquiry into establishment paedophile rings.

A high court judge in Belfast on Tuesday granted a number of former inmates from the Kincora home a judicial review into the decision to keep this scandal out of the investigation, headed by judge Lowell Goddard from New Zealand.

The Kincora boys’ home has been linked to a paedophile ring, some of whose members were allegedly being blackmailed by MI5 and other branches of the security forces during the Troubles.

Until now, the home secretary, Theresa May, has resisted demands from men who were abused at the home – and Amnesty International – that the inquiry be widened to include Kincora.

The campaigners want to establish whether the security services turned a blind eye to the abuse and instead used it to compromise a number of extreme Ulster loyalists guilty of abusing boys at the home.

If you read carefully you will see the abuse victims have won the right to challenge the exclusion of the boys home from a UK wide investigation. A long way from forcing MI5 and other collaborators in sexual abuse of children to provide an accounting in the clear light of day.

Leaked documents, caches of spy cables, spy documents, always show agents of the government protecting war criminals, paedophiles, engaging in torture, including rape and other dishonorable conduct.

My question is why does the mainstream media honors the fiction that government secrets are meant to protect the public? Government secrets are used to protect guilty, the dishonorable and the despicable. What’s unclear about that?