Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

February 23, 2019

USA Confirms Hacking Only Viable Path To Transparency

Filed under: Government,Hacking,Transparency — Patrick Durusau @ 5:12 pm

After years of delays and democratic regression, USA releases weak open government plan from: E Pluribus Unum

From the post:

If the American public wants to see meaningful progress on transparency, accountability or ethics in U.S. government, it should call on Congress to act, not the Trump White House.
With little fanfare or notice, the United States of America has published a fourth National Action Plan for Open Government for the Open Government Partnership (OGP). The USA was automatically placed under review in January, but not because of two years of regression on transparency, accountability, and brazen corruption. The plan was was simply late, after failing to deliver a new plan for the multi-stakeholder initiative for years.
The new “national action plan” is notable for its lack of ambition, specificity or relevance to backsliding on democracy in the USA under the Trump administration.

Calling on the U.S. Congress for “…meaningful progress on transparency, accountability or ethics in U.S. government…” is a jest too cruel for laughter.

The current U.S. president has labored mightly to reduce government transparency but Congress is responsible for the crazy quilt laws enabling agencies to practice secrecy as their default position. Any sane system of transparency starts with transparency as the default setting, putting the burden of secrecy on those who desire it.

You can waste supporter dollars on yearly tilts at the transparency windmill in Congress, or bi-annual elections of members of Congress who promise (but don’t deliver) transparency, or presidential elections every four years. The resulting government structures will not be meaningfully more transparent at any future point in time.

If you see a viable (as in effective) alternative to hacking as a means of making government transparent, please leave it in a comment below.

September 13, 2018

OpenOversight: A public, searchable database of law enforcement officers

Filed under: Government,Transparency — Patrick Durusau @ 2:41 pm

OpenOversight: A public, searchable database of law enforcement officers

From the about page:

OpenOversight is a Lucy Parsons Labs project that aims to improve law enforcement visibility and transparency using public and crowdsourced data. We maintain databases, digital galleries, and profiles of individual law enforcement officers from departments across the United States that consolidate information including names, birthdates, mentions in news articles, salaries, and photographs.

The remarkable resource was forwarded to me by Camille Fassett.

Similar resources for members of legislatures, fracking companies, etc.?

August 21, 2018

Hacking: The hope for corporate and governmental transparency

Filed under: FOIA,Government,Hacking,Transparency — Patrick Durusau @ 1:31 pm

DEF CON 26 (2018) was the source of many headlines, including Hacking the US Midterms? It’s Child’s play., Hacking Medical Protocols to Change Vital Signs, and, Tesla Plans to Open-source its Vehicle Security Software, to say nothing of zero-day bugs and new attacks on old ones.

The most encouraging news, at least for transparency of corporations and governments comes from Breaking Badge – The DEFCON Crazy 8s by NodyaH.

“DEF CON City” is the location of a text-based adventure that can be solved only with interactions between 8 card types (depends on type of attendee) as well as hacking the cards themselves. The goal is to turn all the letters DEFCON green. There are resources at the end of the post, if you already have a badge.

NodyaH does a great job describing the starts, stops and re-tracing steps of participants as they rushed to break the badges.

It’s a fast moving tale so take a few minutes to read it. After having read it, can you name a corporate or governmental agency that would be more difficult to hack than the DEFCON badges?

The solution to grudging transparency and documents that mis-led more than they inform, is not more FOIA. Transparency requires hackers who peel corporate and government agencies like navel oranges.

Are you one of them or aspire to be?

Keep up with DEFCON!

July 18, 2018

Self-Help Transparency – Smoke Loader

Filed under: Cybersecurity,Malware,Transparency — Patrick Durusau @ 8:18 pm

Dissecting Smoke Loader by Michał Praszmo.

From the post:

Smoke Loader (also known as Dofoil) is a relatively small, modular bot that is mainly used to drop various malware families.

Even though it’s designed to drop other malware, it has some pretty hefty malware-like capabilities on its own.

Despite being quite old, it’s still going strong, recently being dropped from RigEK and MalSpam campaigns.

In this article we’ll see how Smoke Loader unpacks itself and interacts with the C2 server.
…

You can go the Freedom of Information Act (FOIA) route to become an “informed citizen,” provided you don’t mind:

  • Indeterminate exchanges to clarify your request
  • Delays and fees by agencies
  • Exemptions
  • Review and editing of documents by those most interested in non-disclosure

If you had access to the agency’s files:

  • No need to clarify your request
  • No delays or fees by the agency
  • No exemptions from disclosure
  • No review and editing of requested documents to prevent disclosure

Not to mention that self-help transparency saves the agency staff time and other resources in answering your request.

The other advantage of self-help transparency is that it works with political PACs, foreign governments, corporations and a host of other groups and institutions with no FOIA traditions.

All of those are incentives for closely attending to this blog post on the Smoke Loader.

Enjoy!

May 4, 2018

Win MuckRock requests and swag!

Filed under: Government,MuckRock,Transparency — Patrick Durusau @ 4:23 pm

Help analyze Donald Rumsfeld’s memos and win MuckRock requests and swag by Michael Morisy.

From the post:

In January, thanks to a five-year fight by the National Security Archive, the Pentagon began releasing massive troves of former Secretary of Defense Donald Rumsfeld’s memos. The memos were so copious that they developed their own legendary status within the Armed Forces.

Rumsfeld himself describes them:

When I returned to the Pentagon in 2001, I continued writing the short memos that had been nicknamed “snowflakes” some years ago. They quickly became a system of communication with the many employees of DoD, as I would initiate a topic with a short memo to the relevant person, who would in turn provide research, background, or a course of action as necessary. In the digital age it was much easier to keep the originals on file so I could track their progress. They quickly grew in number from mere flurries to a veritable blizzard.

The term “snowflake” covers a range of communications, from notes to myself on topics I found interesting, to extended instructions to my associates, to simple requests for a haircut. There was no set template; some are several pages and some just a few words. They were all conceived individually and I had never considered them as a set until I started work on the memoir. I then found that when reviewed together, they give a remarkable sense of the variety of topics that are confronted by a secretary of defense.

Now you can explore the early days of the War on Terror – and potentially earn free MuckRock requests and even swag – by helping analyze what was in them, surfacing the most interesting and historically important memos and sharing the results with everyone.

…

MuckRock is offering prizes so jump to Morisy’s post and get started.

Enjoy!

January 4, 2018

Helping Google Achieve Transparency – Wage Discrimination

Filed under: sexism,Transparency — Patrick Durusau @ 8:36 pm

Google faces new discrimination charge: paying female teachers less than men by Sam Levin.

From the post:

Google, which has been accused of systematically underpaying female engineers and other workers, is now facing allegations that it discriminated against women who taught employees’ children at the company’s childcare center.

A former employee, Heidi Lamar, is alleging in a complaint that female teachers were paid lower salaries than men with fewer qualifications doing the same job.

Lamar, who worked at Google for four years before quitting in 2017, alleged that the technology company employed roughly 147 women and three men as pre-school teachers, but that two of those men were granted higher starting salaries than nearly all of the women.

…

Google did not respond to the Guardian’s request for data on its hiring practices of teachers.
…

As Levin reports, Google is beside itself with denials and other fact free claims for which it offers no data.

If there was no wage discrimination, Google could release all of its payroll and related data and silence all of its critics at once.

Google has chosen to not silence its critics with facts known only to Google.

Google needs help seeing the value of transparency to answer charges of wage discrimination.

Will you be the one that helps Google realize the value of transparency?

December 14, 2017

98% Fail Rate on Privileged Accounts – Transparency in 2018

Filed under: Cybersecurity,Government,Government Data,Security,Transparency — Patrick Durusau @ 9:55 am

Half of companies fail to tell customers about data breaches, claims study by Nicholas Fearn.

From the post:

Half of organisations don’t bother telling customers when their personal information might have been compromised following a cyber attack, according to a new study.

The latest survey from security firm CyberArk comes with the full implementation of the European Union General Data Protection Regulation (GDPR) just months away.

Organisations that fail to notify the relevant data protection authorities of a breach within 72 hours of finding it can expect to face crippling fines of up to four per cent of turnover – with companies trying to hide breaches likely to be hit with the biggest punishments.

The findings have been published in the second iteration the CyberArk Global Advanced Threat Landscape Report 2018, which explores business leaders’ attitudes towards IT security and data protection.

The survey found that, overall, security “does not translate into accountability”. Some 46 per cent of organisations struggle to stop every attempt to breach their IT infrastructure.

And 63 per cent of business leaders acknowledge that their companies are vulnerable to attacks, such as phishing. Despite this concern, 49 per cent of organisations don’t have the right knowledge about security policies.
…

You can download the report cited in Fearn’s post at: Cyberark Global Advanced Threat Landscape Report 2018: The Business View of Security.

If you think that report has implications for involuntary/inadvertent transparency, Cyberark Global Advanced Threat Landscape Report 2018: Focus on DevOps, reports this gem:

…
It’s not just that businesses underestimate threats. As noted above, they also do not seem to fully understand where privileged accounts and secrets exist. When asked which IT environments and devices contain privileged accounts and secrets, responses (IT decision maker and DevOps/app developer respondents) were at odds with the claim that most businesses have implemented a privileged account security solution. A massive 98% did not select at least one of the ‘containers’, ‘microservices’, ‘CI/CD tools’, ‘cloud environments’ or ‘source code repositories’ options. At the risk of repetition, privileged accounts and secrets are stored in all of these entities.
…

A fail rate of 98% on identifying “privileged accounts and secrets?”

Reports like this make you wonder about the clamor for transparency of organizations and governments. Why bother?

Information in 2018 is kept secure by a lack of interest in collecting it.

Remember that for your next transparency discussion.

December 9, 2017

Zero Days, Thousands of Nights [Zero-day – 6.9 Year Average Life Expectancy]

Filed under: Cybersecurity,Government,Security,Transparency — Patrick Durusau @ 11:41 am

Zero Days, Thousands of Nights – The Life and Times of Zero-Day Vulnerabilities and Their Exploits by Lillian Ablon, Timothy Bogart.

From the post:

Zero-day vulnerabilities — software vulnerabilities for which no patch or fix has been publicly released — and their exploits are useful in cyber operations — whether by criminals, militaries, or governments — as well as in defensive and academic settings.

This report provides findings from real-world zero-day vulnerability and exploit data that could augment conventional proxy examples and expert opinion, complement current efforts to create a framework for deciding whether to disclose or retain a cache of zero-day vulnerabilities and exploits, inform ongoing policy debates regarding stockpiling and vulnerability disclosure, and add extra context for those examining the implications and resulting liability of attacks and data breaches for U.S. consumers, companies, insurers, and for the civil justice system broadly.

The authors provide insights about the zero-day vulnerability research and exploit development industry; give information on what proportion of zero-day vulnerabilities are alive (undisclosed), dead (known), or somewhere in between; and establish some baseline metrics regarding the average lifespan of zero-day vulnerabilities, the likelihood of another party discovering a vulnerability within a given time period, and the time and costs involved in developing an exploit for a zero-day vulnerability.

…

Longevity and Discovery by Others

  • Zero-day exploits and their underlying vulnerabilities have a rather long average life expectancy (6.9 years). Only 25 percent of vulnerabilities do not survive to 1.51 years, and only 25 percent live more than 9.5 years.
  • No vulnerability characteristics indicated a long or short life; however, future analyses may want to examine Linux versus other platform types, the similarity of open and closed source code, and exploit class type.
  • For a given stockpile of zero-day vulnerabilities, after a year, approximately 5.7 percent have been publicly discovered and disclosed by another entity.

…

Rand researchers Ablon and Bogart attempt to interject facts into the debate over stockpiling zero-day vulnerabilities. It a great read, even though I doubt policy decisions over zero-day stockpiling will be fact-driven.

As an advocate of inadvertent or involuntary transparency (is there any other honest kind?), I take heart from the 6.9 year average life expectancy of zero-day exploits.

Researchers should take encouragement from the finding that within a given year, only 5.7 of all zero-days vulnerability discoveries overlap. That is 94.3% of zero-day discoveries are unique. That indicates to me vulnerabilities are left undiscovered every year.

Voluntary transparency, like presidential press conferences, is an opportunity to shape and manipulate your opinions. Zero-day vulnerabilities, on the other hand, can empower honest/involuntary transparency.

Won’t you help?

August 25, 2017

Good News For Transparency Phishers

Filed under: Cybersecurity,Government,Phishing for Leaks,Security,Transparency — Patrick Durusau @ 4:45 pm

If you are a transparency phisher, Shaun Waterman has encouraging news for you in: Most large companies don’t use standard email security to combat spoofing.

From the post:

Only a third of Fortune 500 companies deploy DMARC, a widely-backed best-practice security measure to defeat spoofing — forged emails sent by hackers — and fewer than one-in-10 switch it on, according to a new survey.

The survey, carried out by email security company Agari via an exhaustive search of public Internet records, measured the use of Domain-based Message Authentication, Reporting and Conformance, or DMARC.

“It is unconscionable that only eight percent of the Fortune 500, and even fewer [U.S.] government organizations, are protecting the public against email domain spoofing,” said Patrick Peterson, founder and executive chairman, Agari. A similar survey of federal government agencies earlier this month, by the Global Cyber Alliance, found fewer than five percent of federal domains were protected by switched-on DMARC.

The Agari survey found adoption rates similarly low among companies in the United Kingdom’s FTSE and Australia’s ASX 100.

DMARC is the industry standard measure to prevent hackers from spoofing emails — making their messages appear as if they’re sent by someone else. Spoofing is the basis of phishing, a major form of both cybercrime and cyber-espionage, in which an email appearing to a come from a trusted company like a bank or government agency contains malicious links, directing readers to a fake site which will steal their login and password when they sign on.
…

Only eight (8) percent of the Fortune 500 and less than five (5) percent of federal (US) domains have DMARC protection.

I expect DMARC protection rates fall rapidly outside the Fortune 500 and non-federal government domains.

If you are interested in transparency, for private companies or government agencies, the lack of DMARC adoption and use presents a golden opportunity to obtain otherwise hidden information.

As always, who you are and who you are working for, determines the legality of any phishing effort. Consult with an attorney concerning your legal rights and obligations.

July 7, 2017

New York Times, Fact Checking and Dacosta’s First OpEd

Filed under: Government,Journalism,News,Politics,Reporting,Transparency — Patrick Durusau @ 4:44 pm

Cutbacks on editors/fact-checking at the New York Times came at an unfortunate time for Marc Dacosta‘s first OpEd, The President Wants to Keep Us in the Dark (New York Times, 28 June 2017).

DaCosta decries the lack of TV cameras at several recent White House press briefings. Any proof the lack of TV cameras altered the information available to reporters covering the briefings? Here’s DaCosta on that point:

…
But the truth is that the decision to prevent the press secretary’s comments on the day’s most pressing matters from being televised is an affront to the spirit of an open and participatory government. It’s especially chilling in a country governed by a Constitution whose very First Amendment protects the freedom of the press.

Unfortunately, the slow death of the daily press briefing is only part of a larger assault by the Trump administration on a precious public resource: information.
…

DaCosta’s implied answer is no, a lack of TV cameras resulted in no diminishing of information from the press conference. But, his hyperbole gland kicks in, then he cites disjointed events claimed to diminish public access to information.

For example, Trump’s non-publication of visitor records:

…
Immediately after Mr. Trump took office, the administration stopped publishing daily White House visitor records, reversing a practice established by President Obama detailing the six million appointments he and administration officials took at the White House during his eight years in office. Who is Mr. Trump meeting with today? What about Mr. Bannon? Good luck finding out.
…

Really? Mark J. Rozell summarizes the “detailing the six million appointments he and administration officials took…” this way:

…
Obama’s action clearly violated his own pledge of transparency and an outpouring of criticism of his action somewhat made a difference. He later reversed his position when he announced that indeed the White House visitor logs would be made public after all.

Unfortunately, the president decided only to release lengthy lists of names, with no mention of the purpose of White House visits or even differentiation between tourists and people consulted on policy development.

This action enabled the Obama White House to appear to be promoting openness while providing no substantively useful information. If the visitor log listed “Michael Jordan,” there was no way to tell if the basketball great or a same-named industry lobbyist was the person at the White House that day and the layers of inquiry required to get that information were onerous. But largely because the president had appeared to have reversed himself in reaction to criticism for lack of transparency, the controversy died down, though it should not have.

Much of the current reaction to President Trump’s decision has contrasted that with the action of his predecessor, and claimed that Obama had set the proper standard by opening the books. The reality is different though, as Obama’s action set no standard at all for transparency.
…(Trump should open White House visitor logs, but don’t flatter Obama, The Hill, 18 April 2017)

That last line on White House visitor records under Obama is worth repeating:

The reality is different though, as Obama’s action set no standard at all for transparency.

Obama-style opaqueness would not answer the questions:

Who is Mr. Trump meeting with today? What about Mr. Bannon? [Questions by DaCosta.]

A fact-checker and/or editor at the New York Times knew that answer (hint to NYT management).

Even more disappointing is the failure of DaCosta, as the co-founder of Engima, to bring any data to a claim that White House press briefings are of value.

One way to test the value of White House press briefings is to extract the “facts” announced during the briefing and compare those to media reports in the prior twenty-four hours.

If DaCosta thought of such a test, the reason it went unperformed isn’t hard to guess:

…
The Senate had just released details of a health care plan that would deprive 22 million Americans of health insurance, and President Trump announced that he did not, as he had previously hinted, surreptitiously record his conversations with James Comey, the former F.B.I. director.
… (DaCosta)

First, a presidential press briefing isn’t an organ for the US Senate and second, Trump had already tweeted the news about not recording his conversations with James Comey. None of those “facts” broke at the presidential press briefing.

DaCosta is 0 for 2 for new facts at that press conference.

I offer no defense for the current administration’s lack of transparency, but fact-free and factually wrong claims against it don’t advance DaCosta’s cause:

…
Differences of belief and opinion are inseparable from the democratic process, but when the facts are in dispute or, worse, erased altogether, public debate risks breaking down. To have a free and democratic society we all need a common and shared context of facts to draw from. Facts or data will themselves never solve any problem. But without them, finding solutions to our common problems is impossible.
…

We should all expect better of President Trump, the New York Times and Marc DaCosta (@marc_dacosta).

June 29, 2017

Targeting Data: Law Firms

Filed under: Cybersecurity,Security,Transparency — Patrick Durusau @ 12:58 pm

Law Firm Cyber Security Scorecard

From the webpage:

If you believe your law firm is cyber secure, we recommend that you download this report. We believe you will be quite surprised at the state the law firm industry as it relates to cyber security. This report demonstrates three key findings. First, law firms are woefully insecure. Second, billions of dollars are at-risk from corporate and government clients. Third, there exists little transparency between firms and clients about this issue.

How do we know this? LOGICFORCE surveyed and assessed over 200 law firms, ranging in size from 1 to 450+ total attorneys, located throughout the United States, working in a full complement of practice areas. The insights in this study come from critical data points gathered through authorized collection of anonymized LOGICFORCE system monitoring data, responses to client surveys, our proprietary SYNTHESIS E-IT SECURE™ assessments and published industry information.

Key Findings:

  • Every law firm assessed was targeted for confidential client data in 2016-2017. Approximately 40% did not know they were breached.
  • We see consistent evidence that cyber attacks on law firms are non-discriminatory. Size and revenues don’t seem to matter.
  • Only 23% of firms have cybersecurity insurance policies.
  • 95% of assessments conducted by LOGICFORCE show firms are not compliant with their data governance and cyber security policies.
  • 100% of those firms are not compliant with their client’s policy standards.

LOGICFORCE does not want your law firm to make headlines for the wrong reasons. Download this report now so you can understand your risks and begin to take appropriate action.

The “full report,” which I downloaded, is a sales brochure for LOGICFORCE and not a detailed technical analysis. (12 pages including cover front and back.)

It signals the general cyber vulnerability of law firms, but not so much of what works, what doesn’t, security by practice area, etc.

The Panama Papers provided a start on much needed transparency for governments and the super wealthy. That start was the result of a breach at one (1) law firm.

Martindale.com lists over one million (1,000,000) lawyers and law firms from around the world.

The Panama Papers and following fallout were the result of breaching 1 out of 1,000,000+ lawyers and law firms.

Do you ever wonder what lies hidden in the remaining 1,000,000+ lawyers and law firms?

According to Logicforce, that desire isn’t a difficult one to satisfy.

June 19, 2017

Key DoD Officials – September 1947 to June 2017

Filed under: Government,History,Transparency — Patrick Durusau @ 8:20 pm

While looking for a particular Department of Defense official, I stumbled on: Department of Defense Key Officials September 1947–June 2017.

Yes, almost seventy (70) years worth of key office holders at the DoD. It’s eighty (80) pages long, produced by the Historical Office of the Secretary of Defense.

One potential use, aside from giving historical military fiction a ring of authenticity, would be to use this as a starting set of entities to trace through the development of the military/industrial complex.

Everyone, including me, refers to the military/industrial complex as though it is a separate entity, over there somewhere.

But as everyone discovered with the Panama Papers, however tangled and corrupt even world-wide organizations can be, we have the technology to untangle those knots and to shine bright lights into obscure corners.

Interested?

May 23, 2017

Fiscal Year 2018 Budget

Filed under: Government,Government Data,Politics,Transparency — Patrick Durusau @ 7:23 pm

Fiscal Year 2018 Budget.

In the best pay-to-play tradition, the Government Printing Office (GPO) has these volumes for sale:

America First: A Budget Blueprint To Make America Great Again By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00719-9 ISBN: 9780160937620. Price: $10.00.

Budget of the United States Government, FY 2018 (Paperback Book) By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00723-7 ISBN: 9780160939228. Price: $38.00.

Appendix, Budget of the United States Government, FY 2018 By: Executive Office of the President, Office of Management and Budget GPO Stock # 041-001-00720-2 ISBN: 9780160939334. Price: $79.00.

Budget of the United States Government, FY 2018 (CD-ROM) By: Executive Office of the President, Office of Management and Budget GPO Stock # 041-001-00722-9 ISBN: 9780160939358. Price: $29.00.

Analytical Perspectives, Budget of the United States Government, FY 2018 By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00721-1 ISBN: 9780160939341. Price: $56.00.

Major Savings and Reforms: Budget of the United States Government, Fiscal Year 2018 By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00724-5 ISBN: 9780160939457. Price: $35.00.

If someone doesn’t beat me to it (very likely), I will be either uploading the CD-ROM and/or pointing you to a location with the contents of the CD-ROM.

As citizens, whether you voted or not, you should have the opportunity to verify news accounts, charges and counter-charges with regard to the budget.

March 29, 2017

What’s Up With Data Padding? (Regulations.gov)

Filed under: Data Quality,Fair Use,Government Data,Intellectual Property (IP),Transparency — Patrick Durusau @ 10:41 am

I forgot to mention in Copyright Troll Hunting – 92,398 Possibles -> 146 Possibles that while using LibreOffice, I deleted a large number of either N/A only or columns not relevant for troll-mining.zip.

Except as otherwise noted, after removal of “no last name,” these fields had N/A for all records except as noted:

  1. L – Implementation Date
  2. M – Effective Date
  3. N – Related RINs
  4. O – Document SubType (Comment(s))
  5. P – Subject
  6. Q – Abstract
  7. R – Status – (Posted, except for 2)
  8. S – Source Citation
  9. T – OMB Approval Number
  10. U – FR Citation
  11. V – Federal Register Number (8 exceptions)
  12. W – Start End Page (8 exceptions)
  13. X – Special Instructions
  14. Y – Legacy ID
  15. Z – Post Mark Date
  16. AA – File Type (1 docx)
  17. AB – Number of Pages
  18. AC – Paper Width
  19. AD – Paper Length
  20. AE – Exhibit Type
  21. AF – Exhibit Location
  22. AG – Document Field_1
  23. AH – Document Field_2

Regulations.gov, not the Copyright Office, is responsible for the collection and management of comments, including the bulked up export of comments.

From the state of the records, one suspects the “bulking up” is NOT an artifact of the export but represents the storage of each record.

One way to test that theory would be a query on the noise fields via the API for Regulations.gov.

The documentation for the API is out-dated, the Field References documentation lacks the Document Detail (field AI), which contains the URL to access the comment.

The closest thing I could find was:

fileFormats Formats of the document, included as URLs to download from the API

How easy/hard it will be to download attachments isn’t clear.

BTW, the comment pages themselves are seriously puffed up. Take https://www.regulations.gov/document?D=COLC-2015-0013-52236.

Saved to disk: 148.6 KB.

Content of the comment: 2.5 KB.

The content of the comment is 1.6 % of the delivered webpage.

It must have taken serious effort to achieve a 98.4% noise to 1.6% signal ratio.

How transparent is data when you have to mine for the 1.6% that is actual content?

March 26, 2017

Transparency can have a prophylactic effect

Filed under: Journalism,News,Reporting,Transparency — Patrick Durusau @ 4:57 pm

Farai Chideya set out to explore:

…who reported the 2016 election, and whether political teams’ race and gender diversity had any impact on newsrooms.

That’s an important question and Chideya certainly has the qualifications and support ( fellow at Harvard’s Shorenstein Center on Media, Politics and Public Policy) to pursue it.

One problem. For reasons best known to themselves, numerous media organizations refuse to provide diversity date. (full stop)

…
But the most important data point for this project—numbers from newsrooms on their 2016 political team staffing—has been the hardest to collect because very few managers or business-side staff are willing to disclose their data. One company admitted off the record that they were not responding to diversity requests, period. The Wall Street Journal provided the statement that it “declined to provide specific personnel information.” An organization sent numbers for its corporate parent company, whose size is approximately a thousand times the size of the entire news team, let alone the political team. Another news manager promised verbally to cooperate with the inquiry, but upon repeated follow up completely ghosted.
…

Concealment wasn’t the uniform response as Chideya makes clear but useful responses were so few and far. Enough so to provoke her post.

She captures my sentiments writing:

…
If we journalists can’t turn as unsparing a gaze on ourselves as we do on others, it speaks poorly for us and the credibility of our profession. If the press lauds itself for demanding transparency from government but cannot achieve transparency in its newsrooms, that is cowardice. If we say we can cover all of America with representatives of only a few types of communities, we may win battles but lose the war to keep news relevant to a broad segment of Americans. This is as strong a business argument as a moral argument.
…

If you need additional motivation, be aware that Chideya is proceeding in the face of non-cooperation and when her study is published, there will be a list of who has been naughty and nice.

Here’s how to self-report:

…
Whether or not you are a news organization I’ve already contacted, please email me at Farai_Chideya@hks.harvard.edu

For the purposes of the reporting, I’m looking for a race/gender count of 2016-cycle political staffers—full-time or at least 25-hour-per-week contract workers (but not freelancers paid by the story). People come and go during the election season, but these should be people who spent at least six months covering the election between September 2015 and November 2016.

If you want to add to the data you disclose, you can include separate counts for freelancers; or for staff who worked on politics less than six months of the cycle, but those should be broken out separately.

Want bonus points? Produce an org chart showing how your staff diversity played out across the ranks of reporters and editors. Feel free to annotate for self-reported class background or other metrics if you want, too. But race and gender are the minimum.

We’d like on-the-record numbers and interviews from people who we can use as sources in the report: managers, corporate communications staff, anyone authorized to speak on behalf of the newsroom. Please indicate if you are speaking on the record and in what role.

Because we are not getting this information, in many cases, we also welcome interviews and information on background. That is, if you are a staffer and can provide information, please do, and tell us who you are and that you don’t want to be quoted or cited. We’ll take what you provide to us into account as we do our research, but obviously it can’t be the final word. You could also offer quotes about the topic on the record, and your assessment of staff diversity on background.

As we conclude the report, we will release information on who has provided information, and who it was requested from who did not.

Self-reporting beats being on the naughty list and/or your diversity information extracted by a ham-handed hacker who damages your systems as well.

Who knew? Transparency can have a prophylactic effect.

See Chideya’s full post at: One question that turns courageous journalists into cowards

November 21, 2016

MuckRock Needs Volunteers (9 states in particular)

Filed under: FOIA,Government,Politics,Transparency — Patrick Durusau @ 2:02 pm

MuckRock needs your help to keep filing in all 50 states by Beryl Lipton.

From the post:

Election time excitement got you feeling a little more patriotic than usual? Looking for a way to help but not sure you have the time? Well, MuckRock is looking for a few good people to do a big service requiring little effort: serve as our resident proxies.

…

A few states have put up barriers at their borders, limiting required disclosure and response to requests to only residents. One more thing added to the regular rigamarole of requesting public records, it’s huge block to comparative studies and useful, outside accountability.

This is where you come in.

proxymap-460

We’re looking for volunteers in the ten states that can whip out their residency requirements whenever they get the chance:

  • Alabama
  • Arkansas
  • Georgia
  • Missouri
  • Montana.
  • New Hampshire
  • New Jersey
  • Tennessee
  • Virginia

As a MuckRock proxy requester, you’ll serve as the in-state request representative, allowing requests to be submitted in your name and enabling others to continue to demand accountability. In exchange, you’ll get your own Professional MuckRock account – 20 requests a month and all that comes with them – and the gratitude of the transparency community.

Interested in helping the cause? Let us know at info@muckrock.com, or via the from below.

…

Despite my view that government disclosures are previously undisclosed government lies, I have volunteered for this project.

Depending on where you reside, you should too and/or contribute to support MuckRock.

November 11, 2016

“connecting the dots” requires dots (Support Michael Best)

Filed under: Government Data,Politics,Transparency — Patrick Durusau @ 9:45 pm

Michael Best is creating a massive archive of government documents.

From the post:

Since 2015, I’ve published millions of government documents (about 10% of the text items on the Internet Archive, with some items containing thousands of documents) and terabytes of data; but in order to keep going, I need your help. Since I’ve gotten started, no outlet has matched the number of government documents that I’ve published and made freely available. The only non-governmental publisher that rivals the size and scope of the government files I’ve uploaded is WikiLeaks. While I analyze and write about these documents, I consider publishing them to be more important because it enables and empowers an entire generation of journalists, researchers and students of history.

I’ve also pressured government agencies into making their documents more widely available. This includes the more than 13,000,000 pages of CIA documents that are being put online soon, partially in response to my Kickstarter and publishing efforts. These documents are coming from CREST, which is a special CIA database of declassified records. Currently, it can only be accessed from four computers in the world, all of them just outside of Washington D.C. These records, which represent more than 3/4 of a million CIA files, will soon be more accessible than ever – but even once that’s done, there’s a lot more work left to do.
…

Question: Do you want a transparent and accountable Trump presidency?

Potential Answers include:

1) Yes, but I’m going to spend time and resources hyper-ventilating with others and roaming the streets.

2) Yes, and I’m going to support Michael Best and FOIA efforts.

Governments, even Trump’s presidency, don’t spring from ocean foam.

1024px-sandro_botticelli_-_la_nascita_di_venere_-_google_art_project_-_edited-460

The people chosen fill cabinet and other posts have history, in many cases government history.

For example, I heard a rumor today that Ed Meese, a former government crime lord, is on the Trump transition team. Hell, I thought he was dead.

Michael’s efforts produce the dots that connect past events, places, people, and even present administrations.

The dots Michael produces may support your expose, winning story and/or indictment.

Are you in or out?

The TPP Is Dead! Really Most Sincerely Dead! (Celebration Is In Order!)

Filed under: Government,Intellectual Property (IP),Transparency — Patrick Durusau @ 5:47 pm

Obama Administration Gives Up on Pacific Trade Deal by William Mauldin.

From the post:

The Obama administration on Friday gave up all hope of enacting its sweeping Pacific trade agreement, a pact designed to preserve U.S. economic influence in fast-growing Asia that was buried by a wave of antitrade political sentiment that culminated with Tuesday’s presidential election….

Yes!

I have ranted about the largely secret Trans-Pacific Partnership (TPP) trade agreement on several occasions.

Negotiated entirely in secret and even worse, designed to be kept secret from the citizens of signing countries, among the worse provisions (there were several), were those enabling investors to sue sovereign countries if laws diminished their investments.

I don’t know, like health warnings on cigarettes for example.

With the election of Donald Trump, I should say president-elect Donald Trump, the TPP is dead. (full stop)

As the proverb says:

It’s an ill wind that blows nobody any good.

Whatever your feelings about president-elect Donald Trump and any of his decisions/policies as president, the defeat of the TPP is one for the win column.

Hazards and dangers lie ahead, just as they would for any presidency, but take a moment to appreciate this win.

November 4, 2016

Weakly Weaponized Open Data

Filed under: Open Data,Transparency,Weaponized Open Data — Patrick Durusau @ 7:06 pm

Berners-Lee raises spectre of weaponized open data by Bill Camarda.

From the post:

open-data-sabotage-460

Practically everybody loves open data, ie “data that anyone can access, use or share”. And nobody loves it more than Tim Berners-Lee, creator of the World Wide Web, and co-founder of the Open Data Institute (ODI).

Berners-Lee and his ODI colleagues have spent years passionately evangelizing governments and companies to publicly release their non-personal data for use to improve communities.

So when he recently told the Guardian that hackers could use open data to create societal chaos, it might have been this year’s most surprising “man bites dog” news story.

What’s going on here? The growing fear of data sabotage, that’s what.
…

Bill focuses on the manipulation and/or planting of false data, which could result in massive traffic jams, changes in market prices, etc.

In fact, Berners-Lee says in the original Guardian story:

…
“If you falsify government data then there are all kinds of ways that you could get financial gain, so yes,” he said, “it’s important that even though people think about open data as not a big security problem, it is from the point of view of being accurate.”

He added: “I suppose it’s not as exciting as personal data for hackers to get into because it’s public.”
…

Disruptive to some, profitable to others, but what should be called weakly weaponized open data.

Here is one instance of strongly weaponized open data.

Scenario: We Don’t Need No Water, Let The Motherfucker Burn

The United States is currently experiencing a continuing drought. From the U.S. Drought Monitor:

drought-us-460

Keying on the solid red color around Atlanta, GA, Fire Weather, a service of the National Weather Service, estimates the potential impact of fires near Atlanta:

atlanta-fire-weather-prediction-clip

Impacted by a general conflagration around Atlanta:

Population: 2,783,418
Airports: 38
Miles of Interstate: 556
Miles of Rail: 2399
Parks: 4
Area: 27,707 Sq. Miles

Pipelines are missing from the list of impacts. For that, consult the National Pipeline Mapping System where even a public login reveals:

fulton-pipelines-460

The red lines are hazardous liquid pipelines, blue lines are gas transmission pipelines, the yellow lines outline Fulton County.

We have located a likely place for a forest fire, have some details on its probable impact and a rough idea of gas and other pipelines in the prospective burn area.

Oh, we need a source of ignition. Road flares anyone?

wsdot-flares-460-clip

From the WSDOT, Winter Driving Supply Checklist. Emergency kits with flares are available at box stores and online.

Bottom line:

Intentional forest fires can be planned from public data sources. Governments gratuitously suggest non-suspicious methods of transporting forest fire starting materials.

Details I have elided over, such as evacuation routes, fire watch stations, drones as fire starters, fire histories, public events, plus greater detail from the resources cited, are all available from public sources.

What are your Weaponized Open Data risks?

November 3, 2016

Attn: Secrecy Bed-Wetters! All Five Volumes of Bay of Pigs History Released!

Filed under: FOIA,Government,Government Data,Transparency — Patrick Durusau @ 4:06 pm

Hand-wringers and bed-wetters who use government secrecy to hide incompetence and errors will sleep less easy tonight.

All Five Volumes of Bay of Pigs History Released and Together at Last: FRINFORMSUM 11/3/2016 by Lauren Harper.

From the post:

After more than twenty years, it appears that fear of exposing the Agency’s dirty linen, rather than any significant security information, is what prompts continued denial of requests for release of these records. Although this volume may do nothing to modify that position, hopefully it does put one of the nastiest internal power struggles into proper perspective for the Agency’s own record.” This is according to Agency historian Jack Pfeiffer, author of the CIA’s long-contested Volume V of its official history of the Bay of Pigs invasion that was released after years of work by the National Security Archive to win the volume’s release. Chief CIA Historian David Robarge states in the cover letter announcing the document’s release that the agency is “releasing this draft volume today because recent 2016 changes in the Freedom of Information Act (FOIA) requires us to release some drafts that are responsive to FOIA requests if they are more than 25 years old.” This improvement – codified by the FOIA Improvement Act of 2016 – came directly from the National Security Archive’s years of litigation.

The CIA argued in court for years – backed by Department of Justice lawyers – that the release of this volume would “confuse the public.” National Security Archive Director Tom Blanton says, “Now the public gets to decide for itself how confusing the CIA can be. How many thousands of taxpayer dollars were wasted trying to hide a CIA historian’s opinion that the Bay of Pigs aftermath degenerated into a nasty internal power struggle?”

To read all five volumes of the CIA’s Official History of the Bay of Pigs Operation – together at last – visit the National Security Archive’s website.

…

Even the CIA’s own retelling of the story, The Bay of Pigs Invasion, ends with a chilling reminder for all “rebels” being presently supported by the United States.

…
Brigade 2506’s pleas for air and naval support were refused at the highest US Government levels, although several CIA contract pilots dropped munitions and supplies, resulting in the deaths of four of them: Pete Ray, Leo Baker, Riley Shamburger, and Wade Gray.

Kennedy refused to authorize any extension beyond the hour granted. To this day, there has been no resolution as to what caused this discrepancy in timing.

Without direct air support—no artillery and no weapons—and completely outnumbered by Castro’s forces, members of the Brigade either surrendered or returned to the turquoise water from which they had come.

Two American destroyers attempted to move into the Bay of Pigs to evacuate these members, but gunfire from Cuban forces made that impossible.

In the following days, US entities continued to monitor the waters surrounding the bay in search of survivors, with only a handful being rescued. A few members of the Brigade managed to escape and went into hiding, but soon surrendered due to a lack of food and water. When all was said and done, more than seventy-five percent of Brigade 2506 ended up in Cuban prisons.

100% captured or killed. There’s an example of US support.

In a less media savvy time, the US did pay $53 million (in 1962 dollars, about $424 million today) for the release of 1113 members of Brigade 2506.

Another important fact is that fifty-seven (57) years of delay enabled the participants to escape censure and/or a trip to the gallows for their misdeeds and crimes.

Let’s not let that happen with the full CIA Torture Report. Even the sanitized 6,700 page version would be useful. More so the documents upon which it was based.

All of that exists somewhere. We lack a person with access and moral courage to inform their fellow citizens of the full truth about the CIA torture program. So far.

Update: Michael Best, NatSecGeek advises CIA Histories has the most complete CIA history collection. Thanks Michael!

September 12, 2016

Inside the fight to reveal the CIA’s torture secrets [Support The Guardian]

Filed under: Government,Government Data,Journalism,News,Politics,Reporting,Transparency — Patrick Durusau @ 3:19 pm

Inside the fight to reveal the CIA’s torture secrets by Spencer Ackerman.

Part one: Crossing the bridge

Part two: A constitutional crisis

Part three: The aftermath

Ackerman captures the drama of a failed attempt by the United States Senate to exercise oversight on the Central Intelligence Agency (CIA) in this series.

I say “failed attempt” because even if the full 6,200+ page report is ever released, the lead Senate investigator, Daniel Jones, obscured the identities of all the responsible CIA personnel and sources of information in the report.

Even if the full report is serialized in your local newspaper, the CIA contractors and staff guilty of multiple felonies, will be not one step closer to being brought to justice.

To that extent, the “full” report is itself a disservice to the American people, who elect their congressional leaders and expect them to oversee agencies such as the CIA.

From Ackerman’s account you will learn that the CIA can dictate to its overseers, the location and conditions under which it can view documents, decide which documents it is allowed to see and in cases of conflict, the CIA can spy on the Select Senate Committee on Intelligence.

Does that sound like effective oversight to you?

BTW, you will also learn that members of the “most transparent administration in history” aided and abetted the CIA in preventing an effective investigation into the CIA and its torture program. I use “aided and abetted” deliberately and in their legal sense.

I mention in my header that you should support The Guardian.

This story by Spencer Ackerman is one reason.

Another reason is that given the plethora of names and transfers recited in Ackerman’s story, we need The Guardian to cover future breaks in this story.

Despite the tales of superhuman security, nobody is that good.

I leave you with the thought that if more than one person knows a secret, then it it can be discovered.

Check Ackerman’s story for a starting list of those who know secrets about the CIA torture program.

Good hunting!

August 29, 2016

Hunters Bag > 400 Database Catalogs

Filed under: Government,Open Data,Transparency — Patrick Durusau @ 3:42 pm

Transparency Hunters Capture More than 400 California Database Catalogs by Dave Maass.

The post in its entirety:

A team of over 40 transparency activists aimed their browsers at California this past weekend, collecting more than 400 database catalogs from local government agencies, as required under a new state law. Together, participants in the California Database Hunt shined light on thousands upon thousands of government record systems.

California S.B. 272 requires every local government body, with the exception of educational agencies, to post inventories of their “enterprise systems,” essentially every database that holds records on members of the public or is used as a primary source of information. These database catalogs were required to be posted online (at least by agencies with websites) by July 1, 2016.

EFF, the Data Foundation, the Sunlight Foundation, and Level Zero, combined forces to host volunteers in San Francisco, Washington, D.C., and remotely. More than 40 volunteers scoured as many local agency websites as we could in four hours—cities, counties, regional transportation agencies, water districts, etc. Here are the rough numbers:

680 – The number of unique agencies that supporters searched

970 – The number of searches conducted (Note: agencies found on the first pass not to have catalogs were searched a second time)

430 – Number of agencies with database catalogs online

250 – Number of agencies without database catalogs online, as verified by two people

Download a spreadsheet of the local government database catalogs we found: Excel/TSV

Download a spreadsheet of cities and counties that did not have S.B. 272 catalogs: Excel/TSV

Please note that for each of the cities and counties identified as not posting database catalogs, at least two volunteers searched for the catalogs and could not find them. It is possible that those agencies do in fact have S.B. 272-compliant catalogs posted somewhere, but not in what we would call a “prominent location,” as required by the new law. If you represent an agency that would like its database catalog listed, please send an email to dm@eff.org.

We owe a debt of gratitude to the dozens of volunteers who sacrificed their Saturday afternoons to help make local government in California a little less opaque. Check out this 360-degree photo of our San Francisco team on Facebook.

In the coming days and weeks, we plan to analyze and share the data further. Stay tuned, and if you find anything interesting perusing these database catalogs, please drop us a line at dm@eff.org.

Of course, bagging the database catalogs is like having a collection of Christmas catalogs. It’s great, but there are more riches within!

What data products would you look for first?

Updated to mirror changes (clarification) in original.

August 20, 2016

US Army committed $6.5 trillion in accounting fraud in one year (w/correction)

Filed under: Auditing,Government,Transparency — Patrick Durusau @ 8:54 pm

US Army committed $6.5 trillion in accounting fraud in one year by Cory Doctorow.

From the post:

In June, the Defense Department’s Inspector General released a report on the US Army’s accounting, revealing that the Army had invented $6.5 trillion in “improper adjustments” ($2.8T in one quarter!) to make its books appear balanced though it could not account for where the funds had gone.
…

If you are interested in transparent and trackable information systems, that’s a headline that captures your attention!

Except that when you run it back to the original story, U.S. Army fudged its accounts by trillions of dollars, auditor finds by Scot J. Paltrow, which reads in part:

The United States Army’s finances are so jumbled it had to make trillions of dollars of improper accounting adjustments to create an illusion that its books are balanced.

The Defense Department’s Inspector General, in a June report, said the Army made $2.8 trillion in wrongful adjustments to accounting entries in one quarter alone in 2015, and $6.5 trillion for the year. Yet the Army lacked receipts and invoices to support those numbers or simply made them up.
…

You won’t find a reference to the “June report,” as cited by Paltrow. No link, no title, no nothing.

In fact, there is no such June report.

If you look carefully enough at the Inspector General site for the DoD you will find:

07-26-2016
Financial Management
Army General Fund Adjustments Not Adequately Documented or Supported (Project No. D2015-D000FL-0243.000)
DODIG-2016-113

The webpage for that July report, reads in part:

Finding

The Office of the Assistant Secretary of the Army (Financial Management & Comptroller) (OASA[FM&C]) and the Defense Finance and Accounting Service Indianapolis (DFAS Indianapolis) did not adequately support $2.8 trillion in third quarter journal voucher (JV) adjustments and $6.5 trillion in yearend JV adjustments1 made to AGF data during FY 2015 financial statement compilation.2 The unsupported JV adjustments occurred because OASA(FM&C) and DFAS Indianapolis did not prioritize correcting the system deficiencies that caused errors resulting in JV adjustments, and did not provide sufficient guidance for supporting system‑generated adjustments.

In addition, DFAS Indianapolis did not document or support why the Defense Departmental Reporting System‑Budgetary (DDRS-B), a budgetary reporting system, removed at least 16,513 of 1.3 million records during third quarter FY 2015. This occurred because DFAS Indianapolis did not have detailed documentation describing the DDRS-B import process or have accurate or complete system reports.

As a result, the data used to prepare the FY 2015 AGF third quarter and yearend financial statements were unreliable and lacked an adequate audit trail. Furthermore, DoD and Army managers could not rely on the data in their accounting systems when making management and resource decisions. Until the Army and DFAS Indianapolis correct these control deficiencies, there is considerable risk that AGF financial statements will be materially misstated and the Army will not achieve audit readiness by the congressionally mandated deadline of September 30, 2017.
…

Everybody makes mistakes. I’m sure I make several everyday without hardly trying.

However, if you link to original sources, readers stand some chance of discovering and correcting those errors.

If you cite a resource, link to the resource.

PS: Before you use the word “fraud” with regard to military accounting systems, realize financial accounting is not a primary or even secondary concern of a military force. There are possible solutions to military accounting issues but congressional tantrums, a/k/a mandates, aren’t among them.

June 20, 2016

Concealing the Purchase of Government Officials

Filed under: Government,Transparency — Patrick Durusau @ 7:55 am

Fredreka Schouten reports in House approves Koch-backed bill to shield donors’ names the US House of Representatives, has passed a measure to conceal the purchase of government officials.

From the post:

The House approved a bill Tuesday that would bar the IRS from collecting the names of donors to tax-exempt groups, prompting warnings from campaign-finance watchdogs that it could lead to foreign interests illegally infiltrating American elections.

The measure, which has the support of House Speaker Paul Ryan, R-Wis., also pits the Obama administration against one of the most powerful figures in Republican politics, billionaire industrialist Charles Koch. Koch’s donor network channels hundreds of millions of dollars each year into groups that largely use anonymous donations to shape policies on everything from health care to tax subsidies. Its leaders have urged the Republican-controlled Congress to clamp down on the IRS, citing free-speech concerns.

The names of donors to politically active non-profit groups aren’t public information now, but the organizations still have to disclose donor information to the IRS on annual tax returns. The bill, written by Rep. Peter Roskam, R-Ill., would prohibit the tax agency from collecting names, addresses or any “identifying information” about donors.
…

Truth be told, however, “the House” didn’t vote in favor of H.R.5053 – Preventing IRS Abuse and Protecting Free Speech Act.

Rather, two-hundred and forty (240) identified representatives voted in favor of H.R.5053.

Two-hundred and forty representatives purchased by campaign contributions who now wish to keep their contributors secret.

Two-hundred and forty representatives who are as likely as not, guilty of criminal, financial/sexual or other forms of misconduct, that could result in their replacement.

Two-hundred and forty representatives who continue in office only so long as they are not exposed to law enforcement and the public.

Where are you going to invest your time and resources?

Showing solidarity on issues where substantive change isn’t going to happen, or taking back your government from its current purchasers?

PS: In case you think “substantive change” is possible on gun control, consider the unlikely scenario that “assault weapons” are banned from sale. So what? The ones in circulation number in the millions. Net effect of your “victory” would be exactly zero.

June 6, 2016

Breaking Californication (An Act Performed On The Public)

Filed under: Government,Government Data,Transparency — Patrick Durusau @ 4:43 pm

Law Enforcement Lobby Succeeds In Killing California Transparency Bill by Kit O’Connell.

From the post:

A California Senate committee killed a bill to increase transparency in police misconduct investigations, hampering victims’ efforts to obtain justice.

Chauncee Smith, legislative advocate at the ACLU of California, told MintPress News that the state Legislature “caved to the tremendous influence and power of the law enforcement lobby” and “failed to listen to the demands and concerns of everyday Californian people.”

California has some of the most secretive rules in the country when it comes to investigations into police misconduct and excessive use of force. Records are kept sealed, regardless of the outcome, as the ACLU of Northern California explains on its website:

“In places like Texas, Kentucky, and Utah, peace officer records are made public when an officer is found guilty of misconduct. Other states make records public regardless of whether misconduct is found. This is not the case in California.”

“Right now, there is a tremendous cloud of secrecy that is unparalleled compared to many other states,” Smith added. “California is in the minority in which the public do not know basic information when someone is killed or potentially harmed by those are sworn to serve and protect them.”

In February, Sen. Mark Leno, a Democrat from San Francisco, introduced SB 1286, the “Enhance Community Oversight on Police Misconduct and Serious Uses of Force” bill. It would have allowed “public access to investigations, findings and discipline information on serious uses of force by police” and would have increased transparency in other cases of police misconduct, according to an ACLU fact sheet. Polling data cited by the ACLU suggests about 80 percent of Californians would support the measure.

But the bill’s progress through the legislature ended on May 27, when it failed to pass out of the Senate Appropriations committee.

“Today is a sad day for transparency, accountability, and justice in California,” said Peter Bibring, police practices director for the ACLU of California, in a May 27 press release.
…

Mistrust between police officers and citizens makes the job of police officers more difficult and dangerous, while denying citizens the full advantages of a trained police force, paid for by their tax dollars.

The state legislature, finding sowing and fueling mistrust between police officers and citizens has election upsides for them, fans those flames with secrecy over police misconduct investigations.

Open, not secret (read grand jury) proceedings where witnesses can be fairly examined (unlike the deliberately thrown Michael Brown investigation), can go a long way to re-establishing trust between the police and the public.

Members of the community know when someone was a danger to police officers and others. Whether their family members will admit it or not. Likewise, police officers know which officers are far to quick to escalate to deadly force. Want better community policing? What better citizen cooperation? That’s not going to happen with completely secret police misconduct investigations.

So the State of California is going to collect the evidence, statements, etc., in police misconduct investigations, but won’t share that information with the public. At least not willingly.

Official attempts to break illegitimate government secrecy failed. Even if it had succeeded you’d be paying least $0.25 per page plus a service fee.

Two observations about government networks:

  • Secret (and otherwise) government documents are usually printed on networked printers.
  • Passively capturing Ethernet traffic (network tap) captures printer traffic too.

Whistle blowers don’t have to hack heavily monitored systems, steal logins/passwords, leaking illegally withheld documents is within the reach of anyone who can plug in an Ethernet cable.

There’s a bit more to it than that, but remember all those network cables running through the ceiling, walls, closets, the next time your security consultant, assures you of your network’s security.

As a practical matter, if you start leaking party menus and football pools, someone will start looking for a network tap.

Leak when it makes a significant difference to public discussion and/or legal proceedings. Even then, look for ways to attribute the leak to factions within the government.

Remember the DoD’s amused reaction to State’s huffing and puffing over the Afghan diplomatic cables? That sort of rivalry exists at every level of government. You should use it to your advantage.

The State of California would have you believe that government information sharing is at its sufferance.

I beg to differ.

So should you.

May 25, 2016

Hidden Inspector General Report on Clinton’s Emails?

Filed under: Government,Politics,Transparency — Patrick Durusau @ 8:13 pm

If you haven’t heard about the controversy surrounding Hillary Clinton’s handling of emails during her term as Secretary of State, you are one of the lucky ones.

The rest of us have been treated to a literal circus of pettifogging over her “private” email server for years now. Truly a tempest in a teapot.

But, along comes a much awaited report by the Inspector General for the State Department on those same emails, and where can you find it?

Not on the Inspector General for the State Department homepage (as of 25 May 2016, 9:00 PM EST)!

No, you will have to find that report, the one everyone has been waiting for, Office of the Secretary: Evaluation of Email Records Management and Cybersecurity Requirements to be posted by Politico.

I have no objection to Politico having the “scoop” on this report and/or distributing a document of great public interest. All fine and good.

But why does the Inspector General choose to hide this report from the general public?

Is the Inspector General ashamed of the report?

A report that encompasses other secretaries of state, as though to argue bad and/or criminal behavior can be excused because it is customary?

I’m not familiar with the “customary therefore not criminal” defense.

Perhaps that only obtains at Cabinet level positions.

In any event, please help Steve Linick, the current Inspector General for the State Department, own this report now and forever.

May 24, 2016

FOIA – For Algorithms

Filed under: Algorithms,Government,Transparency — Patrick Durusau @ 3:06 pm

We need to know the algorithms the government uses to make important decisions about us by Nicholas Diakopoulos.

From the post:

In criminal justice systems, credit markets, employment arenas, higher education admissions processes and even social media networks, data-driven algorithms now drive decision-making in ways that touch our economic, social and civic lives. These software systems rank, classify, associate or filter information, using human-crafted or data-induced rules that allow for consistent treatment across large populations.

But while there may be efficiency gains from these techniques, they can also harbor biases against disadvantaged groups or reinforce structural discrimination. In terms of criminal justice, for example, is it fair to make judgments on an individual’s parole based on statistical tendencies measured across a wide group of people? Could discrimination arise from applying a statistical model developed for one state’s population to another, demographically different population?

The public needs to understand the bias and power of algorithms used in the public sphere, including by government agencies. An effort I am involved with, called algorithmic accountability, seeks to make the influences of those sorts of systems clearer and more widely understood.

Existing transparency techniques, when applied to algorithms, could enable people to monitor, audit and criticize how those systems are functioning – or not, as the case may be. Unfortunately, government agencies seem unprepared for inquiries about algorithms and their uses in decisions that significantly affect both individuals and the public at large.

…

Nicholas makes a great case for Freedom of Information Act (FOIA) legislation being improved to explicitly include algorithms used by government or on its behalf.

I include “on its behalf” because as Nicholas documents, some states have learned the trick of having algorithms held by vendors, thus making them “proprietary.”

If you can’t see the algorithms behind data results, there is no meaningful transparency.

Demand meaningful transparency!

April 11, 2016

SEMS 2016 (Auditable Spreadsheets – Quick Grab Your Heart Pills)

Filed under: Programming,Spreadsheets,Transparency — Patrick Durusau @ 3:12 pm

3rd International Workshop on Software Engineering Methods in Spreadsheets

July 4, 2016 Vienna, Austria

Abstracts due: April 11th (that’s today!)

Papers due: April 22nd

From the webpage:

SEMS is the #1 venue for academic spreadsheet research since 2014 (SEMS’14, SEMS’15). This year, SEMS’16 is going to be co-located with STAF 2016 in Vienna.

Spreadsheets are heavily used in industry as they are easy to create and evolve through their intuitive visual interface. They are often initially developed as simple tools, but, over time, spreadsheets can become increasingly complex, up to the point they become too complicated to maintain. Indeed, in many ways, spreadsheets are similar to “professional” software: both concern the storage and manipulation of data, and the presentation of results to the user. But unlike in “professional” software, activities like design, implementation, and maintenance in spreadsheets have to be undertaken by end-users, not trained professionals. This makes applying methods and techniques from other software technologies a challenging task.

The role of SEMS is to explore the possibilities of adopting successful methods from other software contexts to spreadsheets. Some, like testing and modeling, have been tried before and can be built upon. For methods that have not yet been tried on spreadsheets, SEMS will serve as a platform for early feedback.

The SEMS program will include an industrial keynote, followed by a brainstorming session about the topic, a discussion panel of industrial spreadsheet usage, presentation of short and long research papers and plenty of lively discussions. The intended audience is a mixture of spreadsheet researchers and professionals.
…

Felienne Hermans pioneered viewing spreadsheets as programming artifacts, a view that can result in easier maintenance and even, gasp, auditing of spreadsheets.

Inspectors General, GAO and other birds of that feather should sign up for this conference.

Remember topic maps for cumulative and customized auditing data. For example, who, by name, was explaining entries that several years later appear questionable? Topic maps can capture as much or as little data as you require.

Attend, submit an abstract today and a paper in two weeks!

December 29, 2015

Nominations by the U.S. President

Filed under: Government,Transparency — Patrick Durusau @ 8:52 pm

Nominations by the U.S. President

From Congress.gov, a faceted listing of all nominations from 1981 to date.

Facets include Congress, Nomination Type (Civilian, Military, Select Only), Status of Nomination, Senate Committee, Nominees with US State or Territory Indicated.

I haven’t spent a lot of time with this resource but it appears to be unnecessarily difficult to use.

For example:

Let’s look up the nomination of Sonia Sotomayor to the United States Supreme Court:

sonia-sotomayor

Did you notice the absence of any hyperlinks to the three days of hearings, July 13-15, 2009? Or the absence of links to the Senate debate on August 4-5, 2009? Or the absence of links to any of the other documents or agreements?

I remember the WWW being around in 2009 and I am damned sure it is available now!

So, what’s with the lack of hyperlinks?

Do you think they are lurking beneath the surface, waiting to be turned on?

Afraid not. Here is a sample of the underlying content for that page:

<tr>
<td class="date">07/16/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held and completed. Hearings printed: S.Hrg. 111-503.</td>
</tr>
<tr>
<td class="date">07/15/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held.</td>
</tr>
<tr>
<td class="date">07/14/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held.</td>
</tr>
<tr>
<td class="date">07/13/2009</td><td class="actions">
   Committee on the Judiciary. Hearings held.</td>
</tr>

I don’t see any hooks for hyperlinking later on. Do you?

Another data hook that is missing is linking historical campaign donations to nominees for offices, particularly in the State Department.

Surely you didn’t think ambassadors were appointed from the professional ranks of the Foreign Service? People who actually speak the languages of the host country and know it customs and habits. What an odd view of American (or any other) government you have.

Some of the larger ambassadorships do require some experience but out of 270 embassies around the world, there are ones that go to mega-donors.

I don’t know the going rate on ambassadorships but linking nominations to donation records could yield a target minimum for donors to shoot for.

Linking nominations to donations would be a non-trivial exercise but certainly doable.

Other suggestions for Congress.gov on these webpages? They respond well to suggestions. Not to say they always agree but they do respond. More than I can say for some government groups.

December 12, 2015

Why the Open Government Partnership Needs a Reboot [Governments Too]

Filed under: Government,Government Data,Open Government,Transparency — Patrick Durusau @ 7:31 pm

Why the Open Government Partnership Needs a Reboot by Steve Adler.

From the post:

The Open Government Partnership was created in 2011 as an international forum for nations committed to implementing Open Government programs for the advancement of their societies. The idea of open government started in the 1980s after CSPAN was launched to broadcast U.S. Congressional proceedings and hearings to the American public on TV. While the galleries above the House of Representatives and Senate had been “open” to the “public” (if you got permission from your representative to attend) for decades, never before had all public democratic deliberations been broadcast on TV for the entire nation to behold at any time they wished to tune in.

…

I am a big fan of OGP and feel that the ideals and ambition of this partnership are noble and essential to the survival of democracy in this millennium. But OGP is a startup, and every startup business or program faces a chasm it must cross from early adopters and innovators to early majority market implementation and OGP is very much at this crossroads today. It has expanded membership at a furious pace the past three years and it’s clear to me that expansion is now far more important to OGP than the delivery of the benefits of open government to the hundreds of millions of citizens who need transparent transformation.

OGP needs a reboot.

The structure of a system produces its own behavior. OGP needs a new organizational structure with new methods for evaluating national commitments. But that reboot needs to happen within its current mission. We should see clearly that the current structure is straining due to the rapid expansion of membership. There aren’t enough support unit resources to manage the expansion. We have to rethink how we manage national commitments and how we evaluate what it means to be an open government. It’s just not right that countries can celebrate baby steps at OGP events while at the same time passing odious legislation, sidestepping OGP accomplishments, buckling to corruption, and cracking down on journalists.
…

Unlike Steve I didn’t and don’t have a lot of faith in governments being voluntarily transparent.

As I pointed out in Congress: More XQuery Fodder, sometime in 2016, full bill status data will be available for all legislation before the United States Congress.

A lot more data than is easy to access now but it is more smoke than fire.

With legislation status data, you can track the civics lesson progression of a bill through Congress, but that leaves you at least 3 to 4 degrees short of knowing who was behind the legislation.

Just a short list of what more would be useful:

  • Visitor/caller list for everyone who spoke to a member of Congress and their staff. With date and subject of the call.
  • All visitors and calls tied to particular legislation and/or classes of legislation
  • All fund raising calls made by members of Congress and/or their staffs, date, results, substance of call.
  • Representative conversations with reconciliation committee members or their staffers about legislation and requested “corrections.”
  • All conversations between a representative or member of their staff and agency staff, identifying all parties and the substance of the conversation
  • Notes, proposals, discussion notes for all agencies decisions

Current transparency proposals are sufficient to confuse the public with mounds of nearly useless data. None of it reflects the real decision making processes of government.

Before someone shouts “privacy,” I would point out that no citizen has a right to privacy when their request is for a government representative to favor them over other citizens of the same government.

Real government transparency will require breaking the mini-star chamber proceedings from the lowest to the highest levels of government.

What we need is a rebooting of governments.

Older Posts »

Powered by WordPress