Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

April 23, 2019

Best OCR Tools – Side by Side

Filed under: Government,Government Data,OCR — Patrick Durusau @ 8:34 pm

Our Search for the Best OCR Tool, and What We Found by Ted Han and Amanda Hickman.

From the post:

We selected several documents—two easy to read reports, a receipt, an historical document, a legal filing with a lot of redaction, a filled in disclosure form, and a water damaged page—to run through the OCR engines we are most interested in. We tested three free and open source options (Calamari, OCRopus and Tesseract) as well as one desktop app (Adobe Acrobat Pro) and three cloud services (Abbyy Cloud, Google Cloud Vision, and Microsoft Azure Computer Vision).

All the scripts we used, as well as the complete output from each OCR engine, are available on GitHub. You can use the scripts to check our work, or to run your own documents against any of the clients we tested.

The quality of results varied between applications, but there wasn’t a stand out winner. Most of the tools handled a clean document just fine. None got perfect results on trickier documents, but most were good enough to make text significantly more comprehensible. In most cases if you need a complete, accurate transcription you’ll have to do additional review and correction.

Since government offices are loathe to release searchable versions of important documents (think Mueller report), reasonable use of those documents requires OCR tools.

Han and Hickman enable you to compare OCR engines on your documents, an important step before deciding on which engine best meets your needs.

Should you find yourself in a hacker forum, no doubt by accident, do mention agencies which force OCR of their document releases. That unnecessary burden on readers and reporters should not go unrewarded.

January 4, 2019

Crypto-Cash for Crypto-Cache : The Dark Overlord

Filed under: Government,Government Data,Hacking,Intelligence — Patrick Durusau @ 8:24 pm
Crypto-Cash for Crypto-Cache

This is the thedarkoverlord here to deliver a message.


Our Official Bitcoin Wallet Address: 192ZobzfZxAkacLGmg9oY4M9y8MVTPxh7U


As the world is aware, we released our first decryption key for the ‘Preview_Documents.container’ Veracrypt container that contained a small sample of documents to continue to verify the authenticity of our claims. The decryption key for this container is: *CZ4=I{YZ456zGecgg9/cCz|zNP5bZ,nCvJqDZKrq@v?O5V$FezCNs26CD;e:%N^

There’s five layers to go. Layer 1, 2, 3, 4, and fine finally Layer 5. Each layer contains more secrets, more damaging materials, more SSI, more SCI, more government investigation materials, and generally just more truth. Consider our motivations (money, specifically Bitcoin), we’re not inclined to leak the juiciest items until we’re paid in full. However, in the interest of public awareness and transparency, we’re officially announcing our tiered compensation plan. …

This press release is reviewed at: Hacker group releases ‘9/11 Papers’, says future leaks will ‘burn down’ US deep state.

Nothing explosive in the initial documents but you have to wonder why they were scrubbed from Reddit, Pastebin, and Twitter, “immediately.”

I don’t see any ethical issue with The Dark Overlord charging for these documents. We are held hostage by utility, cable, ISP, mortgage and other hostiles. It’s a proven money-making model so why the tension over it being used here?

For further details, see the press release by The Dark Overlord. Please consider contributing to fund the release of these documents.

P.S. I rather doubt any document or report is going to bring down the “deep state.” Remember that it employs hundreds of thousands of people and numerous contractors and vendors. Shutting it down would cripple local economies in a number of places. It likely exists because it is needed to exist.

December 4, 2018

Bulk US Congress Bills, Laws in XML

Filed under: Government,Government Data,Law,Legal Informatics,XML — Patrick Durusau @ 8:47 am

GPO Makes Documents Easy To Download and Repurpose in New XML Format

From the news release:

The U.S. Government Publishing Office (GPO) makes available a subset of enrolled bills, public and private laws, and the Statutes at Large in Beta United States Legislative Markup (USLM) XML, a format that makes documents easier to download and repurpose. The documents available in the Beta USLM XML format include enrolled bills and public laws beginning with the 113th Congress (2013) and the Statutes at Large beginning with the 108th Congress (2003). They are available on govinfo, GPO’s one-stop site to authentic, published Government information. https://www.govinfo.gov/bulkdata.

The conversion of legacy formats into Beta USML XML will provide a uniform set of laws for the public to download. This new format maximizes the number of ways the information can be used or repurposed for mobile apps or other digital or print projects. The public will now be able to download large sets of data in one click rather than downloading each file individually, saving significant time for developers and others who seek to repurpose the data.

GPO is collaborating with various legislative and executive branch organizations on this project, including the Office of the Clerk of the House, the Office of the Secretary of the Senate, and the Office of the Federal Register. The project is being done in support of the Legislative Branch Bulk Data Task Force which was established to examine the increased dissemination of Congressional information via bulk data download by non-Governmental groups for the purpose of supporting openness and transparency in the legislative process.

“Making these documents available in Beta USLM XML is another example of how GPO is meeting the technological needs of Congress and the public,“ said GPO Acting Deputy Director Herbert H. Jackson, Jr. “GPO is committed to working with Congress on new formats that provide the American people easy access to legislative information.“

GPO is the Federal Government’s official, digital, secure resource for producing, procuring, cataloging, indexing, authenticating, disseminating, and preserving the official information products of the U.S. Government. The GPO is responsible for the production and distribution of information products and services for all three branches of the Federal Government, including U.S. passports for the Department of State as well as the official publications of Congress, the White House, and other Federal agencies in digital and print formats. GPO provides for permanent public access to Federal Government information at no charge through www.govinfo.gov and partnerships with approximately 1,140 libraries nationwide participating in the Federal Depository Library Program. For more information, please visit www.gpo.gov.

Not that I have lost any of my disdain and distrust for government, but when any government does something good, they should be praised.

Making “enrolled bills, public and private laws, and the Statues at Large in Beta United States Legislative markup (USML) XML” is a step towards to tracing and integrating legislation with those it benefits.

I’m not convinced that if you could trace specific legislation to a set of donations that the outcomes on legislation would be any different. It’s like tracing payments made to a sex worker. That’s their trade, why should they be ashamed of it?

The same holds true for most members of Congress, save that the latest election has swept non-sex worker types into office. It remains to be seen how many will resist the temptation to sell their offices and which will not.

In either case, kudos to the GPO and Lauren Wood, who I understand has been a major driver in this project!

August 28, 2018

Cybersecurity Fails Set To Spread Beyond Beltway Defense Contractors

Filed under: Cybersecurity,Government,Government Data — Patrick Durusau @ 3:01 pm

I’m sure you were as amused as I was to read: U.S. Department Of Defense Awards $37 Million Contract To Cybersecurity Startup Qadium. It’s only fair you know. Startups can fail at cybersecurity just as well as traditional contractors (names omitted to protect the guilty).

In transparency unlike most media outlets, the post includes a disclaimer that the following was written by Qadium:

Cybersecurity startup Qadium has been awarded a $37.6 million contract by the U.S. Department of Defense, making it the latest venture-backed startup from Silicon Valley to win a major federal contract over traditional Beltway defense contractors.

Qadium is the first company to provide real-time monitoring of the entire global Internet for customers’ assets. In a new era of machine-speed attacks, Qadium helps the world’s most sophisticated organizations define and secure their dynamic network edge.

The contract was awarded by the U.S. Navy’s Space and Warfare Command after the Department of Defense validated Qadium’s commercial software. Qadium is now recognized among a small handful of cybersecurity providers, with DoD making its software accessible department-wide.

“The Defense Department used to love to build its own IT, often poorly and at high cost to taxpayers,” said Qadium CEO and CIA veteran Tim Junio. “The times are finally changing. In the face of the greatest cybersecurity challenges in our nation’s history, we’re seeing the government and private tech companies coming together, making both sides better off.”

I can name one side that will be better off, to the tune of $37 Million.

Hackers also benefit from this news, Qadium becoming a known target for social engineering and other attention.

July 28, 2018

Deep Learning … Wireless Jamming Attacks

Filed under: Cybersecurity,Government,Government Data,Hacking — Patrick Durusau @ 8:25 pm

Deep Learning for Launching and Mitigating Wireless Jamming Attacks by Tugba Erpek, Yalin E. Sagduyu, Yi Shi.

Abstract:

An adversarial machine learning approach is introduced to launch jamming attacks on wireless communications and a defense strategy is provided. A cognitive transmitter uses a pre-trained classifier to predict current channel status based on recent sensing results and decides whether to transmit or not, whereas a jammer collects channel status and ACKs to build a deep learning classifier that reliably predicts whether there will be a successful transmission next and effectively jams these transmissions. This jamming approach is shown to reduce the performance of the transmitter much more severely compared with randomized or sensing-based jamming. Next, a generative adversarial network (GAN) is developed for the jammer to reduce the time to collect the training dataset by augmenting it with synthetic samples. Then, a defense scheme is introduced for the transmitter that prevents the jammer from building a reliable classifier by deliberately taking a small number of wrong actions (in form of a causative attack launched against the jammer) when it accesses the spectrum. The transmitter systematically selects when to take wrong actions and adapts the level of defense to machine learning-based or conventional jamming behavior in order to mislead the jammer into making prediction errors and consequently increase its throughput.

As you know, convenience is going to triumph over security, even (especially?) in the context of military contractors. A deep learning approach may be overkill for low-bid contractor targets but it’s good practice for the occasionally more skilled opponent.

Enjoy!

May 8, 2018

Extracting Data From FBI Reports – No Waterboarding Required!

Filed under: FBI,Government,Government Data,R — Patrick Durusau @ 1:01 pm

Wrangling Data Table Out Of the FBI 2017 IC3 Crime Report

From the post:

The U.S. FBI Internet Crime Complaint Center was established in 2000 to receive complaints of Internet crime. They produce an annual report, just released 2017’s edition, and I need the data from it. Since I have to wrangle it out, I thought some folks might like to play long at home, especially since it turns out I had to use both tabulizer and pdftools to accomplish my goal.

Concepts presented:

  • PDF scraping (with both tabulizer and pdftools)
  • asciiruler
  • general string manipulation
  • case_when() vs ifelse() for text cleanup
  • reformatting data for ggraph treemaps

Let’s get started! (NOTE: you can click/tap on any image for a larger version)

Freeing FBI data from a PDF prison, is a public spirited act.

Demonstrating how to free FBI data from PDF prisons, is a virtuous act!

Enjoy!

April 30, 2018

Examining POTUS Executive Orders [Tweets < Executive Orders < Cern Data]

Filed under: Government Data,R,Text Mining,Texts — Patrick Durusau @ 8:12 pm

Examining POTUS Executive Orders by Bob Rudis.

From the post:

This week’s edition of Data is Plural had two really fun data sets. One is serious fun (the first comprehensive data set on U.S. evictions, and the other I knew about but had forgotten: The Federal Register Executive Order (EO) data set(s).

The EO data is also comprehensive as the summary JSON (or CSV) files have links to more metadata and even more links to the full-text in various formats.

What follows is a quick post to help bootstrap folks who may want to do some tidy text mining on this data. We’ll look at EOs-per-year (per-POTUS) and also take a look at the “top 5 ‘first words’” in the titles of the EOS (also by POTUS).

My estimate of the importance of executive orders by American Presidents, “Tweets < Executive Orders < Cern Data,” is only an approximation.

Rudis leaves you plenty of room to experiment with R and processing the text of executive orders.

Enjoy!

December 27, 2017

From the Valley of Disinformation Rode the 770 – Opportunity Knocks

Filed under: Cybersecurity,Environment,Government,Government Data,Journalism,Reporting — Patrick Durusau @ 10:32 am

More than 700 employees have left the EPA since Scott Pruitt took over by Natasha Geiling.

From the post:

Since Environmental Protection Agency Administrator Scott Pruitt took over the top job at the agency in March, more than 700 employees have either retired, taken voluntary buyouts, or quit, signaling the second-highest exodus of employees from the agency in nearly a decade.

According to agency documents and federal employment statistics, 770 EPA employees departed the agency between April and December, leaving employment levels close to Reagan-era levels of staffing. According to the EPA’s contingency shutdown plan for December, the agency currently has 14,449 employees on board — a marked change from the April contingency plan, which showed a staff of 15,219.

These departures offer journalists a rare opportunity to bleed the government like a stuck pig. From untimely remission of login credentials to acceptance of spear phishing emails, opportunities abound.

Not for “reach it to me” journalists who use sources as shields from potential criminal liability. While their colleagues are imprisoned for the simple act of publication or murdered (as of today in 2017, 42).

Governments have not, are not and will not act in the public interest. Laws that criminalize acquisition of data or documents are a continuation of their failure to act in the public interest.

Journalists who serve the public interest, by exposing the government’s failure to do so, should use any means at their disposal to obtain data and documents that evidence government failure and misconduct.

Are you a journalist serving the public interest or a “reach it to me” journalist, serving the public interest when there’s no threat to you?

December 14, 2017

98% Fail Rate on Privileged Accounts – Transparency in 2018

Filed under: Cybersecurity,Government,Government Data,Security,Transparency — Patrick Durusau @ 9:55 am

Half of companies fail to tell customers about data breaches, claims study by Nicholas Fearn.

From the post:

Half of organisations don’t bother telling customers when their personal information might have been compromised following a cyber attack, according to a new study.

The latest survey from security firm CyberArk comes with the full implementation of the European Union General Data Protection Regulation (GDPR) just months away.

Organisations that fail to notify the relevant data protection authorities of a breach within 72 hours of finding it can expect to face crippling fines of up to four per cent of turnover – with companies trying to hide breaches likely to be hit with the biggest punishments.

The findings have been published in the second iteration the CyberArk Global Advanced Threat Landscape Report 2018, which explores business leaders’ attitudes towards IT security and data protection.

The survey found that, overall, security “does not translate into accountability”. Some 46 per cent of organisations struggle to stop every attempt to breach their IT infrastructure.

And 63 per cent of business leaders acknowledge that their companies are vulnerable to attacks, such as phishing. Despite this concern, 49 per cent of organisations don’t have the right knowledge about security policies.

You can download the report cited in Fearn’s post at: Cyberark Global Advanced Threat Landscape Report 2018: The Business View of Security.

If you think that report has implications for involuntary/inadvertent transparency, Cyberark Global Advanced Threat Landscape Report 2018: Focus on DevOps, reports this gem:


It’s not just that businesses underestimate threats. As noted above, they also do not seem to fully understand where privileged accounts and secrets exist. When asked which IT environments and devices contain privileged accounts and secrets, responses (IT decision maker and DevOps/app developer respondents) were at odds with the claim that most businesses have implemented a privileged account security solution. A massive 98% did not select at least one of the ‘containers’, ‘microservices’, ‘CI/CD tools’, ‘cloud environments’ or ‘source code repositories’ options. At the risk of repetition, privileged accounts and secrets are stored in all of these entities.

A fail rate of 98% on identifying “privileged accounts and secrets?”

Reports like this make you wonder about the clamor for transparency of organizations and governments. Why bother?

Information in 2018 is kept secure by a lack of interest in collecting it.

Remember that for your next transparency discussion.

November 27, 2017

eXist-db v3.6.0 [Prediction for 2018: Multiple data/document leak tsunamis. Are You Ready?]

Filed under: eXist,Government,Government Data,XML,XPath,XQuery — Patrick Durusau @ 9:28 pm

eXist-db v3.6.0

From the post:

Features

  • Switched Collation support to use ICU4j.
  • Implemented XQuery 3.1 UCA (Unicode Collation Algorithm).
  • Implemented map type parameters for XQuery F&O 3.1 fn:serialize.
  • Implemented declare context item for XQuery 3.0.
  • Implemented XQuery 3.0 Regular Expression’s support for non-capturing groups.
  • Implemented a type-safe DSL for describing and testing transactional operations upon the database.
  • Implemented missing node kind tests in the XQuery parser when using @ on an AbbrevForwardStep.
  • Added AspectJ support to the IntelliJ project files (IntelliJ Ultimate only).
  • Repaired the dependencies in the NetBeans project files.
  • Added support for Travis macOS CI.
  • Added support for AppVeyor Windows CI.
  • Updated third-party dependencies:
    • Apache Commons Codec 1.11
    • Apache Commons Compress 1.15
    • Apache Commons Lang 3.7
    • Eclipse AspectJ 1.9.0.RC1
    • Eclipse Jetty 9.4.7.v20170914
    • EXPath HTTP Client 20171116
    • Java 8 Functional Utilities 1.11
    • JCTools 2.1.1
    • XML Unit 2.4.0

Performance Improvements

  • Compiled XQuery cache is now multi-threaded; concurrency is now per-source.
  • RESTXQ compiled XQuery cache is now multi-threaded; concurrency is now per-query URI.
  • STX Templates Cache is now multithreaded.
  • XML-RPC Server will now use Streaming and GZip compression if supported by the client; enabled in eXist’s Java Admin Client.
  • Reduced object creation overhead in the XML-RPC Server.

Apps

The bundled applications of the Documentation, eXide, and Monex have all been updated to the latest versions.

Prediction for 2018: Multiple data/document leak tsunamis.

Are you prepared?

How are your XQuery skills and tools?

Or do you plan on regurgitating news wire summaries?

November 15, 2017

How-Keep A Secret, Well, Secret (Brill)

Filed under: Government,Government Data,Politics — Patrick Durusau @ 4:51 pm

Weapons of Mass Destruction: The Top Secret History of America’s Nuclear, Chemical and Biological Warfare Programs and Their Deployment Overseas, edited by Matthew M. Aid, is described as:

At its peak in 1967, the U.S. nuclear arsenal consisted of 31,255 nuclear weapons with an aggregate destructive power of 12,786 megatons – more than sufficient to wipe out all of humanity several hundred times over. Much less known is that hidden away in earth-covered bunkers spread throughout the U.S., Europe and Japan, over 40,000 tons of American chemical weapons were stored, as well as thousands of specially designed bombs that could be filled with even deadlier biological warfare agents.

The American WMD programs remain cloaked in secrecy, yet a substantial number of revealing documents have been quietly declassified since the late 1970s. Put together, they tell the story of how America secretly built up the world’s largest stockpile of nuclear, chemical, and biological weapons. The documents explain the role these weapons played in a series of world crises, how they shaped U.S. and NATO defense and foreign policy during the Cold War, and what incidents and nearly averted disasters happened. Moreover, they shed a light on the dreadful human and ecological legacy left by decades of nuclear, chemical and biological weapons manufacturing and testing in the U.S. and overseas.

This collection contains more than 2,300 formerly classified U.S. government documents, most of them classified Top Secret or higher. Covering the period from the end of World War II to the present day, it provides unique access to previously unpublished reports, memoranda, cables, intelligence briefs, classified articles, PowerPoint presentations, military manuals and directives, and other declassified documents. Following years of archival research and careful selection, they were brought together from the U.S. National Archives, ten U.S. presidential libraries, the NATO Archives in Brussels, the National Archives of the UK, the National Archives of Canada, and the National Archives of the Netherlands. In addition, a sizeable number of documents in this collection were obtained from the U.S. government and the Pentagon using the Freedom of Information Act (FOIA) and Mandatory Declassification Review (MDR) requests.

This collection comes with several auxiliary aids, including a chronology and a historiographical essay with links to the documents themselves, providing context and allowing for easy navigation for both students and scholars.

It’s an online resource of about 21,212 pages.

Although the editor, Aid and/or Brill did a considerable amount of work assembling these document, the outright purchase price: €4.066,00, $4,886.00 or the daily access price: $39.95/day, effectively keeps these once secret documents secret.

Of particular interest to historians and arms control experts, I expect those identifying recurrent patterns of criminal misconduct in governments will find the data of interest as well.

It does occur to me that when you look at the Contents tab, http://primarysources.brillonline.com/browse/weapons-of-mass-destruction#content-tab, each year lists the documents in the archive. Lists that could be parsed for recovery of the documents from other sources on the Internet.

You would still have to index (did I hear someone say topic map?) the documents, etc., but as a long term asset for the research community, it would be quite nice.

If you doubt the need for such a project, toss “USAF, Cable, CINCUSAFE to CSAF, May 6, 1954, Top Secret, NARA” into your nearest search engine.

How do you feel about Brill being the arbiter of 20th century history, for a price?

Me too.

October 26, 2017

What’s New in the JFK Files? [A Topic Map Could Help Answer That Question]

Filed under: Government,Government Data,History,Topic Maps — Patrick Durusau @ 9:07 pm

The JFK Files: Calling On Citizen Reporters

From the webpage:

The government has released long-secret files on John F. Kennedy’s assassination, and we want your help.

The files are among the last to be released by the National Archives under a 1992 law that ordered the government to make public all remaining documents pertaining to the assassination. Other files are being withheld because of what the White House says are national security, law enforcement and foreign policy concerns.

There has long been a trove of conspiracy theories surrounding Kennedy’s murder in Dallas on Nov. 22, 1963, including doubts about whether Lee Harvey Oswald acted alone, as the Warren Commission determined in its report the following year.

Here’s where you come in. Read the documents linked here. If you find news or noteworthy nuggets among the pages, share them with us on the document below. If we use what you find, we’ll be sure to give you a shoutout!

Given the linear feet of existing files, finding new nuggets or aligning them with old nuggets in the original files, is going to be a slow process.

What more, you or I may find the exact nugget needed to connect dots for someone else, but since we all read, search, and maintain our searches separately, effective sharing of those nuggets won’t happen.

Depending on the granularity of a topic map over those same materials, confirmation of Oswald’s known whereabouts and who reported those could be easily examined and compared to new (if any) whereabouts information in these files. If new files confirm what is known, researchers could skip that material and move to subjects unknown in the original files.

A non-trivial encoding task but full details have been delayed pending another round of hiding professional incompetence. A topic map will help you ferret out the incompetents seeking to hide in the last releases of documents. Interested?

August 4, 2017

“This culture of leaking must stop.” Taking up Sessions’ Gage

Filed under: Cybersecurity,Government,Government Data,Security — Patrick Durusau @ 4:12 pm

Jeff Sessions, the current (4 August 2017) Attorney General of the United States, wants to improve on Barack Obama‘s legacy as the most secretive presidency of the modern era.

Sessions has announced a tripling Justice Department probes into leaks and a review of guidelines for subpoenas for members of the news media. Attorney General says Justice Dept. has tripled the number of leak probes. (Media subpoenas are an effort to discover media sources and hence to plug the “leaks.”)

Sessions has thrown down his gage, declaring war on occasional transparency from government leakers. Indirectly, that war will include members of the media as casualties.

Shakespeare penned the best response for taking up Sessions’ gage:

Cry ‘Havoc,’ and let slip the dogs of war;

In case you don’t know the original sense of “Havoc:”

The military order Havoc! was a signal given to the English military forces in the Middle Ages to direct the soldiery (in Shakespeare’s parlance ‘the dogs of war’) to pillage and chaos. Cry havoc and let slip the dogs of war

It’s on all of us to create enough chaos to protect leakers and members of the media who publish their leaks.

Observations – Not Instructions

Data access: Phishing emails succeed 33% of the time. Do they punish would-be leakers who fall for phishing emails?

Exflitration: Tracing select documents to a leaker is commonplace. How do you trace an entire server disk? The larger and more systematic the data haul, the greater the difficulty in pinning the leak on particular documents. (Back to school specials often include multi-terabyte drives.)

Protect the Media: Full drive leaks posted a Torrent or Dark Web server means media can answer subpoenas with: go to: https://some-location. 😉

BTW, full drive leaks provide transparency for the relationship between the leaked data and media reports. Accountability is as important for the media as the government.

One or more of my observations may constitute crimes depending upon your jurisdiction.

Which I guess is why Nathan Hale is recorded as saying:

Gee, that sounds like a crime. You know, I could get arrested, even executed. None for me please!

Not!

Nathan Hale volunteered to be a spy, was caught and executed, having said:

I only regret, that I have but one life to lose for my country.

Question for you:

Are you a ‘dog of war’ making the government bleed data?

PS: As a security measure, don’t write that answer down or tell anyone. When you read about leaks, you can inwardly smile and know you played your part.

August 2, 2017

“But it feels better when I sneak”

Filed under: Cybersecurity,FOIA,Government,Government Data — Patrick Durusau @ 10:37 am

Email prankster tricks White House officials by Graham Cluley is ample evidence for why you should abandon FOIA requests in favor of phishing/hacking during the reign of Donald Trump.

People can and do obtain mountains of information using FOIA requests, but in the words of Parker Ray, “The Other Woman,”:

“Now I hate to have to cheat
But it feels better when I sneak”

In addition to feeling better, not using FOIA requests during the Trump regime results in:

  1. Access to competitor’s data deposited with the government
  2. Avoids the paperwork and delay of the FOIA process
  3. Bidding and contract data
  4. Develop long-term stealth access than spans presidencies
  5. Incompetence of staff gives broad and deep access across agencies
  6. Mine papers of extremely secretive prior presidents, like Obama
  7. Transparency when least expected and most inconvenient

If that sounds wishful, remember Cluley reports the “technique” used by the prankster was: 1) create an email account in the name of a White House staffer, 2) send an email from that account. This has to be a new low bar for “fake” emails.

Can you afford to be a goody two shoes?

June 20, 2017

Manning Leaks — No Real Harm (Database of Government Liars Anyone?)

Filed under: Government,Government Data,Leaks — Patrick Durusau @ 2:56 pm

Secret Government Report: Chelsea Manning Leaks Caused No Real Harm by Jason Leopold.

From the post:

In the seven years since WikiLeaks published the largest leak of classified documents in history, the federal government has said they caused enormous damage to national security.

But a secret, 107-page report, prepared by a Department of Defense task force and newly obtained by BuzzFeed News, tells a starkly different story: It says the disclosures were largely insignificant and did not cause any real harm to US interests.

Regarding the hundreds of thousands of Iraq-related military documents and State Department cables provided by the Army private Chelsea Manning, the report assessed “with high confidence that disclosure of the Iraq data set will have no direct personal impact on current and former U.S. leadership in Iraq.”

The 107 page report, redacted, runs 35 pages. Thanks to BuzzFeed News for prying that much of a semblance of the truth out of the government.

It is further proof that US prosecutors and other federal government representatives lie to the courts, the press and the public, whenever its suits their purposes.

Anyone with transcripts from the original Manning hearings, should identify statements by prosecutors at variance with this report, noting the prosecutor’s name, rank and recording the page/line reference in the transcript.

That individual prosecutors and federal law enforcement witnesses lie is a commonly known fact. What I haven’t seen, is a central repository of all such liars and the lies they have told.

I mention a central repository because to say one or two prosecutors have lied or been called down by a judge grabs a headline, but showing a pattern over decades of lying by the state, that could move to an entirely different level.

Judges, even conservative ones (especially conservative ones?), don’t appreciate being lied to by anyone, including the state.

The state has chosen lying as its default mode of operation.

Let’s help them wear that banner.

Interested?

June 19, 2017

DoD Audit Ready By End of September (Which September? Define “ready.”)

Filed under: Government,Government Data — Patrick Durusau @ 8:04 pm

For your Monday amusement: Pentagon Official: DoD will be audit ready by end of September by Eric White.

From the post:

In today’s Federal Newscast, the Defense Department’s Comptroller David Norquist said the department has been properly preparing for its deadline for audit readiness.

The Pentagon’s top financial official said DoD will meet its deadline to be “audit ready” by the end of September. DoD has been working toward the deadline for the better part of seven years, and as the department pointed out in its most recent audit readiness update, most federal agencies haven’t earned clean opinions until they’ve been under full-scale audits for several years. But newly-confirmed comptroller David Norquist said now’s the time to start. He said the department has already contracted with several outside accounting firms to perform the audits, both for the Defense Department’s various components and an overarching audit of the entire department.

I’m reminded of the alleged letter by the Duke of Wellington to Whitehall:

Gentlemen,

Whilst marching from Portugal to a position which commands the approach to Madrid and the French forces, my officers have been diligently complying with your requests which have been sent by H.M. ship from London to Lisbon and thence by dispatch to our headquarters.

We have enumerated our saddles, bridles, tents and tent poles, and all manner of sundry items for which His Majesty’s Government holds me accountable. I have dispatched reports on the character, wit, and spleen of every officer. Each item and every farthing has been accounted for, with two regrettable exceptions for which I beg your indulgence.

Unfortunately the sum of one shilling and ninepence remains unaccounted for in one infantry battalion’s petty cash and there has been a hideous confusion as the the number of jars of raspberry jam issued to one cavalry regiment during a sandstorm in western Spain. This reprehensible carelessness may be related to the pressure of circumstance, since we are war with France, a fact which may come as a bit of a surprise to you gentlemen in Whitehall.

This brings me to my present purpose, which is to request elucidation of my instructions from His Majesty’s Government so that I may better understand why I am dragging an army over these barren plains. I construe that perforce it must be one of two alternative duties, as given below. I shall pursue either one with the best of my ability, but I cannot do both:

1. To train an army of uniformed British clerks in Spain for the benefit of the accountants and copy-boys in London or perchance.

2. To see to it that the forces of Napoleon are driven out of Spain.

Your most obedient servant,

Wellington

The primary function of any military organization is suppression of the currently designated “enemy.”

Congress should direct the Department of Homeland Security (DHS) to auditing the DoD.

Instead of chasing fictional terrorists, DHS staff would be chasing known to exist dollars and alleged expenses.

June 17, 2017

OpSec Reminder

Filed under: Cybersecurity,Government Data,Security — Patrick Durusau @ 9:59 am

Catalin Cimpanu covers a hack of the DoD’s Enhanced Mobile Satellite Services (EMSS) satellite phone network in 2014 in British Hacker Used Home Internet Connection to Hack the DoD in 2014.

The details are amusing but the most important part of Cimpanu’s post is a reminder about OpSec:


In a statement released yesterday, the NCA said it had a solid case against Caffrey because they traced back the attack to his house, and found the stolen data on his computer. Furthermore, officers found an online messaging account linked to the hack on Caffrey’s computer.

Caffrey’s OpSec stumbles:

  1. Connection traced to his computer (No use of Tor or VPN)
  2. Data found on his hard drive (No use of encryption and/or storage elsewhere)
  3. Online account used in hack operated from his computer (Again, no use of Tor or VPN)

I’m sure the hack was a clever one but Caffrey’s OpSec was less so. Decidedly less so.

PS: The National Criminal Agency (NCA) report on Caffrey.

June 16, 2017

FOIA Success Prediction

Filed under: FOIA,Government,Government Data — Patrick Durusau @ 7:57 pm

Will your FOIA request succeed? This new machine will tell you by Benjamin Mullin.

From the post:

Many journalists know the feeling: There could be a cache of documents that might confirm an important story. Your big scoop hinges on one question: Will the government official responsible for the records respond to your FOIA request?

Now, thanks to a new project from a data storage and analysis company, some of the guesswork has been taken out of that question.

Want to know the chances your public records request will get rejected? Plug it into FOIA Predictor, a probability analysis web application from Data.World, and it will provide an estimation of your success based on factors including word count, average sentence length and specificity.

Accuracy?

Best way to gauge that is experience with your FOIA requests.

Try starting at MuckRock.com.

Enjoy!

June 9, 2017

Real Talk on Reality (Knowledge Gap on Leaking)

Filed under: Cybersecurity,Government Data,Leaks,NSA — Patrick Durusau @ 8:32 pm

Real Talk on Reality : Leaking is high risk by the grugq.

From the post:

On June 5th The Intercept released an article based on an anonymously leaked Top Secret NSA document. The article was about one aspect of the Russian cyber campaign against the 2016 US election — the targeting of election device manufacturers. The relevance of this aspect of the Russian operation is not exactly clear, but we’ll address that in a separate post because… just hours after The Intercept’s article went live the US Department of Justice released an affidavit (and search warrant) covering the arrest of Reality Winner — the alleged leaker. Let’s look at that!

You could teach a short course on leaking from this one post but there is one “meta” issue that merits your attention.

The failures of Reality Winner and the Intercept signal users need educating in the art of information leaking.

With wide spread tracking of web browsers, training on information leaking needs to be pushed to users. It would stand out if one member of the military requested and was sent an email lesson on leaking. An email that went to everyone in a particular command, not so much.

Public Service Announcements (PSAs) in web zines, as ads, etc. with only the barest of tips, is another mechanism to consider.

If you are very creative, perhaps “Mr. Bill” claymation episodes with one principle of leaking each? Need to be funny enough that viewing/sharing isn’t suspicious.

Other suggestions?

Raw FBI Uniform Crime Report (UCR) Files for 2015 (NICAR Database Library)

Filed under: FBI,Government Data,Journalism,News,Reporting — Patrick Durusau @ 7:31 pm

IRE & NICAR to freely publish unprocessed data by Charles Minshew.

From the post:

Inspired by our members, IRE is pleased to announce the first release of raw, unprocessed data from the NICAR Database Library.

The contents of the FBI’s Uniform Crime Report (UCR) master file for 2015 are now available for free download on our website. The package contains the original fixed-width files, data dictionaries for the tables as well as the FBI’s UCR user guide. We are planning subsequent releases of other raw data that is not readily available online.

The yearly data from the FBI details arrest and offense numbers for police agencies across the United States. If you download this unprocessed data, expect to do some work to get it in a useable format. The data is fixed-width, across multiple tables, contains many records on a single row that need to be unpacked and in some cases decoded, before being cleaned and imported for use in programs like Excel or your favorite database manager. Not up to the task? We do all of this work in the version of the data that we will soon have for sale in the Database Library.

I have peeked at the data and documentation files and “raw” is the correct term.

Think of it as great exercise for when an already cleaned and formatted data set isn’t available.

More to follow on processing this data set.

(Legal) Office of Personnel Management Data!

Filed under: Government,Government Data,Open Data,Open Government — Patrick Durusau @ 4:30 pm

We’re Sharing A Vast Trove Of Federal Payroll Records by Jeremy Singer-Vine.

From the post:

Today, BuzzFeed News is sharing an enormous dataset — one that sheds light on four decades of the United States’ federal payroll.

The dataset contains hundreds of millions of rows and stretches all the way back to 1973. It provides salary, title, and demographic details about millions of U.S. government employees, as well as their migrations into, out of, and through the federal bureaucracy. In many cases, the data also contains employees’ names.

We obtained the information — nearly 30 gigabytes of it — from the U.S. Office of Personnel Management, via the Freedom of Information Act (FOIA). Now, we’re sharing it with the public. You can download it for free on the Internet Archive.

This is the first time, it seems, that such extensive federal payroll data is freely available online, in bulk. (The Asbury Park Press and FedsDataCenter.com both publish searchable databases. They’re great for browsing, but don’t let you download the data.)

We hope that policy wonks, sociologists, statisticians, fellow journalists — or anyone else, for that matter — find the data useful.

We obtained the information through two Freedom of Information Act requests to OPM. The first chunk of data, provided in response to a request filed in September 2014, covers late 1973 through mid-2014. The second, provided in response to a request filed in December 2015, covers late 2014 through late 2016. We have submitted a third request, pending with the agency, to update the data further.

Between our first and second requests, OPM announced it had suffered a massive computer hack. As a result, the agency told us, it would no longer release certain information, including the employee “pseudo identifier” that had previously disambiguated employees with common names.

What a great data release! Kudos and thanks to BuzzFeed News!

If you need the “pseudo identifiers” for the second or following releases and/or data for the employees withheld (generally the more interesting ones), consult data from the massive computer hack.

Or obtain the excluded data directly from the Office of Personnel Management without permission.

Enjoy!

May 23, 2017

Fiscal Year 2018 Budget

Filed under: Government,Government Data,Politics,Transparency — Patrick Durusau @ 7:23 pm

Fiscal Year 2018 Budget.

In the best pay-to-play tradition, the Government Printing Office (GPO) has these volumes for sale:

America First: A Budget Blueprint To Make America Great Again By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00719-9 ISBN: 9780160937620. Price: $10.00.

Budget of the United States Government, FY 2018 (Paperback Book) By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00723-7 ISBN: 9780160939228. Price: $38.00.

Appendix, Budget of the United States Government, FY 2018 By: Executive Office of the President, Office of Management and Budget GPO Stock # 041-001-00720-2 ISBN: 9780160939334. Price: $79.00.

Budget of the United States Government, FY 2018 (CD-ROM) By: Executive Office of the President, Office of Management and Budget GPO Stock # 041-001-00722-9 ISBN: 9780160939358. Price: $29.00.

Analytical Perspectives, Budget of the United States Government, FY 2018 By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00721-1 ISBN: 9780160939341. Price: $56.00.

Major Savings and Reforms: Budget of the United States Government, Fiscal Year 2018 By: Executive Office of the President, Office of Management and Budget. GPO Stock # 041-001-00724-5 ISBN: 9780160939457. Price: $35.00.

If someone doesn’t beat me to it (very likely), I will be either uploading the CD-ROM and/or pointing you to a location with the contents of the CD-ROM.

As citizens, whether you voted or not, you should have the opportunity to verify news accounts, charges and counter-charges with regard to the budget.

March 29, 2017

What’s Up With Data Padding? (Regulations.gov)

Filed under: Data Quality,Fair Use,Government Data,Intellectual Property (IP),Transparency — Patrick Durusau @ 10:41 am

I forgot to mention in Copyright Troll Hunting – 92,398 Possibles -> 146 Possibles that while using LibreOffice, I deleted a large number of either N/A only or columns not relevant for troll-mining.zip.

Except as otherwise noted, after removal of “no last name,” these fields had N/A for all records except as noted:

  1. L – Implementation Date
  2. M – Effective Date
  3. N – Related RINs
  4. O – Document SubType (Comment(s))
  5. P – Subject
  6. Q – Abstract
  7. R – Status – (Posted, except for 2)
  8. S – Source Citation
  9. T – OMB Approval Number
  10. U – FR Citation
  11. V – Federal Register Number (8 exceptions)
  12. W – Start End Page (8 exceptions)
  13. X – Special Instructions
  14. Y – Legacy ID
  15. Z – Post Mark Date
  16. AA – File Type (1 docx)
  17. AB – Number of Pages
  18. AC – Paper Width
  19. AD – Paper Length
  20. AE – Exhibit Type
  21. AF – Exhibit Location
  22. AG – Document Field_1
  23. AH – Document Field_2

Regulations.gov, not the Copyright Office, is responsible for the collection and management of comments, including the bulked up export of comments.

From the state of the records, one suspects the “bulking up” is NOT an artifact of the export but represents the storage of each record.

One way to test that theory would be a query on the noise fields via the API for Regulations.gov.

The documentation for the API is out-dated, the Field References documentation lacks the Document Detail (field AI), which contains the URL to access the comment.

The closest thing I could find was:

fileFormats Formats of the document, included as URLs to download from the API

How easy/hard it will be to download attachments isn’t clear.

BTW, the comment pages themselves are seriously puffed up. Take https://www.regulations.gov/document?D=COLC-2015-0013-52236.

Saved to disk: 148.6 KB.

Content of the comment: 2.5 KB.

The content of the comment is 1.6 % of the delivered webpage.

It must have taken serious effort to achieve a 98.4% noise to 1.6% signal ratio.

How transparent is data when you have to mine for the 1.6% that is actual content?

January 31, 2017

Executive Orders (Bulk Data From Federal Register)

Filed under: Government,Government Data,Politics — Patrick Durusau @ 2:23 pm

Executive Orders

From the webpage:

The President of the United States manages the operations of the Executive branch of Government through Executive orders. After the President signs an Executive order, the White House sends it to the Office of the Federal Register (OFR). The OFR numbers each order consecutively as part of a series, and publishes it in the daily Federal Register shortly after receipt.

Executive orders issued since 1994 are available as a single bulk download and as a bulk download by President, or you can browse by President and year from the list below. More details about our APIs and other developer tools can be found on our developer pages.

Don’t ignore the developer pages.

Whether friend or foe of the current regime in Washington, the FederalRegister.gov API enables access to all the regulatory material published in the Federal Register. Use it.

It should be especially useful in light of Presidential Executive Order on Reducing Regulation and Controlling Regulatory Costs, which provides in part:


Sec. 2. Regulatory Cap for Fiscal Year 2017. (a) Unless prohibited by law, whenever an executive department or agency (agency) publicly proposes for notice and comment or otherwise promulgates a new regulation, it shall identify at least two existing regulations to be repealed.

Disclaimer: Any resemblance to an executive order is purely coincidental:

January 18, 2017

The CIA’s Secret History Is Now Online [Indexing, Mapping, NLP Anyone?]

Filed under: Government,Government Data,Politics — Patrick Durusau @ 3:59 pm

The CIA’s Secret History Is Now Online by Jason Leopold.

From the post:

Decades ago, the CIA declassified a 26-page secret document cryptically titled “clarifying statement to Fidel Castro concerning assassination.”

It was a step toward greater transparency for one of the most secretive of all federal agencies. But to find out what the document actually said, you had to trek to the National Archives in College Park, Maryland, between the hours of 9 a.m. and 4:30 p.m. and hope that one of only four computers designated by the CIA to access its archives would be available.

But today the CIA posted the Castro record on its website along with more than 12 million pages of the agency’s other declassified documents that have eluded the public, journalists, and historians for nearly two decades. You can view the documents here.

The title of the Castro document, as it turns out, was far more interesting than the contents. It includes a partial transcript of a 1977 transcript between Barbara Walters and Fidel Castro in which she asked the late Cuban dictator whether he had “proof” of the CIA’s last attempt to assassinate him. The transcript was sent to Adm. Stansfield Turner, the CIA director at the time, by a public affairs official at the agency with a note highlighting all references to CIA.

But that’s just one of the millions documents, which date from the 1940s to 1990s, are wide-ranging, covering everything from Nazi war crimes to mind-control experiments to the role the CIA played in overthrowing governments in Chile and Iran. There are also secret documents about a telepathy and precognition program known as Star Gate, files the CIA kept on certain media publications, such as Mother Jones, photographs, more than 100,000 pages of internal intelligence bulletins, policy papers, and memos written by former CIA directors.

Michael Best, @NatSecGeek has pointed out the “CIA de-OCRed at least some of the CREST files before they uploaded them.”

Spy agency class petty. Grant public access but force the restoration of text search.

The restoration of text search work is underway so next steps will be indexing, NLP, mapping, etc.

A great set of documents to get ready for future official and unofficial leaks of CIA documents.

Enjoy!

PS: Curious if any of the search engine vendors will use CREST as demonstration data? Non-trivial size, interesting search issues, etc.

Ask at the next search conference.

November 28, 2016

CIA Cartography [Comparison to other maps?]

Filed under: Cartography,Government Data,Intelligence,Maps — Patrick Durusau @ 10:15 pm

CIA Cartography

From the webpage:

Tracing its roots to October 1941, CIA’s Cartography Center has a long, proud history of service to the Intelligence Community (IC) and continues to respond to a variety of finished intelligence map requirements. The mission of the Cartography Center is to provide a full range of maps, geographic analysis, and research in support of the Agency, the White House, senior policymakers, and the IC at large. Its chief objectives are to analyze geospatial information, extract intelligence-related geodata, and present the information visually in creative and effective ways for maximum understanding by intelligence consumers.

Since 1941, the Cartography Center maps have told the stories of post-WWII reconstruction, the Suez crisis, the Cuban Missile crisis, the Falklands War, and many other important events in history.

There you will find:

Cartography Tools 211 photos

Cartography Maps 1940s 22 photos

Cartography Maps 1950s 14 photos

Cartography Maps 1960s 16 photos

Cartography Maps 1970s 19 photos

Cartography Maps 1980s 12 photos

Cartography Maps 1990s 16 photos

Cartography Maps 2000s 16 photos

Cartography Maps 2010s 15 photos

The albums have this motto at the top:

CIA Cartography Center has been making vital contributions to our Nation’s security, providing policymakers with crucial insights that simply cannot be conveyed through words alone.

President-elect Trump is said to be gaining foreign intelligence from sources other than his national security briefings. Trump is ignoring daily intelligence briefings, relying on ‘a number of sources’ instead. That report is based on a Washington Post account, which puts its credibility somewhere between a conversation overhead in a laundry mat and a stump speech by a member of Congress.

Assuming Trump is gaining intelligence from other sources, just how good are other sources of intelligence?

This release of maps by the CIA, some 160 maps spread from the 1940’s to the 2010’s, provides one axis for evaluating CIA intelligence versus what was commonly known at the time.

As a starting point, may I suggest: Image matching for historical maps comparison by C. Balletti and F. Guerrae, Perimetron, Vol. 4, No. 3, 2009 [180-186] www.e-perimetron.org | ISSN 1790-3769?

Abstract:

In cartographic heritage we suddenly find maps of the same mapmaker and of the same area, published in different years, or new editions due to integration of cartographic, such us in national cartographic series. These maps have the same projective system and the same cut, but they present very small differences. The manual comparison can be very difficult and with uncertain results, because it’s easy to leave some particulars out. It is necessary to find an automatic procedure to compare these maps and a solution can be given by digital maps comparison.

In the last years our experience in cartographic data processing was opted for find new tools for digital comparison and today solution is given by a new software, ACM (Automatic Correlation Map), which finds areas that are candidate to contain differences between two maps. ACM is based on image matching, a key component in almost any image analysis process.

Interesting paper but it presupposes a closeness of the maps that is likely to be missing when comparing CIA maps to other maps of the same places and time period.

I am in the process of locating other tools for map comparison.

Any favorites you would like to suggest?

November 21, 2016

OPM Farce Continues – 2016 Inspector General Report

Filed under: Cybersecurity,Government,Government Data,NSA,Security — Patrick Durusau @ 4:59 pm

U.S. Office of Personnel Management – Office of the Inspector General – Office of Audits

The Office of Personnel Management hack was back in the old days when China was being blamed for every hack. There’s no credible evidence of that but the Chinese were blamed in any event.

The OMP hack illustrated the danger inherent in appointing campaign staff to run mission critical federal agencies. Just a sampling of the impressive depth of Archuleta’s incompetence, read Flash Audit on OPM Infrastructure Update Plan.

The executive summary of the current report offers little room for hope:

This audit report again communicates a material weakness related to OPM’s Security Assessment and Authorization (Authorization) program. In April 2015, the then Chief Information Officer issued a memorandum that granted an extension of the previous Authorizations for all systems whose Authorization had already expired, and for those scheduled to expire through September 2016. Although the moratorium on Authorizations has since been lifted, the effects of the April 2015 memorandum continue to have a significant negative impact on OPM. At the end of fiscal year (FY) 2016, the agency still had at least 18 major systems without a valid Authorization in place.

However, OPM did initiate an “Authorization Sprint” during FY 2016 in an effort to get all of the agency’s systems compliant with the Authorization requirements. We acknowledge that OPM is once again taking system Authorization seriously. We intend to perform a comprehensive audit of OPM’s Authorization process in early FY 2017.

This audit report also re-issues a significant deficiency related to OPM’s information security management structure. Although OPM has developed a security management structure that we believe can be effective, there has been an extremely high turnover rate of critical positions. The negative impact of these staffing issues is apparent in the results of our current FISMA audit work. There has been a significant regression in OPM’s compliance with FISMA requirements, as the agency failed to meet requirements that it had successfully met in prior years. We acknowledge that OPM has placed significant effort toward filling these positions, but simply having the staff does not guarantee that the team can effectively manage information security and keep OPM compliant with FISMA requirements. We will continue to closely monitor activity in this area throughout FY 2017.

It’s illegal but hacking the OPM remains easier than the NSA.

Hacking the NSA requires a job at Booz Allen and a USB drive.

November 11, 2016

“connecting the dots” requires dots (Support Michael Best)

Filed under: Government Data,Politics,Transparency — Patrick Durusau @ 9:45 pm

Michael Best is creating a massive archive of government documents.

From the post:

Since 2015, I’ve published millions of government documents (about 10% of the text items on the Internet Archive, with some items containing thousands of documents) and terabytes of data; but in order to keep going, I need your help. Since I’ve gotten started, no outlet has matched the number of government documents that I’ve published and made freely available. The only non-governmental publisher that rivals the size and scope of the government files I’ve uploaded is WikiLeaks. While I analyze and write about these documents, I consider publishing them to be more important because it enables and empowers an entire generation of journalists, researchers and students of history.

I’ve also pressured government agencies into making their documents more widely available. This includes the more than 13,000,000 pages of CIA documents that are being put online soon, partially in response to my Kickstarter and publishing efforts. These documents are coming from CREST, which is a special CIA database of declassified records. Currently, it can only be accessed from four computers in the world, all of them just outside of Washington D.C. These records, which represent more than 3/4 of a million CIA files, will soon be more accessible than ever – but even once that’s done, there’s a lot more work left to do.

Question: Do you want a transparent and accountable Trump presidency?

Potential Answers include:

1) Yes, but I’m going to spend time and resources hyper-ventilating with others and roaming the streets.

2) Yes, and I’m going to support Michael Best and FOIA efforts.

Governments, even Trump’s presidency, don’t spring from ocean foam.

1024px-sandro_botticelli_-_la_nascita_di_venere_-_google_art_project_-_edited-460

The people chosen fill cabinet and other posts have history, in many cases government history.

For example, I heard a rumor today that Ed Meese, a former government crime lord, is on the Trump transition team. Hell, I thought he was dead.

Michael’s efforts produce the dots that connect past events, places, people, and even present administrations.

The dots Michael produces may support your expose, winning story and/or indictment.

Are you in or out?

November 3, 2016

Attn: Secrecy Bed-Wetters! All Five Volumes of Bay of Pigs History Released!

Filed under: FOIA,Government,Government Data,Transparency — Patrick Durusau @ 4:06 pm

Hand-wringers and bed-wetters who use government secrecy to hide incompetence and errors will sleep less easy tonight.

All Five Volumes of Bay of Pigs History Released and Together at Last: FRINFORMSUM 11/3/2016 by Lauren Harper.

From the post:

After more than twenty years, it appears that fear of exposing the Agency’s dirty linen, rather than any significant security information, is what prompts continued denial of requests for release of these records. Although this volume may do nothing to modify that position, hopefully it does put one of the nastiest internal power struggles into proper perspective for the Agency’s own record.” This is according to Agency historian Jack Pfeiffer, author of the CIA’s long-contested Volume V of its official history of the Bay of Pigs invasion that was released after years of work by the National Security Archive to win the volume’s release. Chief CIA Historian David Robarge states in the cover letter announcing the document’s release that the agency is “releasing this draft volume today because recent 2016 changes in the Freedom of Information Act (FOIA) requires us to release some drafts that are responsive to FOIA requests if they are more than 25 years old.” This improvement – codified by the FOIA Improvement Act of 2016 – came directly from the National Security Archive’s years of litigation.

The CIA argued in court for years – backed by Department of Justice lawyers – that the release of this volume would “confuse the public.” National Security Archive Director Tom Blanton says, “Now the public gets to decide for itself how confusing the CIA can be. How many thousands of taxpayer dollars were wasted trying to hide a CIA historian’s opinion that the Bay of Pigs aftermath degenerated into a nasty internal power struggle?”

To read all five volumes of the CIA’s Official History of the Bay of Pigs Operation – together at last – visit the National Security Archive’s website.

Even the CIA’s own retelling of the story, The Bay of Pigs Invasion, ends with a chilling reminder for all “rebels” being presently supported by the United States.


Brigade 2506’s pleas for air and naval support were refused at the highest US Government levels, although several CIA contract pilots dropped munitions and supplies, resulting in the deaths of four of them: Pete Ray, Leo Baker, Riley Shamburger, and Wade Gray.

Kennedy refused to authorize any extension beyond the hour granted. To this day, there has been no resolution as to what caused this discrepancy in timing.

Without direct air support—no artillery and no weapons—and completely outnumbered by Castro’s forces, members of the Brigade either surrendered or returned to the turquoise water from which they had come.

Two American destroyers attempted to move into the Bay of Pigs to evacuate these members, but gunfire from Cuban forces made that impossible.

In the following days, US entities continued to monitor the waters surrounding the bay in search of survivors, with only a handful being rescued. A few members of the Brigade managed to escape and went into hiding, but soon surrendered due to a lack of food and water. When all was said and done, more than seventy-five percent of Brigade 2506 ended up in Cuban prisons.

100% captured or killed. There’s an example of US support.

In a less media savvy time, the US did pay $53 million (in 1962 dollars, about $424 million today) for the release of 1113 members of Brigade 2506.

Another important fact is that fifty-seven (57) years of delay enabled the participants to escape censure and/or a trip to the gallows for their misdeeds and crimes.

Let’s not let that happen with the full CIA Torture Report. Even the sanitized 6,700 page version would be useful. More so the documents upon which it was based.

All of that exists somewhere. We lack a person with access and moral courage to inform their fellow citizens of the full truth about the CIA torture program. So far.


Update: Michael Best, NatSecGeek advises CIA Histories has the most complete CIA history collection. Thanks Michael!

September 19, 2016

Hackers May Fake Documents, Congress Publishes False Ones

Filed under: Cybersecurity,Government,Government Data — Patrick Durusau @ 12:47 pm

I pointed out in Lions, Tigers, and Lies! Oh My! that Bruce Schneier‘s concerns over the potential for hackers faking documents to be leaked pales beside the mis-information distributed by government.

Executive Summary of Review of the Unauthorized Disclosures of Former National Security Agency Contractor Edward Snowden (their title, not mine), is a case in point.

Barton Gellman in The House Intelligence Committee’s Terrible, Horrible, Very Bad Snowden Report leaves no doubt the House Permanent Select Committee on Intelligence (HPSCI) report is a sack of lies.

Not mistakes, not exaggerations, not simply misleading, but actual, factual lies.

For example:


Since I’m on record claiming the report is dishonest, let’s skip straight to the fourth section. That’s the one that describes Snowden as “a serial exaggerator and fabricator,” with “a pattern of intentional lying.” Here is the evidence adduced for that finding, in its entirety.

“He claimed to have obtained a high school degree equivalent when in fact he never did.”

I do not know how the committee could get this one wrong in good faith. According to the official Maryland State Department of Education test report, which I have reviewed, Snowden sat for the high school equivalency test on May 4, 2004. He needed a score of 2250 to pass. He scored 3550. His Diploma No. 269403 was dated June 2, 2004, the same month he would have graduated had he returned to Arundel High School after losing his sophomore year to mononucleosis. In the interim, he took courses at Anne Arundel Community College.

See Gellman’s post for more examples.

All twenty-two members of the HPSCI signed the report. To save you time in the future, here’s a listing of the members of Congress who agreed to report these lies:

Republicans

Democrats

I sorted each group in to alphabetical order. The original listings were in an order that no doubt makes sense to fellow rodents but not to the casual reader.

That’s twenty-two members of Congress who are willing to distribute known falsehoods.

Does anyone have an equivalent list of hackers?

Older Posts »

Powered by WordPress