Archive for the ‘Leaks’ Category

Manning Leaks — No Real Harm (Database of Government Liars Anyone?)

Tuesday, June 20th, 2017

From the post:

In the seven years since WikiLeaks published the largest leak of classified documents in history, the federal government has said they caused enormous damage to national security.

But a secret, 107-page report, prepared by a Department of Defense task force and newly obtained by BuzzFeed News, tells a starkly different story: It says the disclosures were largely insignificant and did not cause any real harm to US interests.

Regarding the hundreds of thousands of Iraq-related military documents and State Department cables provided by the Army private Chelsea Manning, the report assessed “with high confidence that disclosure of the Iraq data set will have no direct personal impact on current and former U.S. leadership in Iraq.”

The 107 page report, redacted, runs 35 pages. Thanks to BuzzFeed News for prying that much of a semblance of the truth out of the government.

It is further proof that US prosecutors and other federal government representatives lie to the courts, the press and the public, whenever its suits their purposes.

Anyone with transcripts from the original Manning hearings, should identify statements by prosecutors at variance with this report, noting the prosecutor’s name, rank and recording the page/line reference in the transcript.

That individual prosecutors and federal law enforcement witnesses lie is a commonly known fact. What I haven’t seen, is a central repository of all such liars and the lies they have told.

I mention a central repository because to say one or two prosecutors have lied or been called down by a judge grabs a headline, but showing a pattern over decades of lying by the state, that could move to an entirely different level.

Judges, even conservative ones (especially conservative ones?), don’t appreciate being lied to by anyone, including the state.

The state has chosen lying as its default mode of operation.

Let’s help them wear that banner.

Interested?

Real Talk on Reality (Knowledge Gap on Leaking)

Friday, June 9th, 2017

Real Talk on Reality : Leaking is high risk by the grugq.

From the post:

On June 5th The Intercept released an article based on an anonymously leaked Top Secret NSA document. The article was about one aspect of the Russian cyber campaign against the 2016 US election — the targeting of election device manufacturers. The relevance of this aspect of the Russian operation is not exactly clear, but we’ll address that in a separate post because… just hours after The Intercept’s article went live the US Department of Justice released an affidavit (and search warrant) covering the arrest of Reality Winner — the alleged leaker. Let’s look at that!

You could teach a short course on leaking from this one post but there is one “meta” issue that merits your attention.

The failures of Reality Winner and the Intercept signal users need educating in the art of information leaking.

With wide spread tracking of web browsers, training on information leaking needs to be pushed to users. It would stand out if one member of the military requested and was sent an email lesson on leaking. An email that went to everyone in a particular command, not so much.

Public Service Announcements (PSAs) in web zines, as ads, etc. with only the barest of tips, is another mechanism to consider.

If you are very creative, perhaps “Mr. Bill” claymation episodes with one principle of leaking each? Need to be funny enough that viewing/sharing isn’t suspicious.

Other suggestions?

Practical Suggestions For Improving Transparency

Tuesday, May 2nd, 2017

A crowd wail about Presidents Obama, Trump, opacity, lack of transparency, loss of democracy, freedom of the press, the imminent death of civilization, etc., isn’t going to improve transparency.

I have two practical suggestions for improving transparency.

First suggestion: Always re-post, tweet, share stories with links to leaked materials. If the story you read doesn’t have such a link, seek out one that does to re-post, tweet, share.

Some stories of leaks include a URL to the leaked material, like Hacker leaks Orange is the New Black new season after ransom demands ignored by Sean Gallagher, or NSA-leaking Shadow Brokers just dumped its most damaging release yet by Dan Goodin, both of Ars Technica

Some stories of the same leaks do not include a URL to the leaked material,The Netflix ‘Orange is the New Black’ Leak Shows TV Piracy Is So 2012 (which does have the best strategy for fighting piracy I have ever read) or, Shadow Brokers leak trove of NSA hacking tools.

Second suggestion: If you encounter leaked materials, post, tweet and share them as widely as possible. (Translations are always needed.)

Improving transparency requires only internet access and the initiative to do so.

Are you game?

Other Methods for Boiling Vault 7: CIA Hacking Tools Revealed?

Friday, March 24th, 2017

You may have other methods for boiling content out of the Wikileaks Vault 7: CIA Hacking Tools Revealed.

To that end, here is the list of deduped files.

Warning: The Wikileaks pages I have encountered are so malformed that repair will always be necessary before using XQuery.

Enjoy!

Efficient Querying of Vault 7: CIA Hacking Tools Revealed

Friday, March 24th, 2017

This week we have covered:

1. Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 1) Eliminated duplication and editorial artifacts, 1134 HTML files out of 7809 remain.
2. Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 2 – The PDF Files) Eliminated public and placeholder documents, 114 arguably CIA files remain.
3. CIA Documents or Reports of CIA Documents? Vault7 All of the HTML files are reports of possibly CIA material but we do know HTML file != CIA document.
4. Boiling Reports of CIA Documents (Wikileaks CIA Vault 7 CIA Hacking Tools Revealed) The HTML files contain a large amount of cruft, which can be extracted using XQuery and common tools.

Interesting, from a certain point of view, but aside from highlighting bloated leaking from Wikileaks, why should anyone care?

Good question!

Let’s compare the de-duped but raw with the de-duped but boiled document set.

De-duped but raw document set:

De-duped and boiled document set:

In raw count, boiling took us from 2,131,135 words/tokens to 665,202 words/tokens.

Here’s a question for busy reporters/researchers:

Has the CIA compromised the Tor network?

In the raw files, Tor occurs 22660 times.

In the boiled files, Tor occurs 4 times.

Here’s a screen shot of the occurrences:

With TextSTAT, select the occurrence in the concordance and another select (mouse click to non-specialists) takes you to:

In a matter of seconds, you can answer as far as the HTML documents of Vault7 Part1 show, the CIA is talking about top of rack (ToR), a switching architecture for networks. Not, the Tor Project.

What other questions do you want to pose to the boiled Vault 7: CIA Hacking Tools Revealed document set?

Tooling up for efficient queries

First, you need: Boiled Content of Unique CIA Vault 7 Hacking Tools Revealed Files.

Second, grab a copy of: TextSTAT – Simple Text Analysis Tool runs on Windows, GNU/Linux and MacOS. (free)

When you first open TextSTAT, it will invite you to create a copora.

The barrel icon to the far left creates a new corpora. Select it and:

Once you save the new corpora, this reminder about encodings pops up:

I haven’t explored loading Windows files while on a Linux box but will and report back. Interesting to see inclusion of PDF. Something we need to explore after we choose which of the 124 possibly CIA PDF files to import.

Finally, you are at the point of navigating to where you have stored the unzipped Boiled Content of Unique CIA Vault 7 Hacking Tools Revealed Files:

Select the first file, scroll to the end of the list, press shift and select the last file. Then choose OK. It takes a minute or so to load but it has a progress bar to let you know it is working.

Observations on TextSTAT

As far as I can tell, TextSTAT doesn’t use the traditional stop list of words but enables you to set of maximum and minimum occurrences in the Word Form window. Along with wildcards as well. More flexible than the old stop list practice.

BTW, the context right/left on the Concordance window refers to characters, not words/tokens. Another departure from my experience with concordances. Not a criticism, just an observation of something that puzzled me at first.

Conclusion

The benefit of making secret information available, a la Wikileaks cannot be over-stated.

But making secret information available isn’t the same as making it accessible.

Investigators, reporters, researchers, the oft-mentioned public, all benefit from accessible information.

Next week look for a review of the probably CIA PDF files to see which ones I would incorporate into the corpora. (You may include more or less.)

PS: I’m looking for telecommuting work, editing, research (see this blog), patrick@durusau.net.

Boiling Reports of CIA Documents (Wikileaks CIA Vault 7 CIA Hacking Tools Revealed)

Thursday, March 23rd, 2017

Before you read today’s installment on the Wikileaks CIA Vault 7 CIA Hacking Tools Revealed, you should check out the latest drop from Wikileaks: CIA Vault 7 Dark Matter. Five documents and they all look interesting.

I started with a fresh copy of the HTML files in a completely new directory and ran Tidy first, plus fixed:

page_26345506.html:<declarations><string name="½ö"></string></declarations><p>›<br>

which I described in: Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 1).

So with a clean and well-formed set of files, I modified the XQuery to collect all of the references to prior versions. Reasoning that any file that was a prior version, we can ditch that, leaving only the latest files.

 for $doc in collection('collection.xml')//a[matches(.,'^\d+$')] return ($doc/@href/string(), ' ')  Unlike the count function we used before, this returns the value of the href attribute and appends a new line, after each one. I saved that listing to priors.txt and then (your experience may vary on this next step): xargs rm < priors.txt WARNING: If your file names have spaces in them, you may delete files unintentionally. My data had no such spaces so this works in this case. Once I had the set of files without those representing “previous versions,” I’m down to the expected 1134. That’s still a fair number of files and there is a lot of cruft in them. For variety I did look at XSLT, but these are some empty XSLT template statements needed to clean these files: <xsl:template match="style"/>  <xsl:template match="//div[@id = 'submit_wlkey']" /> <xsl:template match="//div[@id = 'submit_help_contact']" /> <xsl:template match="//div[@id = 'submit_help_tor']" /> <xsl:template match="//div[@id = 'submit_help_tips']" /> <xsl:template match="//div[@id = 'submit_help_after']" /> <xsl:template match="//div[@id = 'submit']" /> <xsl:template match="//div[@id = 'submit_help_buttons']" /> <xsl:template match="//div[@id = 'efm-button']" /> <xsl:template match="//div[@id = 'top-navigation']" /> <xsl:template match="//div[@id = 'menu']" /> <xsl:template match="//footer" /> <xsl:template match="//script" />  <xsl:template match="//li[@class = 'comment']" /> Compare the XQuery query, on the command line no less: for file in *.html; do java -cp /home/patrick/saxon/saxon9he.jar net.sf.saxon.Query -s:"$file" -qs:"/html/body//div[@id = 'uniquer']" -o:"$file.new" done (The line break in front of -qs: is strictly for formatting for this post.) The files generated here will not be valid HTML. Easy enough to fix with another round of Tidy. After running Tidy, I was surprised to see a large number of very small files. Or at least I interpret 296 files of less than 1K in size to be very small files. I created a list of them, linked back to the Wikileaks originals (296 Under 1K Wikileaks CIA Vault 7 Hacking Tools Revealed Files) so you can verify that I capture the content reported by Wikileaks. Oh, and here are the files I generated as well, Boiled Content of Unique CIA Vault 7 Hacking Tools Revealed Files. In case you are interested, boiling the 1134 files took them from 38.6 MB to 8.8 MB of actual content for indexing, searching, concordances, etc. Using the content only files, tomorrow I will illustrate how you can correlate information across files. Stay tuned! Leak Publication: Sharing, Crediting, and Re-Using Leaks Wednesday, March 22nd, 2017 If you substitute “leak” for “data” in this essay by Daniella Lowenberg, does it work for leaks as well? From the post: In the most basic terms- Data Publishing is the process of making research data publicly available for re-use. But even in this simple statement there are many misconceptions about what Data Publications are and why they are necessary for the future of scholarly communications. Let’s break down a commonly accepted definition of “research data publishing”. A Data Publication has three core features: 1 – data that are publicly accessible and are preserved for an indefinite amount of time, 2 – descriptive information about the data (metadata), and 3 – a citation for the data (giving credit to the data). Why are these elements essential? These three features make research data reusable and reproducible- the goal of a Data Publication. As much as I admire the work of the International Consortium of Investigative Journalists (ICIJ, especially its Panama Papers project, sharing data beyond the confines of their community isn’t a value, much less a goal. As all secret keepers, government, industry, organizations, ICIJ has “reasons” for its secrecy, but none that I find any more or less convincing than those offered by other secret keepers. Every secret keeper has an agenda their secrecy serves. Agendas that which don’t include a public empowered to make judgments about their secret keeping. The ICIJ proclaims Leak to Us. A good place to leak but include with your leak a demand, an unconditional demand, that your leak be released in its entirely within a year or two of its first publication. Help enable the public to watch all secrets and secret keepers, not just those some secret keepers choose to expose. CIA Documents or Reports of CIA Documents? Vault7 Wednesday, March 22nd, 2017 As I tool up to analyze the 1134 non-duplicate/artifact HTML files in Vault 7: CIA Hacking Tools Revealed, it occurred to me those aren’t “CIA documents.” Take Weeping Angel (Extending) Engineering Notes as an example. Caveat: My range of experience with “CIA documents” is limited to those obtained by Michael Best and others using Freedom of Information Act requests. But that should be sufficient to identify “CIA documents.” Some things I notice about Weeping Angel (Extending) Engineering Notes: 1. A Wikileaks header with donation button. 2. “Vault 7: CIA Hacking Tools Revealed” 3. Wikileaks navigation 4. reported text 5. More Wikileaks navigation 6. Ads for Wikileaks, Tor, Tails, Courage, bitcoin I’m going to say that the 1134 non-duplicate/artifact HTML files in Vault7, Part1, are reports of portions (which portions is unknown) of some unknown number of CIA documents. A distinction that influences searching, indexing, concordances, word frequency, just to name a few. What I need is the reported text, minus: 1. A Wikileaks header with donation button. 2. “Vault 7: CIA Hacking Tools Revealed” 3. Wikileaks navigation 4. More Wikileaks navigation 5. Ads for Wikileaks, Tor, Tails, Courage, bitcoin Check in tomorrow when I boil 1134 reports of CIA documents to get something better suited for text analysis. Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 2 – The PDF Files) Tuesday, March 21st, 2017 You may want to read Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 1) before reading this post. In Part 1, I walk you through obtaining a copy of Wikileaks’ Vault 7: CIA Hacking Tools Revealed so you can follow and check my analysis and conclusions. Fact checking applies to every source, including this blog. I proofed my listing of the 357 PDF files in the first Vault 7 release and report an increase in arguably CIA files and a slight decline in public documents. An increase from 114 to 125 for the CIA and a decrease from 109 to 98 for public documents. 1. Arguably CIA – 125 2. Public – 98 3. Wikileaks placeholders – 134 The listings to date: For public documents, I created hyperlinks whenever possible. (Saying a fact and having evidence are two different things.) Vendor documentation that was not marked with a security classification I counted as public. All I can say for the Wikileaks placeholders, some 134 of them, is to ignore them unless you like mining low grade ore. I created notes in the CIA listing to help narrow your focus down to highly relevant materials. I have more commentary in the works but I wanted to release these listings in case they help others make efficient use of their time. Enjoy! PS: A question I want to start addressing this week is how the dilution of a leak impacts the use of same? Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 1) Monday, March 20th, 2017 Executive Summary: If you reported Vault 7: CIA Hacking Tools Revealed as containing: 8,761 documents and files from an isolated, high-security network situated inside the CIA’s Center for Cyber Intelligence in Langley, Virgina…. (Vault 7: CIA Hacking Tools Revealed) you failed to check your facts. I detail my process below but in terms of numbers: 1. Of 7809 HTML files, 6675 are duplicates or Wikileaks artifacts 2. Of 357 PDF files, 134 are Wikileaks artifacts (for materials not released). Of the remaining 223 PDF files, 109 of them are public information, the GNU Make Manual for instance. Out of the 357 pdf files, Wikileaks has delivered 114 arguably from the CIA and some of those are doubtful. (Part 2, forthcoming) Wikileaks haters will find little solace here. My criticisms of Wikileaks are for padding the leak and not enabling effective use of the leak. Padding the leak is obvious from the inclusion of numerous duplicate and irrelevant documents. Effective use of the leak is impaired by the padding but also by teases of what could have been released but wasn’t. Getting Started To start on common ground, fire up a torrent client, obtain and decompress: Wikileaks-Year-Zero-2017-v1.7z.torrent. Decompression requires this password: SplinterItIntoAThousandPiecesAndScatterItIntoTheWinds The root directory is year0. When I run a recursive ls from above that directory: ls -R year0 | wc -l My system reports: 8820 Change to the year0 directory and ls reveals: bootstrap/ css/ highlighter/ IMG/ localhost:6081@ static/ vault7/ Checking the files in vault7: ls -R vault7/ | wc -l returns: 8755 Change to the vault7 directory and ls shows: cms/ files/ index.html logo.png The directory files/ has only one file, org-chart.png. An organization chart of the CIA but with sub-departments are listed with acronyms and “???.” Did the author of the chart not know the names of those departments? I point that out as the first of many file anomalies. Some 7809 HTML files are found under cms/. The cms/ directory has a sub-directory files, plus main.css and 7809 HTML files (including the index.html file). Duplicated HTML Files I discovered duplication of the HTML files quite by accident. I had prepared the files with Tidy for parsing with Saxon and compressed a set of those files for uploading. The 7808 files I compressed started at 296.7 MB. The compressed size, using 7z, was approximately 3.5 MB. That’s almost 2 order of magnitude of compression. 7z is good, but it’s not quite that good. 😉 Checking my file compression numbers You don’t have to take my word for the file compression experience. If you select all the page_*, space_* and user_* HTML files in a file browser, it should report a total size of 296.7 MB. Create a sub-directory to year0/vault7/cms/, say mkdir htmlfiles and then: cp *.html htmlfiles Then: cd htmlfiles and, 7z a compressedhtml.7z *.html Run: ls -l compressedhtml.7z Result: 3488727 Mar 16 16:31 compressedhtml.7z Tom Harris, in How File Compression Works, explains that: Most types of computer files are fairly redundant — they have the same information listed over and over again. File-compression programs simply get rid of the redundancy. Instead of listing a piece of information over and over again, a file-compression program lists that information once and then refers back to it whenever it appears in the original program. If you don’t agree the HTML file are highly repetitive, check the next section where one source of duplication is demonstrated. Demonstrating Duplication of HTML files Let’s start with the same file as we look for a source of duplication. Load Cinnamon Cisco881 Testing at Wikileaks into your browser. Scroll to near the bottom of the file where you see: Yes! There are 136 prior versions of this alleged CIA file in the directory. Cinnamon Cisco881 Testinghas the most prior versions but all of them have prior versions. Are we now in agreement that duplicated versions of the HTML pages exist in the year0/vault7/cms/ directory? Good! Now we need to count how many duplicated files there are in year0/vault7/cms/. Counting Prior Versions of the HTML Files You may or may not have noticed but every reference to a prior version takes the form: <a href=”filename.html”>integer</a*gt; That going to be an important fact but let’s clean up the HTML so we can process it with XQuery/Saxon. Preparing for XQuery Before we start crunching the HTML files, let’s clean them up with Tidy. Here’s my Tidy config file: output-xml: yes quote-nbsp: no show-warnings: no show-info: no quiet: yes write-back: yes In htmlfiles I run: tidy -config tidy.config *.html Tidy reports two errors:  line 887 column 1 - Error: is not recognized! line 887 column 15 - Error: is not recognized!  Grepping for “declarations>”: grep "declarations" *.html Returns: page_26345506.html:<declarations><string name="½ö"></string></declarations><p>›<br> The string element is present as well so we open up the file and repair it with XML comments: <!-- <declarations><string name="½ö"></string></declarations><p>›<br> --> <!-- prior line commented out to avoid Tidy error, pld 14 March 2017--> Rerun Tidy: tidy -config tidy.config *.html Now Tidy returns no errors. XQuery Finds Prior Versions Our files are ready to be queried but 7809 is a lot of files. There are a number of solutions but a simple one is to create an XML collection of the documents and run our XQuery statements across the files as a set. Here’s how I created a collection file for these files: I did an ls in the directory and piped that to collection.xml. Opening the file I deleted index.html, started each entry with <doc href=" and ended each one with "/>, inserted <collection> before the first entry and </collection> after the last entry and then saved the file. Your version should look something like: <collection> <doc href="page_10158081.html"/> <doc href="page_10158088.html"/> <doc href="page_10452995.html"/> ... <doc href="user_7995631.html"/> <doc href="user_8650754.html"/> <doc href="user_9535837.html"/> </collection>  The prior versions in Cinnamon Cisco881 Testing from Wikileaks, have this appearance in HTML source: <h3>Previous versions:</h3> <p>| <a href=”page_17760540.html”>1</a> <span class=”pg-tag”><i>empty</i></span> | <a href=”page_17760578.html”>2</a> <span class=”pg-tag”></span> ….. | <a href=”page_23134323.html”>135</a> <span class=”pg-tag”>[Xetron]</span> | <a href=”page_23134377.html”>136</a> <span class=”pg-tag”>[Xetron]</span> |</p> </div> You will need to spend some time with the files (I have obviously) to satisfy yourself that <a> elements that contain only numbers are exclusively used for prior references. If you come across any counter-examples, I would be very interested to hear about them. To get a file count on all the prior references, I used: let$count := count(collection('collection.xml')//a[matches(.,'^\d+$')]) return$count

Run that script to find: 6514 previous editions of the base files

Unpacking the XQuery

Rest assured that’s not how I wrote the first XQuery on this data set! 😉

Without exploring all the by-ways and alleys I traversed, I will unpack that query.

First, the goal of the query is to identify every <a> element that only contains digits. Recalling that previous versions link have digits only in their <a> elements.

A shout out to Jonathan Robie, Editor of XQuery, for reminding me that string expressions match substrings unless they are have beginning and ending line anchors. Here:

'^\d+$' The \d matches only digits, the + enables matching 1 or more digits, and the beginning ^ and ending $ eliminate any <a> elements that might start with one or more digits, but also contains text. Like links to files, etc.

Expanding out a bit more, [matches(.,'^\d+$')], the [ ] enclose a predicate that consist of the matches function, which takes two arguments. The . here represents the content of an <a> element, followed by a comma as a separator and then the regex that provides the pattern to match against. Although talked about as a “code smell,” the //a in //a[matches(.,'^\d+$')] enables us to pick up the <a> elements wherever they are located. We did have to repair these HTML files and I don’t want to spend time debugging ephemeral HTML.

Almost there! The collection file, along with the collection function, collection('collection.xml') enables us to apply the XQuery to all the files listed in the collection file.

Finally, we surround all of the foregoing with the count function: count(collection('collection.xml')//a[matches(.,'^\d+$')]) and declare a variable to capture the result of the count function: let$count :=

So far so good? I know, tedious for XQuery jocks but not all news reporters are XQuery jocks, at least not yet!

Then we produce the results: return \$count.

But 6514 files aren’t 6675 files, you said 6675 files

Yes, your right! Thanks for paying attention!

I said at the top, 6675 are duplicates or Wikileaks artifacts.

Where are the others?

If you look at User #71477, which has the file name, user_40828931.html, you will find it’s not a CIA document but part of Wikileaks administration for these documents. There are 90 such pages.

If you look at Marble Framework, which has the file name, space_15204359.html, you find it’s a CIA document but a form of indexing created by Wikileaks. There are 70 such pages.

Don’t forget the index.html page.

When added together, 6514 (duplicates), 90 (user pages), 70 (space pages), index.html, I get 6675 duplicates or Wikileaks artifacts.

In Fact Checking Wikileaks’ Vault 7: CIA Hacking Tools Revealed (Part 2), I look under year0/vault7/cms/files to discover: