Archive for the ‘Open Source Intelligence’ Category

Open Source Data Jeopardizing Cleared Personnel:… (School Yearbooks?)

Wednesday, May 17th, 2017

Open Source Data Jeopardizing Cleared Personnel: Intelligence Operations Outsmarted by Technology by Alexander H. Georgiades.

Abstract:

The availability and accessibility of Open Source Intelligence (OSINT) combined with the information from data breaches has affected cleared personnel in the United States Intelligence Community (IC) and Department of Defense (DoD) who conduct and support intelligence operations. This information when used in conjunction with biometric detection technology at border crossings has greatly improved the likelihood of cleared personnel from the United States Government (USG) of being identified and targeted by adversaries. The shift from traditional Tactics, Techniques, and Procedures (TTPs) used by cleared personnel (either operating in an overt or covert status) during the Cold War when biometric technology was not an obstacle, has caught the United States government intelligence services off-guard when conducting sensitive missions Outside of the Continental United States (OCONUS).

The consequences of not maintaining updated software and hardware standards have already affected U.S. intelligence operations and exposed cleared personnel. The computer breach at the Office of Personnel and Management (OPM), where millions of sensitive records from cleared personnel in the private and public sectors is the most recent example. This unprecedented loss of Personally Identifiable Information (PII) has been the unfortunate wakeup call needed for decision makers in the United States government to reevaluate how they handle, collect, store, and protect the information of cleared personnel in this digital age.

The analysis of competing hypothesis and other predictive analytical methods will be used to evaluate the data available to adversaries who target cleared personnel and the intelligence operations they support. Case studies, news articles, books, government, and industry reports will be used as supporting evidence to illustrate how the growth in biometric detection technology use in conjunction with the availability of OSINT and material from data breaches adversely affect intelligence operations.

The amount of information available to adversaries is at an unprecedented level. Open source forums provide detailed information about cleared personnel and government TTPs that can be used by adversaries to unravel intelligence operations, target cleared personnel, and jeopardize USG equities (such as sources and methods) in the field. The cleared workforce must learn from mistakes of complacency and poor tradecraft in the past to develop new methodologies to neutralize the effectiveness of adversaries who use OSINT and biometric technology to their advantage.

Social media use by cleared employees who reveal too much operational information about themselves or the projects they work on is one of the gateways that can be easily closed to adversaries. Cleared personnel must be mandated to limit the amount of information they publish online. By closing the door to social media and preventing the personal and professional lives of the cleared workforce from being used to target them, adversaries would not be as effective in jeopardizing or exposing intelligence operations overseas. Increased Operational Security (OPSEC) procedures must also be mandated to protect the programs and operations these cleared personnel work on, with an emphasis on covert officers who use false personas when operating overseas.

The information bridges that were created after September 11, 2001 to increase collaboration must be reevaluated to determine if the relaxation of classified information safeguards and storage of sensitive information is now becoming detrimental to USG intelligence operations and cleared personnel.

As you know, I have little sympathy for the Intelligence Community (IC), creators of the fishbowl in which we commonly reside. Members of the IC sharing that fate, has a ring of justice to it.

This thesis offers a general overview of the problem and should be good to spark ideas of open source intelligence that can be used to corroborate or contradict other sources of intelligence.

By way of example, educational records are easy enough to edit and convincing to anyone not aware they have been edited.

On the other hand, original and digitized year books or similar contemporary resources, are not so easily manipulated.

As I say that, tracking every child from first grade through the end of their academic career, is eminently doable, with the main obstacle being acquisition of the original yearbooks.

Cross-referencing other large collections of photos and the project starts to sound useful to any number of governments, especially those worried about operatives from Western countries.

Are you worried about Western operatives?

Building a Keyword Monitoring Pipeline… (Think Download Before Removal)

Wednesday, April 19th, 2017

Building a Keyword Monitoring Pipeline with Python, Pastebin and Searx by Justin Seitz.

From the post:

Having an early warning system is an incredibly useful tool in the OSINT world. Being able to monitor search engines and other sites for keywords, IP addresses, document names, or email addresses is extremely useful. This can tell you if an adversary, competitor or a friendly ally is talking about you online. In this blog post we are going to setup a keyword monitoring pipeline so that we can monitor both popular search engines and Pastebin for keywords, leaked credentials, or anything else we are interested in.

The pipeline will be designed to alert you whenever one of those keywords is discovered or if you are seeing movement for a keyword on a particular search engine.

Learning of data that was posted but is no longer available, is a sad thing.

Increase your odds of grabbing data before removal by following Justin’s post.

A couple of caveats:

  • I would not use GMail, preferring a Tor mail solution, especially for tracking Pastebin postings.
  • Use and rotate at random VPN connections for your Searx setup.

Going completely dark takes more time and effort than most of us can spare, but you can avoid being like a new car dealership with search lights crossing the sky.

Dark Web OSINT With Python Part Three: Visualization

Thursday, September 1st, 2016

Dark Web OSINT With Python Part Three: Visualization by Justin.

From the post:

Welcome back! In this series of blog posts we are wrapping the awesome OnionScan tool and then analyzing the data that falls out of it. If you haven’t read parts one and two in this series then you should go do that first. In this post we are going to analyze our data in a new light by visualizing how hidden services are linked together as well as how hidden services are linked to clearnet sites.

One of the awesome things that OnionScan does is look for links between hidden services and clearnet sites and makes these links available to us in the JSON output. Additionally it looks for IP address leaks or references to IP addresses that could be used for deanonymization.

We are going to extract these connections and create visualizations that will assist us in looking at interesting connections, popular hidden services with a high number of links and along the way learn some Python and how to use Gephi, a visualization tool. Let’s get started!

Jason tops off this great series on OnionScan by teaching the rudiments of using Gephi to visualize and explore the resulting data.

Can you map yourself from the Dark Web to visible site?

If so, you aren’t hidden well enough.

Dark Web OSINT with Python Part Two: … [Prizes For Unmasking Government Sites?]

Wednesday, August 10th, 2016

Dark Web OSINT with Python Part Two: SSH Keys and Shodan by Justin.

From the post:

Welcome back good Python soldiers. In Part One of this series we created a wrapper around OnionScan, a fantastic tool created by Sarah Jamie Lewis (@sarajamielewis). If you haven’t read Part One then go do so now. Now that you have a bunch of data (or you downloaded it from here) we want to do some analysis and further intelligence gathering with it. Here are a few objectives we are going to cover in the rest of the series.

  1. Attempt to discover clearnet servers that share SSH fingerprints with hidden services, using Shodan. As part of this we will also analyze whether the same SSH key is shared amongst hidden services.
  2. Map out connections between hidden services, clearnet sites and any IP address leaks.
  3. Discover clusters of sites that are similar based on their index pages, this can help find knockoffs or clones of “legitimate” sites. We’ll use a machine learning library called scikit-learn to achieve this.

The scripts that were created for this series are quick little one-offs, so there is some shared code between each script. Feel free to tighten this up into a function or a module you can import. The goal is to give you little chunks of code that will teach you some basics on how to begin analyzing some of the data and more importantly to give you some ideas on how you can use it for your own purposes.

In this post we are going to look at how to connect hidden services by their SSH public key fingerprints, as well as how to expand our intelligence gathering using Shodan. Let’s get started!

Expand your Dark Web OSINT intell skills!

Being mindful that if you can discover your Dark Web site, so can others.

Anyone awarding Black Hat conference registrations for unmasking government sites on the Dark Web?

How-To Track Projects Like A Defense Contractor

Sunday, July 31st, 2016

Transparency Tip: How to Track Government Projects Like a Defense Contractor by Dave Maass.

From the post:

Over the last year, thousands of pages of sensitive documents outlining the government’s intelligence practices have landed on our desktops.

One set of documents describes the Director of National Intelligence’s goal of funding “dramatic improvements in unconstrained face recognition.” A presentation from the Navy uses examples from Star Trek to explain its electronic warfare program. Other records show the FBI was purchasing mobile phone extraction devices, malware and fiber network-tapping systems. A sign-in list shows the names and contact details of hundreds of cybersecurity contractors who turned up a Department of Homeland Security “Industry Day.” Yet another document, a heavily redacted contract, provides details of U.S. assistance with drone surveillance programs in Burundi, Kenya and Uganda.

But these aren’t top-secret records carefully leaked to journalists. They aren’t classified dossiers pasted haphazardly on the Internet by hacktivists. They weren’t even liberated through the Freedom of Information Act. No, these public documents are available to anyone who looks at the U.S. government’s contracting website, FBO.gov. In this case “anyone,” is usually just contractors looking to sell goods, services, or research to the government. But, because the government often makes itself more accessible to businesses than the general public, it’s also a useful tool for watchdogs. Every government program costs money, and whenever money is involved, there’s a paper trail.

Searching FBO.gov is difficult enough that there are firms that offer search services to assist contractors with locating business opportunities.

Collating FBO.gov data with topic maps (read adding non-FBO.gov data) will be a value-add to watchdogs, potential contractors (including yourself), or watchers watching watchers.

Dave’s post will get you started on your way.

Dark Web OSINT With Python and OnionScan: Part One

Saturday, July 30th, 2016

Dark Web OSINT With Python and OnionScan: Part One by Justin.

When you tire of what passes for political discussion on Twitter and/or Facebook this weekend, why not try your hand at something useful?

Like looking for data leaks on the Dark Web?

You could, in theory at least, notify the sites of their data leaks. 😉

One of the aspects of announced leaks that never ceases to amaze me are reports that read:

Well, we pawned the (some string of letters) database and then notified them of the issue.

Before getting a copy of the entire database? What’s the point?

All you have accomplished is making another breach more difficult and demonstrating your ability to breach a system where the root password was most likely “god.”

Anyway, Justin gets you started on seeking data leaks on the Dark Web saying:

You may have heard of this awesome tool called OnionScan that is used to scan hidden services in the dark web looking for potential data leaks. Recently the project released some cool visualizations and a high level description of what their scanning results looked like. What they didn’t provide is how to actually go about scanning as much of the dark web as possible, and then how to produce those very cool visualizations that they show.

At a high level we need to do the following:

  1. Setup a server somewhere to host our scanner 24/7 because it takes some time to do the scanning work.
  2. Get TOR running on the server.
  3. Get OnionScan setup.
  4. Write some Python to handle the scanning and some of the other data management to deal with the scan results.
  5. Write some more Python to make some cool graphs. (Part Two of the series)

Let’s get started!

Very much looking forward to Part 2!

Enjoy!

Automatically Finding Weapons…

Wednesday, January 13th, 2016

Automatically Finding Weapons in Social Media Images Part 1 by Justin Seitz.

From the post:

As part of my previous post on gangs in Detroit, one thing had struck me: there are an awful lot of guns being waved around on social media. Shocker, I know. More importantly I began to wonder if there wasn’t a way to automatically identify when a social media post has guns or other weapons contained in them. This post will cover how to use a couple of techniques to send images to the Imagga API that will automatically tag pictures with keywords that it feels accurately describe some of the objects contained within the picture. As well, I will teach you how to use some slicing and dicing techniques in Python to help increase the accuracy of the tagging. Keep in mind that I am specifically looking for guns or firearm-related keywords, but you can easily just change the list of keywords you are interested in and try to find other things of interest like tanks, or rockets.

This blog post will cover how to handle the image tagging portion of this task. In a follow up post I will cover how to pull down all Tweets from an account and extract all the images that the user has posted (something my students do all the time!).

This rocks!

Whether you are trying to make contact with a weapon owner who isn’t in the “business” of selling guns or if you are looking for like-minded individuals, this is a great post.

Would make an interesting way to broadly tag images for inclusion in group subjects in a topic map, awaiting further refinement by algorithm or humans.

This is a great blog to follow: Automating OSINT.

Open Source Intelligence Techniques:… (review)

Wednesday, June 10th, 2015

Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information by CyberWarrior.

From the post:

Author Michael Bazzell has been well known and respected in government circles for his ability to locate personal information about any target through Open Source Intelligence (OSINT). In this book, he shares his methods in great detail. Each step of his process is explained throughout sixteen chapters of specialized websites, application programming interfaces, and software solutions. Based on his live and online video training at IntelTechniques.com, over 250 resources are identified with narrative tutorials and screen captures.

This book will serve as a reference guide for anyone that is responsible for the collection of online content. It is written in a hands-on style that encourages the reader to execute the tutorials as they go. The search techniques offered will inspire analysts to “think outside the box” when scouring the internet for personal information.

On the flip side, Open Source Intelligence Techniques is must reading for anyone who is charged with avoiding disclosure of information that can be matched with other open source intelligence.

How many people has your agency outed today?