DHS' Pilots for Social
Media Screening Need
Increased Rigor to Ensure
Scalability and Long-term
Success
(Redacted)
February 27, 2017
OIG-17-40
FOR OFFICIAL USE ONLY
DHS OIG HIGHLIGHTS
DHS’ Pilots for Social Media Screening Need Increased
Rigor to Ensure Scalability and Long-term Success
February 27,
2017
Why We
Did This
Inspection
Following the December
2015 terrorist attack in
San Bernardino,
California, Congress
raised concerns about
the use of social media
by terrorist groups and
requested that the
Department of Homeland
Security expand social
media background
checks. We conducted
this inspection to review
DHS’ social media pilot
programs.
What We
Recommend
We made one
recommendation to help
ensure DHS develops an
effective social media
screening program.
For Further Information:
Contact our Office of Public Affairs at
(202) 254-4100, or email us at
DHS-OIG.OfficePublicAffai[email protected]
What We Found
DHS has established a task force for using social media to
screen applicants for immigration benefits. In connection
with that effort, U.S. Citizenship and Immigration Services
(USCIS) began pilots to expand social media screening of
immigration applicants. Additionally, Immigration and
Customs Enforcement (ICE) independently began a pilot to
use social media screening during the visa issuance process.
However, these pilots, on which DHS plans to base future
department-wide use of social media screening, lack criteria
for measuring performance to ensure they meet their
objectives. Although the pilots include some objectives, such
as determining the effectiveness of an automated search tool
and assessing data collection and dissemination procedures,
it is not clear DHS is measuring and evaluating the pilots’
results to determine how well they are performing against set
criteria. Absent measurement criteria, the pilots may provide
limited information for planning and implementing an
effective, department-wide future social media screening
program.
DHS Response
I&A, on behalf of itself, USCIS, and ICE, concurred with the
recommendation and has begun taking corrective actions.
www.oig.dhs.gov OIG-17-40
FOR OFFICIAL USE ONLY
FOR
OFFICIAL
USE
ONLY
OFFICE
OF
INSPECTOR
GENERAL
Department
of
Homeland
Security
Washington
,
DC
20528
/
www
.
oig.dhs
.gov
February
27
,
2017
MEMORANDUM FOR: David
J.
Glawe
Acting
Under
Secretary
Office
of
Intelligenoe
and
Analysis
Lori Scia1abba
Acting
Director
U.S.
Citizenship
and
Immigration
Services
Thomas
D. Homan
__
_
Acting
Director
U.S.
Immigration
and
Customs
Enforcement
FROM
:
John
Roth
~k~h_
Inspector
General
SUBJECT
:
DHS'
Pilots
for
Social Media Screening
Need
Increased
Rigor to
Ensure
Scalability
and
Long-term Success -
For
Officia1. Use
Omy
For
your
action
is
our
final
report
,
DHS'
Pilots
for
Social
Media
Screening Need
Increased Rigor to En.sure Scalability
and
Long-term Success. We
incorporated
the
formal!
comments
provided
by
you
r office.
The
report
contains
one
recommendation
aimed
at
improving
DHS
'
social
media
scrieening
pilots
.
The
Office
of
Intelligence
and
Analysis
il~
concurred
with
tlte
recommendation.
Based
on
information
provided
in
the
response
to
the
draft
:repon,
we
consider
the
recommendation
open
and
resolved.
Once
the
recommendation
has
been
fully
implemented,
p
lea5e
submit
a
formal
doseout
letter
to
us
within
30
days
so
that
we
may
close
the
recommendation
.
The
memorandum
sh
o
uld
be
accompanied
by
evidence
of
completion
of
agreed
-
upon
corrective
actions.
Please
send
your
response
or
closure
request
to
OIGi
nspec
tionsFollov.n..ipg)oig.dhs.gov.
--
--
··
-- ·-·
Consistent
\vi.th
our
respons
ibility
under
the
Inspector General Act.
we
will
provide
copies
of
our
report
to
congressional
committees
with
oversight
and
appropriation
responsibility
over
the
Department
of
Home
land
Security. We will
post
a
redacted
version
of
the
report
on
our
website.
Please
call
me
with
any
questions,
or
your
staJf
may
amt.act
Laurel
Rim.on,
Acting Ac;sistant
Inspector
General
for
Inspections
and
Evaluations
or
Angela
Garvin
.
Deputy
Assistant
Inspector
General,
at
f202
)
254
-
4100.
www.oig.dhs.gov
OIG-
17-40
FOR
OFFICIPaL
USE
ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
Background
Following the December 2, 2015 terrorist attack in San Bernardino, California,
lawmakers became increasingly concerned about the use of social media by
terrorist groups. On December 15, 2015, Senator Jeanne Shaheen and 24
other Senators sent a letter to the Secretary of Homeland Security requesting
that, as soon as possible, DHS expand social media background checks,
focused on possible connections to terrorist activity, to screening for visa
determinations for visitors and immigrants. In response, the then DHS
Secretary and Deputy Secretary asked the Under Secretary for Intelligence and
Analysis (I&A) to lead a task force to review the Department’s current use of
social media and identify options to optimize its use across DHS. The task force
comprises senior representatives and staff from DHS, U.S. Citizenship and
Immigration Services (USCIS), U.S. Immigration and Customs Enforcement
(ICE), U.S. Customs and Border Protection, the Transportation Security
Administration, and DHS oversight offices including the Office of General
Counsel, Privacy Office, and Office for Civil Rights and Civil Liberties, which
have a vested interest in DHS’ use of social media.
According to the task force’s implementation plan, the first priority was to
conduct a pilot at USCIS to test systematic screening of applicants for
immigration benefits.
1
USCIS had used social media in a limited capacity but
had no experience using it as a large-scale screening tool. In December 2015,
USCIS started screening the social media accounts of a limited number of
applying for status, using both manual and
automated screening. The task force intended to use the December 2015 pilot
and lessons learned from other DHS components’ use of social media to
develop policies and processes for standardized use of social media
department-wide. Additionally, to expand social media screening across all
DHS components, the task force recommended, and the Secretary approved,
creation of a DHS Social Media Center of Excellence (COE).
2
(Appendix B
contains details about the COE.) In April 2016, USCIS expanded social media
screening, testing another automated tool together with manual screening. In
1
USCIS grants immigration benefits to people who meet the eligibility requirements to stay in
the United States temporarily or permanently. These benefits include granting U.S. citizenship
to eligible individuals, authorizing individuals to reside permanently in the United States, and
authorizing aliens to work in the United States.
2
During our review it was called the “Center of Excellence”; however, the name was later
changed to “Shared Social Media Screening Service”. We refer to it as the “Center of Excellence”
in this report.
www.oig.dhs.gov 1 OIG-17-40
FOR OFFICIAL USE ONLY
USCIS started a pilot in December 2015 to screen the social media accounts of
and applicants for status. The pilot’s objective was to
examine the feasibility of using social media screening with an automated
search tool called and determine whether useful information for
adjudicating refugee applications could be obtained.
6
Although the pilot had an
objective, it did not define what would constitute a successful outcome, nor did
it identify metrics against which to benchmark its findings. For example, the
pilot discovered that individuals had confirmed social media
accounts; likely had a social media account; individuals had unconfirmed
3
American embassies and consulates issue nonimmigrant visas authorizing temporary visits to
the United States for individuals who do not intend to become permanent residents.
4
GAO-12-208G, January 2012
5
GAO-09-45, November 2008
6
, a social media analytics tool, covers a large number of social media platforms,
has access to third-party information providers, and can access web-based information.
www.oig.dhs.gov 2 OIG-17-40
FOR OFFICIAL USE ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
August 2016, ICE began using a different automated tool to screen the social
media of nonimmigrant visa holders .
3
Results of Inspection
DHS has established a task force for using social media to screen applicants
for immigration benefits. In connection with that effort, USCIS began pilots to
expand social media screening of immigration applicants. Additionally, ICE
independently began a pilot to use social media screening during the visa
issuance process. However, these pilots, on which DHS plans to base future
department-wide use of social media screening, lack criteria for measuring
performance to ensure they meet their objectives. Although the pilots include
some objectives, such as determining the effectiveness of an automated search
tool and assessing data collection and dissemination procedures, it is not clear
DHS is measuring and evaluating the pilots’ results to determine how well they
are performing. Absent measurement criteria, the pilots will be of limited use in
planning and implementing an effective, department-wide future social media
screening program.
The Government Accountability Office (GAO) has issued best practices for an
effective pilot phase of a program. According to GAO, “The pilot phase allows
for a check on whether program operations occur as expected.”
4
GAO has also
stated that pilot programs should have well-defined, clear, and measurable
objectives; criteria or standards for determining pilot performance; and a plan
to track the pilot’s performance and evaluate the final results.
5
Following these
best practices can increase the rigor of the pilots to ensure scalability and long-
term success.
accounts; and individuals had no identified social media account. USCIS
concluded that the individuals had a minimal presence on
social media platforms accessible through . Because the pilot did
not have metrics to measure success, it is difficult to conclude whether finding
individuals with confirmed social media accounts constitutes success.
The purpose of another USCIS pilot, which began in April 2016, was to expand
screening to an additional , predominately , applicants
through use of the Defense Advanced Research Projects Agency (DARPA)
tool.
7
The applicants were asked to voluntarily give their social media user
names.
8
USCIS then screened the user names against
using the tool; USCIS also manually screened the user names against
. USCIS assessed identified accounts to determine whether the
refugees were linked to derogatory social media information that could impact
their eligibility for immigration benefits or admissibility into the United States.
Using the tool and manual screening, USCIS identified
individuals with confirmed social media accounts and individuals with
unconfirmed accounts. In reviewing the pilot, USCIS concluded that the
tool was not a viable option for automated social media screening and that
manual review was more effective at identifying accounts. USCIS based its
conclusion on the tool’s low “match confidence.” Because the resulting
accounts identified by the tool did not always match up with the applicants,
officers had to manually check the results. However, USCIS did not establish
match benchmarks for the tool, so it does not know what level of match
confidence would signify success or failure.
In August 2016, ICE independently began a pilot to screen the social media
activity of nonimmigrant visa applicants from
. The intent of this pilot is to
evaluate the capability of to conduct initial
screening of social media activity during visa application and continue social
media monitoring during the .
9
According to the pilot plan, this approach will complement existing background
7
DARPA is a social media screening platform that screens against .
8
National Security Council staff requested USCIS and the Department of State, which also
participated, to pilot the elicitation of the applicants’ social media identifiers. USCIS is
to access social media platforms by DHS Privacy
regulations and using government accounts on social media platforms may introduce
additional risk to government systems. D HS and its components have of
identifying social media account for individuals that may not be
appropriate or viable for large scale screening of applicants for immigration benefits.
9
is a web search tool that specializes in social media data
exploitation by analyzing social media data and funneling it into actionable information.
www.oig.dhs.gov 3 OIG-17-40
FOR OFFICIAL USE ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
checks conducted in conjunction with the Department of State and help
identify potential derogatory information not found in Government databases.
Although ICE has identified several measurable criteria for data that will be
collected during the pilot, it has not articulated a plan to use these metrics to
evaluate performance and determine overall program effectiveness. In our
opinion, a clearly defined performance evaluation is essential to determining
whether, how, and when to integrate pilot activities into overall efforts and will
enable the Department to implement a more effective social media screening
program.
DHS and the task force are planning department-wide use of social media for
screening based on these pilots. Because components are not measuring their
pilots against clear success criteria, DHS may not be able to make informed
decisions when it designs its social media screening program. The task force
needs to better manage ongoing efforts to pilot social media screening to help
ensure an effective and successful social media screening program in the
future.
Recommendation
We recommend that the Under Secretary of Intelligence and Analysis
coordinate with USCIS and ICE to develop and implement a plan to evaluate
the performance of social media screening pilots that includes features such as
well-defined, clear, and measurable objectives and standards for determining
pilot performance to help ensure DHS develops an effective social media
screening program.
Management Comments and OIG Analysis
I&A responded on behalf of itself, ICE, and USCIS. The components concurred
with our recommendation and provided comments to the draft report. A
summary of the management response and our analysis follows. We have
included a copy of the management comments in their entirety in appendix A.
I&A also provided technical comments to our report, which we incorporated, as
appropriate.
I&A’s Response to the Recommendation: I&A concurred with the
recommendation and stated that DHS has taken steps to improve its social
media screening pilots by implementing a four-pronged approach that
measures performance to develop consistent benchmarks and continue
improving performance to ensure rigor and scalability for long-term success.
This approach includes using qualitative and quantitative criteria for
measuring tool performance; collecting and analyzing comprehensive
www.oig.dhs.gov 4 OIG-17-40
FOR OFFICIAL USE ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
performance metrics of ongoing research and development pilots; reporting
project milestones to the task force; and reporting select metrics measuring
pilot performance in a weekly task force agenda. Additionally, DHS will
continue to identify and refine new and existing metrics to measure social
media screening pilots’ performance, to include identifying benchmarks for the
various tools tested.
OIG Analysis: I&A’s planned actions are partially responsive to this
recommendation. This recommendation is resolved, but will remain open until
I&A provides documentation of well-defined, clear, and measurable objectives
and standards for determining pilot performance. I&A’s response indicated that
a new social media screening pilot was scheduled to begin in January 2017.
I&A should take this opportunity to develop and implement GAO’s best
practices for an effective pilot program to ensure scalability and long-term
success of the pilot.
I&A also emphasized that, at the launch of the task force, neither the private
sector nor the U.S. Government possessed the capabilities for large-scale social
media screening. OIG recognizes that DHS has made strides in identifying and
assessing social media screening technology and conducting pilots that will
enable the development of such capabilities. The Homeland Security Systems
Engineering & Development Institute report, Social Media Analytics Capability
Testing: Independent Assessment, referenced in I&A’s response, provides
evidence of some of the progress that has been made in leveraging technology
to meet the Department’s needs. The report was not received by OIG until
January 6, 2017, and therefore was not included in our analysis. We note that
this independent assessment includes many metrics that may be useful in
establishing benchmarks and determining performance of future pilot
programs. The assessment included seven steps in conducting evaluations for
quantitative analysis: (1) define the purpose of the evaluation; (2) elaborate a
task model; (3) define top-level quality characteristics; (4) produce system
requirements; (5) define metrics to measure requirements; (6) define techniques
to measure metrics; and (7) carry out and interpret the evaluation. This
framework was followed for the independent assessment and should be
continued with testing pilots.
Objective, Scope, and Methodology
DHS OIG was established by the Homeland Security Act of 2002 (Public Law
107−296) by amendment to the Inspector General Act of 1978.
The initial objective of this inspection was to evaluate DHS’ progress in
developing social media screening policies, procedures, and capabilities.
www.oig.dhs.gov 5 OIG-17-40
FOR OFFICIAL USE ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
However, upon discovering the lack of performance measures in USCIS’ and
ICE’s pilots for social media screening for immigration benefits, we adjusted
our scope and reviewed DHS’ social media task force and DHS’ use of pilots to
help create a social media screening program. Although we reviewed DHS
policies relevant to social media screening, including DHS Instruction Number
110–01-001, Privacy Policy for Operational Use of Social Media, we could not
evaluate specific policies and procedures for the pilots because they had not
been written. Likewise, we reviewed the task force’s concept of operations for
the role of the COE, but could not evaluate the COE at this time because it has
not been set up or funded.
We interviewed DHS staff from the Office of Policy, USCIS’ Fraud Detection and
National Security Directorate, ICE’s Visa Security Program and Homeland
Security Investigations, Science and Technology Directorate, Privacy Office,
Office for Civil Rights and Civil Liberties, and the Chair of the Social Media task
force to determine their roles in social media screening.
We conducted this inspection between July and October 2016 pursuant to the
Inspector General Act of 1978, as amended, and according to the Quality
Standards for Inspection and Evaluation issued by the Council of the Inspectors
General on Integrity and Efficiency.
The Office of Inspections and Evaluations major contributors to this report are
William McCarron, Chief Inspector; Natalie Fussell Enclade, Lead Inspector;
Carie Mellies, Lead Inspector; Stephen Farrell, Inspector; Adam Robinson,
Inspector; James Alver, Inspector; Kelly Herberger, Communications Analyst;
and Elizabeth Kingma, Independent Referencer.
www.oig.dhs.gov 6 OIG-17-40
FOR OFFICIAL USE ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
Appendix B
Center of Excellence
The COE concept of operations envisions the center conducting social media
activities on behalf of components that lack resources or expertise and
supporting component-level social media screening programs. The COE will
also identify new social media tools and opportunities, test and evaluate new
and emerging technologies, and identify best practices for the appropriate
sharing and collaborative use of social media within DHS and with external
partners. The COE will set standards for social media use in relevant DHS
operations while ensuring privacy and civil rights and civil liberties protections.
The task force also developed a proposal for the creation of a social media
board and social media council to replace the existing task force and oversee
the future governance of DHS’ social media use and the COE. According to the
proposed charter, the social media board will be a senior-level independent
group that will ensure awareness of social media use and capabilities among
DHS leadership and control DHS’ social media use programs. The social media
council will ensure working-level communication for social media use and
capabilities, develop new policies and procedures, oversee performance
management, and promote information sharing.
www.oig.dhs.gov 11 OIG-17-40
FOR OFFICIAL USE ONLY
FOR OFFICIAL USE ONLY
OFFICE OF INSPECTOR GENERAL
Department of Homeland Security
Appendix C
Report Distribution
Department of Homeland Security
Secretary
Deputy Secretary
Chief of Staff
Deputy Chiefs of Staff
General Counsel
Executive Secretary
Director, GAO/OIG Liaison Office
Assistant Secretary for Office of Policy
Assistant Secretary for Office of Public Affairs
Assistant Secretary for Office of Legislative Affairs
Office of Intelligence and Analysis Audit Liaison
USCIS Audit Liaison
ICE Audit Liaison
S&T Audit Liaison
Office of Management and Budget
Chief, Homeland Security Branch
DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
www.oig.dhs.gov 12 OIG-17-40
FOR OFFICIAL USE ONLY
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports, please visit our website at: www.oig.dhs.gov.
For further information or questions, please contact Office of Inspector General Public Affairs
at: DHS-OIG.OfficeP[email protected]hs.gov. Follow us on Twitter at: @dhsoig.
OIG HOTLINE
To report fraud, waste, or abuse, visit our website at www.oig.dhs.gov and click on the red
"Hotline" tab. If you cannot access our website, call our hotline at (800) 323-8603, fax our
hotline at (202) 254-4297, or write to us at:
Department of Homeland Security
Office of Inspector General, Mail Stop 0305
Attention: Hotline
245 Murray Drive, SW
Washington, DC 20528-0305
