You may be right to be paranoid about government surveillance and tracking, because the FBI may not be playing by even its own rules.
A report released yesterday (June 15) by the U.S. Government Accountability Office (GAO) revealed that the FBI's facial-recognition systems have access to 411.9 million photos of individuals, far more than the 29.7 million previously disclosed. If you've got a passport, or hold a driver's license from Illinois, Michigan, Texas, North Carolina or a dozen other states, you're probably in there. The GAO report also criticized the FBI for not adequately performing required assessments of the systems' impact on privacy.
The 382 million additional images the public was not aware of until yesterday were obtained by the FBI from the Department of State, which provided photos from passports and visa applications; the Department of Defense, which provided photos of persons detained by American military forces; and from driver's-license, arrest and prison records held by 16 states. (Notably, there's no mention of any participation by the Department of Homeland Security, which has a reputation for strict enforcement of privacy guidelines.)
Until yesterday, the FBI had disclosed the existence only of the 29.7 million photos in the bureau's Next Generation Identification-Interstate Photo System (NGI-IPS). The NGI photos are mostly mug shots provided by participating federal, state and local law enforcement agencies, and are designed to be accessible by those agencies when searching for the identities of suspects across state lines.
By contrast, the much bigger data set belongs to the FBI's Facial Analysis, Comparison and Evaluation (FACE) system, which is for the FBI's own investigative use. Presumably, local and state police won't get access to the passport, visa, military-detainee and driver's-license photos that aren't in the NGI-IPS database.
FACE is still being developed. Despite what we've seen on TV or in movies, the FBI apparently doesn't yet have instant, infallible facial-recognition technology. (It's possible that other government agencies do.)
The GAO report demands that the Department of Justice slap the FBI on the wrist for failing to follow the DOJ's Privacy Impact Assessment (PIA) procedures, which mandate informing the American public of programs and changes to programs that might affect personal privacy. According to the GAO, the FBI should have told the public when it added the new sets of photos from State, DOD and the states.
With 140 million passport and visa photos, and at least 200 million photos from driver's licenses, the FACE database includes photos of millions of individuals, both U.S. residents and foreigners, who have never been arrested. The GAO report worries about accurate the FACE system really is, and suggests it may lead to false arrest of innocent persons.
However, the database does not hold data on some 400 million individuals, as there are certainly many duplicates. Take the images obtained from the Illinois driver's license database, for example: the GAO report states that the FBI pulled 43 million photos, but the US Census estimates that the state's population is a smidge shy of 13 million. Likewise, Vermont's contribution of 1.8 million driver's-license photos is about three times the number of Vermont residents.