running ingest again: duplicates?

Any Autopsy specific discussions, events, module releases, that don't fall into the other categories.

Moderator: carrier

running ingest again: duplicates?

Postby CyCriGuy » Wed Aug 17, 2016 6:35 am

Hi everybody!

Yesterday autopsy crashed while running an ingest. Adding files from .e01 worked fine, but a few hours later autopsy crashed (probably because the hard drive containing the .e01 died).
The ingest modules did not complete, but I already do have a lot of results.

If I run the ingest modules again over the original data source already added to autopsy and the extracted virtual machine, will I have duplicate entries in the case?

Thank you!

My setup:
client /ingesting-machine: Autopsy 4.1.0 on windows 7 64bit
Server: PostgreSQL 9.3.13 and activemq 5.6.0 on Ubuntu 14 LTS, Bitnami Solr 4.10.3 running on client
CyCriGuy
 
Posts: 2
Joined: Wed Aug 17, 2016 6:21 am

Re: running ingest again: duplicates?

Postby Hoyt » Tue Sep 20, 2016 3:40 pm

Some modules will duplicate, some won't. I've researched some modules, but not all. Another consideration is deciding which modules you run and when. Some modules depend on the output of other modules. Some of those modules are written to call their dependent module, but I don't think all are. Here are some of my findings by module if it helps:

1. EO1 Verifier = Can be ran anytime; nothing depends on it; depends on source date being EW format; doesn't duplicate since it doesn't populate Tree Viewer.

2. Recent Activity = Can be ran anytime; nothing depends on it; calls own dependencies; duplicates Tree Viewer stats.

3. PhotoRec Carver = Can be ran anytime, but output only addressed by subsequent modules; nothing depends on it per se; depends on nothing manually; Tree Viewer duplication unknown, but probably does.

4. Embedded File Extractor = Run after or concurrent with #3 to include those results; nothing depends on it per se; depends on nothing manually; duplicates Tree Viewer stats.

5. Hash Lookup = Run after or concurrent with #3 and #4 to include those results; nothing depends on it; depends on nothing manually; doesn't duplicate Tree Viewer stats.

6. File Type Identification = Run after or concurrent with #3 and #4 to include those results; nothing depends on it that doesn't call it automatically; calls own dependencies; doesn't duplicate Tree Viewer stats.

7. Keyword Search = Also runs indexer, so run after or concurrent with #3 and #4; nothing depends on Keyword Search, but Interesting Files Finder may depend on indexer, so run Keyword Search before that even without specifying any keyword lists; doesn't add duplicates to index, but does add duplicates to keywords results in Tree Viewer.

8. Interesting Files Finder = Run after or concurrent with #3, #4, and #7; nothing depends on it; may depend on Keyword Search; Tree Viewer duplication unknown.

9. Extension Mismatch Detector = Run after or concurrent with #3, #4, and #6; nothing depends on it; may depend on Scalpel module, but that's not user defined; duplicates Tree Viewer results.

10. Exif Parser = Run after or concurrent with #3 and #4; nothing depends on it; may depend on File Type Identification (untested); doesn't duplicate.

11. Email Parser = Run after or concurrent with #3 and #4; nothing depends on it; may depend on File Type Identification (untested); doesn't duplicate.

12. Android Analyzer = Run after or concurrent with #3 and #4; nothing depends on it; may depend on File Type Identification (untested); doesn't duplicate.

IMPORTANT: These are cursory observations and are subject to error. Verify for yourself before relying on this information. Also, there are new features in 4.1.1 that helps with this, notably an indicator that lets you know if you've already ran a module in a case. This might not work as expected after an Autopsy crash, however.

I've tinkered with database de-duplication with varying results, none satisfying. I do believe that this issue can be ultimately avoided. Separating the indexer from the keyword search module would be a plus, I think, but would be a non-trivial refactor. For now, at least, it's just something to be aware of. You'll likely need to put in some extra work regarding your use of bookmarks/tags so that your final report represents a more accurate picture without too much artificial stat inflation.


Hoyt
Hoyt
 
Posts: 61
Joined: Thu Dec 11, 2014 4:02 am
Location: Little Rock, AR


Return to Autopsy General

Who is online

Users browsing this forum: No registered users and 2 guests

cron