Autopsy 4 Development on GNU/Linux & OS X

[Cube]

Hi guys!

I'm in!! Here my github profile: https://github.com/MalfurionStormrage

I'd like to see an ebuild package for linux gentoo distribution but now, imho, a full review of ant configuration is more useful.
I was thinking to starting the full check just today.

bye!

[Hoyt]

Here's an update:

Both Linux Mint 18 and Autopsy 4.1.0 are out, so testing Autopsy 4.1.0 using TSK 4.3 and libewf_20140608 on Linux Mint 18 Sarah continues. Mac OS X 10.11 El Capitan is next...

[Hoyt]

Another update...

I've forked Basis Tech's GitHub repository, which I've linked to at the bottom of this post. If you want to work with me on this, you'll need to fork that repository, clone it to your local machine, work on it, push changes back to your fork, then send me pull requests through GitHub. You'll see that the first commit I've made isn't signed, but all the rest after that will be. You'll need to sign your commits before I'll bring them in. I'll make pull requests back to sleuthkit/autopsy as we go along.

Before you start, you'll need to make sure your tool chains are set up. The Sleuth Kit and Autopsy documentation cover this. We can talk about it more here if needs be. You'll also want Oracle's Java 8 with the correct variables in ~/.bashrc. You'll also need the TSK source variable there as well. I'm using NetBeans 8.1 for Autopsy proper.

I decided to use AFFLIB after all. Others may have AFF images they're working with, so I used it. I went with afflib-dbg in the repositories. Remember that Linux Mint 18 is based on Ubuntu 16.04 LTS, so you should find it with either OS. It will pull in afflib-tools, libafflib-dev, and libafflib0v5. You only need the dev version and can go with just that, but the other packages are nice if you decide to work with the tools independently. I think we mentioned ITT or elsewhere on the forum that the stable versions of AFFLIB are deprecated and AFF4 isn't ready for prime time yet. This is obviously a deprecated version, but Mint and Ubuntu package managers put everything in the right place (/usr/) so TSK can find things.

I also decided to use libewf from the repositories as well. For Xenial, it's the latest stable version and also gets put into the right place. I couldn't find documentation on the other packages included with libewf-dbg, so I installed that, along with libewf-dev, python-libewf, and libewf2. One or more of those may drag in the others, but it's easy enough to just select all of them and let dpkg sort it out. libewf also gets put in the right places in /usr/. Again, you only need the dev libraries. The tools don't hurt anything and give you other options if you decide to use them independently.

TSK in the repos is one minor version behind, so I built 4.3. I called 'configure' with both AFFLIB and libewf options. You can stop after running 'make', since Autopsy will make calls to your source directory anyway. If you run 'make install', you can use TSK tools independently as well, but this isn't needed by Autopsy. I used /usr/local/src/, added a sleuthkit directory, and cloned the 4.3 release source into that. Use /usr/ when specifying AFFLIB and libewf directories. TSK will 'make install' to /usr/local/ by default and finds AFFLIB and libewf in /usr/ just fine, so go with that. You will need to rename a file after 'make' finishes. At path [your sleuthkit source directory]/bindings/java/dist/, you'll find a file named 'Tsk_DataModel.jar'. Rename this to 'Tsk_DataModel_PostgreSQL.jar' to prevent 'file not found' errors when compiling Autopsy later. Otherwise, I think everything is fine as far as I know right now.

Other Linux/Autopsy builders here have been using Apache Ant from a bash script. I wanted to use NetBeans and that's what I'm doing. Up until now, none of the source code I used had to be changed, but Autopsy does. The latest Java version for Linux from Oracle is 1.8.0_101 and fails Autopsy's version test. The suffix is three digits and the test only accounts for up to two. I added the additional test, then compiled. There were plenty of warning messages, but it built. I'm still getting runtime errors for some parts and there are warning flags in some of the modules, so there's still work to be done.

That's my update for now. Here's the repository link:

https://github.com/Positronikal/autopsy/tree/master

Hoyt

[mary]

I would love to help out with this. I have been wanting to make this happen for a while.

[Hoyt]

After some minor use of the force, i.e. running NetBeans as root (see here), everything compiles without error using the IDE. Running it became a different matter.

Initial run wasn't too bad. I could get it to run even with the previous compile errors, but had at least one obvious runtime problem related to null pointer references. Running NetBeans as root and compiling that way eliminated the compile errors as well as the obvious runtime errors when I ran it. I assume for the time being that root privileges are extended to run and I'm not experiencing some timeout issues under sudo. I may try again as sudo su and see if there's a difference.

As I said, new runtime issues popped up when I attempted to try a test case. For this, I'm using Barry Grundy's ntfs_pract.E01 file available here. Initial ingest without selecting any available modules went fine as far as I can tell. I didn't see anything in the logs to indicate a problem, but there's very little that actually happens under those circumstances and the operations that do happen seem to work alright. I tried running a few ingest modules after initial ingest and ran into some mayhem. I haven't had time to analyze those results in depth yet and I'm not near my test equipment to recall exactly which modules I ran. Some errors appeared to me to be issues related to the fact that Barry's disk image isn't a complete operating system image, so some things weren't found. I don't get errors when I run that image on a Windows workstation, so I'm guessing for now that the way Autopsy et al handles those situations in a Windows environment doesn't translate perfectly to a UNIX environment. Other issues appear at first glance to be deeper and it will take a while to sort those out.

Nothing so far has changed in the repository I linked to above, so just make sure you've cloned it after the date of the last push and you should be able to get to the same place as I am. In any event, the beat goes on...

Hoyt

[Hoyt]

Distinguished forum members... it's time to check your pulse to see if you're still alive out there. It feels awfully lonely here when no one else posts. At least let me know you're interested so I don't decide to stop updating this. Also, no one's obligated to use my repository. The only advantage is that you might get a change before it's merged with the upstream master. There's no guarantee that anything I pr will get merged, either, so there's that. So far, no real changes have occurred and what has occurred has been already merged. Both sleuthkit/autopsy:master and Positronikal/autopsy:master are even as of right now.

Speaking of, here's an update...

As you probably know by now, 4.1.1 came out a couple of days ago. This was an update release to address some module issues after 4.1.0 hit the streets. I ran a build in NetBeans on Mint 18, then diff'd the build log against one of Richard's 4.1.0 build logs from Windows. I've still got the same warnings I had after the first successful 4.1.0 build, but - and this is the good news - they're the same warnings Richard had in his. Everything else is the same, considering the expected differences due to different build platforms. To check your own build logs, find the directory where you stored your project. Mine is in /dev in my home directory. The path is:

[path-to-your-development-directory]/autopsy/build/testuserdir/var/log/

You'll find several different logs there. The one you're looking for is "messages.log". If you're following along and run into snags other than the ones I'm covering, copy/paste the pertinent snippet in a post in this thread with any commentary you have about it. If you use the "Code" button at the top of the editor window when posting, it will look something like this:

Code: Select all

Scanning for modules in /home/hharness/dev/autopsy/netbeans-plat/8.1/harness
Scanning for modules in /home/hharness/dev/autopsy/netbeans-plat/8.1/java
Scanning for modules in /home/hharness/dev/autopsy/netbeans-plat/8.1/platform
Scanning for modules in suite /home/hharness/dev/autopsy
No such dir /home/hharness/dev/autopsy/CoreLibs/test/unit/src; should not define test deps
org.sleuthkit.autopsy.corelibs.download-ivy:
org.sleuthkit.autopsy.corelibs.init-ivy:
org.sleuthkit.autopsy.corelibs.build-native-libs:
Expanding: /home/hharness/dev/autopsy/thirdparty/sigar/1.6.4/sigar-native.zip into home/hharness/dev/autopsy/CoreLibs/release/modules/lib

I'm planning to test modules one at a time today against Barry's image and see what happens then. I first hoped that the module runtime issues that prompted the 4.1.1 update would account for the lion's share of issues I had with 4.1.0 modules, but Brian's email notification does say that the affected modules were Python modules. Again, we'll see. I know the PhotoRec module will break, so I'm saving that for last. If all goes at least reasonably well, I'll try building and running on OS X at least by the end of the week.

I might mention that I think I understand that Brian may have gotten a version of Autopsy 4 to successfully build on a version of OS X. I have no details about that. Maybe he'll post some details when he's ready.

Hoyt

[Hoyt]

Results from the run test...

I tested all user-accessible modules, even though the test image I'm using (Barry Grundy's ntfs_pract.E01) doesn't have all the test data I need. I'll switch to a different image when I have more time to test.

I started with the last successful clean build, then ran it. All went well with that and I brought the case into Autopsy without error. I ran each module in the below order individually and one at a time:
* E01 Verifier = completed successfully.
* Recent Activity = registry exam failed with 3 errors:

Code: Select all

Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.ExtractRegistry analyzeRegistryFiles
INFO: Registry- Now getting registry information from /home/hharness/dev/411test_20160825/411test_20160825/Temp/RecentActivity/reg/NTUSER.DAT
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.ExtractRegistry ripRegistryFile
INFO: Writing RegRipper results to: /home/hharness/dev/411test_20160825/411test_20160825/ModuleOutput/RecentActivity/reg/NTUSER.DAT-regripper-0-autopsy.txt
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.ExtractRegistry executeRegRipper
SEVERE: Unable to run RegRipper
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.ExtractRegistry ripRegistryFile
INFO: Writing Full RegRipper results to: /home/hharness/dev/411test_20160825/411test_20160825/ModuleOutput/RecentActivity/reg/NTUSER.DAT-regripper-0-full.txt
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.ExtractRegistry executeRegRipper
SEVERE: Unable to run RegRipper
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.RAImageIngestModule process
SEVERE: Exception occurred in Registry
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.recentactivity.SearchEngineURLQueryAnalyzer complete
INFO: Search Engine URL Query Analyzer has completed.
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline process
INFO: Recent Activity analysis of ntfs_pract.E01 (jobId=1) finished
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.ingest.DataSourceIngestJob finishFirstStage
INFO: Finished first stage analysis for ntfs_pract.E01 (jobId=2)
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.ingest.DataSourceIngestJob finish
INFO: Finished analysis for ntfs_pract.E01 (jobId=2)
Thu Aug 25 12:01:18 CDT 2016 org.sleuthkit.autopsy.ingest.IngestManager finishIngestJob
INFO: Ingest job 2 completed

* Embedded File Extractor = successful
* Hash Lookup = successful (hash set successfully created in Autopsy also)
* Keyword Search/Indexer & individual keyword search = successful (keyword list successfully created also)
* Extension Mismatch Detector = successful (no optional exceptions selected)
* Exif Parser = successful
* Email Parser = ran without error, but no test data to work against. errors still showing in NetBeans.
* Android Analyzer = ran without error, but no test data to work against
* File Type Identification = successful
*Virtual Machine Extractor = ran without error, but no test data to work against
* PhotoRec Carver = failed - Windows only (this was expected)
* Image Gallery = runs, but does not display graphics
* Timeline = failed with the following:

Code: Select all

java.lang.RuntimeException: ControlsFX Error: ControlsFX 8.40.10 requires at least Java Version 8 Update 40
   at impl.org.controlsfx.version.VersionChecker.doVersionCheck(VersionChecker.java:96)
   at org.controlsfx.control.ControlsFXControl.<init>(ControlsFXControl.java:35)
   at org.controlsfx.control.RangeSlider.<init>(RangeSlider.java:175)
   at org.sleuthkit.autopsy.timeline.ui.ViewFrame.<init>(ViewFrame.java:161)
   at org.sleuthkit.autopsy.timeline.TimeLineTopComponent.initFXComponents(TimeLineTopComponent.java:263)
   at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
   at java.security.AccessController.doPrivileged(Native Method)
   at com.sun.javafx.application.PlatformImpl.lambda$runLater$174(PlatformImpl.java:294)
[catch] at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
   at com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
   at com.sun.glass.ui.gtk.GtkApplication.lambda$null$49(GtkApplication.java:139)
   at java.lang.Thread.run(Thread.java:745)

Additional notes:
* Bookmarks seem to work fine.
* No "Media" tab to display documents, pictures, etc.
* Thumbnail view and show in external viewer both work
* HTML report builds fine. I didn't test the other report types.
* The UI seems a bit wonky, but it's hard to explain why. A "Reset windows to default positions" option might be a good idea.

Some of the above errors have to do with the fact that there are 26 .exe files that Autopsy calls here and there, depending on what it's doing. Linux and Mac won't run those natively, which affects any operation where calls are made to RegRipper, PhotoRec, TestDisk, and others. The solution, at least in my mind, is to include UNIX executables, then write in tests so that the correct ones may be called when needed. It's not as simple as that, though. For example, Autopsy uses two versions of RegRipper - the normal set of plug-ins and a second set customized for Autopsy to produce XML structure that better lends itself to inclusion as Blackboard artifacts. This would require that the same thing be done for UNIX variants.

That's all for now...

Hoyt

[Hoyt]

Up to now I've been working in the master branch. I saw this morning when I browsed my Github news feed that a commit had been made in the develop branch that should fix the controlsfx issue regarding Java version checking. So... today I synced my local develop, pushed that online, and tried a clean build with run tests. There were indeed fixes. The runtime differences I saw compared to my previous post were:
* Timeline = successful (issues related to RegRipper executable notwithstanding)
* Image Gallery = successful, no built-in views for video files
* Media tab on the main UI shows up now. Picture files can be seen in the UI, but not videos (same as Image Gallery)

This is very good news.

Issues remaining:
- Native views for documents, i.e. pdf. doc, docx, etc.
- Native views for videos
- Regripper/PhotoRec/TestDisk .exe problem. These files are:
- photorec_win.exe = main PhotoRec executable for Windows
- qphotorec_win.exe = same thing, but for Qt4 (may not be used per se)
- testdisk_win.exe = main TestDisk executable (may not be used)
- rr.exe = RegRipper as a Windows GUI
- rip.exe = RegRipper as a Windows CLI

This is all the time I'll have for this for a while probably.

Hoyt

[fdicarlo]

Here's an update:

Both Linux Mint 18 and Autopsy 4.1.0 are out, so testing Autopsy 4.1.0 using TSK 4.3 and libewf_20140608 on Linux Mint 18 Sarah continues. Mac OS X 10.11 El Capitan is next...

Hi Hoyt, I didn't find your github repository but I would love to test and help you, since I'm interested in the same. I setup a VM with a minimal Ubuntu and my main system is Linux Mint 18 as you. Please let me know how to help.

[Hoyt]

Hello fdiacarlo...

I've been a bit distracted lately and haven't worked on this since OSDFCon. From there I was working on the RegRipper scripts trying to get them to run properly under Linux. There's that, along with the other binaries, such as PhotoRec, that have to be addressed. The big question in my head at the time was about whether or not it would be better to simply make calls to those after they've been natively installed instead of trying to run them as proper modules. When you start thinking about it, it can become quite a dilemma either way.

In any case, here's my Github:

https://github.com/Positronikal/autopsy

Beyond that, I'll be back at work on this soon. I've got a few irons in the fire that are hotter. I'm glad you're pitching in. Open discussions about the above issues and where we generally should be going with it are the neediest needs we have right now, I think.

Hoyt