Ir al contenido

Advanced Techniques and Data Ingestion with Plaso (log2timeline.py)

Expert techniques to optimize data ingestion performance, collection filters, and YARA signature analysis in Plaso.

0views

This document assumes you have already mastered the concepts covered in the Plaso fundamentals manual (basic timeline creation, time zones, parsers, and VSS).

We will focus on advanced use cases: how to maximize CPU performance, how to filter data before processing it (during file ingestion), how to extract signature information via YARA rules, and how to manage error logs in large-scale forensic investigations.


Analyzing disks of 1 TB or larger can take entire days. Optimizing Plaso's parallel engine reduces this critical incident response time to just a few hours.

Plaso is natively multi-process. We can explicitly define the number of CPU worker processes (workers) and modulate the size of the memory buffers:

  • ✔ --workers <NÚMERO>: Specifies the number of simultaneous processes to launch for low-level extraction and analysis.
    • Guideline: It is recommended to define a number of workers equivalent to the total number of your CPU's physical cores (e.g., --workers 8). In compressed E01 images, the operation may be limited by the decompression processor bottleneck. On RAW disk drives mounted on fast SSDs, a higher number of workers results in a higher I/O rate.
  • ✔ --buffer_size <TAMAÑO>: Controls the capacity in megabytes of the intermediate transfer buffer used by each worker. By default, it is assigned 1024MB.
    • Guideline: On analytical systems equipped with large amounts of RAM (e.g., 64 GB), we can increase this rate to --buffer_size 2048M to improve performance and minimize hot disk writes.

Example of an optimized command for analytical machines (16 workers, 64GB RAM):

Sección titulada «Ejemplo de comando optimizado para máquinas analíticas (16 workers, 64GB RAM):»
Ventana de terminal
log2timeline.py --workers 16 --buffer_size 2048M --status_view linear /opt/forensics/cases/case-001/my_timeline.plaso /opt/forensics/cases/case-001/image.E01

In immediate response interventions or triage-style analysis, processing millions of temporary files, browser cache files, or irrelevant system files is highly inefficient. We are only interested in execution artifacts and the Windows registry.

Through the --file_filter parameter, we can instruct Plaso to index and analyze only the paths that match a collection filter template:

Ventana de terminal
--file_filter <CAMINO_AL_ARCHIVO_FILTRO>
  • Note: This filter does not work with complex regular expressions; instead, it validates whether the file path contained in the image starts with any of the lines in the filter text file.
  1. Create a directive text file called /opt/forensics/cases/case-001/triage_filter.txt with the following list of key paths:
    /Windows/System32/config/
    /Users/%/NTUSER.DAT
    /Users/%/AppData/Local/Microsoft/Windows/UsrClass.dat
    /Users/%/AppData/Roaming/Microsoft/Windows/Recent/
    /Users/%/AppData/Local/Microsoft/Windows/WebCache/
    /Windows/Prefetch/
  2. Run Plaso ingestion by calling the prepared filter:
    Ventana de terminal
    log2timeline.py --file_filter /opt/forensics/cases/case-001/triage_filter.txt /opt/forensics/cases/case-001/my_triage_timeline.plaso /opt/forensics/cases/case-001/image.E01

With this action, 90% of irrelevant user and system files will be skipped, reducing processing time from 24 hours to less than 30 minutes.


4. Case Study 3: YARA Signature-Based Filtering

Sección titulada «4. Casuística 3: Filtrado Basado en Firmas YARA»

It is required to automatically classify, detect, and tag any file that matches malicious rules or signatures during the same ingestion and timeline construction phase.

Plaso can load YARA rules on the fly and evaluate each analyzed file before indexing it:

Ventana de terminal
# Ejecutar ingesta evaluando firmas yara
log2timeline.py --yara_rules /opt/forensics/rules/custom-signatures.yar /opt/forensics/cases/case-001/my_timeline.plaso /opt/forensics/cases/case-001/image.E01

If a binary (e.g., c:\temp\mimikatz.exe) matches any of the rules contained in the signature file, Plaso will immediately generate an event with the data_type “fs:yara_match” in the timeline. This allows direct correlation of the threat creation date with nearby network connections or registry modifications.


5. Case Study 4: Extended Extraction of Hashes and Resources

Sección titulada «5. Casuística 4: Extracción Ampliada de Hashes y Recursos»

In the case of deep malware analysis or the need to correlate the descriptive text of Windows event IDs (which by default only show numeric codes).

We can force file hash calculation during ingestion and force string resolution from EVTX files:

Ventana de terminal
# Calcular hashes SHA-1 y SHA-256 de todos los archivos en la ingesta
log2timeline.py --hashers "sha1,sha256" ...
# Extraer texto y descripciones completas de los mensajes de los logs EVTX de Windows
log2timeline.py --extract_winevt_resources ...

If the task fails due to log corruption or intermittent locks, isolate the process or disable the ZeroMQ queue (especially useful in virtual machines experiencing bottlenecks in local sockets):

Ventana de terminal
# Redirigir todas las salidas de depuración a un log físico detallado
log2timeline.py --log_file /opt/forensics/cases/case-001/plaso_debug.log ...
# Desactivar la cola de red interna de ZeroMQ si experimentas congelaciones
log2timeline.py --disable_zeromq ...