Advanced Techniques and Data Ingestion with Plaso (log2timeline.py)
Expert techniques to optimize data ingestion performance, collection filters, and YARA signature analysis in Plaso.
Expert Techniques for Specific Use Cases
Sección titulada «Técnicas de Experto para Casuísticas Específicas»1. Overview
Sección titulada «1. Visión General»This document assumes you have already mastered the concepts covered in the Plaso fundamentals manual (basic timeline creation, time zones, parsers, and VSS).
We will focus on advanced use cases: how to maximize CPU performance, how to filter data before processing it (during file ingestion), how to extract signature information via YARA rules, and how to manage error logs in large-scale forensic investigations.
2. Case Study 1: CPU Optimization and Processing Performance
Sección titulada «2. Casuística 1: Optimización de CPU y Rendimiento de Procesamiento»The Challenge
Sección titulada «El Desafío»Analyzing disks of 1 TB or larger can take entire days. Optimizing Plaso's parallel engine reduces this critical incident response time to just a few hours.
Engineering Solution
Sección titulada «Solución de Ingeniería»Plaso is natively multi-process. We can explicitly define the number of CPU worker processes (workers) and modulate the size of the memory buffers:
- ✔
--workers <NÚMERO>: Specifies the number of simultaneous processes to launch for low-level extraction and analysis.- Guideline: It is recommended to define a number of workers equivalent to the total number of your CPU's physical cores (e.g.,
--workers 8). In compressed E01 images, the operation may be limited by the decompression processor bottleneck. On RAW disk drives mounted on fast SSDs, a higher number of workers results in a higher I/O rate.
- Guideline: It is recommended to define a number of workers equivalent to the total number of your CPU's physical cores (e.g.,
- ✔
--buffer_size <TAMAÑO>: Controls the capacity in megabytes of the intermediate transfer buffer used by each worker. By default, it is assigned 1024MB.- Guideline: On analytical systems equipped with large amounts of RAM (e.g., 64 GB), we can increase this rate to
--buffer_size 2048Mto improve performance and minimize hot disk writes.
- Guideline: On analytical systems equipped with large amounts of RAM (e.g., 64 GB), we can increase this rate to
Example of an optimized command for analytical machines (16 workers, 64GB RAM):
Sección titulada «Ejemplo de comando optimizado para máquinas analíticas (16 workers, 64GB RAM):»log2timeline.py --workers 16 --buffer_size 2048M --status_view linear /opt/forensics/cases/case-001/my_timeline.plaso /opt/forensics/cases/case-001/image.E013. Case Study 2: Ingest Filtering (Rapid File Triage)
Sección titulada «3. Casuística 2: Filtrado en la Ingesta (Triaje Rápido de Archivos)»The Challenge
Sección titulada «El Desafío»In immediate response interventions or triage-style analysis, processing millions of temporary files, browser cache files, or irrelevant system files is highly inefficient. We are only interested in execution artifacts and the Windows registry.
Engineering Solution
Sección titulada «Solución de Ingeniería»Through the --file_filter parameter, we can instruct Plaso to index and analyze only the paths that match a collection filter template:
--file_filter <CAMINO_AL_ARCHIVO_FILTRO>- Note: This filter does not work with complex regular expressions; instead, it validates whether the file path contained in the image starts with any of the lines in the filter text file.
Practical Example for Rapid Windows Triage:
Sección titulada «Ejemplo Práctico para Triaje Rápido de Windows:»- Create a directive text file called
/opt/forensics/cases/case-001/triage_filter.txtwith the following list of key paths:/Windows/System32/config//Users/%/NTUSER.DAT/Users/%/AppData/Local/Microsoft/Windows/UsrClass.dat/Users/%/AppData/Roaming/Microsoft/Windows/Recent//Users/%/AppData/Local/Microsoft/Windows/WebCache//Windows/Prefetch/ - Run Plaso ingestion by calling the prepared filter:
Ventana de terminal log2timeline.py --file_filter /opt/forensics/cases/case-001/triage_filter.txt /opt/forensics/cases/case-001/my_triage_timeline.plaso /opt/forensics/cases/case-001/image.E01
With this action, 90% of irrelevant user and system files will be skipped, reducing processing time from 24 hours to less than 30 minutes.
4. Case Study 3: YARA Signature-Based Filtering
Sección titulada «4. Casuística 3: Filtrado Basado en Firmas YARA»The Challenge
Sección titulada «El Desafío»It is required to automatically classify, detect, and tag any file that matches malicious rules or signatures during the same ingestion and timeline construction phase.
Engineering Solution
Sección titulada «Solución de Ingeniería»Plaso can load YARA rules on the fly and evaluate each analyzed file before indexing it:
# Ejecutar ingesta evaluando firmas yaralog2timeline.py --yara_rules /opt/forensics/rules/custom-signatures.yar /opt/forensics/cases/case-001/my_timeline.plaso /opt/forensics/cases/case-001/image.E01If a binary (e.g., c:\temp\mimikatz.exe) matches any of the rules contained in the signature file, Plaso will immediately generate an event with the data_type “fs:yara_match” in the timeline. This allows direct correlation of the threat creation date with nearby network connections or registry modifications.
5. Case Study 4: Extended Extraction of Hashes and Resources
Sección titulada «5. Casuística 4: Extracción Ampliada de Hashes y Recursos»The Challenge
Sección titulada «El Desafío»In the case of deep malware analysis or the need to correlate the descriptive text of Windows event IDs (which by default only show numeric codes).
Engineering Solution
Sección titulada «Solución de Ingeniería»We can force file hash calculation during ingestion and force string resolution from EVTX files:
# Calcular hashes SHA-1 y SHA-256 de todos los archivos en la ingestalog2timeline.py --hashers "sha1,sha256" ...
# Extraer texto y descripciones completas de los mensajes de los logs EVTX de Windowslog2timeline.py --extract_winevt_resources ...6. Case Study 5: Debugging and Error Management
Sección titulada «6. Casuística 5: Depuración y Gestión de Errores»If the task fails due to log corruption or intermittent locks, isolate the process or disable the ZeroMQ queue (especially useful in virtual machines experiencing bottlenecks in local sockets):
# Redirigir todas las salidas de depuración a un log físico detalladolog2timeline.py --log_file /opt/forensics/cases/case-001/plaso_debug.log ...
# Desactivar la cola de red interna de ZeroMQ si experimentas congelacioneslog2timeline.py --disable_zeromq ...
