Version 7.33
New features
- Update sensor to xenial kernel
- ICAP Processing log
- Improved user interface for file submission
- Display labels for the classification of samples in file download and mail attachment views
- Support for sensor groups
- Upgrade to Suricata 4.0
- Leverage malware analysis information and labels in Enterprise product
- EVE-based parsing of Suricata events
Update sensor to xenial kernel
This release installs a major kernel upgrade for sensor appliances, updating the currently used Ubuntu Trusty kernel (3.3.x) to the kernel distributed in the Ubuntu Xenial (4.4.x). The kernel upgrade brings significant improvements to the kernel network stack, which are particularly relevant on sniffing sensors.
The upgrade will be automatically installed on the sensor but will not disrupt the normal sensor functionality. Runs of lastline_test_appliance will however issue a warning to inform about the availability of a major kernel upgrade. For the upgrade to be effective, a reboot will be required. The appliance will however be fully operational even before the reboot, which can therefore be performed whenever convenient.
This change was tracked internally as FEAT-2728
ICAP Processing log
An additional log is now available to detail transaction processing information for sensors set up as ICAP responders or as explicit proxies. The log, located in /var/log/c-icap/processing.log, provides detailed information on the analysis decisions taken during the analysis of each HTTP transaction.
The log format contains the following information:
This change was tracked internally as FEAT-2530
Improved user interface for file submission
With this release, the functionality for submitting files for analysis in the Lastline Portal has been revamped, so users now can:
- Select multiple files to be submitted for analysis
- Drag and drop files to be analyzed into the page
- View the status of multiple submissions directly in the submission page
This change was tracked internally as FEAT-2521
Display labels for the classification of samples in file download and mail attachment views
The display of mail attachments and file downloads in the Lastline Portal now includes additional information on the classification of the samples.
This change extends to a number of views: - mail attachments - mail URLS - file downloads - analysed URLs
Specifically, the tables in these views now have two additional columns:
- Av class. This is the general classification of this sample according to antivirus technology, and has values such as "trojan", "ransomware", "adware".
- Malware: this is the malware name attributed to this sample
This change was tracked internally as FEAT-2470
Support for sensor groups
With this release, we are introducing support for sensor groups. By configuring a number of sensors to be part of a group, users can enable all of Lastline's current and future correlation functionality to work across sensors within the group. Specifically, configuring a group of sensors has two main effects:
- We can correlate detections that happened on different sensors within a group.
- We can assume that local network IP addresses observed by different sensors within a group are consistent. That is, if different sensors see a local IP address such as 192.168.1.1, this IP is assumed to refer to the same host.
Users are now able to configure sensor groups in the sensor groups configuration page. Furthermore, sensor group information is now available as an additional column in the existing sensor listing.
This change was tracked internally as FEAT-2347
Upgrade to Suricata 4.0
This release upgrades the version of Suricata used by the sensor to 4.0.1. The upgrade reworks and addresses a number of limitations and inefficiencies in previous versions of the Lastline sensor.
- Performance improvements: the adoption of Intel Hyperscan (https://01.org/hyperscan) speeds up matching performance when loading large amounts of rules.
- Improvements to SMB2 file extraction: SMB file extraction is now triggered by filestore signatures. Differently from previous releases, only artifacts of relevant filetypes will be extracted for analysis, addressing a number of previously known problems in large network deployments.
- Pcap snipping improvements: the generation of network captures in case of signature hits has now become more reliable.
This change was tracked internally as FEAT-2297
Leverage malware analysis information and labels in Enterprise product
This release adds a number of features that allow us to make better use of malware analysis information within the Enterprise product.
Specifically, this leverages information on:
- the Antivirus class and family of analyzed files or URLs
- malware label information based on the network traffic observed within analysis
- the specific activities observed
This information was already available within an analysis report, but is now propagated into the Enterprise product, where it can be used for searching within the protected network, by making use of the new "analysis tag" filter.
This release introduces support for filtering based on analysis tags in:
- the downloads tab of the portal
- the mail attachments and mail urls views of the portal
Similarly, ability to search for these analysis tags throughout the protected network has been added to the search tab of the portal. For this, just select the "Analysis tags" type for the search. Both filtering and search support auto-completion to help users find values to search for.
This allows users to ask general questions such as "which ransomware samples were seen in my network?", as well as much more specific questions such as "which samples were seen in my network exhibiting a specific evasion behavior?".
Finally, the analysis report overview page has been extended with links to the search tab. This allows a user who is viewing an analysis report to quickly identify other samples that detected in the protected network that share a classification or a specific behavior with the sample being viewed.
This change was tracked internally as FEAT-2204
EVE-based parsing of Suricata events
The Suricata event processing pipeline has been completely reworked and now makes full use of the Extensible Event (EVE) Format. The sensor no longer relies on the llidsupload daemon for Suricata event processing, and the daemon is no longer installed on the appliances. The sniffing events archive (previously located in /var/lib/llidsupload/archive) is now moved to /var/lib/suricata-eve/archive/suricata-lastline. Also, all the archived event logs are now in json format following the Suricata EVE standard (http://suricata.readthedocs.io/en/latest/output/eve/eve-json-format.html).
This change was tracked internally as FEAT-2156
Bug Fixes and Improvements
- SENT-791: Updated ICAP integration guide
- SENT-790: A major issue was identified in the AF_PACKET support for i40e NICs (Intel X710). A bug in the kernel driver was preventing Suricata from accessing the NIC rings. The problem has been fixed by updating the i40e driver to version 2.4.6.
- SENT-779: This release improves the sensor ability to correctly configure the sniffing service in presence of multiple NICs. The required network configuration is now being inferred by checking the NIC model associated to each sniffing interface. The presence of an unused NIC on the appliance will no longer affect the correct configuration of the sensor. This also increases the ability of the sensor to cope with unsupported NICs when AF_PACKET is enabled.
- LLMAIL-420: More robust handling of email URL extraction (better handling of non-lowercase schemas)
- FEAT-2703: MTA email analysis: it is now possible to configure via the web UI a list of regular expressions for recipient domains for which the Sensor should accept mails. If the list is non-empty, emails will be accepted only if the recipient domain matches at least one of the regular expressions. If the recipient domain does not match any of the regular expressions, the email will be rejected with a 521 SMTP error. If the domain list is empty, all emails will be accepted for analysis/forwarding.
- CC-2207: To address a compatibility issue with the upgrade to kernel 4.4.x, the PF_RING drivers have been updated to version 6.4.1.
Deprecation of API methods
Since release 7.24, all methods of the legacy API (/ll_api/ll_api) are deprecated. The Lastline API documentation includes a deprecation schedule for methods in the legacy API, as well as information on how to replace usage of these deprecated methods with supported methods.
Released appliance versions
As part of this release, we are making available the following versions of Lastline appliances for use with Lastline Enterprise Hosted:
Lastline Sensor version 730.1
Deprecation of appliance versions
Since release 7.28, sensor versions before 720 are no longer compatible with Lastline backend.
Since release 7.24, sensor versions before 717 are no longer compatible with Lastline backend.
Distribution Upgrade
Sensor version 730.1, which is being made available as part of this release, does not support Ubuntu Precise as the underlying operating system distribution. Before upgrading to the latest sensor versions, sensors that are still on Ubuntu Precise will need to be upgraded to Ubuntu Trusty.
Users can check the distribution in use by an appliance in the Appliance Status view of the portal. The "Base Distribution" listed should be "trusty". If it is "precise", the appliance distribution needs to be upgraded.
For complete information regarding the upgrade process please refer to the Lastline Support Knowledge Base.