Sensor on AWS Deployment and Administration

This document describes the deployment and administration of the Sensor in the AWS environment.

About the Sensor

The Sensor examines your network traffic in order to identify a variety of network events that can be of interest to the VMware NSX Network Detection and Response. This ranges from file transfers (for example, executables, documents, or email messages) to suspicious network interactions, to metadata on network activities observed in the environment (for example, netflow, pdns, or webrequests). All this information is extracted by the Sensor and streamed to the VMware backend that processes and presents the data to the user.

The Sensor is available as a software ISO that you install on your own hardware or in a VMware ESXi VM, as an Amazon Machine Image (AMI), or as an Azure VM.

Network Connectivity

The installation and update services need to connect to external servers for downloading software and data bundles (such as sandbox images). All hosts that are contacted for such downloads are listed in this section.

To increase the availability and reduce download times, the system can be configured to download large files from content distribution network (CDN) servers. As such hosts are geographically distributed, the contacted hosts may vary from system to system, and hosts outside the documented list may be contacted for downloads.

The use of CDNs is enabled by default. You can also explicitly enable or disable this feature with the lastline_register command (see Register the Sensor, 11).

If you explicitly enable the use of CDNs or choose to accept the default, ensure that you adjust your firewall rules to allow access to the CDN servers.

Domain Names

For a hosted installation using the NSX Cloud, the server hosting the Sensor needs to be able to connect to:

You can add FQDNs such as the CDN domain for Google. For further details and information about VMware NSX Network Detection and Response CDN operation, see VMware Knowledge Base article NSX Lastline CDN Usage (900006).

Expected IP Addresses

The domain names above may resolve to any IP addresses within the following ranges:

  • 38.95.226.0/24

  • 38.142.33.16/28

  • 199.91.71.80/28

  • 46.244.5.64/28

  • 66.170.109.0/24

Deploy the Sensor

The following topics describe how to deploy the Sensor on Amazon Web Services and then register it with the VMware backend including the User Portal. In addition, an example is provided showing how to use the Sensor to be an AWS traffic mirroring target.

Deploy the Sensor AMI

Before you can deploy the Sensor in AWS, you must have a VPC defined and configured.

Deploy a Sensor instance based on the official VMware NSX Network Detection and Response AMI in the AWS Marketplace. Configure the Sensor instance for the AWS environment.

  1. Select the AMI

    Navigate to the "Bring Your Own License (BYOL)" AMI in the AWS Marketplace. Click Continue to Subscribe.

  2. Select an EC2 Region

    Click the Launch with EC2 Dashboard button beside your desired region.

  3. Select the instance type

    On the Choose an Instance Type page, select an instance type.

    The Sensor will automatically configure itself to reflect the specifications assigned to it. For applications involving significant amount of packet processing, VMware recommends you use m5.4xlarge. However, a smaller instance type may be selected for smaller tasks, for example, dedicating a Sensor AMI for smaller networks with limited throughput.

    Click Next: Configure Instance Details.

  4. Fill in the instance details

    On the Configure Instance Details page, fill in the following items:

    • Set the Network to your VPC.

    • Set the Subnet to the public subnet defined in the VPC.

    • Set the Auto-assign Public IP to Enable.

    Then click Next: Add Storage.

  5. Optional: Select the amount of storage

    On the Add Storage page, set the Size (GiB) to 500 (this is the default). Then click Next: Add Tags.

  6. Optional: Add tags

    Define the relevant tags for the Sensor instance.

    Click Next: Configure Security Group.

  7. Optional: Create a new security group

    On the Configure Security Group page, for the Assign a security group option, select the Create a new security group radio button. Set a Security group name and Description. Then in the table, create any needed entries.

    Click Review and Launch.

  8. Launch the Sensor instance

    From the Key pair pop-up, generate and download a new key pair (private key file, for example keypair.pem) or select an existing key pair. Ensure the key pair file is read-only by the owner:

    yourhost$ chmod 400 keypair.pem
    Note:

    On the EC2 Dashboard, you can name/rename the Sensor instance.

Registration and Configuration

To register and apply the software configuration to the Sensor, you must login to the server console.

Register the Sensor

  1. SSH to the Sensor instance

    Login to the Sensor using the default user account, lastline. Password authentication is disabled for this user, only key-based authentication is possible using the keys that are automatically configured by Amazon based on your account configuration.

    yourhost$ ssh -i "keypair.pem" lastline@ec2-sensor.region.compute.amazonaws.com
lastline@lastline-sensor:~$

    The keypair.pem file was downloaded from AWS when you launched the Sensor instance (see Deploy the Sensor AMI, 8). In the URL, sensor is an AWS hostname, for example, ec2-18-222-42-84 and region is the AWS region, for example, us-west-2.

  2. Start the registration process

    Run the lastline_register command.

    lastline@lastline-sensor:~$ lastline_register
  3. Select the VMware backend

    The registration process prompts you to select the "Manager Backend". Your choice is "Use Lastline cloud" or "Use On-Premises Manager".

    Unless this installation is part of an On-Premises deployment, select "Use Lastline cloud". Then select <Ok> or press Enter.

  4. Select the primary network interface and network address

    The registration process prompts you to select the "Primary network interface". It presents a list of interfaces discovered during the validation process. Select the interface that is used by the server to communicate with the other hosts on the network.

    Then you are prompted to select how the server will obtain its network address. Your choice is "Obtain via DHCP" or "Enter static address". You must select "Obtain via DHCP".

    To continue, select <Ok> or press Enter.

  5. Optional: Configure an HTTP proxy

    The newly installed Sensor must have access to the NSX Cloud servers or the local Manager.

    Configure an HTTP proxy if it is required to access the Internet via HTTPS. Enter the address of the proxy server. This address can be a FQDN or an IP address. Specify the port for the proxy server. Examples of valid proxy configurations:

    proxy.example.com:3128

    192.168.0.1:8080

    Otherwise if no proxy configuration is required, leave this field empty.

    To continue, select <Ok> or press Enter.

  6. Select the traffic capture interface

    The Sensor requires at least one interface to use to monitor network traffic for malicious activity. In the AWS deployment, the main interface is also used as the traffic capture interface.

    To continue, select <Ok> or press Enter.

  7. Select the reset interface

    Select a network interface to be used for injection of network reset packets. This interface must be able to communicate with the other hosts of the monitored network.

    To continue, select <Ok> or press Enter.

    The network configuration is applied to the Sensor.

  8. Specify the hosted Manager location

    If you selected "Use Lastline cloud" when choosing the VMware backend, then you must specify the hosted manager location. Select from "EMEA" (Eurasia) or "US" (Americas).

    If you selected "Use On-Premises Manager", then you must specify the address of the Manager. This address can be a FQDN or an IP address. For example, manager.lastline.example.com or 192.20.24.42.

    To continue, select <Ok> or press Enter.

  9. Configure an NTP server

    The Network Time Protocol (NTP) is used to set the correct time for the Sensor. Enter the address of the NTP server. This address can be a FQDN or an IP address.

    Note:

    The selected NTP server must be reachable over UDP port 123. Unless you must use a specific NTP server, use the default value, ntp.lastline.com.

    To continue, select <Ok> or press Enter.

    The network configuration is tested to check for connectivity to the NTP server and VMware backend. This test may take a while.

  10. Define the CDN rules

    The registration process prompts you to define the CDN rules for your installation. Select one of the following:

    • Explicitly enable the CDN servers. This is the default behavior.

    • Explicitly disable the CDN servers.

    To continue, select <Ok> or press Enter.

  11. Enter your VMware username and password

    As the first stage to applying your license to the Sensor you are prompted for your VMware username. Enter your username and then select <Ok> or press Enter.

    Note:

    This is your User Portal username. To use the Sensor with a "Bring your own license" (BYOL), you first must have signed up for a VMware user account. Using the information in the VMware welcome email message, point your browser to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/) and then login. For your initial login, use the Forgot your password? link and follow the subsequent instructions.

    Enter your VMware password and then select <Ok> or press Enter.

  12. Select the correct license

    If the credentials you provided are valid, the registration process displays a list of the available license keys. Use the UP and DOWN keys to select the correct license.

    Note:

    If there are no valid licenses associated with your credentials or your list of license keys is not retrieved correctly, contact VMware Support. Provide the error message the registration process displayed in your request.

    The Sensor license consists of a Sensor key, which is a string you create, concatenated to your VMware generated customer License key (for example, ABCDEFGHIJ0123456789:sensor-1). This license structure allows the On-Premises Manager or the VMware backend to quickly and correctly identify the Sensor when it connects.

    You need a Sensor license for each Sensor you deploy. The usual process to create another license is to use the User Portal (navigate to the Admin → Licensing page and select the Sensors tab).

    Another option is to create a new Sensor license immediately at this step. Select Create a new sensor key:

    1. Select customer license key

      If you have more than one License key, you are prompted to select the License key under which you want to create the Sensor license. Use the UP and DOWN keys to make your selection.

      To continue, select <Ok> or press Enter.

      Otherwise your customer License key is displayed and the registration process asks if you want to use it. Select <Yes> to use that license and continue.

      If you select <No>, you will have to provide another username.

    2. Create a sensor key

      Enter a Sensor key. The sensor key is restricted to alpha-numeric characters, dot (.), and dash (-).

      To continue, select <Ok> or press Enter.

    3. Optional: Provide a sensor name

      Enter a Sensor name. This user-friendly name is displayed on some of the screens of the User Portal.

      The name is restricted to alpha-numeric characters and the following special characters; (, ), [, ], -, :, ., ,, ;, _, @, ~, /, #, %, !, |, $, and ^. The name should be between 1 and 64 characters in length.

      To continue, select <Ok> or press Enter.

    The registration process displays a prompt: "Registration completed successfully".

    To continue, select <Ok> or press Enter.

The registration process runs some tests to check hardware compatibility. The configuration is then applied to the machine. This process may take a while (20-40 minutes) depending on your network connectivity and system characteristics.

After the completed prompt is displayed, select <Ok> or press Enter to exit from the registration process.

Re-registration

If the Sensor needs to be replaced or reinstalled, the existing appliance needs to be deregistered first before your new registration will succeed.

  1. Login to the Web UI

    Using your Web browser, login to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/).

  2. Access the Appliances page

    From the Main navigation menu, click Admin. On the Admin page, select Admin from left sidebar menu. For most users, the Appliances page is displayed by default.

  3. View the appliance status

    On the Appliances page, click the Status tab.

  4. Optional: Select an appliance

    If no appliance is currently selected, click the Appliance: None Selected link. From the Select Appliance pop-up tick the box for the appliance you want to use, then click Select Appliance.

  5. Deregister the existing Sensor

    To deregister a Sensor, click the cogs (multiple actions) button and select Deregister from the drop-down menu.

  6. Register the reinstalled Sensor

    To replace or reinstall a Sensor, you must run the lastline_register command again from the server console (see Register the Sensor).

Delete the Sensor

Before you can successfully delete the Sensor from the User Portal it must be offline. This is done from the EC2 Dashboard.

To delete the Sensor, it needs to be offline and deregistered.

  1. Shutdown the appliance

    On the EC2 Dashboard, click Running Instances. Select the Sensor instance. Then click the Actions button and select Interface state → Stop to shutdown the instance.

  2. Login to the Web UI

    Using your Web browser, login to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/).

  3. Access the Appliances page

    From the Main navigation menu, click Admin. On the Admin page, select Admin from left sidebar menu. For most users, the Appliances page is displayed by default.

  4. View the appliance status

    On the Appliances page, click the Status tab.

  5. Deregister the Sensor

    Click the cogs (multiple actions) button and select Deregister from the drop-down menu.

  6. Delete the appliance

    Click the Overview tab to return to the initial view. In the appliances listing, the Status of the Sensor must be Deregistered.

    In the Actions column, click the Quick links icon and select Delete. A confirmation pop-up is displayed. Click Delete appliance to dismiss the pop-up. The Sensor is permanently deleted.

Configure AWS Traffic Mirroring

AWS traffic mirroring allows you to route the traffic generated by a load balancer or monitored appliance to the Sensor AMI for processing. This feature provides certain advantages:

  • VPC NAT gateways are not required to monitor outgoing traffic from private networks.

  • Traffic within a private network can be mirrored, including AWS DNS requests.

  • The Sensor does not need to be inline, reducing the risk of failures.

  • The general network setup is less complex. Traffic can be mirrored to an internal network load balancer which then distributes it to the Sensor.

  • As your workloads increase, scaling can be easily implemented by simply adding or removing the number of appliances registered at the network load balancer.

Note:

The mirrored traffic is tunneled using the VXLAN encapsulation protocol. The Sensor automatically accepts and parses these tunneled packets and analyzes the traffic.

  1. Obtain the source and target addresses

    Open the EC2 Dashboard. Click Network Interfaces and note the ENI of the source and target instances.

  2. Access the traffic mirroring page

    Open the VPC console. Scroll down to Traffic Mirroring section and click Mirror Targets.

  3. Create a traffic mirror

    On the Traffic mirror targets page, click Create traffic mirror target.

    On the Create traffic mirror target page, enter the following:

    • An optional Name tag and Description.

    • Select the Target type from the pull-down menu. Choose Network Interface or Network Load Balancer.

      To enable scaling, we recommend you configure a load balancer as the mirror target.

    • Select the Target from the pull-down menu. This menu is populated with the ENI of the appropriate devices.

    • Add optional tags to the traffic mirror target. If you create a Name tag above, it is displayed here.

    Click Create when you are done.

  4. Create mirroring filters

    Create a filter to select specific traffic to be mirrored. Click Mirror Filters in the Traffic Mirroring section.

    On the Traffic mirror filters page, click Create traffic mirror filter.

    On the Create traffic mirror filter page, enter the following:

    • An optional Name tag and Description.

    • Optionally select amazon-dns.

    • In the Inbound rules section, click Add rule. Then select the rule action (accept or deny), select the desired protocols and related ports, and provide a description.

    • In the Outbound rules section, click Add rule. Then select the rule action (accept or deny), select the desired protocols and related ports, and provide a description.

    • Add optional tags to the traffic mirror filter. If you created a Name tag above, it is displayed here.

    Note:

    To capture all traffic from the source, create an inbound and outbound rule each set to All protocols with their source and destination CIDR blocks set to 0.0.0.0/0.

    Click Create when you are done.

  5. Start the traffic mirroring session

    In the Traffic Mirroring section, click Mirror Sessions.

    On the Traffic mirror sessions page, click Create traffic mirror session.

    On the Create traffic mirror session page, enter the following:

    • An optional Name tag and Description.

    • Select the Mirror source and Mirror target.

    • Define a Session number. This can be an arbitrary number. The VMware NSX Network Detection and Response software does not rely on this value.

    • Provide a VNI. This is a unique VXLAN network identifier that is included in the encapsulated mirrored packet that is sent to the target. The VMware NSX Network Detection and Response software does not rely on this value.

    • Select the Packet length. By default, the entire packet is sent to the target.

    • Select the Filter.

    • Add optional tags to the traffic mirror session. If you created a Name tag above, it is displayed here.

    Click Create when you are done.

Traffic from the mirror source that matches the (optional) filter is encapsulated as specified in RFC 7348 and delivered to the mirror target. If the target is a load balancer, the traffic gets distributed to the Sensor instances for processing. The load balancer must be configured to handle VXLAN packets. See Traffic Mirror Targets for further information.

Administer the Sensor

The Sensor was developed to require as little maintenance and administration as possible.

The following topics describe how to customize and configure some of the advanced features of the Sensor.

Configuration Tool

Use the VMware NSX Network Detection and Response configuration tool, lastline_setup, to administer and manage the Sensor.

  1. Start the configuration tool

    Execute the lastline_setup command.

    lastline@lastline-sensor:~$ lastline_setup

    If you are prompted for the sudo password, use the password for the default lastline user account.

  2. Run the help option

    To view all the supported options, type help.

    -> help
Documented commands (type help <topic>):
========================================
EOF                      https_proxy                   ntp_server
anonymization_password   inject_interface              ntp_servers
appliance_state          inline_interfaces             save
appliance_uuid           license_key                   sensor_subkey
disable_support_channel  manager                       sentinel_subkey
edit                     monitoring_user_password      show
exit                     network                       sniffing_interfaces
help                     new_monitoring_user_password
    Tip:

    For any option, type the first few unique characters of its name then type Tab. The lastline_setup command will auto-complete the name for you.

  3. View help details

    To view a detailed description of individual options, type help topic, where topic is the name of a specific option.

    -> help network
 network <variable> [<new-value>]
        Get/set network settings.
            network interface <iface>: interface used for network access
            network method dhcp|static: use DHCP or static IP address
                configuration for network access
        When static configuration is used, these values must also be set:
            network address <address>: IPv4 address of the interface
            network netmask <netmask>: dotted-quad netmask for the address
            network gateway <gateway>: default gateway for network access; if
                specified value is -, set gateway to None
            network dns_nameservers <nameserver> ...: space-separated list of
                DNS nameservers, if specified value is -, set dns_nameservers to
                None
  4. Exit the configuration tool

    To quit from the configuration tool without saving your changes, type exit.

    -> exit
lastline@lastline-sensor:~$
Important:

If you encounter an error running any of the lastline_setup command options, make a note of the error message returned and contact VMware Support.

Network Configuration

You can easily change the network configuration of the Sensor. This may be needed if its assigned IP address changes (for example, upon a reconfiguration of the network).

Update Fully Qualified Domain Name

You can update the FQDN of the Sensor.

  1. Login to the console

    Login to the console.

  2. Run the registration process with the change FQDN option

    Execute the lastline_register command, providing the new local FQDN for the Sensor in its arguments. 

    lastline@lastline-sensor:~$ lastline_register --change-local-fqdn new_sensor.lastline.example.com
    Note:

    If you are prompted for the sudo password, use the password for the default lastline user account.

Test the Sensor

Check the state of the Sensor with the lastline_test_appliance command.

  1. Start the test appliance tool

    Execute the lastline_test_appliance command.

    lastline@lastline-sensor:~$ lastline_test_appliance

> Lastline appliance check and fix utility.
> 
> Version 2.0
> 
> :Copyright:
>     Copyright 2014 Lastline, Inc.  All Rights Reserved.
...

    The lastline_test_appliance command takes for a few minutes to perform its analysis of the Sensor. When it is done, it provides a summary of the conditions it uncovered, if any.

  2. Optional: Fix any reported configuration errors

    The test appliance tool checks for signs of common configuration errors and can help you with fixing them.

    Note:

    Contact VMware Support if the test appliance tool displays an error that you cannot fix. Provide the error message that was displayed.

Disable Automatic Updates

VMware periodically releases appliance updates or hotfixes. By default, automatic updates are enabled on newly installed appliances. As long as the appliance has automatic updates enabled, these updates and fixes will transparently be applied to the system.

If you prefer to manually update the Sensor, follow these steps to disable automatic updates.

  1. Login to the Web UI

    Using your Web browser, login to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/).

  2. Access the Appliances page

    From the Main navigation menu, click Admin. On the Admin page, select Admin from left sidebar menu. For most users, the Appliances page is displayed by default.

  3. View the appliance configuration

    On the Appliances page, click Configuration tab.

  4. Optional: Select an appliance

    If no appliance is currently selected, click the Appliance: None Selected link. From the Select Appliance pop-up tick the box for the appliance you want to use, then click Select Appliance.

  5. Access the System tab

    Click the System tab.

  6. Disable automatic updates

    Toggle the Auto Update button to Disabled.

    The appliance will no longer automatically apply updates and hotfixes when released by VMware. You must apply those manually.

Manual Updates

If you have disabled automatic updates for your appliances you must apply updates and hotfixes manually.

Follow these steps to manually update an appliance.

  1. Login to the Web UI

    Using your Web browser, login to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/).

  2. Access the Appliances page

    From the Main navigation menu, click Admin. On the Admin page, select Admin from left sidebar menu. For most users, the Appliances page is displayed by default.

  3. View the appliance status

    On the Appliances page, click the Status tab.

  4. Optional: Select an appliance

    If no appliance is currently selected, click the Appliance: None Selected link. From the Select Appliance pop-up tick the box for the appliance you want to use, then click Select Appliance.

  5. Update the selected appliance

    To update an appliance, click the cogs (multiple actions) button and select Upgrade from the drop-down menu.