AWS flow logs

About the Sensor

The Sensor examines your network traffic in order to identify a variety of network events that can be of interest to the VMware NSX Network Detection and Response. This ranges from file transfers (for example, executables, documents, or email messages) to suspicious network interactions, to metadata on network activities observed in the environment (for example, netflow, pdns, or webrequests). All this information is extracted by the Sensor and streamed to the VMware backend that processes and presents the data to the user.

The Sensor is available as a software ISO that you install on your own hardware or in a VMware ESXi VM, as an Amazon Machine Image (AMI), or as an Azure VM.

About AWS VPC flow logs

AWS Flow logs capture information about the IP traffic traversing an AWS VPC. The flow log data can be published to Amazon S3 logs and Amazon CloudWatch. You can then integrate the flow logs for ingestion by the VMware NSX Network Detection and Response.

Before the system can ingest flow logs, you must configure AWS credentials.

Configure AWS credentials

To obtain the Access key ID and Secret access key, login to your AWS IAM dashboard and select the appropriate account. On the Summary page, select the Security credentials tab. Click Create access key to generate a new Access key ID and Secret access key. Amazon encourages you to download these credentials in csv format. There is no subsequent way of obtaining the secret access key. However, you can always create another key pair.

The IAM role should be configured with the minimal security policy recommended by ScoutSuite. An IAM role can only be used with a Sensor running as an AWS instance.

Configure AWS authentication on the AWS credentials tab. These credentials are used for the collection of VPC flow logs or the acquisition of AWS Cloud Asset data. You can authenticate to AWS using an access key or an IAM role.

Login to the User Portal

With your Web browser, go to the User Portal and login. For Hosted customers, go to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/). For On-Premises installations, use the portal running on the Manager, for example, https://user.lastline.example.com/.
Navigate to the AWS credentials tab

From the Main navigation menu, click Admin. On the Admin page, select Data sources from left sidebar menu. Then on the Data sources page, click AWS credentials.
Select a sensor

Click the Appliance button at the top right of the tab, then select the appropriate appliance from the Select appliances pop-up. Click the icon in the Name column to select a specific appliance. Then click Apply to dismiss the pop-up.
Add new AWS credentials

Click the button to create a new AWS credentials entry.
1. Enter a profile name
  
  In the Profile name field enter a unique name for the AWS account. This must be the same name you used on AWS.
  
  For an account that has the both credential types, you must create two profiles. In this case, you should use the same name for each profile.
2. Select the type of credentials
  
  In the Credential type field, click the underlined text and select Access and Secret Key or IAM Role from the pull-down menu. Then click to save your selection or to cancel.
  
  Note:
  An IAM role can only be used with an AWS Sensor instance.
3. Enter the access key
  
  Enter the Access key ID generated by AWS for the account. Then click to save your selection or to cancel.
  
  This column is always N/A for the IAM Role.
4. Enter the secret access key
  
  Enter the Secret access key generated by AWS for the account. Then click to save your selection or to cancel.
  
  This column is always N/A for the IAM Role.
5. Set the discovery default
  
  Click the Discovery default toggle to select the specified profile as the cloud asset discovery default. Only one profile from the credentials list can be set as default. If the credentials for the selected profile are incomplete, it cannot be selected.
  
  Note:
  You must explicitly select the cloud asset discovery default profile.
Optional: Delete the AWS credentials

To delete an AWS credential, click the icon in the Actions column.
Deploy the AWS credentials

Click the Deploy changes button to save changes and retrigger a device configuration. Any changes that have not been deployed will be lost.

Publish flow logs to Amazon S3

On AWS, configure your VPC to send flow logs to Amazon S3.

Note:

Refer to the Amazon documentation Publishing Flow Logs to Amazon S3 and Create a Bucket for more details.

Create an S3 bucket

Open the S3 console and select Create bucket.
Give the bucket a unique name

In the Create bucket pop-up, enter a Bucket name.

Note:
The name you provide must be unique across all existing bucket names in Amazon S3.

Select the region where you want the bucket to reside.

Click Create to dismiss the pop-up.
View your configured VPCs

Open the VPC console. Select Your VPCs from the Virtual Private Cloud side bar.
Select the VPC to monitor

On the VPCs page, select one or more VPCs you want to capture flow logs from. Click Create flow log from the Actions pull-down menu.

Note:
You can also select the Flow logs tab in the lower half of the page and click the Create flow log button.
Select the flow log options
On the Create flow log page, set the following options:
- Set the Filter to All. This will log accepted and rejected traffic.
- Click the radio button to set the Destination to Send to an S3 bucket.
- Set the S3 bucket ARN to the Amazon Resource Name of the S3 bucket. Acquire the ARN by selecting the bucket on the S3 buckets page then clicking the Copy Bucket ARN button.
Click the Create button to complete the flow log creation.

Set permissions on the S3 bucket

The S3 bucket must have the s3:GetBucketAcl and s3:PutObject permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSLogDeliveryAclCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::bucket_name"
    },
    {
      "Sid": "AWSLogDeliveryWrite",
      "Effect": "Allow",
      "Principal": {
                "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket_name/optional_folder/AWSLogs/account_id/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

In addition, the user account that creates the VPC flow log must have the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogDelivery",
        "logs:DeleteLogDelivery"
      ],
      "Resource": "*"
    }
  ]
}

This can be the same user who owns the S3 bucket.

To access the S3 bucket, the user account configured on the User Portal must, at a minimum, have the following IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bucket_name",
        "arn:aws:s3:::bucket_name/*"
      ]
    }
  ]
}

Ingest flow logs from Amazon S3

Configure the User Portal to ingest flow logs from Amazon S3.

Login to the User Portal

With your Web browser, go to the User Portal and login. For Hosted customers, go to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/). For On-Premises installations, use the portal running on the Manager, for example, https://user.lastline.example.com/.
Navigate to the VPC flow logs tab

From the Main navigation menu, click Admin. On the Admin page, select Data sources from left sidebar menu. Then on the Data sources page, click VPC flow logs.
Select a sensor

Click the Appliance button at the top right of the tab, then select the appropriate appliance from the Select appliances pop-up. Click the icon in the Name column to select a specific appliance. Then click Apply to dismiss the pop-up.
Configure an S3 collector

On the VPC flow logs tab, click over the S3 list to add an S3 collector.
1. Select the profile name
  
  In the Profile name field, click the underlined Select text. From the pull-down menu, select the unique ID created on the AWS credentials tab. Then click to save your selection or to cancel.
  
  This provides the credentials the collector uses to access the S3 bucket.
2. Enter a collector label
  
  In the Label field, enter a unique name that is used to identify the collector.
3. Enter the S3 bucket name
  
  In the Location field, enter the name of the S3 bucket.
4. Optional: Enter AWS regions
  
  In the Regions field, enter the regions that can sent flow logs to the S3 bucket.
  
  AWS allows you to configure multiple VPC flow logs, from multiple regions, all using the same S3 bucket. This optional parameter is used to filter flow logs by the specified regions.
  
  Refer to Flow Log Files for the structure of the flow log folders.
5. Optional: Enter user accounts
  
  In the Accounts enter one or more account IDs. Similar to the previous, this optional parameter is used to filter flow logs by the account ID of the bucket folder.
Optional: Delete the S3 collector

To delete an S3 collector, click the icon in the Actions column.
Save the S3 collector

Click Save when you are done. The Sensor will be reconfigured with the options provided.

Publish flow logs to CloudWatch logs

On AWS, configure your VPC to send flow logs to CloudWatch logs.

Note:

Refer to the Amazon documentation for more details.

View your configured VPCs

Open the VPC console. Select Your VPCs from the Virtual Private Cloud side bar.
Select the VPC to monitor

On the VPCs page, select one or more VPCs you want to capture flow logs from. Click Create flow log from the Actions pull-down menu.

Note:
You can also select the Flow logs tab in the lower half of the page and click the Create flow log button.
Select the flow log options
On the Create flow log page, set the following options:
- Set the Filter to All. This will log accepted and rejected traffic.
- Click the radio button to set the Destination to Send to CloudWatch Logs.
- Set the Destination log group to the name of a group in CloudWatch Logs to which the flow logs are to be published. If you specify the name of a log group that does not exist, AWS attempts to create the group for you.
- For IAM role, click the Set Up Permissions link to specify an IAM role that has permissions to publish to CloudWatch Logs.

Create a new IAM role

On the IAM set up permission page, the IAM Role is Create a new IAM Role and the Role Name is flowlogsRole. These permissions are required to push data to the CloudWatch Logs. You can toggle the View Policy Document link to display the IAM role policy:

{
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Click the Allow button to create the role.

Note:

In addition, the user account must have the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "vpc-flow-logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

It must also have permissions to use the iam:PassRole action:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["iam:PassRole"],
      "Resource": "arn:aws:iam::account-id:role/flow-log-role-name"
    }
  ]
}

Refer to the Amazon documentation for more details.

The user account configured on the User Portal must, at a minimum, have the following IAM permissions to access the CloudWatch Logs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "logs:GetLogEvents",
        "logs:FilterLogEvents"
      ],
      "Resource": "arn:aws:logs:*:account-id:log-group:log_group_name:*:*"
    }
  ]
}

Create the flow log

Back on the Create flow log page, confirm that the IAM role is set to flowlogsRole then click the Create button to complete the flow log creation.

Ingest flow logs from Amazon CloudWatch

Configure the User Portal to ingest flow logs from Amazon CloudWatch.

Login to the User Portal

With your Web browser, go to the User Portal and login. For Hosted customers, go to the User Portal at https://user.lastline.com/ (for EMEA customers https://user.emea.lastline.com/). For On-Premises installations, use the portal running on the Manager, for example, https://user.lastline.example.com/.
Navigate to the VPC flow logs tab

From the Main navigation menu, click Admin. On the Admin page, select Data sources from left sidebar menu. Then on the Data sources page, click VPC flow logs.
Select a sensor

Click the Appliance button at the top right of the tab, then select the appropriate appliance from the Select appliances pop-up. Click the icon in the Name column to select a specific appliance. Then click Apply to dismiss the pop-up.
Configure a CloudWatch collector

On the VPC flow logs tab, click over the CloudWatch list to add an Amazon CloudWatch collector.
1. Select the profile name
  
  In the Profile name field, click the underlined Select text. From the pull-down menu, select the unique ID created on the AWS credentials tab. Then click to save your selection or to cancel.
  
  This provides the credentials the collector uses to access Amazon CloudWatch.
2. Enter a collector label
  
  In the Label field, enter a unique name that is used to identify the collector.
3. Enter the region
  
  In the Regions field, enter the region the flow log was created in.
4. Enter log group
  
  In the Log group field, enter the CloudWatch group name.
Optional: Delete the CloudWatch collector

To delete a CloudWatch collector, click the icon in the Actions column.
Save the CloudWatch collector

Click Save when you are done. The Sensor will be reconfigured with the options provided.

VMware NSX Network Detection and Response
Home
Legal
Support

VMware, Inc.

3401 Hillview Ave.

Palo Alto, CA 94304

www.vmware.com

Ingesting AWS Flow Logs

About the Sensor

About AWS VPC flow logs

Configure AWS credentials

Publish flow logs to Amazon S3

Ingest flow logs from Amazon S3

Publish flow logs to CloudWatch logs

Ingest flow logs from Amazon CloudWatch