AWS Elasticsearch Service Lab

Posted on Edited on In , , , , , , , Symbols count in article: 7.8k Reading time ≈ 7 mins.

AWS OpenSearch (Previous Elasticsearch) Service

AWS Elasticsearch Service Lab

AWS OpenSearch

Kibana Music

Getting started with Amazon OpenSearch Service

Lab Details

  • Create S3 bucket and OpenSearch cluster
  • Create mock mp3 sentiment data from customer calls
  • Bulk load the call data into OpenSearch
  • Use Kibana data discovery to gether data from expanded time horizon
  • Create OpenSearch index pattern in Kibana
  • Visualize call data
  • Create a Kibana dashboard

S3 Configuration

  1. Create your S3 bucket. My bucket was enabled public access for minimizing lab complexity. Make sure your bucket policy obeys certain requirements.

  2. Create a folder Elasticsearch for Kibana in S3 bucket

  1. Create a folder Music for Kibana in Elasticsearch folder

OpenSearch Configuration

Create an OpenSearch Cluster

AWS CLI

If you have any permission error, see AWS CLI

Use AWS Command Line to create OpenSearch cluster.
Replace the the following AWS CLI parameters to your:

  • --domain-name * 2
  • AWS Account ID & AWS User * 2
  • Region * 2
  • elasticsearch version: 6.8 is the latest version on 2022-07-21, you can also upgrade it on AWS console after the installation.
sh
1
aws es create-elasticsearch-domain --domain-name music --elasticsearch-version 6.8 --elasticsearch-cluster-config  InstanceType=t3.medium.elasticsearch,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=10 --access-policies '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::542892269888:user/ZacksShen"},"Action":"es:*","Resource":"arn:aws:es:us-west-2:542892269888:domain/music/*"}]}' --region us-west-2

Press Q to quit

Services -> OpenSearch

Refresh the page and you can see music is on the welcome page and the navigation panel
Click on music on the navigation panel

Wait until the Domain status switches from Loading to Active.
It may take around 10 minutes.

Click on View details

Copy OpenSearch Cluster’s Domain endpoint

The Domain endpoin is end with es.amazonaws.com, such as https://search-music-mtcellyreh5pg4di5r4hhgxnjq.us-west-2.es.amazonaws.com

Upload data to OpenSearch cluster

Install OpenSearch through Python PIP.
If you would like to run Python in virtual environment, see Pipenv or venv.

For Pipenv:

Shell
1
2
3
4
5
6
7
8
9
10
11
# Make a directory AWS for your AWS labs
mkdir AWS
# Change directory to AWS
cd AWS
# Make an empty Pipfile for Pipenv if you are in a subdirectory.
# You can skip this step if dir AWS is not a subdirectory.
touch Pipfile
# Activate Pipenv shell
pipenv shell
# To deactive Pipenv shell
# exit

Shell
1
2
# Install boto3
pipenv install boto3

Shell
1
2
# Install opensearch-py
pipenv install opensearch-py

Shell
1
2
# Review your installed packages
pipenv graph

For pip (SKIP this step if you already perform the above Pipenv step):

Shell
1
2
3
4
# Install boto3
pip install boto3
# Install opensearch-py
pip install opensearch-py
Shell
1
2
# Review your installed packages
pip list

OpenSearch Documentation
Sample code for Amazon OpenSearch Service

Create a .py file such as aws-es-music.py with the following code.

  • Replace the host to your OpenSearch Cluster Domain Endpoint (without https://) such as search-music-mtcellyreh5pg4di5r4hhgxnjq.us-west-2.es.amazonaws.com

  • Replace the region to your OpenSearch Service region and S3 region.

aws-es-music.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3
import json

# Define host and aws region
host = 'search-music-mtcellyreh5pg4di5r4hhgxnjq.us-west-2.es.amazonaws.com' # change to your OpenSearch host name
region = 'us-west-2' # the region where you created OpenSearch cluster

# Define credentials, and auth
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)

# Define index
index_name = 'music'

# Define local file path
with open('music_bulk.json', 'r') as f:
    bulk_file = f.read()

# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = auth,
    use_ssl = True,
    verify_certs = True,
    ssl_assert_hostname = False,
    ssl_show_warn = False,
    connection_class = RequestsHttpConnection
)

# Upload files in a single request
response = client.bulk(bulk_file)
print(json.dumps(response, indent=4, sort_keys=True))

Download the dataset music_bulk.json and save it to the directory of aws-es-music.py

Run aws-es-music.py for uploading dataset music_bulk.json to OpenSearch Cluster.

If you can see the http response from the Python output, you have successfully uploaded the JSON file to your OpenSearch cluster.

OpenSearch Upgrade

If you encounter any incompatibility in the following steps, you can upgrade your OpenSearch cluster to the latest or any version on AWS OpenSearch console.

For example, I am upgrading my OpenSearch cluster from 6.2 to 6.8 for enabling Fine-grained access control under security configuration for public access.

Kibana Configuration

Services -> OpenSearch Service

Click on Domains -> music -> Security configuration

Click on Edit under Security configuration

Access policy

I get a “User: anonymous is not authorized” error when I try to access my Amazon OpenSearch Service cluster
Identity and Access Management in Amazon OpenSearch Service - IP-based policies

  • Domain access policy: Configure domain level access policy

  • Effect: “Allow”

  • AWS: * or your IAM user arn.

  • Resource: No need to change. It should be your recourse.

  • aws:SourceIp: Your local computer public IP address.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:us-west-2:542892269888:domain/music/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "your ip address 1",
            "your ip address 2"
          ]
        }
      }
    }
  ]
}

Click on Save changes

Wait until the Domain status switches from Processing to Active.

Click on Domain endpoint.

The page should be like:

Click on Kibana URL.

Click on Explore on my own

Click on Discover on the navigation panel

Discover
  • Index pattern: test-data*

Click on Next step

  • Time Filter field name: timestamp

Click on Create index pattern

You can see the indexes of the data that we uploaded to OpenSearch.
Click on Discover on the navigation panel.

Because there is No results found, we click on the opening the time picker

We want to use the time picker to pick the proper time range.
Click on Last 5 years

Now we have all of our data.
Click on Visualize on the navigation panel

Kibana Visualization

Pie

Click on Create a visualization

Click on Pie

Click on test-data

  • Buckets type: Split Slices
  • Aggregation: Terms
  • Field: sentiment.keyword

Click on Apply changes

We can see there are three sentiments: NEGATIVE, NEUTRAL, and POSITIVE
Click on Visualize on the navigation panel.

Horizontal Bar

Click on Visualize -> Create a visualization -> Horizontal Bar -> test-data

  • Buckets: Split Series
  • Aggregation: Terms
  • Field: keywords.keyword
  • Size: 20

Click on Apply changes

We can see there are 12 keywords

Dashboard

Click on Dashboard on the navigation panel.
Click on Create a dashboard

Click on Add

Click on Add new visualization

Pie

Add the same pie chart we did on above steps:

  • Select buckets type: Split Slices
  • Aggregation: Terms
  • Field: sentiment.keyword

Click on Apply changes
Click on Inspect on the top of the page.

Click on View: Data -> Requests

Statistics

Request

Response

Click on Save on the top of the page
Give it a name Pie Chart Visualization
Click on Confirm Save

Horizontal Bar

Click on Add on the top of the page

Repeat the same steps to add new visualization —- a Horizontal Bar chart.

  • Select buckets type: Split Series
  • Aggregation: Terms
  • Field: keywords.keyword
  • Size: 20

Click on Apply changes
Click on Save on the top of the page
Give it a name Horizontal Bar Visualization
Click on Confirm Save

Dashboard Manipulating

You can drag the visualization for reordering the dashboard

Click on Save on the top of the page

  • Title: Music Dashboard

Click on Save

You can filter the data on dashboard.
Click on Add a filter.

OR remove the previous filter then just select an area of a chart.

Remember the filer is just a SQL query statement.

Share Dashboard

Click on Share on the top of the page

Embed the Embedded code into your website, you can view and manipulate the dashboard

For example, I embedded the iframe of Music Dashboard into my blog.

You can also share your Kibana dashboard through Permalinks.

Warning

Services -> OpenSearch Service

Delete domain to avoid bankrupt.
Click on Delete

Click on Delete