Main Purpose

  • This idea has been initiated for preparing the INCORE workshop at Nov 15, 2023. 
  • There will be more than 50 users in the workshop
  • There will be chances that 50 users access Geoserver and IN-CORE services at the same time 
  • Needed to check the performance to see if the Geoserver and IN-CORE services are good enough to provide the prompt response with many user requests
  • This documentation might be helpful to others to do the similar load or stress test for there own project

Test Method

  • Test was done by Locust
    • An open source load testing tool that allows users to write test scenarios in Python code, to simulate large numbers of users interacting with the given application. 
    • Simulated 70 users hit the server at the same time
  • In case of IN-CORE GeoServer

  • In case of IN-CORE Services

    • Various services are tested:
      • Data services
      • DFR3 services
      • Hazard services, including several operations:
        • Fetch hazard
        • Create model-based hazard
        • Create dataset-based hazard
        • Update hazard values
    • The implementation and description of this locust test are in IN-CORE GitHub repository. Please see here for more details.

  • Steps for testing

    • Install locust  

      pip install locust

    • Create a Python script (Please see here for more implementation details)

      import json
import logging
import mimetypes
import time
from http.client import HTTPConnection
from locust import HttpUser, task, between
from codecs import encode

class MyUser(HttpUser):
    wait_time = between(1, 5)  # Add the desired wait time between tasks

    headers = {
        "Authorization": ""
    }

    ##### Debugging #####
    # HTTPConnection.debuglevel = 1
    # logging.basicConfig()
    # logging.getLogger().setLevel(logging.DEBUG)
    # requests_log = logging.getLogger("requests.packages.urllib3")
    # requests_log.setLevel(logging.DEBUG)
    # requests_log.propagate = True

    @task
    def geoserver(self):
        # Joplin building inventory
        self.client.get("/geoserver/incore/wms?service=WMS&version=1.1.0&request=GetMap&layers=incore%3A60622b01c57ada48e48d7013&bbox=-94.58364094982268%2C37.01554491871878%2C-94.40576902948776%2C37.14723277660234&width=768&height=568&srs=EPSG%3A4326&styles=&format=image/png")

        # Lumberton flood
        self.client.get("/geoserver/incore/wms?service=WMS&version=1.1.0&request=GetMap&layers=incore%3A5f4d02e352147a614c71960b&bbox=-79.212607077%2C34.562496808%2C-78.973249171%2C34.705963648&width=768&height=460&srs=EPSG%3A4326&styles=&format=image/png")

    @task
    def dataset(self):
        # Get tornado dataset (60a44ae77da24b7b5ba0e86f)
        self.client.get("/data/api/datasets/60a44ae77da24b7b5ba0e86f", headers=self.headers)

        # Get building dataset (5dbc8478b9219c06dd242c0d)
        self.client.get("/data/api/datasets/5dbc8478b9219c06dd242c0d", headers=self.headers)

        # Get socio_demographic_data = "60fb4241544e944c3cedb507"
        self.client.get("/data/api/datasets/60fb4241544e944c3cedb507", headers=self.headers)

    @task
    def dfr3(self):
        # GET /fragilities/{id}
        self.client.get("/dfr3/api/fragilities/5b47b2d7337d4a36187c61c9", headers=self.headers)

        # GET /mappings/{id}
        self.client.get("/dfr3/api/mappings/5b47b2d9337d4a36187c7564", headers=self.headers)

        # GET /repairs/{id}
        self.client.get("/dfr3/api/repairs/6513058862ba036575da5fca", headers=self.headers)

    @task
    def hazard_earthquake(self):
        id = "5b902cb273c3371e1236b36b";
        url = "/hazard/api/earthquakes/"
        earthquake_model_json = {...}
        earthquake_dataset_json = {...}
        points = "..."

        files = [
            {"name": "eq-dataset-SA.tif",
             "location": "/data/hazard/earthquake/eq-dataset-SA.tif"},
            {"name": "eq-dataset-PGA.tif",
             "location": "/data/hazard/earthquake/eq-dataset-PGA.tif"}
        ]

        # GET /earthquakes/{id}
        self.client.get(f"{url}{id}", headers=self.headers)

        # POST /earthquake for model based
        self.post_hazard_model(url, self.headers, earthquake_model_json, "earthquake")

        # POST /earthquake for dataset based
        self.post_hazard_dataset(url, self.headers, earthquake_dataset_json, "earthquake", files)

        # POST /earthquakes/{id}/values
        self.post_hazard_values(url, id, self.headers, points)

    • Start the Locust Master 

      locust -f <<filename_of_above_code>>
    • Configure and start a test
      • Go to http://localhost:8089 
      • The web browser shows the user interface
        • user means total number of the user to test
        • spawn is how many new users are added per second to the test. See number of users below.
    • Monitor and analyze the results

What to test

  • In case of IN-CORE GeoServer
    • Memory

      • Tested with JAVA memory heap size

      • Fortunately there were env variable for setting this.

      • Changed the variables in helm chart and deployed multiple times with various setting to incore dev cluster

      • Tested with the heap size of minimum of 256m, 512m, 1024m and maximum to 1G

      • Confirmed if the variable change actually modified geoserver's memory size by 

        • Exec into geoserver container and do ps -wwaux 

        • In Geoserver GUI, go to About & Status  → Server Status  in the left side menu

      • Testing with the various maximum memory changes did not make any effect either

        • Tested with minimum 1024 and maximum 2G and it was almost identical as the other tests

    • Result

      • Changing memory size doesn't improve any performance in the test


instance

spawn

heap size

min average time (ms)

max average time (ms)

average time after 15 sec (ms)

joplin buildings

dev

50 users / 1 new user per sec

256

140

10344

2239

Lumberton flood

dev

50 users / 1 new user per sec

256

174

10468

1716

Joplin buildings

dev

50 users / 1 new user per sec

512

171

14565

2360

Lumberton flood

dev

50 users / 1 new user per sec

512

178

10390

1674

Joplin buildings

dev

50 users / 1 new user per sec

1024

186

10596

1753

Lumberton flood

dev

50 users / 1 new user per sec

1024

167

9954

1665

  • In case of IN-CORE Services
    • Make sure there are no failures and the response time is reasonable.
    • Monitor the cluster performance in Rancher during the load test.
    • We found a problem with MongoDB using a lot more resources than others (shown by the orange line in Rancher performance monitoring), so we added a missing index before the workshop to fix it.
  • No labels