Description:

Manages pool of Spark sessions over HTTP

Tags:

Livy, REST, Spark, http

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the Expression Language Guide.

Name

Default Value

Allowable Values

Description

Livy Host

The hostname (or IP address) of the Livy server.

Supports Expression Language: true (will be evaluated using variable registry only)


Livy Port

8998 The port number for the Livy server.

Supports Expression Language: true (will be evaluated using variable registry only)


Session Pool Size

2 Number of sessions to keep open

Supports Expression Language: true (will be evaluated using variable registry only)


Session Type

spark * spark
* pyspark
* pyspark3
* sparkr
The type of Spark session to start (spark, pyspark, pyspark3, sparkr, e.g.)

Session Manager Status Interval

2 sec The amount of time to wait between requesting session information updates.

Supports Expression Language: true (will be evaluated using variable registry only)


SSL Context Service

Controller Service API: 


SSLContextService

Implementations: 

StandardRestrictedSSLContextService


StandardSSLContextService


The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.

Connection Timeout

5 secs Max wait time for connection to remote service.
Session JARs JARs to be used in the Spark session.

Supports Expression Language: true (will be evaluated using variable registry only)


Session Files Files to be used in the Spark session.

Supports Expression Language: true (will be evaluated using variable registry only)


Kerberos Credentials Service

Controller Service API: 


KerberosCredentialsService

Implementation: 

KeytabCredentialsService


Specifies the Kerberos Credentials Controller Service that should be used for authenticating with Kerberos

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified.