mtap.deployment

Deploying Processors

Module for deploying a set of processing services and the events server all at once.

See python/mtap/examples/exampleDeploymentConfiguration.yml for an example of the yaml deployment configuration which can be loaded using from_yaml_file()

class mtap.deployment.Deployment(global_settings: GlobalSettings | None = None, events_deployment: EventsDeployment | None = None, shared_processor_config: SharedProcessorConfig | None = None, processors: List[ProcessorDeployment] | None = None)

An automatic deployment configuration which launches a configurable set of MTAP services.

global_settings: GlobalSettings | None = None

Settings shared among all services.

events_deployment: EventsDeployment | None = None

Deployment settings for the events service.

shared_processor_config: SharedProcessorConfig | None = None

Shared configuration settings for all processors.

processors: List[ProcessorDeployment] = None

Configurations for individual processors.

static from_dict(conf: Dict) → Deployment

Creates a deployment object from a configuration dictionary.

Parameters:

conf – The configuration dictionary.

Returns:

Deployment object created.

static from_yaml_file(conf_path: str | bytes | PathLike) → Deployment

Loads a deployment configuration from a yaml file.

Parameters:

conf_path – The path to the yaml configuration file.

Returns:

Deployment object created from the configuration.

run_servers() → Tuple[List[str], List[List[str]]]

A context manager that starts all the configured services in subprocesses and returns.

Raises:

ServiceDeploymentException – If one or more of the services fails to launch.

Examples

>>> deploy = Deployment.from_yaml_file('deploy_config.yml')
>>> with deploy.run_servers():
>>>     # do something that requires the servers.
>>> # servers are automatically shutdown / terminated

run_servers_and_wait()

Starts the specified servers and blocks until KeyboardInterrupt, SIGINT, or SIGTERM are received.

Raises:

ServiceDeploymentException – If one or more of the services fails to launch.

class mtap.deployment.GlobalSettings(host: str | None = None, mtap_config: str | None = None, log_level: str | None = None, register: bool | None = None)

Settings shared by event service and all processors.

host: str | None = None

The global host override, forces all services to use a specific host name.

mtap_config: str | None = None

The path to an MTAP config file to load for all services.

log_level: str | None = None

A python logging level to pass to all services.

register: bool | None = None

Whether services should register with service discovery.

static from_dict(conf: Dict[str, Any] | None) → GlobalSettings

Creates a global settings object from a configuration dictionary.

Keyword Arguments:

conf – The configuration dictionary.

Returns:

The global settings object.

class mtap.deployment.EventsDeployment(enabled: bool = False, address: str | None = None, addresses: Sequence[str] | None = None, workers: int | None = None, register: bool | None = None, mtap_config: str | None = None, log_level: str | None = None, startup_timeout: int = 10)

Deployment configuration for the events service.

enabled: bool = False

Whether an events service should be created.

address: str | None = None

The host address of one events service to launch.

addresses: Sequence[str] | None = None

The host addresses of multiple events services to launch.

workers: int | None = None

The number of worker threads the events service should use.

register: bool | None = None

Whether to register the events service with discovery.

mtap_config: str | None = None

Path to an mtap configuration file.

log_level: str | None = None

The log level for the events service.

startup_timeout: int = 10

The startup timeout for events deployment.

static from_dict(conf: Dict | None) → EventsDeployment

Creates the EventsDeployment configuration option from a configuration dictionary.

Parameters:

conf – The configuration dictionary

Returns:

EventsDeployment or None from the configuration dictionary.

class mtap.deployment.SharedProcessorConfig(events_addresses: List[str] | None = None, workers: int | None = None, additional_args: List[str] | None = None, jvm_args: List[str] | None = None, java_classpath: str | None = None, java_additional_args: List[str] | None = None, startup_timeout: float = 30, mp_spawn_method: str | None = None)

Configuration that is shared between multiple processor services.

events_addresses: List[str] | None = None

An optional GRPC-compatible target for the events service to be used by all processors.

workers: int | None = None

The default number of worker threads which will perform processing.

additional_args: List[str] | None = None

A list of additional arguments that should be appended to every processor.

jvm_args: List[str] | None = None

A list of JVM arguments for all Java processors.

java_classpath: str | None = None

A classpath string that will be passed to all Java processors.

java_additional_args: List[str] | None = None

A list of additional arguments that is added to only Java processors.

startup_timeout: float = 30

The default startup timeout for processors.

mp_spawn_method: str | None = None

A multiprocessing.get_context() argument to create the multiprocessing context.

static from_dict(conf: Dict[str, Any] | None) → SharedProcessorConfig

Builds a configuration from a dictionary representation.

Parameters:

conf – The configuration dictionary.

Returns:

A shared processor config object.

class mtap.deployment.ProcessorDeployment(implementation: str, entry_point: str, enabled: bool = True, instances: int = 1, host: str | None = None, port: int | None = None, workers: int | None = None, register: bool | None = None, mtap_config: str | None = None, log_level: str | None = None, name: str | None = None, sid: str | List[str] | None = None, pre_args: List[str] | None = None, additional_args: List[str] | None = None, startup_timeout: float | None = None, mp: bool | None = None, mp_processes: int | None = None, mp_start_method: str | None = None)

Deployment configuration for an MTAP processor.

Used to construct the command for launching the processor. The processor should be a Java Class with a main method or a Python module with a main block. It should accept the standard MTAP processor deployment arguments and launch an MTAP processor using mtap.run_processor() or the equivalent Java method.

implementation: str

Either “java” or “python”.

entry_point: str

Either the java main class, or the python main module.

enabled: bool = True

Whether the processor should be launched as part of deployment.

instances: int = 1

The number of instances of the processor to launch.

host: str | None = None

The listening host for the processor service.

port: int | None = None

The listening port for the processor service.

workers: int | None = None

The number of worker threads per instance.

register: bool | None = None

Whether the processor should register with the discovery service specified in the MTAP config file.

mtap_config: str | None = None

Path to the MTAP configuration file.

log_level: str | None = None

The log level for the processor.

name: str | None = None

An optional service name override to use for registration.

sid: str | List[str] | None = None

An optional service instance unique identifier. If instances is 1 this should be a string, if instances is more than 1 this should be a list of strings, one for each instance.

pre_args: List[str] | None = None

Arguments that occur prior to the MTAP service arguments (like host, port, etc).

additional_args: List[str] | None = None

Arguments that occur after the MTAP service arguments.

startup_timeout: float | None = None

Optional override startup timeout.

mp: bool | None = None

Whether to use the multiprocessing pool based processor server.

mp_processes: int | None = None

If using multiprocessing host, the number of worker processes.

mp_start_method: str | None = None

A multiprocessing.get_context start method to use.

static from_dict(conf: Dict) → ProcessorDeployment

Creates an MTAP processor deployment configuration from a configuration dictionary.

Parameters:

conf – The configuration dictionary.

Returns:

ProcessorDeployment object that can be used to construct the call for the processor.