Transports

Abstract transport and state machine for hosts state.

class cumin.transports.Command(command, timeout=None, ok_codes=None)[source]
Bases: object

Class to represent a command.

Command constructor.

Parameters:

command (str) -- the command to execute.

timeout (int, optional) -- the command's timeout in seconds.

ok_codes (list, optional) -- a list of exit codes to be considered successful for the command. The exit code zero is considered successful by default, if this option is set it override it. If set to an empty list [], it means that any code is considered successful.

__repr__()[source]

Return the representation of the Command.

The representation allow to instantiate a new Command instance with the same properties.

Returns:

the representation of the object.

Return type:

str

__str__()[source]

Return the string representation of the command.

Returns:

the string representation of the object.

Return type:

str

__eq__(other)[source]

Equality operation. Allow to directly compare a Command object to another or a string.

Parameters:

according to Python's Data model object.__eq__().

Returns:

True if the other object is equal to this one, False otherwise.

Return type:

bool

Raises:

exceptions.ValueError -- if the comparing object is not an instance of Command or a str.

__ne__(other)[source]

Inequality operation. Allow to directly compare a Command object to another or a string.

Parameters:

according to Python's Data model object.__ne__().

Returns:

True if the other object is different to this one, False otherwise.

Return type:

bool

Raises:

exceptions.ValueError -- if the comparing object is not an instance of Command or a str.
shortened(max_len: int = 35) → str[source]
Return the shortened version of the command up to the given length, omitting the central part.

Examples
>>> command = Command('long-command --with --many --options and arguments')
>>> command.shortened()
'long-command --w...ns and arguments'
>>> command.shortened(20)
'long-comm...rguments'
>>> command.shortened(50)
'long-command --with --many --options and arguments'
>>> command.shortened(5)
'l...s'
Parameters:

max_len (int) -- the maximum length of the returned string, must be at least 5.

Raises:

cumin.transports.WorkerError -- if the max_len argument is smaller than 5.

Returns:

the shortened version of the command, if it's longer than max_len.

Return type:

str
property timeout

Timeout of the Command.

Getter:

Returns the current timeout or None if not set.

Setter:

float, int, None: the timeout in seconds for the execution of the command on each host. Both float and int are accepted and converted internally to float. If None the timeout is reset to its default value.

Raises:

cumin.transports.WorkerError -- if trying to set it to an invalid value.

property ok_codes

List of exit codes to be considered successful for the execution of the Command.

Getter:

Returns the current ok_codes or a list with the element 0 if not set.

Setter:

list[int], None: list of exit codes to be considered successful for the execution of the command on each host. Must be a list of int in the range 0-255 included, or None to unset it. The exit code 0 is considered successful by default, but it can be overriden setting this property. Set it to an empty list to consider any exit code successful.

Raises:

cumin.transports.WorkerError -- if trying to set it to an invalid value.
class cumin.transports.State(init=None)[source]

Bases: object

State machine for the state of a host.

current

cumin.transports.HostState: the current state.

is_pending

bool: True if the current state is cumin.transports.HostState.PENDING, False otherwise.

is_scheduled

bool: True if the current state is cumin.transports.HostState.SCHEDULED, False otherwise.

is_running

bool: True if the current state is cumin.transports.HostState.RUNNING, False otherwise.

is_success

bool: True if the current state is cumin.transports.HostState.SUCCESS, False otherwise.

is_failed

bool: True if the current state is cumin.transports.HostState.FAILED, False otherwise.

is_timeout

bool: True if the current state is cumin.transports.HostState.TIMEOUT, False otherwise.

State constructor. The initial state is set to pending it not provided.

Parameters:

init (cumin.transports.HostState, optional) -- the initial state from where to start. The pending state will be used if not set.

Raises:

cumin.transports.InvalidStateError -- if init is an invalid state.

allowed_state_transitions = {HostState.FAILED: (), HostState.PENDING: (HostState.SCHEDULED,), HostState.RUNNING: (HostState.RUNNING, HostState.SUCCESS, HostState.FAILED, HostState.TIMEOUT), HostState.SCHEDULED: (HostState.RUNNING,), HostState.SUCCESS: (HostState.PENDING,), HostState.TIMEOUT: ()}

Dictionary with {valid state: tuple of valid states} mapping of the allowed transitions between all the possile states.

This is the diagram of the allowed transitions:

Type:

dict

__getattr__(name)[source]

Attribute accessor.

Accessible properties:

current (cumin.transports.HostState): retuns the current state.

is_{valid_state_name} (bool): for each valid state name, returns True if the current state matches the state in the variable name. False otherwise.

Parameters:

according to Python's Data model object.__getattr__().

Raises:

exceptions.AttributeError -- if the attribute name is not available.

__repr__()[source]

Return the representation of the State.

The representation allow to instantiate a new State instance with the same properties.

Returns:

the representation of the object.

Return type:

str

__str__()[source]

Return the string representation of the state.

Returns:

the string representation of the object.

Return type:

str

update(new)[source]

Transition the state from the current state to the new one, if the transition is allowed.

Parameters:

new (cumin.transports.HostState) -- the new state to set. Only specific state transitions are allowed.

Raises:

cumin.transports.StateTransitionError -- if the transition is not allowed, see allowed_state_transitions.

exception cumin.transports.WorkerError[source]

Bases: CuminError

Custom exception class for worker errors.

exception cumin.transports.StateTransitionError[source]

Bases: CuminError

Exception raised when an invalid transition for a node's State was attempted.

exception cumin.transports.InvalidStateError[source]

Bases: CuminError

Exception raised when an invalid transition for a node's State was attempted.

exception cumin.transports.OutputsMismatchError[source]

Bases: CuminError

Exception raised when attempting to get a single output result but there isn't a single unique output.

exception cumin.transports.SingleOutputMissingHostsError[source]

Bases: CuminError

Exception raised when attempting to get a single output result but the output doesn't cover all hosts.

exception cumin.transports.HostNotFoundError[source]

Bases: CuminError

Exception raised when attempting to get results for a host that is not part of the execution.

exception cumin.transports.SingleCommandOnlyError[source]

Bases: CuminError

Exception raised by methods reserved for a single command execution when there are multiple commands.

class cumin.transports.HostState(*values)[source]

Bases: StrEnum

StrEnum to describe all the possible states for a host.

PENDING = 'pending'

Pending state, not yet scheduled.

Type:: str

SCHEDULED = 'scheduled'

Scheduled for execution state.

Type:: str

RUNNING = 'running'

Running state, during the execution.

Type:: str

SUCCESS = 'success'

Execution completed with success state.

Type:: str

FAILED = 'failed'

Exectution failed state.

Type:: str

TIMEOUT = 'timeout'

Execution timed out state.

Type:: str

static _generate_next_value_(name, start, count, last_values): Return the lower-cased version of the member name.

class cumin.transports.Target(hosts, batch_size=None, batch_size_ratio=None, batch_sleep=None)[source]

Bases: object

Targets management class.

Constructor, inizialize the Target with the list of hosts and additional parameters.

Parameters:

hosts (ClusterShell.NodeSet.NodeSet, list) -- hosts that will be targeted, both ClusterShell.NodeSet.NodeSet and list are accepted and converted automatically to ClusterShell.NodeSet.NodeSet internally.
batch_size (int, optional) -- set the batch size so that no more that this number of hosts are targeted at any given time. It must be a positive integer. If greater than the number of hosts it will be auto-resized to the number of hosts.
batch_size_ratio (float, optional) -- set the batch size with a ratio so that no more that this fraction of hosts are targeted at any given time. It must be a float between 0 and 1 and will raise exception if after rounding it there are 0 hosts selected.
batch_sleep (float, optional) -- sleep time in seconds between the end of execution of one host in the batch and the start in the next host. It must be a positive float.

Raises:

cumin.transports.WorkerError -- if the hosts parameter is empty or invalid, if both the batch_size and batch_size_ratio parameters are set or if the batch_size_ratio selects no hosts.

property first_batch

First batch of the hosts to target.

Getter:: Returns a ClusterShell.NodeSet.NodeSet of the first batch of hosts, according to the batch_size.

_compute_batch_size(batch_size, hosts)[source]

Compute the batch_size based on the hosts size and return the value to be used.

Parameters:

batch_size (int, None) -- a positive integer to indicate the batch_size to apply when executing the worker or None to get its default value of all the hosts. If greater than the number of hosts, the number of hosts will be used as value instead.
hosts (ClusterShell.NodeSet.NodeSet) -- the list of hosts to use to calculate the batch size.

Returns:

the effective batch_size to use.

Return type:

int

static _compute_batch_sleep(batch_sleep)[source]

Validate batch_sleep and return its value or a default value.

Parameters:: batch_sleep (float, None) -- a positive float indicating the sleep in seconds to apply between one batched host and the next, or None to get its default value.
Returns:: the effective batch_sleep to use.
Return type:: float

class cumin.transports.BaseWorker(config, target)[source]

Bases: object

abstract Worker interface to be extended by concrete workers.

Worker constructor. Setup environment variables and initialize properties.

Parameters:

config (dict) -- a dictionary with the parsed configuration file.
target (Target) -- a Target instance.

run_one() → CommandResults[source]

Execute a single command on all the targets using the handler.

This method should be used when there is only one command to execute as it returns directly a cumin.transports.CommandResults that is easier to manage for the single command use case than a cumin.transports.ExecutionResults one as returned by the cumin.transports.BaseWorker.run() method.

Returns:: the results instance for the single command execution.
Return type:: cumin.transports.CommandResults
Raises:: cumin.transports.SingleCommandOnlyError -- if there is more than one command set for the execution.

abstractmethod run() → ExecutionResults[source]

Execute all the configured commands on all the targets using the handler.

When there is only one command to execute, the cumin.transports.BaseWorker.run_one() method can be used instead for an easier access to the results.

Returns:: the results instance for all the commands executed.
Return type:: cumin.transports.ExecutionResults

abstract property results: ExecutionResults

Property to access the results instance after having executed the commands.

Although both cumin.transports.BaseWorker.run_one() and cumin.transports.BaseWorker.run() methods returns already their results, this property can be convient in some cases when in need to access the results at a later point or to get the partial results in case of a catched exception.

Returns:: the execution results instance if the execution is completed.
Return type:: cumin.transports.ExecutionResults
Raises:: cumin.transports.WorkerError -- if unable to return results, for example because the execution has not started yet.

abstractmethod execute()[source]

Execute the task as configured.

Returns:: 0 on success, a positive integer on failure.
Return type:: int
Raises:: cumin.transports.WorkerError -- if misconfigured.

abstractmethod get_results()[source]

Iterate over the results (generator).

Yields:: tuple -- with (hosts, result) for each host(s) of the current execution.

property commands

Commands for the current execution.

Getter:: Returns the current command list or an empty list if not set.
Setter:: list[Command], list[str]: a list of Command objects or str to be executed in the hosts. The elements are converted to Command automatically.
Raises:: cumin.transports.WorkerError -- if trying to set it with invalid data.

abstract property handler

Get and set the handler for the current execution.

Getter:: Returns the current handler or None if not set.
Setter:: str, EventHandler, None: an event handler to be notified of the progress during execution. Its interface depends on the actual transport chosen. Accepted values are: * None => don't use an event handler (default) * str => a string label to choose one of the available default EventHandler classes in that transport, * an event handler class object (not instance)

property timeout

Global timeout for the current execution.

Getter:: int: returns the current timeout or 0 (no timeout) if not set.
Setter:: int, None: timeout for the current execution in seconds. Must be a positive integer or None to reset it.
Raises:: cumin.transports.WorkerError -- if trying to set it to an invalid value.

property success_threshold

Success threshold for the current execution.

Getter:: float: returns the current success_threshold or 1.0 (100%) if not set.
Setter:: float, None: The success ratio threshold that must be reached to consider the run successful. A float between 0 and 1 or None to reset it. The specific meaning might change based on the chosen transport.
Raises:: cumin.transports.WorkerError -- if trying to set it to an invalid value.

cumin.transports.validate_list(property_name, value, allow_empty=False)[source]

Validate a list.

Parameters:

property_name (str) -- the name of the property to validate.
value (list) -- the value to validate.
allow_empty (bool, optional) -- whether to consider an empty list valid.

Raises:

cumin.transports.WorkerError -- if trying to set it to an invalid value.

cumin.transports.validate_positive_integer(property_name, value)[source]

Validate a positive integer or None.

Parameters:

property_name (str) -- the name of the property to validate.
value (int, None) -- the value to validate.

Raises:

cumin.transports.WorkerError -- if trying to set it to an invalid value.

cumin.transports.validate_positive_float(property_name, value)[source]

Validate a positive float or None.

Parameters:

property_name (str) -- the name of the property to validate.
value (float, None) -- the value to validate.

Raises:

cumin.transports.WorkerError -- if trying to set it to an invalid value.

cumin.transports.raise_error(property_name, message, value)[source]

Raise a WorkerError exception.

Parameters:

property_name (str) -- the name of the property that raised the exception.
message (str) -- the message to use for the exception.
value (mixed) -- the value that raised the exception.

class cumin.transports.BaseExecutionProgress[source]

Bases: object

abstract Listener interface to consume notification of the status of successful / failed hosts.

The listener needs to be notified of the total number of hosts when the operation starts, and then notified of successes and failures.

abstractmethod init(num_hosts: int) → None[source]

Initialize the progress bars.

Parameters:: num_hosts (int) -- the total number of hosts

abstractmethod close() → None[source]: Closes the progress bars.

abstractmethod update_success(num_hosts: int = 1) → None[source]

Updates the number of successful hosts.

Parameters:: num_hosts (int) -- increment to the number of hosts that have completed successfully

abstractmethod update_failed(num_hosts: int = 1) → None[source]

Updates the number of failed hosts.

Parameters:: num_hosts (int) -- increment to the number of hosts that have completed in error

class cumin.transports.TqdmProgressBars[source]

Bases: BaseExecutionProgress

Progress bars based on TQDM.

Create the progress bars.

Note

the progress bars themselves are not initalized at object creation. init() needs to be called before using the progress bars.

init(num_hosts: int) → None[source]

Initialize the progress bars.

Parameters:: num_hosts (int) -- the total number of hosts

close() → None[source]: Closes the progress bars.

update_success(num_hosts: int = 1) → None[source]

Updates the number of successful hosts.

Parameters:: num_hosts (int) -- increment to the number of hosts that have completed successfully

update_failed(num_hosts: int = 1) → None[source]

Updates the number of failed hosts.

Parameters:: num_hosts (int) -- increment to the number of hosts that have completed in error

class cumin.transports.NoProgress[source]

Bases: BaseExecutionProgress

Used as a null object to disable the display of execution progress.

init(num_hosts: int) → None[source]: Does nothing.

close() → None[source]: Does nothing.

update_success(num_hosts: int = 1) → None[source]: Does nothing.

update_failed(num_hosts: int = 1) → None[source]: Does nothing.

class cumin.transports.ExecutionStatus(*values)[source]

Bases: IntEnum

Enum to define the possible execution statuses.

SUCCEEDED = 0: The execution completed successfully on all hosts.

COMPLETED_WITH_FAILURES = 1: The execution completed for some hosts, but there were some failures.

FAILED = 2: The execution failed, no host executed all commands successfully.

UNKNOWN = 3: The execution status is unknown, this is the initial status.

TIMEDOUT = 4: The execution timed out due to the global timeout.

INTERRUPTED = 5: The execution has been interrupted.

class cumin.transports.CommandOutputResult(*, splitted_stderr: bool, command: Command, command_index: int, _stdout: MsgTreeElem, _stderr: MsgTreeElem = <factory>)[source]

Bases: object

Dataclass to represent a single result output.

Notes

The instances of this class are read-only.

Parameters:

splitted_stderr (bool) -- whether the capture of stdout and stderr was done separately or all the output is gathered in the output property.
command (cumin.transports.Command) -- the command that generated this output.
command_index (int) -- the index of the command that generated this output in the list of commands.
_stdout (ClusterShell.MsgTree.MsgTreeElem) -- the data structure that recorded the output. Client's code should call the cumin.transports.CommandOutputResult.stdout() method to get the command's standard output.
_stderr (ClusterShell.MsgTree.MsgTreeElem, optional) -- the data structure that recorded the output. Client's code should call the cumin.transports.CommandOutputResult.stderr() method to get the command's standard error output.

stdout(*, encoding: str = 'utf-8') → str[source]

Get the standard output of the executed command.

It contains both stdout and stderr when the cumin.transports.CommandOutputResult.splitted_stderr property is False or just stdout when is True.

Parameters:: encoding (str, optional) -- the encoding to use to convert the output from binary to string.
Returns:: the caputred output string.
Return type:: str

stderr(*, encoding: str = 'utf-8') → str[source]

Get the standard error of the executed command.

Parameters:: encoding (str, optional) -- the encoding to use to convert the output from binary to string.
Returns:: the output string.
Return type:: str
Raises:: cumin.transports.WorkerError -- when the cumin.transports.CommandOutputResult.splitted_stderr property is set to False.

format(*, encoding: str = 'utf-8', colored: bool = False) → str[source]

String representation of the command outputs.

Parameters:

encoding (str, optional) -- the encoding to use to convert the output from binary to string.
colored (bool, optional) -- whether to return a color-encoded output with the same colors of the CLI.

Returns:

a pre-formatted output with all the command captured output, splitting standard output and error if they where recorded separately.

Return type:

str

class cumin.transports.HostsOutputResult(*, hosts: NodeSet, output: CommandOutputResult)[source]

Bases: object

Dataclass to represent a single result output for a group of hosts.

Notes

The instances of this class are read-only.

Parameters:

hosts (ClusterShell.NodeSet.NodeSet) -- the hosts that generated the output.
output (cumin.transports.CommandOutputResult) -- the command result.

class cumin.transports.HostResults(*, name: str, state: HostState, commands: tuple[Command, ...], last_executed_command_index: int, return_codes: tuple[int, ...], outputs: tuple[CommandOutputResult, ...])[source]

Bases: object

Dataclass to represent all the output of all the executed commands for a given host.

Notes

The instances of this class are read-only.

Parameters:

name (str) -- the hostname used to connect to it, usually the FQDN.
state (cumin.transports.HostState) -- the final execution state of the host.
commands (tuple) -- the list of cumin.transports.Command instances, one for each executed command.
last_executed_command_index (int) -- the index of the last executed command from the commands tuple on this host. Has the value of -1 if no commands were executed on this host.
return_codes (tuple) -- a tuple of int objects with the return code of each executed commands. It can be shorter than the commands tuple if some commands were not executed.
outputs (tuple) -- a tuple of cumin.transports.CommandOutputResult instances with the results of the commands execution. Each position in this tuple represent the output of the command with the same index in the commands tuple. It can be shorter than the commands tuple if some commands were not executed.

property completed: bool

Whether all commands were executed and completed execution.

Returns:: if the host has completed the execution of all commands.
Return type:: bool

class cumin.transports.TargetedHostsMembers(*, all: NodeSet, by_state: Mapping[HostState, NodeSet], by_return_code: Mapping[int, NodeSet])[source]

Bases: object

Class to represent the targeted hosts based on their state and return code after the execution.

Each instance of this class is meant to be directly related to a specific command executed.

Notes

The instances of this class are read-only.

Parameters:

all (ClusterShell.NodeSet.NodeSet) -- the set of all targeted hosts.
by_state (Mapping) -- a mapping that for each cumin.transports.HostState member returns the set of targeted hosts that ended up in that state after the execution of the command.
by_return_code (Mapping) -- a mapping that for each return code returns the set of targeted hosts that returned that return code after the execution of the command.

class cumin.transports.TargetedHostsCounters(*, total: int, by_state: Mapping[HostState, int], by_return_code: Mapping[int, int])[source]

Bases: object

Class to represent the targeted hosts count based on their state and return code after the execution.

Each instance of this class is meant to be directly related to a specific command executed.

Notes

The instances of this class are read-only.

Parameters:

total (int) -- the total number of hosts that were scheduled to execute this command.
by_state (Mapping) -- a mapping that for each cumin.transports.HostState member returns the number of targeted hosts that ended up in that state after the execution of the command.
by_return_code (Mapping) -- a mapping that for each return code returns the number of targeted hosts that returned that return code after the execution of the command.

class cumin.transports.TargetedHosts(*, hosts: TargetedHostsMembers, counters: TargetedHostsCounters)[source]

Bases: object

Class to represent the targeted hosts based on their state and return code after the execution.

Each instance of this class is meant to be directly related to a specific command executed.

Notes

The instances of this class are read-only.

Parameters:

hosts (cumin.transports.TargetedHostsMembers) -- the targeted hosts with their name.
counter (cumin.transports.TargetedHostsCounters) -- the targeted hosts counters.

cumin.transports.get_targeted_hosts(*, all_hosts: NodeSet, by_state: Mapping[HostState, NodeSet], by_return_code: Mapping[int, NodeSet]) → TargetedHosts[source]

Helper function to get a cumin.transports.TargetedHosts instance.

Parameters:

all_hosts (ClusterShell.NodeSet.NodeSet) -- the set of all targeted hosts.
by_state (Mapping) -- a mapping that for each cumin.transports.HostState member returns the set of targeted hosts that ended up in that state after the execution of the command.
by_return_code (Mapping) -- a mapping that for each return code returns the set of targeted hosts that returned that return code after the execution of the command.

Returns:

the targeted hosts instance with members and counters.

Return type:

cumin.transports.TargetedHosts

class cumin.transports.CommandResults(*, command: Command, command_index: int, targets: TargetedHosts, outputs: tuple[HostsOutputResult, ...])[source]

Bases: object

Class to expose the results of the execution of a single command.

Notes

The instances of this class are read-only.

Parameters:

command (cumin.transports.Command) -- the command executed.
command_index (int) -- the index of this command in the list of commands executed.
targets (cumin.transports.TargetedHosts) -- the targeted hosts instance with members and counters.
outputs (tuple) -- a tuple of cumin.transports.HostsOutputResult instances with the recorded outputs. If all hosts had no output for this command the tuple will be empty.

property has_no_outputs: bool

Property that tells if the execution has produced no outputs.

Returns:: True if the execution has produced no outputs, False otherwise.
Return type:: bool

property has_single_output: bool

Property that tells if the execution has produced the same output across all targeted hosts.

Returns:: True if the execution has produced a single output that is the same for all targeted hosts.
Return type:: bool

get_single_output() → CommandOutputResult[source]

Quicker access to get the output instance in case all hosts have produced the same output.

Returns:

the output instance.

Return type:

cumin.transports.CommandOutputResult

Raises:

cumin.transports.OutputsMismatchError -- when there are multiple outputs.
cumin.transports.SingleOutputMissingHostsError -- when there is a single output but some hosts had no output.

get_host_output(name: str) → CommandOutputResult | None[source]

Get the output instance for a given host or None if it had no output.

Parameters:: name (str) -- the hostname used to connect to it, usually the FQDN.
Returns:: when the host had no output. cumin.transports.CommandOutputResult: the host output instance when there is any output.
Return type:: None

class cumin.transports.ExecutionResults(*, status: ExecutionStatus, last_executed_command_index: int, commands_results: tuple[CommandResults, ...], hosts_results: Mapping[str, HostResults])[source]

Bases: object

Class to expose to the final results of an execution run.

Notes

The instances of this class are read-only.

Parameters:

status (cumin.transports.ExecutionStatus) -- the final status of the execution run.
last_executed_command_index (int) -- the index of the last executed command in the list of commands to be executed.
commands_results (tuple) -- the tuple of cumin.transports.CommandResults instances for each command to be executed.
hosts_results (mapping) -- the mapping of hosts results with string keys (hostnames or FQDN) and instances of cumin.transports.HostResults as values.

property return_code

Property to easily access the return code of the whole execution run.

Returns:: the return code of the execution.
Return type:: int

property has_no_outputs: bool

Property that tells if the execution has produced no outputs.

Returns:: True if the execution has produced no outputs, False otherwise.
Return type:: bool

Available transports

ClusterShell