`klio.metrics.native`¶

Klio supports Apache Beam’s metrics via NativeMetricsClient.

When running on Dataflow, only Counter and Distribution type metrics are emitted to Dataflow’s monitoring. See documentation for more information.

This client is used by default regardless of runner, no configuration needed. Configuration is supported for the unit of measure for timers objects in klio-job.yaml:

job_config:
  metrics:
    native:
      # Default timer unit is s/seconds; available
      # options include `s` or `seconds`, `ms` or `milliseconds`,
      # `us` or `microseconds`, and `ns` or `nanoseconds`.
      timer_unit: ms

To configure a different unit of measure than the default for specific timers, pass in the desired unit when instantiating. For example:

class MyTransform(KlioBaseTransform):
    def __init__(self):
        self.my_timer = self._klio.metrics.timer(
            name="my-timer", timer_unit="ns"
        )

Caution

When running on Dataflow, in order for the Native metrics client to be able to report metrics to Stackdriver, the following experiment must be added to klio-job.yaml:

# <--snip-->
pipeline_options:
  experiments:
    - enable_stackdriver_agent_metrics
# <--snip-->

klio.metrics.native.TIMER_UNIT_MAP = {'microseconds': 'us', 'milliseconds': 'ms', 'ms': 'ms', 'nanoseconds': 'ns', 'ns': 'ns', 's': 's', 'seconds': 's', 'us': 'us'}¶: Map of supported measurement units to shorthand for NativeTimer.

class klio.metrics.native.NativeMetricsClient(klio_config)¶

Metrics client for Beam-native metrics collection.

Intended to be instantiated by klio.metrics.client.MetricsRegistry and not by itself.

Parameters: klio_config (klio_core.config.KlioConfig) – the job’s configuration.

DEFAULT_TIME_UNIT = 's'¶: Default unit of measurement for timer metrics.

counter(name, transform=None, **kwargs)¶

Create a NativeCounter object.

Parameters

name (str) – name of counter
transform (str) – transform the counter is associated with. Defaults to the job’s name.

Returns

a native counter instance

Return type

NativeCounter

gauge(name, transform=None, **kwargs)¶

Create a NativeGauge object.

Parameters

name (str) – name of gauge
transform (str) – transform the gauge is associated with. Defaults to the job’s name.

Returns

a native gauge instance

Return type

NativeGauge

timer(name, transform=None, timer_unit=None, **kwargs)¶

Create a NativeTimer object.

Parameters

name (str) – name of timer
transform (str) – transform the timer is associated with. Defaults to the job’s name.
timer_unit (str) – timer unit; defaults to configured value in klio-job.yaml, or NativeMetricsClient.DEFAULT_TIME_UNIT. Options: TIMER_UNIT_MAP.

Returns

a native timer instance

Return type

NativeTimer

class klio.metrics.native.NativeCounter(name, namespace)¶

Counter metric using Beam’s counter-type metric.

Parameters

name (str) – name of counter
namespace (str) – Name of namespace the counter belongs to (e.g. the counter’s transform).

class klio.metrics.native.NativeGauge(name, namespace)¶

Gauge metric using Beam’s gauge-type metric.

Parameters

name (str) – name of gauge
namespace (str) – Name of namespace the gauge belongs to (e.g. the gauge’s transform).

class klio.metrics.native.NativeTimer(name, namespace, timer_unit='ns')¶

Timer metric using Beam’s distribution-type metric.

Parameters

name (str) – name of timer
namespace (str) – Name of namespace the timer belongs to (e.g. the timer’s transform).
timer_unit (str) – Unit of measurement. Options: TIMER_UNIT_MAP. Default: ns (nanoseconds).

klio.metrics.native¶

`klio.metrics.native`¶