klio.metrics.native
Klio supports Apache Beam’s metrics via NativeMetricsClient.
NativeMetricsClient
When running on Dataflow, only Counter and Distribution type metrics are emitted to Dataflow’s monitoring. See documentation for more information.
This client is used by default regardless of runner, no configuration needed. Configuration is supported for the unit of measure for timers objects in klio-job.yaml:
klio-job.yaml
job_config: metrics: native: # Default timer unit is s/seconds; available # options include `s` or `seconds`, `ms` or `milliseconds`, # `us` or `microseconds`, and `ns` or `nanoseconds`. timer_unit: ms
To configure a different unit of measure than the default for specific timers, pass in the desired unit when instantiating. For example:
class MyTransform(KlioBaseTransform): def __init__(self): self.my_timer = self._klio.metrics.timer( name="my-timer", timer_unit="ns" )
Caution
When running on Dataflow, in order for the Native metrics client to be able to report metrics to Stackdriver, the following experiment must be added to klio-job.yaml:
experiment
# <--snip--> pipeline_options: experiments: - enable_stackdriver_agent_metrics # <--snip-->
klio.metrics.native.
TIMER_UNIT_MAP
Map of supported measurement units to shorthand for NativeTimer.
NativeTimer
Metrics client for Beam-native metrics collection.
Intended to be instantiated by klio.metrics.client.MetricsRegistry and not by itself.
klio.metrics.client.MetricsRegistry
klio_config (klio_core.config.KlioConfig) – the job’s configuration.
DEFAULT_TIME_UNIT
Default unit of measurement for timer metrics.
counter
Create a NativeCounter object.
NativeCounter
name (str) – name of counter
transform (str) – transform the counter is associated with. Defaults to the job’s name.
a native counter instance
gauge
Create a NativeGauge object.
NativeGauge
name (str) – name of gauge
transform (str) – transform the gauge is associated with. Defaults to the job’s name.
a native gauge instance
timer
Create a NativeTimer object.
name (str) – name of timer
transform (str) – transform the timer is associated with. Defaults to the job’s name.
timer_unit (str) – timer unit; defaults to configured value in klio-job.yaml, or NativeMetricsClient.DEFAULT_TIME_UNIT. Options: TIMER_UNIT_MAP.
NativeMetricsClient.DEFAULT_TIME_UNIT
a native timer instance
Counter metric using Beam’s counter-type metric.
namespace (str) – Name of namespace the counter belongs to (e.g. the counter’s transform).
Gauge metric using Beam’s gauge-type metric.
namespace (str) – Name of namespace the gauge belongs to (e.g. the gauge’s transform).
Timer metric using Beam’s distribution-type metric.
namespace (str) – Name of namespace the timer belongs to (e.g. the timer’s transform).
timer_unit (str) – Unit of measurement. Options: TIMER_UNIT_MAP. Default: ns (nanoseconds).
ns