Data Manipulation Utilities

List of Functions

• extract_chars()

• extract_digits()

• percent_to_decimal()

• round_half_up()

• get_current_date_string()

• convert_date_string()

• IndexGenerator

Getting Started

import_helper("data_utils")

Import the data_utils integration helper in the Integration Before Hook to expose the helper functions to the integration’s namespace.

extract_chars()

data_utils.extract_chars(field, start=0, stop=None, step=None)

Takes str or Flex(None) input and returns spliced output using typical python string splicing logic. Returns None for Flex(None)

data_utils.extract_chars("test123abc", 4) -> "test" 
data_utils.extract_chars("test123abc", 7, 100) -> "abc" 
data_utils.extract_chars("test123abc", 0, 7, 2) -> "ts13" 
data_utils.extract_chars("") -> "" 
data_utils.extract_chars(None, 0, 7, 2) -> None

extract_digits()

data_utils.extract_digits(field)

Given a str or Flex(None) input, and outputs all digits.

data_utils.extract_digits("abc123") -> "123"
data_utils.extract_digits("123-456-7890") -> "1234567890"
data_utils.extract_digits("") -> ""
data_utils.extract_digits(None) -> None

percent_to_decimal()

data_utils.percent_to_decimal(percent)

Given an int, float, Decimal, or number in str format and returns value in decimal format.

data_utils.percent_to_decimal(8) -> Decimal("0.08")
data_utils.percent_to_decimal("") -> None
data_utils.percent_to_decimal(None) -> None

round_half_up()

data_utils.round_half_up(input_num, decimal_places=2)

Given an input of float, int, Decimal, or number in str format, outputs a Decimal number rounded to n decimal places. This implements the typical round half up strategy where 0.5 rounds up.

data_utils.round_half_up(0.875, decimal_places=2) -> Decimal("0.88")
data_utils.round_half_up("", decimal_places=2) -> None
data_utils.round_half_up(None, decimal_places=2) -> None

get_current_date_string()

data_utils.get_current_date_string(timezone, date_format="%Y-%m-%d")

Given a time zone and datetime format input, returns the current date str in the specified format.

data_utils.get_current_date_string("US/Pacific", "%Y-%m-%d") -> "2024-12-22"

All timezones in the pytz package are supported. Here are the major US time zones:

• US/Central

• US/Eastern

• US/Mountain

• US/Pacific

You can use the following Python to capture the full list:

import pytz

# total number of timezones
len(pytz.all_timezones)

# list all timezones
for tz in pytz.all_timezones:
    print(tz)

convert_date_string()

data_utils.convert_date_string(input_date, in_format, out_format="%Y-%m-%d")

Given a date string and it’s date time format, converts to the specified output format as a date string. (default output format is "%Y-%m-%d")\

data_utils.convert_date_string("2024-12-22", "%Y-%m-%d", "%Y/%m/%d") -> "2024/12/22"

IndexGenerator

custom_idx = data_utils.IndexGenerator(start=0)

This class is intended to generate numbers sequentially. A typical use case would be to define field mapping paths with next() and .last so that the Field Mapping paths are dynamically generated and an Include For Each isn’t necessary.

custom_idx.last → None
next(custom_idx) → 0
custom_idx.last → 0
next(custom_idx) → 1
custom_idx.last → 1
...

Be sure to keep() the instance of IndexGenerator in the relevant hook or an AttributeError exception will be thrown. e.g:keep(custom_idx = data_utils.IndexGenerator())

Be aware that Integration Gateway evaluates the Value Column on the Field Mapping table before the Field column which can result in unintended behavior. e.g.:

if custom_idx == 11;

Field Path: some.field.path[next(custom_idx)] → some.field.path[12]
Value Column: f”some_value{custom_idx.last}” → “some_value11“