Utils | EVL Tool Suite

Version: 2.8

16 Utils

EVL Utils are standalone command line utilities which can be split by the purpose.

Other Utils

evl_increment_run_id – increment and return unique job run ID, each job invocation will have it’s unique ID
guess-timestamp-format – read an input dates, datetimes, timestamps and guess format string

16.1 csv2evd

(since EVL 2.2)

Read <file.csv> or standard input, and guess:

data types,
field separator (unless option ‘--separator=<char>’ is used),
if strings are quoted (unless option ‘--quote=<char>’ or ‘--optional-quote=<char>’ is used),
end-of-line character(s) (unless option ‘--dos-eol’ or ‘--lin-eol’ or ‘--mac-eol’ is used)

and write EVD to standard output or to <file.evd>.

It uses header line for field names, spaces are replaced by underscores.

Separator is trying to be guessed in this order: ‘,’ (comma), ‘;’ (semi-colon), ‘|’ (pipe), ‘\t’ (tab), ‘:’ (colon), (space).

Quotation character is guessed in this order: double quotes, single quotes.

EVD is EVL data definition file, for details see man 5 evd.

Synopsis

csv2evd
  [<file.csv>] [-o|--output=<file.evd>]
  [--inline]
  [-d|--date=<format>]
  [-h|--header=<field_name>,...]
  [-n|--no-header]
  [-l|--null=<string>]
  [-q|--quote=<char> | --optional-quote=<char>]
  [-s|--separator=<char>]
  [-t|--datetime=<format>]
  [--timestamp=<format>]
  [--dos-eol | --lin-eol | --mac-eol]
  [-v|--verbose]

csv2evd
  ( --help | --usage | --version )

Options

-d, --date=<format>
by default it tries only ‘%Y-%m-%d’, then ‘%d.%m.%Y’

-h, --header=<field_name>,...
use comma separated list of field names instead of header line, for example when there is no header in csv file (option ‘-n’ must be used) or when other field names would be used

--inline
output EVD in the inline format (for example to use EVD by other component with ‘-d’ option)

--dos-eol
do not guess end-of-line character(s), but suppose the input is text with CRLF as end of line,

--lin-eol
do not guess end-of-line character(s), but suppose the input is text with LF as end of line

--mac-eol
do not guess end-of-line character(s), but suppose the input is text with CR as end of line

-n, --no-header
with this option it suppose there is no header. Fields will be named ‘field_001’, ‘field_002’, etc.

-l, --null=<string>
to specify what string is used for NULL values in CSV, empty string is allowed

-o, --output=<file.evd>
write output into file <file.evd> instead of standard output

--optional-quote=<char>
suppose optional quote character <char>, must be used together with ‘--separator’

-q, --quote=<char>
do not guess if fields are quoted, but suppose <char> as quotation character

-s, --separator=<char>
do not guess the separator, but use <char> instead

-t, --datetime=<format>
by default it tries only ‘%Y-%m-%d %H:%M:%S’

--timestamp=<format>
by default it tries only ‘%Y-%m-%d %H:%M:%S.%E*f’

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

Examples

Having table.csv:

id;started;value
1;2019-06-06;some string

This command:

csv2evd table.csv

will try to guess data types, field separator and if strings are quoted or not, and use header line for field names, to produce EVD to standard output:

id       int              null="" sep=";"
started  date("%Y-%m-%d") null="" sep=";"
value    string           null="" sep="\n"

Just an alternative invocation forwording output EVD to a file:
```
csv2evd < table.csv > table.evd
```

To skip header and use different field names:

csv2evd --header="first_field,other_field,last_one" \
  table.csv > table.evd

Case when there is no header in CSV file, but use specified field names:

csv2evd --no-header --header="first_field,other_field,last_one" \
  table.csv > table.evd

No header in CSV and use generated field names ‘field_001’, ‘field_002’, etc.:
```
csv2evd --no-header table.csv > table.evd
```
Consider specific date format, here day of year (‘001..366’), and ‘|’ as a field separator:
```
csv2evd --date="%j" -s '|' table.csv > table.evd
```

16.2 csv2qvd

(since EVL 2.2)

Read <file.csv> with sturcture defined either in <evd> file or by <inline_evd> or guess

data types,
field separator (unless option ‘--separator=<char>’ is used),
if strings are quoted (unless option ‘--quote=<char>’ or ‘--optional-quote=<char>’ is used),
end-of-line character(s) (unless option ‘--dos-eol’ or ‘--lin-eol’ or ‘--mac-eol’ is used)

and write QVD file to <file.qvd> or standard output. For guessing data types (EVD) it uses utility ‘csv2evd’.

EVD is EVL data definition file, for details see man 5 evd.

Synopsis

csv2qvd
  <file.csv>
  [-o|--output=<file.qvd>]
  [-d|--date=<format>]
  [-h|--header=<field_name>,...]
  [-n|--no-header]
  [-l|--null=<string>]
  [-q|--quote=<char> | --optional-quote=<char>]
  [-s|--separator=<char>]
  [-t|--datetime=<format>]
  [--timestamp=<format>]
  [--dos-eol | --lin-eol |--mac-eol]
  [-v|--verbose]

csv2qvd
  <file.csv> (<evd>|-d <inline_evd>)
  [-o|--output=<file.qvd>]
  [--dos-eol | --lin-eol |--mac-eol]
  [-v|--verbose]

csv2qvd
  ( --help | --usage | --version )

Options

Standard options:

-d, --data-definition=<inline_evd>
either this option or the file <evd> must be presented to use already defined EVD

--dos-eol
do not guess end-of-line character(s), but suppose the input is text with CRLF as end of line,

--lin-eol
do not guess end-of-line character(s), but suppose the input is text with LF as end of line,

--mac-eol
do not guess end-of-line character(s), but suppose the input is text with CR as end of line,

-o, --output=<file.qvd>
write output into <file.qvd> instead of standard output

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

EVD options:

--date=<format>
by default it tries only ‘%Y-%m-%d’, then ‘%d.%m.%Y’

-n, --no-header
with this option it suppose there is no header. Fields will be named ‘field_001’, ‘field_002’, etc.

-l, --null=<string>
to specify what string is used for NULL values in CSV, empty string is allowed

--optional-quote=<char>
suppose optional quote character <char>, must be used together with ‘--separator’

-q, --quote=<char>
do not guess if fields are quoted, but suppose <char> as quotation character

-s, --separator=<char>
do not guess the separator, but use <char> instead

-t, --datetime=<format>
by default it tries only ‘%Y-%m-%d %H:%M:%S’

--timestamp=<format>
by default it tries only ‘%Y-%m-%d %H:%M:%S.%E*f’

Examples

Having ‘some.csv’:

id;started;value
1;2019-06-06;some string

The command:

csv2qvd --null="NULL" some.csv > some.qvd

will produce some.qvd file with these field:

id       int               null="NULL"  sep=";"
started  date("%Y-%m-%d")  null="NULL"  sep=";"
value    string            null="NULL"  sep="\n"

Following invocation will guess data types, field separator and if strings are quoted or not, and use header line for field names:
```
csv2qvd table.csv > table.qvd
```
With the ‘--verbose’ option it will write to standard error the whole EVD file which was used:
```
csv2qvd --verbose table.csv > table.qvd
```

To skip header and use different field names:

csv2qvd --header="first_field,other_field,last_one"
  table.csv > table.qvd

Case when there is no header in CSV file, but use specified field names:

csv2qvd --no-header --header="first_field,other_field,last_one" \
  table.csv > table.qvd

No header in CSV and use generated field names ‘field_001’, ‘field_002’, etc.:
```
csv2qvd --no-header table.csv > table.qvd
```
Consider specific date format, here day of year (‘001..366’), and ‘|’ as a field separator:
```
csv2qvd --date="%j" -s '|' table.csv > table.qvd
```
To use own (specific or already generated) EVD file (i.e. data types definition):
```
csv2qvd table.csv table.evd > table.qvd
```

16.3 evd2sql

(since EVL 2.6)

Read the EVL data definition (a.k.a. EVD) from <table.evd> and write to standard output (unless ‘--output’ option is used) ‘CREATE TABLE’ statement specific for given SQL dialect: ANSI, MS SQL, PostgreSQL, Redshift, etc.

When more than one <table.evd> files specified, then write to ‘*.sql’ files named accordingly with the same basename.

The SQL statement looks like this in general:

CREATE TABLE [IF NOT EXISTS] ["<schema_name>".]"<table_name>" (
    <column_1_based_on_evd>
  , <column_2_based_on_evd>
  , <column_3_based_on_evd>
  , ...
[<table_constraints>]
)
[<table_attributes>]
;

EVL data types mapping:

EVL	Postgres/Redshift	MS SQL
‘`char`’	‘`"CHAR"`’	‘`SMALLINT`’
‘`uchar`’	‘`BOOLEAN`’	‘`TINYINT`’
‘`short`’	‘`SMALLINT`’	‘`SMALLINT`’
‘`ushort`’	‘`SMALLSERIAL`’	‘`INT`’
‘`int`’	‘`INTEGER`’	‘`INT`’
‘`uint`’	‘`SERIAL`’	‘`BIGINT`’
‘`long`’	‘`BIGINT`’	‘`BIGINT`’
‘`ulong`’	‘`BIGSERIAL`’	‘`DECIMAL(20,0)`’
‘`int128`’	‘`NUMERIC(38,0)`’	‘`DECIMAL(38,0)`’
‘`utint128`’	‘`NUMERIC(38,0)`’	‘`DECIMAL(38,0)`’
‘`float`’	‘`REAL`’	‘`REAL`’
‘`double`’	‘`DOUBLE PRECISION`’	‘`FLOAT`’
‘`decimal(m,n)`’	‘`NUMERIC(m,n)`’	‘`DECIMAL(m,n)`’
‘`string`’	‘`TEXT`’	‘`VARCHAR`’
‘`ustring`’	‘`TEXT`’	‘`NVARCHAR`’
‘`date`’	‘`DATE`’	‘`DATE`’
‘`time`’	‘`TIME`’	‘`TIME`’
‘`interval`’	‘`INTERVAL`’	N/A
‘`datetime`’	‘`TIMESTAMP(0)`’	‘`DATETIME2(0)`’
‘`timestamp`’	‘`TIMESTAMP(6)`’	‘`DATETIME2(6)`’

Synopsis

evd2sql
  ( <table.evd>... | -i|--input <table.evd> )
  [-d|--sql-dialect <database> ]
  [--if-not-exists]
  [-o|--output ( <table.sql> | <target_dir> ) ]
  [-s|--schema <schema_name>]
  [-t|--table <table_name>]
  [--table-attributes <table_attributes>]
  [--table-constraints <table_constraints>]
  [--varchar <length>]
  [-v|--verbose]

evd2sql
  ( --help | --usage | --version )

Options

-d, --sql-dialect=<database>
currently these SQL types are supported:

ansi (default)
mssql
postgres
redshift

--if-not-exists
use ‘CREATE TABLE IF NOT EXISTS’ instead of default ‘CREATE TABLE’

-i, --input=<table.evd>
read file <table.evd>

-o, --output=<path>
if <path> is an existing directory, it writes output there. If it is not a directory, it is considered as an output file name.

-s, --schema=<schema_name>
add <schema_name> to table name

-t, --table=<table_name>
by default basename of <table.evd> from ‘--input’ option is used as table name in ‘CREATE TABLE’ statement, this option can overwrite it. When reading EVD from standard input, this option is recommended, otherwise table name will be empty

--table-attributes=<table_attributes>
string to be added right after closing bracket, e.g. for Redshift it might be ‘SORTKEY (some_id,other_col)’

--table-constraints=<table_constraints>
string to be added right after column list, e.g. ‘, PRIMARY KEY (some_id)’

--varchar=<length>
specify the default VARCHAR length, default is 256

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

Examples

Having an EVD file ‘some.evd’:

id       int             sep=";"
started  date    null="" sep=";"
value    string  null="" sep="\n"

This command:

evd2sql -s postgres -i some.evd --if-not-exists

will produce:

CREATE TABLE IF NOT EXISTS "some" (
    id      INTEGER NOT NULL
  , started DATE
  , value   TEXT
);

16.4 guess-timestamp-format

(since EVL 2.4)

Read line by line standard input or an input <file> and try to guess format string of the date, datetime or timestamp.

It uses the <config_file> with the list of format strings like:

%Y-%m-%d %H:%M:%S
%Y-%m-%dT%H:%M:%S
%Y-%m-%d
%d/%m/%y %H:%M
%-m/%-d/%y %H:%M
%d.%m.%y %H:%M
%d.%m.%y %-H:%M
%d.%m.%y %-H:%M:%S

Unless ‘--config’ option is used, it uses a file timestamp-formats-order.csv from the same folder as this script (try ‘which guess-timestamp-format’).

Synopsis

guess-timestamp-format
  [-i|--input=<file>] [-c|--config=<config_file>] [-d|--with-data-type]
  [-v|--verbose]

guess-timestamp-format
  ( --help | --usage | --version )

Options

-c, –config=<config_file>

-d, –with-data-type

-i, –input=<file>

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

Examples

Let us have this file ‘timestamps.csv’:
```
03/12/2022 11:20:00
03/12/2022 01:02:00
03/13/2022 03:24:55
03/14/2022 11:20:59
```
Following comman recognize the format:
```
guess-timestamp-format < timestamps.csv
```
and returns:
```
%m/%d/%Y %H:%M:%S
```
With the option ‘--with-data-type’ it returns full data type information:
```
datetime("%m/%d/%Y %H:%M:%S")
```

16.5 json2evd

(since EVL 2.4)

Read <file.json> or standard input, guess data types, and write EVD to standard output or to <file.evd>.

Synopsis

json2evd
  [<file.json>] [-o|--output=<file.evd>]
  [-d|--date=<format>]
  [-l|--null=<string>]
  [-q|--quote=<char>]
  [-s|--separator=<char>]
  [-t|--datetime=<format>]
  [-v|--verbose]

json2evd
  ( --help | --usage | --version )

Options

-d, --date=<format>
by default it tries only ‘%Y-%m-%d’, then ‘%Y%m%d’, then ‘%d.%m.%Y’

-l, --null=<string>
to specify what string is used for NULL values in JSON, empty string is allowed

-o, --output=<file.evd>
write output into file <file.evd> instead of standard output

-q, --quote=<char>
do not guess if fields are quoted, but suppose <char> as quotation character

-s, --separator=<char>
do not guess the separator, but use <char> instead

-t, --datetime=<format>
by default it tries only ‘%Y-%m-%d %H:%M:%S’, then ‘%Y%m%d%H%M%S’

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

Examples

Having some.json:

TBA

This command:

json2evd --null="" some.json

will produce:

id       int              null="" sep=";"
started  date("%Y-%m-%d") null="" sep=";"
value    string           null="" sep="\n"

16.6 pg2evd

(since EVL 2.6)

Read the definition of PostgreSQL table and write EVD to standard output or to <file>.

Password is taken:

from file ‘$EVL_PASSFILE’, which is by default ‘$HOME/.evlpass’,

from file ‘$PGPASSFILE’, which is by default ‘$HOME/.pgpass’.

When such file has not permissions 600, it is ignored! For details see ‘evl-password’.

Synopsis

pg2evd
  [<schema>.]<table> [-o|--output=<file>] [-f|--output-format (evd|json)]
  [-b|--dbname=<database>] [-h|--host=<hostname>] [-p|--port=<port>]
  [-u|--username=<pguser>] [--psql=<psql_options>]
  [-d|--date=<format>]
  [-l|--null[=<string>]]
  [-q|--quote=<char>]
  [-r|--record-separator=<char>]
  [-s|--field-separator=<char>]
  [-t|--datetime=<format>]
  [-v|--verbose]

pg2evd
  ( --help | --usage | --version )

Options

-d, --date=<format>
by default it produce no format for date, so the EVL_DEFAULT_DATE_PATTERN is then used (by default it is ‘"%Y-%m-%d"’)

-l, --null=<string>
add ‘null="<string>"’ to every field

-o, --output=<file>
write output into file <file> instead of standard output

-f, --output-format=(evd|json)
write output in given file format, by default write ‘evd’

-q, --quote=<char>
add ‘quote="<char>"’ to every field

-s, –separator=<char>

-r, --record-separator=<char>
use ‘sep="<char>"’ for last field

-s, --field-separator=<char>
add ‘sep="<char>"’ to each field, except the last one

-t, --datetime=<format>
by default it produces no format for datetime, so the EVL_DEFAULT_DATETIME_PATTERN is then used (which is by default set to ‘"%Y-%m-%d %H:%M:%S"’)

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

’psql’ options:

-b, --dbname=<database>
either this or environment variable ‘PGDATABASE’ should be provided, if not, then current system username is used as psql database. If also ‘PGDATABASE’ environment variable is set, this option has preference. (This option is provided to ‘psql’ command.)

-h, --host=<hostname>
either this or environment variable ‘PGHOST’ should be provided when connecting to other host than localhost. If also ‘PGHOST’ variable is set, this option has preference. (This option is provided to ‘psql’ command.)

-p, --port=<port>
either this or environment variable ‘PGPORT’ should be provided when using other then standard port ‘5432’. (This option is provided to ‘psql’ command.)

--psql=<psql_options>
all other options to be provides to psql command. See ‘man psql’ for details.

-u, --username=<pguser>
either this or environment variable ‘PGUSER’ should be provided, if not, then current system username is used as psql user. If variable ‘PGUSER’ is set, this option has preference. (This option is provided to ‘psql’ command.)

Examples

Having ‘some_table’ in databese ‘some_db’:

id      integer
started date
value   varchar(20)

This command:

pg2evd --null="" --separator=";" some_db.some_table

will produce:

id       int     null="" sep=";"
started  date    null="" sep=";"
value    string  null="" sep="\n"

16.7 qvd2csv

(since EVL 2.4)

Read <file.qvd> and write CSV file to <file.csv> or standard output. It uses data types from QVD header or from existing <evd> file or from <inline_evd>.

EVD is EVL data definition file, for details see man 5 evd.

Synopsis

qvd2csv
  <file.qvd>
  [-o|--output=<file.csv>]
  [--all-as-string | --real-as-decimal[=<precision>,<scale>]]
  [-d|--date=<format>]
  [-h|--header=<field_name>,...]
  [-n|--no-header]
  [-l|--null=<string>]
  [-q|--quote=<char>]
  [-s|--separator=<char>]
  [-t|--datetime=<format>]
  [-a|--dos-eol | -b|--mac-eol]
  [--filter=<condition>]
  [--first-record=<n>]
  [--guess-uniform-symbol-size]
  [--low-memory]
  [-v|--verbose]

qvd2csv
  <file.qvd> (<evd>|-d <inline_evd>)
  [-m|--match-fields]
  [-o|--output=<file.csv>]
  [-h|--header=<field_name>,...]
  [-n|--no-header]
  [-a|--dos-eol | -b|--mac-eol]
  [--filter=<condition>]
  [--first-record=<n>]
  [--guess-uniform-symbol-size]
  [--low-memory]
  [-v|--verbose]

qvd2csv
  ( --help | --usage | --version )

Options

--all-as-string
interpret all fields as strings. (Since EVL 2.5.)

-d, --data-definition=<inline_evd>
either this option or the file <evd> must be presented to use already defined (custom) EVD

-a, --dos-eol
output DOS end-of-line, i.e. CR+LF (‘\r\n’)

-b, --mac-eol
output Mac end-of-line, i.e. CR (‘\r’)

--date=<format>
to specify a <format> for date data type

--filter=<condition>
read only records with given <condition>. (Since EVL 2.6.)

--first-record=<n>
start to read from the record number <n>. (Since EVL 2.6.)

--guess-uniform-symbol-size
might speed up indexing of dictionary, but it could not work in all cases. Use only in special cases when need really good performance. (Since EVL 2.6.)

-h, --header=<field_name>,...
use comma separated list of field names instead of header line, for example when you don’t want to use field names from QVD header.

--low-memory
do not read dictionary into memory. This could save memory consumption, but slows down reading the source file. (Since EVL 2.6.)

-l, --null=<string>
to specify what string is used for NULL values in CSV, empty string is allowed

-m, --match-fields
to read only a subset of fields from QVD file or to read them in different order

-n, --no-header
with this option it produces no header line

-o, --output=<file.csv>
write output into <file.csv> instead of standard output

-q, --quote=<char>
to use quoted fields for the CSV output. When data contains such <char>, all of them are escaped by duplicating them. For example using ‘--quote="\""’ will serve data like ‘some "text"’ as ‘"some ""text"""’.

--real-as-decimal[=<precision>,<scale>]
interpret ‘real’ data types as ‘decimal(<precision>,<scale>)’. When no <precision> or <scale> is specified, use values from environment variables ‘EVL_DEFAULT_DECIMAL_PRECISION’ and ‘EVL_DEFAULT_DECIMAL_SCALE’, which are by default set to 18 and 2. (Since EVL 2.5.)

-s, --separator=<char>
to use <char> as field separator for the CSV output

-t, --datetime=<format>
to specify a <format> for datetime data type

-v, --verbose
print to standard error output info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

Examples

Having ‘some.qvd’, the command to produce CSV file with empty strings representing NULL values, dates in format ‘DD.MM.YYYY’ and with Windows end-of-line (i.e. CRLF):
```
qvd2csv --null="" --date="%d.%m.%Y" --dos-eol some.qvd > some.csv
```
To filter only particular records from ‘large.qvd’, for example we would like to read only latest records represented by field ‘invoice_id’:
```
qvd2csv --filter="invoice_id>7654000" large.qvd > latest.csv
```
To cut only particular columns from ‘large.qvd’, for example only column ‘invoice_id’:
```
qvd2csv --match-fields -d 'invoice_id int null=""' large.qvd > latest.csv
```
To read only after by some number of rows:
```
qvd2csv --first-record=1234000 huge.qvd > latest.csv
```
This could be quite useful when reading a huge QVD file.

16.8 qvd2evd

(since EVL 2.4)

Read header of <file.qvd> or standard input, guess data types, and write EVD to standard output or to <file.evd>.

EVD is EVL data definition file, for details see man 5 evd.

Synopsis

qvd2evd
  [<file.qvd>] [-o|--output=<file.evd>]
  [--all-as-string | --real-as-decimal[=<precision>,<scale>]]
  [-d|--date=<format>]
  [--inline]
  [-l|--null[=<string>]]
  [-q|--quote=<char>]
  [-r|--record-separator=<char>]
  [-s|--field-separator=<char>]
  [-t|--datetime=<format>]
  [-v|--verbose]

qvd2evd
  ( --help | --usage | --version )

Options

--all-as-string
produce EVD with all fields as strings. (Since EVL 2.5.)

-d, --date=<date_format>
use format argument for date data type

--inline
output EVD in the inline format (for example to use EVD by other component with ‘-d’ option)

-l, --null=<string>
to specify what string is used for NULL values in QVD, empty string is allowed

-o, --output=<file.evd>
write output into file <file.evd> instead of standard output

-q, --quote=<char>
to use a quote argument in EVD

--real-as-decimal[=<precision>,<scale>]
produce EVD with ‘decimal(<precision>,<scale>)’ instead of ‘double’. When no <precision> or <scale> is specified, it uses values from environment variables ‘EVL_DEFAULT_DECIMAL_PRECISION’ and ‘EVL_DEFAULT_DECIMAL_SCALE’, which are by default set to 18 and 2. (Since EVL 2.5.)

-r, --record-separator=<char>
use ‘sep="<char>"’ for last field

-s, --field-separator=<char>
add ‘sep="<char>"’ to each field, except the last one

-t, --datetime=<format>
use format for datetime data type by default it produces no format for datetime, so the EVL_DEFAULT_DATETIME_PATTERN is then used (which is by default set to ‘"%Y-%m-%d %H:%M:%S"’)

-v, --verbose
print to STDERR info/debug messages

--help
print this help and exit

--usage
print short usage information and exit

--version
print version and exit

Examples

Having ‘some.qvd’, this command:

qvd2evd --null -r '\n' -s ';' -d '%d.%m.%Y' some.qvd

will produce:

id       int              null="" sep=";"
started  date("%d.%m.%Y") null="" sep=";"
value    string           null="" sep="\n"

16.9 evl_increment_run_id

(since EVL 2.0)

Let’s have defined variable EVL_RUN_ID_FILE with the path to the file which contains the high watermark of some incremental ID. (Let’s call such value RUN_ID as the purpose is usually having unique ID of each job.)

The invocation of the command ‘evl_increment_run_id’ takes the value from the file defined by EVL_RUN_ID_FILE, increase it by one and write it back to file and also to ‘stdout’.

So one can use it this way:

In each job, there would be

export EVL_RUN_ID=$(evl_increment_run_id)

and then the value can be used in the mapping for example:

static long run_id = atol(std::getenv("EVL_RUN_ID"));

Data type of RUN_ID is ‘long’.

If EVL_RUN_ID_FILE is not defined, then ‘evl_increment_run_id’ command fail. If it is defined, but the file doesn’t exist or is empty, then the file is created with the initial value~0.

When the file EVL_RUN_ID_FILE is locked by other process, ‘evl_increment_run_id’ will try every 300\ms to get access. It will wait at most EVL_RUN_ID_FILE_LOCK_WAIT seconds before fail. Default is 30 seconds.

16.10 qvd-header

(since EVL 2.3)

Take the header of <file.qvd> or standard input and produce to standard output particular information, for example EVL data definition file or number of records.

Synopsis

qvd-header
  [<file.qvd>] --output=evd
  [--all-as-string | --real-as-decimal[=<precision>,<scale>]]
  [-d|--date=<format>]
  [--inline]
  [-l|--null=<string>]
  [-q|--quote=<char>]
  [-r|--record-separator=<char>]
  [-s|--field-separator=<char>]
  [-t|--datetime=<format>]
  [-v|--verbose]

qvd-header
  [<file.qvd>] --output=(json|xml)
  [--fields]
  [-v|--verbose]

qvd-header
  [<file.qvd>]
  [ --table-name | --no-of-records | --fields | --tag=<xml_tag_name> ]
  [-v|--verbose]

qvd-header
  ( --help | --usage | --version )

Options

--no-of-records
return the value of ‘NoOfRecords’ tag

--fields
provide only fields’ information

--table-name
return the value of ‘TableName’

--tag=<xml_tag_name>
return the value of <xml_tag_name>

Output Options:

--output=evd
return EVD data types definition

--output=json
return information as JSON

--output=xml
return information as XML

EVD options:

--all-as-string
produce EVD with all fields as strings. (Since EVL 2.5.)

-d, --date=<date_format>
use format argument for date data type

--inline
output EVD, XML or JSON in the inline format (for example to use EVD by other component with ‘-d’ option)

-l, --null=<string>
to specify what string is used for NULL values in EVD

-q, --quote=<char>
to use a quote argument in EVD

-r, --record-separator=<char>
use ‘sep="<char>"’ for last field

-s, --field-separator=<char>
use ‘sep="<char>"’ for each field

-t, --datetime=<date_format>
use format for datetime data type

Standard options:

--help
print this help and exit

--usage
print short usage information and exit

-v, --verbose
print to stderr info/debug messages of the component

--version
print version and exit

Examples

EVD example:

qvd-header some.qvd --output=evd --record-separator="\n" \
                    --null="" --date="%Y-%m-%d"

will produce for example:

id          int              null=""
some_stamp  datetime         null=""
some_date   date("%Y-%m-%d") null=""
value       string           null="" sep="\n"

JSON example:

qvd-header some.qvd --output=json --fields

will produce for example:

{
  "fields":
  [
    {
      name: "REQUEST_HOUR",
      type: "timestamp",
      format: "%Y-%m-%d"
    },
    ...
  ]
}

And:

qvd-header some.qvd --output=json

will produce for example:

{
  "name": "Table1",
  "records": 3615,
  "fields":
  [
    {
      name: "REQUEST_HOUR",
      type: "timestamp",
      format: "%Y-%m-%d",
      "tags": [
        "$numeric",
        "$timestamp"
      ]
    },
    ...
  ]
}

16 Utils​

CSV Utils​

EVD Utils​

JSON Utils​

QVD Utils​

Other Utils​

16.1 csv2evd​

Synopsis​

Options​

Examples​

16.2 csv2qvd​

Synopsis​

Options​

Standard options:​

EVD options:​

Examples​

16.3 evd2sql​

Synopsis​

Options​

Examples​

16.4 guess-timestamp-format​

Synopsis​

Options​

Examples​

16.5 json2evd​

Synopsis​

Options​

Examples​

16.6 pg2evd​

Synopsis​

Options​

’psql’ options:​

Examples​

16.7 qvd2csv​

Synopsis​

Options​

Examples​

16.8 qvd2evd​

Synopsis​

Options​

Examples​

16.9 evl_increment_run_id​

16.10 qvd-header​

Synopsis​

Options​

Output Options:​

EVD options:​

Standard options:​

Examples​

16 Utils

CSV Utils

EVD Utils

JSON Utils

QVD Utils

Other Utils

16.1 csv2evd

Synopsis

Options

Examples

16.2 csv2qvd

Synopsis

Options

Standard options:

EVD options:

Examples

16.3 evd2sql

Synopsis

Options

Examples

16.4 guess-timestamp-format

Synopsis

Options

Examples

16.5 json2evd

Synopsis

Options

Examples

16.6 pg2evd

Synopsis

Options

’psql’ options:

Examples

16.7 qvd2csv

Synopsis

Options

Examples

16.8 qvd2evd

Synopsis

Options

Examples

16.9 evl_increment_run_id

16.10 qvd-header

Synopsis

Options

Output Options:

EVD options:

Standard options:

Examples