Skip to main content
Version: 2.5

Release-Notes

2 Release Notes ​

Versions numbering: EVL x.y.z

x – major release, i.e. big changes must happen to advance this number

y – minor releases, i.e. introduce new features

z – bugfixes

Overview ​

Version 1.0 (2017/07)
Initial version of “classical” ETL tool with HDFS support.
New components: Aggreg, Cat, Comp, Cut, Filter, Join, Map, Read, Sort, Tee, Trash, Write.
New commands: Mv, Mkdir.

Version 1.1 (2017/11)
Read/Write-Avro (in flat format only), Read/Write-Kafka, Read/Write-Teradata. Validation functions within mapping: ‘is_valid_*’.
New components: Head, Readavro, Readkafka, ReadTD, Tail, Validate, Writeavro, Writekafka, WriteTD.

Version 1.2 (2018/01)
Nested fields, Partitioning/parallelism, Read/Write JSON/Avro with full nested fields support.
New utilities: evd_remove_comments, evd_to_avro_scheme.
New components: Assign, Depart, Gather, Merge, Partition, Readjson, Watcher, Writejson.

Version 1.3 (2018/03)
Read/Write XML, Records generator.
New components: Generate, Readxml, Writexml.
New command: Cp.

Version 2.0 (2018/11)
EVL Job Manager renamed to EVL Worklow, man pages, Read/Write PostgreSQL, standalone components, ‘in_to_out()’ enhanced, Shared lookup tables, Read/Write Parquet, sort within a group.
New utilities: evl_increment_run_id – unique EVL run ID.
New components: Echo, Lookup, Readparq, ReadPG, RunPG, Sortgroup, Writeparq, WritePG.
New commands: Fr, Ls, Rm, Spark.

Version 2.1 (2019/04)
Microservices introduced, Read/Write Oracle, Write XLSX and QVX, Docker image available.
New components: ReadOra, RunOra, WriteOra, Writeqvx, Writexlsx.
New commands: Mail, Test.

Version 2.2 (2019/10)
EVL Manager – graphical web UI, output to Qlikview/Qliksense’s QVD file,
New utilities: csv2evd, csv2qvd.
New components: Readxls, Readxlsx, Tac, Writeqvd.

Version 2.3 (2020/04)
EVL Manager enhancements, Read and Write supports also sftp://, gs://, s3://, Clang compiler can be used, debug mode.
New utilities: qvd_header.
New components: Readqvd.
New commands: Chmod.

Version 2.4 (2020/10)
EVL Manager – Microsevices integration, crontab generation, anonymize IBAN, IP addresses manipulation, MySQL/MariaDB, password handling.
New components: Readmysql, Runmysql, Writemysql.
New commands: cancel, crontab, info, manager, Sleep.
New utilities: json2evd, qvd2evd.

Version 2.5 (2021/04)
Read and Write supports also Samba and database URIs, randomize IBAN, EVD manipulation components.
New components: Readevd, Writeevd.

Version 2.6 (2021/10)
New options for Readqvd to speed up reading subset of QVD file, secret password handling.
New mapping functions: rsa_encrypt(), rsa_decrypt(), str_to_base64(), base64_to_str().
New commands: key generate, Rmdir.
New utilities: pg2evd.

Version 2.7 (2022/04)
Read and Write supports also Google Drive: sdrive://.
New utilities: evd2sql, guess-timestamp-format.
New commands: Touch.
New components: Readsqlite, Runsqlite, Writesqlite.

Version 2.8 (2023/04)
New math functions: abs(), ceil(), floor(), max(), min(), pow(), round(), sqrt(), trunc()
New mapping functions: fail(), is_equal(), log(), warn()
New data types: time, interval.
New command: Calendar.
New component: Wc.

Version 1.0 ​

Released
2017/07

Changes
First official version was released in the summer 2017 after more than a year of design and development and after first industry implementation in T-Mobile CZ.

New features

  • ‘Lookup tables’ – lookup loaded into memory and used in mappings.
  • ‘Checksum functions’ – standard checksum function for strings:
    ‘md5’, ‘sha224’, ‘sha256’, ‘sha384’, ‘sha512’.
  • HDFS support
  • Spark code generation – Parquet and Impala integration
  • Job Manager

New components

  • ‘Aggreg’ – do aggregation for groups of records.
  • ‘Cat’ – concatenate several input flows into single output one.
  • ‘Comp’ – use custom component, which is actually another job.
  • ‘Cut’ – omit fields from input by the output data definition.
  • ‘Filter’ – for simple one- or two-way switch. For more complex use ‘Map’.
  • ‘Join’ – join two input flows by the key. Catch left/right or even unmatched records.
  • ‘Map’ – transform input fields and write into output fields.
  • ‘Read’ – read file(s) into output flow, uncompress if needed.
  • ‘Sort’ – sort, deduplicate, check sort; simply the output is always sorted by the key.
  • ‘Tee’ – replicate one input flow to several output ones.
  • ‘Trash’ – like /dev/null.
  • ‘Write’ – write the flow into file, compress if needed.

New commands

  • ‘Mkdir’
  • ‘Mv’

Version 1.1 ​

Released
2017/11

New features

  • Avro data format support (only flat structures)
  • Produce/consume Kafka data stream
  • Teradata FastExport and FastLoad integration
  • Validation functions within mapping

New components

  • ‘Head’
  • ‘Readavro’
  • ‘Readkafka’
  • ‘ReadTD’
  • ‘Tail’
  • ‘Validate’
  • ‘Writeavro’
  • ‘Writekafka’
  • ‘WriteTD’

Version 1.2 ​

Released
2018/01

New features

  • Nested fields
  • Partitioning and parallelism
  • Read/write JSON files with full nested fields support
  • Read/write Avro files with full nested fields support

New utilities

  • ‘evd_remove_comments’
  • ‘evd_to_avro_scheme’

New components

  • ‘Assign’ – assign content of the flow into variable
  • ‘Depart’
  • ‘Gather’
  • ‘Merge’
  • ‘Partition’
  • ‘Readjson’
  • ‘Watcher’ – makes debugging easier
  • ‘Writejson’

Bugfixes
Fixed issue with standard C++ library that was very rarely and on some systems only, causing memory/data corruption when copying partially overlapping memory regions.

Version 1.3 ​

Released
2018/03

New features

  • Read/write XML
  • Sample data generator
  • New string manipulation functions

New components

  • ‘Generate’
  • ‘Readxml’
  • ‘Writexml’

New commands

  • ‘Cp’

Version 2.0 ​

Released
2018/11

New features

  • EVL Job Manager renamed to EVL Workflow
  • Man pages
  • Standalone components (i.e. components can be used from command-line)
  • Handy mapping functions applied for group of columns – ‘in_to_out()’
  • PostgreSQL connectivity
  • String manipulation functions added:
    ‘length’, ‘starts_with’, ‘ends_with’, ‘substr’,
    ‘hex_to_str’, ‘str_to_hex’,
    ‘str_compress’, ‘str_uncompress’
    ‘str_index’, ‘str_rindex’
  • Shared lookup tables – one lookup can be used in several mappings
  • Sort within a group – sort already grouped data
  • Read/Write Parquet files
  • Incremental unique Run ID

New utilities

  • ‘evl_increment_run_id’

New components

  • ‘Echo’
  • ‘Lookup’
  • ‘Readparq’
  • ‘ReadPG’
  • ‘RunPG’
  • ‘Sortgroup’
  • ‘Writeparq’
  • ‘WritePG’

New commands

  • ‘Fr’ – to handle File Registration
  • ‘Ls’
  • ‘Rm’
  • ‘Spark’ – to run Spark jobs

Version 2.1 ​

Released
2019/04

New features

  • EVL Workflow enhanced
  • Microservices – EVL parts bundled for specific purposes
  • Oracle connectivity
  • String manipulation functions added (‘str_pad_left’, ‘str_pad_right’)
  • Docker images available
  • New outputs available: XLSX, QVX

New components

  • ‘ReadOra’
  • ‘RunOra’
  • ‘WriteOra’
  • ‘Writeqvx’ – Write QlikView’s QVX file
  • ‘Writexlsx’ – Write Excel sheets

New commands

  • ‘Mail’ – to have e-mailing easier
  • ‘Test’ – to handle test command also on HDFS

Version 2.2 ​

Released
2019/10

New features

  • EVL Manager – initial version of monitoring tool with web UI
  • Full Unicode support (ICU)
  • Helpers: ‘csv2evd’, ‘csv2qvd’

New utilities

  • ‘csv2evd’
  • ‘csv2qvd’

New components

  • ‘Readxls’ – read (oldstyle) Excel sheets
  • ‘Readxlsx’ – read Excel sheets
  • ‘Tac’ – write records in reverse order
  • ‘Writeqvd’ – write Qlik’s QVD file

Version 2.3 ​

Released
2020/04

New features

  • EVL Manager – various GUI enhancements
  • ‘Read’ and ‘Write’ components support (next to ‘hdfs://’) also ‘sftp://’, ‘gs://’, and ‘s3://’ URI
  • next to GCC, also Clang compiler can be used
  • DEBUG mode

New utilities

  • ‘qvd_header’ – get QVD’s header XML and provide various information, in JSON, XML or EVD

New components

  • ‘Readqvd’ – read Qlik’s QVD file

New commands

  • ‘Chmod’

Version 2.4 ​

Released
2020/10

New features

  • EVL Manager – EVL Microservices integrated
  • Anonymize IBAN
  • MariaDB/MySQL connectivity
  • Functions: ‘str_join’, ‘is_in’
  • IPv4 and IPv6 manipulation functions
  • Crontab generation for scheduling workflows
  • Integration of secret passwords handling

New utilities

  • ‘json2evd’ – guess data types based on JSON file and produce EVD file
  • ‘qvd2evd’ – generate EVD file based on QVD

New commands

  • ‘cancel’
  • ‘crontab’
  • ‘info’
  • ‘manager’
  • ‘Sleep’ – to have sleep command integrated

New components

  • ‘Readmysql’
  • ‘Runmysql’
  • ‘Writemysql’

Version 2.5 ​

Released
2021/04

New features

  • Read/write EVD files
  • ‘Read’ and ‘Write’ components support also Samba and databases, so recognize URIs: ‘smb://’, ‘mysql://’, ‘oracle://’, ‘postgres://’, ‘teradata://’

New components

  • ‘Readevd’
  • ‘Writeevd’

Version 2.6 ​

Released
2021/10

New features

  • RSA encryption/decryption by mapping function: rsa_encrypt(), rsa_decrypt(), str_to_base64(), base64_to_str()
  • New utility pg2evd to create EVL data structure (EVD file) based on PostgreSQL table.

Version 2.7 ​

Released
2022/04

New features

  • Read/Write SQLite table and run SQL in SQLite DB
  • ‘Read’ and ‘Write’ components support also Google Drive, available by URI schema ‘gdrive://’ and SQLite DB, with URI schema ‘sqlite://’

New components

  • ‘Readsqlite’
  • ‘Runsqlite’
  • ‘Writesqlite’

New utilities

  • ‘evd2sql’ – produce CREATE TABLE statement based on EVD
  • ‘guess-timestamp-format’ – return C-style timestamp format string based on the input timestamps

Version 2.8 ​

Released
2023/04

New mapping functions

  • ‘fail()’, ‘is_equal()’, ‘log()’, ‘warn()’
  • Math functions: ‘abs()’, ‘ceil()’, ‘floor()’, ‘max()’, ‘min()’, ‘pow()’, ‘round()’, ‘sqrt()’, ‘trunc()’

New command

  • ‘Calendar’ – To specify a calendar based on which the job or workflow will be fired

New component

  • ‘Wc’ – Count records and bytes

New data types

  • ‘interval’
  • ‘time’