Flexter automates the conversion of complex XML/JSON to a relational or tabular format. It stores the output in text files (CSV, TSV), a relational database (Oracle, SQL Server etc.), or big data formats on Hadoop/Spark (Avro, Parquet, ORC).
The idea for Flexter was born from our frustration of working with XML and JSON in data warehouse and business intelligence projects.
Whenever we came across one of these data formats the project into the same set of problems.
The data analysts took forever to make sense of the XML/JSON structures.
The developers had to write endless lines of code or ETL data flows.
The complex requirements resulted in buggy and hard to maintain code.
Data quality issues were rampant.
Once all the code was written it turned out that it was not fast enough or could not handle the large volumes of data.
Traditionally ETL tools have been used for converting XML/JSON to a database. However, the core strength of ETL tools is to transform structured data and work with relational databases. They often struggle with semi-structured data in XML or JSON. While most ETL tools offer functionality to handle simple and flat XML files at low volumes, they have serious limitations:
They don’t automate the conversion process. ETL developers still need to create data flows (potentially hundreds for complex XMLs) and data pipelines. A significant development effort indeed.
ETL tools don’t scale beyond a single server for XML processing.
Most ETL tools can’t handle XML files in batches. They process one XML/JSON file at a time. This has a significant impact on performance as we need to iterate over these files again and again
Flexter addresses all of these problems
Flexter automates the whole XML/JSON conversion process. It automatically generates the relational target schema, the relationships between tables (foreign and primary keys), the mappings from source elements to target attributes, and last but not least it auto-populates the target schema. Flexter saves up to 80% of development costs.
Flexter uses in-memory technology and is extremely fast at parsing XML files. Companies can meet any SLAs with Flexter.
Flexter handles any volume of XML/JSON data. Clients can run Flexter on multiple machines in parallel. Using big data frameworks and distributed computing it scales linearly with the volumes of data.
Many XML projects fail or are delayed. Flexter eliminates project risk. We have seen many XML conversion projects fail. The failure rate grows exponentially with the complexity of the XML and the data volumes that need to be converted.
Flexter can be installed and configured in a day or two and clients can start processing XML/JSON straight away. Developers can focus on data engineering tasks that add value to the business rather than having to wrangle with XML/JSON.
- XML converter