Embedded Files
Research and Presentations
Tue, July 8, 2025, Alexandre Bijaye
This project was presented at Safe Software’s Peak of Data and AI and was rebroadcasted as a webinar due to its success (video link below). We built FME AI CHECKER for Veremes, a partner of Safe Software and French distributor of the FME Platform. FME AI CHECKER is an intelligent assistant that reviews workflows, checks for errors, and even suggests improvements.
FME AI CHECKER: A New Way to Review Your Workflows?
Imagine getting instant, intelligent feedback on your FME workflows. What if an AI could review your pipeline, check for errors, and even suggest improvements? That's the power behind FME AI CHECKER, a creative AI focused project we built on the FME platform.
Let's dive into the project's details, but more importantly, we’ll explore the critical process of communicating with AI within an ETL pipeline. While we used FME, the principles of robustly calling an AI and managing its response are universal and can be applied to any other tools since it’s simple string manipulation. We'll first take a look at the FME AI CHECKER itself, and then we'll break down the technical challenges and solutions for building an AI-powered pipeline.
The Project: Instant Workflow Validation
FME AI CHECKER was built on the FME platform as its name indicates. This project aims to provide a tool for instant review of any FME workflow. Imagine providing your freshly built project to an intelligent review tool to get immediate verification of Transformers' configuration, Logic and expressions, global workflow design.
The user can choose one of two modes:
Annotation Mode: The tool comments on every step of the pipeline.
Error Checking Mode: Review of all transformer parameters and highlights potential errors or misconfigurations with attached annotations exactly where issues were identified.
Now, let's take a look at the engineering behind the project.
Parsing the FME Workflow
How are FME projects are stored? Every FME workflow (.fmw) is saved as an XML-like file. XML is a hierarchical, tree-like organization of tagged elements allowing for structured data representation.
In a FME Workflow (.fmw file), many tags are found such as <DATASETS>, <COMMENTS>, <TRANSFORMERS> etc (see image 1).
Image 1 - FMW File structure
The <TRANSFORMERS> section here stores every single parameter of our transformer.
By reviewing these parameters, we can identify missing critical inputs or flawed logic.
It is in this section of our XML that we need to do our magic in order to be able to analyze our pipeline. Reading the FME project as a simple text file allows us to store the entire workflow as a single, long string. This string is easily manipulated using FME’s string tools or python.
Handling API Limitations
We used OpenAI’s ChatGPT for this project, which in our setting, had a character limit of 25K characters per call. This limitation forced us to process our workflow in batches. Using a custom Python script (in PythonCaller), we isolated full transformers (in <TRANSFORMERS> section) from the main XML, ensuring we never cut a transformer in half. The script calculated the length of each batch to keep it under the 25K character limit, ensuring correct AI communication (see image 2).
Image 2 - Example of a transformer as present in our XML file
The Art of Prompt Engineering
Since this AI communication would be embedded in an ETL pipeline, we need the AI's response structure to be consistent every time. A different structure would make the rest of our pipeline fragile and unreliable. This is why defining a clear output structure is key before we even start with the main prompt.
The choice of the following structure was made:
- WHAT: which issue is encountered
- FIX: what should we do to fix it
- IDENTIFIER: each transformer has an ID - this is needed to attach the annotation to its respective transformer
- POSITION: where is the transformer located within the canvas
- |EOC end of comment tag
Here's the rule we used to train our AI (see image 3):
Image 3 - Rule part of our prompt
We taught our little GPT to always answer in the same manner whenever it received a fragment of XML containing transformers for review. Any answer that didn't follow this rule was discarded, and the call was executed again.
With the output structure defined, we wrote the main task (see image 4):
Image 4 - Task part of our prompt
This is the result of a call for a faulty HTTPCaller (see image 5):
(HTTPCaller is a transformer used to communicate with external APIs)
Image 5 - HTTPCaller comments
You may notice the "|EOC" delimiter at the end. Interestingly, when we tried using "END" or "BREAK" as a delimiter, AI would sometimes interpret it as a stopping order and halt its analysis. Using “|EOC”, a set of characters that bring no special meaning for "End Of Comment”, completely removed this behavior since it doesn't bring any direct meaning, making the whole process more robust.
Refining the AI's Output
One scenario we had to consider was when the AI would flag minor, optional FME parameters as critical issues.
To solve this, all answers from our first AI call were sent back to the AI. This second call's purpose was to analyze the nature of the comments and filter out useless information, ensuring only truly important insights were kept.
Other than communicating with the AI, the rest of the FME project remains relatively simple, consisting only of transformers that manipulate strings and insert the AI's comments into the <COMMENTS> tag of our original FME workflow file. See below (Image 6) for global workflow design:
Image 6 - Global FME Workflow design
Let's take a look at some results!
Images 7, 8, 9, 10 - Results obtained
Is This a Replacement for Human Review?
While we observed that some issues were easier for the AI to identify than others, this project still provided instant feedback on errors and suggested potential improvements 65% of the time. It also successfully annotated entire workflows 80% of the time.
However, over 100 uses, we did encounter some inconsistencies and "laziness" in less than 15% of the answers. The AI would fail to adhere to the required answering structure or halt its analysis midway. We also observed some "hallucinations" in less than 5% of the results, where the service would invent parameters to be corrected that simply do not exist.
Of course, potential improvements would be to waive the token limit, which currently forces us to dissect our FME project into small chunks. There may be issues in Reader/Writers, Scripted parameters, etc… and these won’t be reviewed by this project.
Training our own custom model would also bring the domain-specific expertise a common ChatGPT lacks, allowing it to better understand FME transformers and workflows.
You can download the entire slide deck and watch FME AI CHECKER webinar here (timestamp 18:08) : Peak of Data & AI Encore: AI for Metadata and Smarter Workflows
NARBOdata extends its sympathies to Veremes, a trusted partner who made this research project possible.
We also thank Safe Software for their interest in FME AI CHECKER and for featuring the project in their Peak of Data and AI Encore rebroadcast series.
Google Sites
Report abuse