or: How I Learned to Stop Worrying and Love the BOM
I actually don’t love the BOM but a corny movie title reference (Dr. Strangelove) was irresistible.
We noticed odd behavior when using curl to push CSV data to a custom Maximo API. For unknown reasons, the operation would intermittently fail and suggest a key attribute has not been supplied in the data. This was baffling, because the key attribute was the first header of the file.
We found that we could paste the CSV data into Notepad and resave the file to resolve the issue. We figured we were getting unprintable characters in the data somehow and this process magically filtered them. Pasting the ‘bad’ data into Postman worked fine as well.
Spoiler/Summary: This sequence of bytes is called a Byte Order Marker and it’s a way to explicitly declare the rest of the bytes represent text that is UTF-8 encoded. Specifically, this is called UTF-8-BOM or UTF-8-SIG. This is a problem is because we’re using curl to POST a CSV file with the –data-binary argument to avoid other problems. Normally something in the ‘signal chain’ would understand those three bytes and transparently handle them. The initial root cause is how the files are saved from Excel; the first option creates a UTF-8-BOM file (instead of regular UTF-8). Fixing this problem involves re-encoding the files OR filtering the BOM in the API script.
Assuming this was something easy, I confidently opened the file, hit the ‘show all characters’ option, and saw… nothing. No unprintable characters, unescaped delimiters, fancy quotations, CR without LFs, etc. Clearly there was nothing to be filtered, so why would copy-pasting fix the data?
I opened the file in a hex editor, thinking nothing can hide from such scrutiny. Sure enough, the first three bytes were 0xEF, 0xBB, 0xBF, making the first header “INSPFORMNUM” which is clearly not a valid Attribute.
Examining other problematic files showed the same three bytes. Clearly, this was not random data corruption; this was happening for a reason. Just not a reason I could think of.
Not understanding the problem at first, I wrote some PowerShell to recursively search a folder and report on which CSV files were ‘tainted’. To fix them, I simply skipped the first three bytes and wrote the rest out back to disk.
“That was easy”, I thought, “almost too easy. I’ll just adapt this to Python and paste it into the API script. Mischief managed.” I was right; it was too easy.
Maximo Jython is based on Python 2.7 and has two mutually exclusive types: str and unicode. In 3.x there’s no difference, but in 2.7, str is a byte array and unicode is a slightly more complex abstraction that handles encoding for you (so you don’t have to get clever with byte manipulation). As a result, a unicode reference cannot be cast into a str reference or a byte array.
This meant I just couldn’t manipulate the header row at the byte level. Python is too nice to let me suffer the indignity of such low-level manipulation. Attempts to leverage encoding/decoding methods failed because the underlying class was treating those three bytes as the literal characters “”. Which, fittingly, looks like a new grawlix.
I eventually realized that slicing the unicode object produced single-character objects, and I could compare the value by casting the hex value to a unicode object. Now that I could detect the BOM, I could conditionally strip it out.
It’s worth noting most people specifically advise against using UTF-8-BOM encoding for this exact reason; generally its easier just to forgo the BOM and stick with UTF-8.
So, where did the BOMs come from in the first place? Initial root cause is actually the default behavior of Excel. When you open a Save File dialog and browse the file types, the first relevant option is “CSV UTF-8”. This does create a UTF-8 file but this also adds the BOM. The next option, “CSV”, also produces UTF-8 encoded files but does not include the BOM.
For those who want to guard against this problem in Maximo Automation Script, here’s a snippet of code that will filter out BOMs:
This presumes a unicode ‘line’ from extracted from the CSV file contents. Since the BOM can be expressed differently, we look for both forms. And yes, processing CSV data as a string isn’t the best approach but there are reasons.
Finally, it’s helpful to know that Notepad++ has an ‘Encoding’ tab, which allows you to change encodings. This presumes you’ve determined the underlying problem is with encoding, which hopefully this article has assisted with. The more you know 😊
About
A3J Group is a leading solutions and consulting firm based in Tampa, Florida, specializing in Enterprise Asset Management (EAM) with a focus on fully developed Computerized Maintenance Management System (CMMS) solutions like IBM Maximo.
In addition to our suite of proprietary products—including MxMobile, our powerful mobile solution for Maximo—we provide traditional consulting services that help organizations plan, implement, optimize, and support their EAM strategies. Whether clients need hands-on configuration, strategic guidance, or ongoing system support, our team delivers the expertise to drive long-term success.
With over 50 years of combined experience, our specialized team is dedicated to our core mission: to serve as excellent stewards of our clients’ trust and investments, while driving measurable results through tailored, high-impact solutions and intelligent consulting.
Visit us at www.a3jgroup.com. Or drop us a line at info@a3jgroup.com.
