On my way into work I decided I would create a repository for various code bits that I had worked on over the years. Nothing proprietary of course, just general functions that have proved useful. Among them was a small helper library that I wrote in MASM32 to help workaround the lack of unsigned integers and longs in VB6.

I had a roughly-done sample application that I had used to remind me of the calling conventions. It thought, ok, I will just polish this up a little bit before publishing it. Unfortunately it wasn’t as simple as that. The final function I had added to that DLL was suddenly causing my program to crash. Yet I know it used to work. As I fiddled with it, even cosmetically, some tests would fail consistently and other succeed. For instance at one point when it was working, I added messagebox before invoking the library, and this change by itself would cause a crash. I recompiled the library and could get it work if the two byte parameters were instead passed in as a single integer. But it would fail if I passed them as two integers. Was this an alignment issue? Isn’t the flat model supposed to take care of that? I was stumped.

Eventually, I came across an old paper titled Writing DLL In Assembler for External Calling in Maple by Dr. Milailovs where he gave me a hint: “In Windows and Linux, registers eax, ecx, and edx can be arbitrarily modified by programs, but other registers including ebx should be preserved…” EBX? Uhh… Stackoverflow agreed.

Here was the original version of the troublesome function:

ParseDword proc inDword:DWORD, inStart:BYTE, inLength:BYTE
    mov eax,inDword    ;initialize register
    or  ebx,0FFFFFFFFh ;initialize mask
    mov cl,inLength    ;get mask size
    shl ebx,cl         ;create invert mask in lower portion of ebx
    not ebx            ;mask now in lower portion of ebx
    mov cl,inStart     ;get desired position of mask
    shl ebx,cl         ;mask of correct size, in correct location
    and eax,ebx        ;apply mask
    shr eax,cl         ;shift result to bit pos 0 for returning
    ret
ParseDword Endp

I definitely was not treating EBX with such reverence here. Could that really be the issue? I’ll humor the idea and use EDX instead:

ParseDword proc inDword:DWORD, inStart:BYTE, inLength:BYTE
    mov eax,inDword    ;initialize register
    or  edx,0FFFFFFFFh ;initialize mask
    mov cl,inLength    ;get mask size
    shl edx,cl         ;create invert mask in lower portion of edx
    not edx            ;mask now in lower portion of edx
    mov cl,inStart     ;get desired position of mask
    shl edx,cl         ;mask of correct size, in correct location
    and eax,edx        ;apply mask
    shr eax,cl         ;shift result to bit pos 0 for returning
    ret
ParseDword Endp

…and it works. Argh. Wow, well I guess it sometimes takes a few hours of debugging to remember something as trivial as that.

Now back to figuring out how I want to layout the repository. (And maybe clean up those code blocks with better formatting.)