Potential FFI Problems #1

Open
opened 2026-02-26 14:27:49 +01:00 by soegaard · 8 comments

I don't know whether this is helpful or not, but I asked Codex to look at potential problems.


Yes, there are binding-level issues that could plausibly cause memory corruption.

  1. libFLAC init signature likely wrong (high risk ABI mismatch)
    In libflac-ffi.rkt, FLAC__stream_decoder_init_file is bound with 5 args.
    Current FLAC C API uses a 6th client_data pointer. If your installed libFLAC expects that, this is an ABI mismatch.

  2. TagLib complex-property pointer depth looks wrong (high risk)
    Your own comment shows TagLib_Complex_Property_Attribute***, but binding uses a single pointer type in taglib-ffi.rkt and function defs at lines 236/239/244.
    If the real API is triple-pointer here, this can definitely corrupt memory.

  3. ao_shutdown return type is incorrect
    In libao-ffi.rkt, comment says void ao_shutdown(); but binding is -> _int at line 71.
    Usually this won’t always crash, but it is still an incorrect ABI declaration.

  4. Potential lifetime issue in picture extraction path
    In taglib-ffi.rkt, bytes are produced from a pointer, then taglib_complex_property_free is called at line 279.
    This is only safe if that cast makes a full copy; if not, it becomes use-after-free.


I don't know whether this is helpful or not, but I asked Codex to look at potential problems. --- Yes, there are binding-level issues that could plausibly cause memory corruption. 1. **`libFLAC` init signature likely wrong (high risk ABI mismatch)** In [`libflac-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libflac/libflac-ffi.rkt:461), `FLAC__stream_decoder_init_file` is bound with 5 args. Current FLAC C API uses a 6th `client_data` pointer. If your installed `libFLAC` expects that, this is an ABI mismatch. 2. **TagLib complex-property pointer depth looks wrong (high risk)** Your own comment shows `TagLib_Complex_Property_Attribute***`, but binding uses a single pointer type in [`taglib-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib-ffi.rkt:234) and function defs at lines 236/239/244. If the real API is triple-pointer here, this can definitely corrupt memory. 3. **`ao_shutdown` return type is incorrect** In [`libao-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libao/libao-ffi.rkt:70), comment says `void ao_shutdown();` but binding is `-> _int` at line 71. Usually this won’t always crash, but it is still an incorrect ABI declaration. 4. **Potential lifetime issue in picture extraction path** In [`taglib-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib-ffi.rkt:274), bytes are produced from a pointer, then `taglib_complex_property_free` is called at line 279. This is only safe if that `cast` makes a full copy; if not, it becomes use-after-free. ---
Owner
  1. libFLAC init signature likely wrong (high risk ABI mismatch)
    In libflac-ffi.rkt, FLAC__stream_decoder_init_file is bound with 5 args.
    Current FLAC C API uses a 6th client_data pointer. If your installed libFLAC expects that, this is an ABI mismatch.

This is indeed correct, I hopefully corrected this.

> 1. **`libFLAC` init signature likely wrong (high risk ABI mismatch)** > In [`libflac-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libflac/libflac-ffi.rkt:461), `FLAC__stream_decoder_init_file` is bound with 5 args. > Current FLAC C API uses a 6th `client_data` pointer. If your installed `libFLAC` expects that, this is an ABI mismatch. This is indeed correct, I hopefully corrected this.
Owner
  1. ao_shutdown return type is incorrect
    In libao-ffi.rkt, comment says void ao_shutdown(); but binding is -> _int at line 71.
    Usually this won’t always crash, but it is still an incorrect ABI declaration.

I corrected this.

> 3. **`ao_shutdown` return type is incorrect** > In [`libao-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libao/libao-ffi.rkt:70), comment says `void ao_shutdown();` but binding is `-> _int` at line 71. > Usually this won’t always crash, but it is still an incorrect ABI declaration. I corrected this.
Owner
  1. TagLib complex-property pointer depth looks wrong (high risk)
    Your own comment shows TagLib_Complex_Property_Attribute***, but binding uses a single pointer type in taglib-ffi.rkt and function defs at lines 236/239/244.
    If the real API is triple-pointer here, this can definitely corrupt memory.

This is not really a problem as this is treated purely as a pointer handle to call api functions of tagLib.

  1. Potential lifetime issue in picture extraction path
    In taglib-ffi.rkt, bytes are produced from a pointer, then taglib_complex_property_free is called at line 279.
    This is only safe if that cast makes a full copy; if not, it becomes use-after-free.

cast will make a copy that can be garbage collected.

> 2. **TagLib complex-property pointer depth looks wrong (high risk)** > Your own comment shows `TagLib_Complex_Property_Attribute***`, but binding uses a single pointer type in [`taglib-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib-ffi.rkt:234) and function defs at lines 236/239/244. > If the real API is triple-pointer here, this can definitely corrupt memory. This is not really a problem as this is treated purely as a pointer handle to call api functions of tagLib. > 4. **Potential lifetime issue in picture extraction path** > In [`taglib-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib-ffi.rkt:274), bytes are produced from a pointer, then `taglib_complex_property_free` is called at line 279. > This is only safe if that `cast` makes a full copy; if not, it becomes use-after-free. `cast` will make a copy that can be garbage collected.
Author

Although 1. and 3. were bugs, I don't see them as relevant for the problem you were seeing with memory corruption.
I noticed that you said the error only appears on Linux, not on Windows.
That might be hint for someone know more about the FFI.

Although 1. and 3. were bugs, I don't see them as relevant for the problem you were seeing with memory corruption. I noticed that you said the error only appears on Linux, not on Windows. That might be hint for someone know more about the FFI.
Owner

I've run the play-test.rkt program, in racket, and it does not generate this problem.
I'll test my more complex rktplayer program (which also uses a gui).

I've run the `play-test.rkt` program, in racket, and it does not generate this problem. I'll test my more complex rktplayer program (which also uses a gui).
Author

FWIW - here is a new list of potential problems.
Here 2. and 4. are the same as before.


Yes. I still see several plausible FFI-boundary problems:

  1. Likely use-after-free in FLAC callbacks (high risk)
    In libflac-ffi.rkt, write-callback stores raw frame/buffer pointers into write-data; meta-callback stores meta pointers at line 501.
    Those pointers are consumed later (process-write-data/process-meta-data), outside the callback frame. If libFLAC only guarantees validity during callback, this is classic memory corruption territory.

  2. TagLib complex-property pointer depth still suspicious (high risk)
    taglib-ffi.rkt binds taglib_complex_property_get as a single pointer, but the nearby C comment says TagLib_Complex_Property_Attribute***.
    If the real ABI is triple-indirection, this remains an incorrect binding and can corrupt memory.

  3. Possible null-pointer call into TagLib (medium risk)
    In taglib.rkt, result of taglib_file_new is used immediately in taglib_file_is_valid (line 70) without null check.
    If creation fails and returns null, behavior depends on TagLib internals; this can crash.

  4. Ownership/lifetime mismatch in TagLib picture extraction path (medium risk)
    taglib-ffi.rkt casts picture data to bytes, then frees complex properties at line 279.
    This is safe only if cast performs an eager copy. If not, it is use-after-free.

FWIW - here is a new list of potential problems. Here 2. and 4. are the same as before. --- Yes. I still see several plausible FFI-boundary problems: 1. **Likely use-after-free in FLAC callbacks (high risk)** In [`libflac-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libflac/libflac-ffi.rkt:497), `write-callback` stores raw `frame`/`buffer` pointers into `write-data`; `meta-callback` stores `meta` pointers at line 501. Those pointers are consumed later (`process-write-data`/`process-meta-data`), outside the callback frame. If libFLAC only guarantees validity during callback, this is classic memory corruption territory. 2. **TagLib complex-property pointer depth still suspicious (high risk)** [`taglib-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib-ffi.rkt:236) binds `taglib_complex_property_get` as a single pointer, but the nearby C comment says `TagLib_Complex_Property_Attribute***`. If the real ABI is triple-indirection, this remains an incorrect binding and can corrupt memory. 3. **Possible null-pointer call into TagLib (medium risk)** In [`taglib.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib.rkt:69), result of `taglib_file_new` is used immediately in `taglib_file_is_valid` (line 70) without null check. If creation fails and returns null, behavior depends on TagLib internals; this can crash. 4. **Ownership/lifetime mismatch in TagLib picture extraction path (medium risk)** [`taglib-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib-ffi.rkt:274) casts picture data to bytes, then frees complex properties at line 279. This is safe only if `cast` performs an eager copy. If not, it is use-after-free.
Owner

It was a ao problem. Too many closing of ao-device handles. A finalizer that worked on a wrong administration on progam exit.

It was a ao problem. Too many closing of ao-device handles. A finalizer that worked on a wrong administration on progam exit.
Owner
  1. Likely use-after-free in FLAC callbacks (high risk)
    In libflac-ffi.rkt, write-callback stores raw frame/buffer pointers into write-data; meta-callback stores meta pointers at line 501.
    Those pointers are consumed later (process-write-data/process-meta-data), outside the callback frame. If libFLAC only guarantees validity during callback, this is classic memory corruption territory.

This is probably true, given the documentation of libFLAC, although I've not seen problems here, maybe because I'm using very little functionality of the library. For complex metadata, I'm using libtag.

  1. Possible null-pointer call into TagLib (medium risk)
    In taglib.rkt, result of taglib_file_new is used immediately in taglib_file_is_valid (line 70) without null check.
    If creation fails and returns null, behavior depends on TagLib internals; this can crash.

That could be true, but is unlikely to happen. I'll fix it though.

> 1. **Likely use-after-free in FLAC callbacks (high risk)** > In [`libflac-ffi.rkt`](/Users/soegaard/tmp/racket-sound/libflac/libflac-ffi.rkt:497), `write-callback` stores raw `frame`/`buffer` pointers into `write-data`; `meta-callback` stores `meta` pointers at line 501. > Those pointers are consumed later (`process-write-data`/`process-meta-data`), outside the callback frame. If libFLAC only guarantees validity during callback, this is classic memory corruption territory. This is probably true, given the documentation of libFLAC, although I've not seen problems here, maybe because I'm using very little functionality of the library. For complex metadata, I'm using libtag. > 3. **Possible null-pointer call into TagLib (medium risk)** > In [`taglib.rkt`](/Users/soegaard/tmp/racket-sound/libtag/taglib.rkt:69), result of `taglib_file_new` is used immediately in `taglib_file_is_valid` (line 70) without null check. > If creation fails and returns null, behavior depends on TagLib internals; this can crash. That could be true, but is unlikely to happen. I'll fix it though.
Sign in to join this conversation.