How to fix registering audio callback as NaN?

[Lebei2046]

I wrote voice-to-text(real-time and wav-file) for chatroom-perry.

• add audio callback mechanism for voice-to-text processing in a patched perry
• add an extention to integrate vosk in chatroom-perry

Converting a wav-file to text works fine. However, real-time voice-to-text does not work.

Here is the Speech Recognition Architecture

Bug: Register audio callback as NaN

[SpeechRecognizer] Session started, testing audio system
[SpeechRecognizer] Session started, registering audio callback
[LIB] perry_system_audio_register_callback called! callback=NaN
[audio] register_audio_callback called, callback: NaN
[audio] AUDIO_CALLBACK mutex is being updated
[audio] AUDIO_CALLBACK has been set to: Some(NaN)
[SpeechRecognizer] Starting audio
[SpeechRecognizer] Returning true

Add audio callback mechanism for voice-to-text processing

• Add register_audio_callback FFI function for real-time audio processing
• Implement js_closure_call2 for passing audio samples to JS callbacks
• Update compile/link/resolve commands to support voice-to-text workflow
• Add TypeScript bindings for audio callback registration

pub unsafe extern "C" fn register_audio_callback(callback: f64) {
    eprintln!("[audio] register_audio_callback called, callback: {:?}", callback);
    eprintln!("[audio] AUDIO_CALLBACK mutex is being updated");
    *AUDIO_CALLBACK.lock().unwrap() = Some(callback);
    eprintln!("[audio] AUDIO_CALLBACK has been set to: {:?}", *AUDIO_CALLBACK.lock().unwrap());
}

Check audio.rs for the implementation of register_audio_callback.

Register audio callback in TypeScript

console.log("[SpeechRecognizer] Session started, registering audio callback");
audioRegisterCallback((samplesPtr: any, numSamples: number) => {
    this.sampleCount += numSamples;
    if (this.sampleCount % (48000 * 2) === 0) {
        console.log("[SpeechRecognizer] Audio callback called, total samples:", this.sampleCount);
    }
    voskProcessSamples(samplesPtr, numSamples);
});
console.log("[SpeechRecognizer] Starting audio");
audioStart();
console.log("[SpeechRecognizer] Returning true");

Check SpeechRecognizer.ts for the call of audioRegisterCallback.

[proggeramlug]

The callback=NaN log is a red herring — that's expected, not the bug.

In Perry, function values are NaN-boxed: the f64 bit pattern is 0x7FFD_0000_0000_0000 | (ptr & 0xFFFF_FFFF_FFFF). That's a quiet NaN by IEEE-754 construction, so eprintln!("{:?}", cb_f64) prints NaN for any function/object/string value. The bits underneath still hold the pointer. (Switch your logs to eprintln!("{:#x}", cb.to_bits()) and you'll see 7ffd… for closures, 7fff… for strings, etc.)

The actual bug is in your invocation in audio.rs:

let cb_bits = cb.to_bits();
let closure_ptr = cb_bits as *const u8;          // ← keeps the 0x7FFD tag in the high bits
js_closure_call2(closure_ptr, samples_ptr_val, num_samples_val);

closure_ptr is 0x7FFD_xxxx_xxxx_xxxx — way outside any real heap address — so the call jumps to garbage. You need to strip the tag with js_nanbox_get_pointer first, the same way the GTK4 button widget does in crates/perry-ui-gtk4/src/widgets/button.rs:

extern "C" {
    fn js_nanbox_get_pointer(value: f64) -> i64;
    fn js_nanbox_pointer(ptr: i64) -> f64;
    fn js_closure_call2(closure: *const u8, a0: f64, a1: f64) -> f64;
}

let closure_ptr = unsafe { js_nanbox_get_pointer(cb) } as *const u8;

// Both args to js_closure_call2 are NaN-boxed JS values, not raw C values:
let samples_arg = unsafe { js_nanbox_pointer(samples_ptr as i64) }; // box pointer w/ POINTER_TAG
let count_arg   = num_samples as f64;                                // plain numbers ride f64 directly

unsafe { js_closure_call2(closure_ptr, samples_arg, count_arg); }

Three rules of thumb for FFI like this:

1. Print bits, not values, when debugging NaN-boxed handles: {:#x} on to_bits().
2. Strip the tag with js_nanbox_get_pointer before treating a closure/object/string f64 as a pointer.
3. Re-box args going into js_closure_call*: pointers via js_nanbox_pointer, strings via js_nanbox_string, numbers as plain f64. Don't pass raw C values — the JS callback expects NaN-boxed inputs.

One more thing to check on the TS side: in your callback signature (samplesPtr: any, numSamples: number), samplesPtr typed as any will be treated as a JS value (pointer-tagged). If you later hand that off to voskProcessSamples, make sure that extern's Rust side also calls js_nanbox_get_pointer to recover the raw *const f32 (or whatever sample type).