Binary parameter handling for GeoDjango

[eduzen]

Description

hey folks, this is my very first PR in the django-commons. I did this because I faced the issue myself in one project that I'm working on. I don't know how I can test this in that project. So I will wait for some of you to tell me how I can do that... but in the meantime this is the fix for the SQL Explain functionality for GeoDjango queries with binary parameters.

Fixes #423

SQL Explain feature throws a 500 error when used with GeoDjango queries containing binary parameters. This affects any query using PostGIS functions like ST_GeomFromEWKB(), ST_Distance_Sphere(), etc.
I'm trying to create a robust binary parameter handling system:

1. Enhanced Parameter Encoding (tracking.py)

• Detects binary data (bytes, bytearray) during parameter logging
• Encodes binary data as base64 with a special marker: {"djdt_binary": "base64_data"}
• Preserves exact binary integrity through JSON serialization

2. Parameter Reconstruction (forms.py)

• Added _reconstruct_params() function to decode stored parameters
• Recursively handles nested structures (lists, dicts) containing binary data
• Restores original binary data from base64 encoding

3. Updated SQL Operations

• Modified select(), explain(), and profile() methods to use parameter reconstruction
• Added helper method _get_query_params() to eliminate code duplication

Checklist:

• I have added the relevant tests for this change.
• I have added an item to the Pending section of docs/changes.rst.

[eduzen]

@tim-schilling I added some commits following what you were pointing out, let me know what do you think!

[tim-schilling]

We should move this logic into a shared utility that way we don't have the magic value "__djdt_binary__" floating around the code-base. Plus it would be clearer how the logic gets paired together.

I'm also curious if we should be using this with the Store logic. If we do, then we could potentially ignore this and handle it within DebugToolbarJSONEncoder and then have DebugToolbarJSONDecoder as above. What do you think?

[eduzen]

Thank you for the feedback @tim-schilling ! I think an encoders.py or moving even everything into utils.py is a more consistent way.
About your second question, what do you mean with using the store logic?

[tim-schilling]

The Store logic converts all the panels data into JSON. So this binary parameter would eventually be converted to JSON to be added to the Store. So if the Store handles binary data properly, it may remove the need for the SQL panel to deal with it itself. Does that make more sense?

[tim-schilling]

Thank you for continuing to work on this! I have a few directional changes. If you want more specific feedback, I can provide it. I didn't want to be overly prescriptive.

[eduzen]

Thank you for continuing to work on this! I have a few directional changes. If you want more specific feedback, I can provide it. I didn't want to be overly prescriptive.

No please! Tell me more, because I see that you are pointing to a deeper idea that I did not catch yet.

[tim-schilling]

@eduzen oh, I meant I was writing out the solution when I did the last round of review. It seemed like you may be interested in working through the issues to arrive at your own implementation and I didn't want to deprive you of the opportunity. So I should have said was, "if you'd like me to make these edits myself, please let me know."

[eduzen]

Super clear @tim-schilling let me take another look and I let you know!

[tim-schilling]

Hey @eduzen no rush on this! Just wanted to check-in to see how things are going. Take care!

[eduzen]

@tim-schilling thank you for asking. Busy days 😅 let me see if I can take a look the upcoming days!

[eduzen]

Hey @tim-schilling, I’m adding a page to the example project so I can see the issue and the fix with my own eyes (in addition to the tests). Do you think I should include that in this MR as well?

I’m still cooking the solution. While playing with the UI, I ran into a couple of issues that I’m working on now. Thank you!

[eduzen]

@tim-schilling I've addressed your feedback 🤞. I also updated is_select_query in utils.py because I noticed the view and explain buttons weren't working correctly for this query in the sample app.

[tim-schilling]

Thank you @eduzen for continuing on this. I finally got around to rereviewing. Most of my comments are minimal, though I pulled this down and the tests have a few failures yet.

FAIL: test_serialize_logs_on_failure (tests.test_store.SerializationTestCase.test_serialize_logs_on_failure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/schillingt/Projects/debug_toolbar/tests/test_store.py", line 18, in test_serialize_logs_on_failure
    self.assertEqual(
AssertionError: '{"hello": {"foo": {"__djdt_binary__": "YmFy"}}}' != '{"hello": {"foo": "bar"}}'
- {"hello": {"foo": {"__djdt_binary__": "YmFy"}}}
+ {"hello": {"foo": "bar"}}

[tim-schilling]

@eduzen I'm assuming you're using an LLM for the tests judging from the fact that this is in it's own file, the doc string comment, and the comments on very obvious lines of code serialized = serialize(params). Can you revisit these tests and remove the redundant pieces? (Comments, extraneous tests, extraneous assertions). We haven't fully adopted the LLM approach of "let it test everything it can think of" yet on this project. Please also move these tests into the test_sql.py file.

[eduzen]

Sure, Tim. Sorry for the delay. I was using indeed llms, for asking question for iterating and also for testing. But I also wrote code myself, test it.

[eduzen]

Let me know what do you think now 😄

[tim-schilling]

@eduzen hey I finally got back around to this. It appears the tests are still failing (you can review the logs on GitHub). I also set this up with a project using GeoDjango and the explain button is still causing a 500 error. You can pull that project down and run it with docker and uv here: https://github.com/tim-schilling/geodjango_test

Please let me know if I'm misunderstanding something or holding it wrong.