What is the way to tell stable diffusion to output deterministic results?

[StudenteChamp2]

I am using build 5a34bc7. Here the command that i use:
sd-cli.exe --backend vulkan0 --diffusion-model flux-2-klein-4b-Q8_0.gguf --llm Qwen3-4B-UD-Q4_K_XL.gguf --vae flux2.full_encoder_small_decoder.safetensors -p "turn the image into a high quality photograph" -r input_image.png -o output_photo.png --cfg-scale 2 --steps 4 --offload-to-cpu --fa --seed 42

Let's take this example

I get this result:

With a change of the camera position/orientation:

I get this:

As you can see the scene changed. For instance the color of the entities circled in green changed! And objects in red in first render simply vanished.

I MADE another test. This time telling stable diffusion what to do.
Here the test scene:

The prompt:
Turn the image into a high quality photograph. Apply a white porcelain material on the teapot! Use white material for the water pitcher. Use light wood texture for the table. Use dark wood texture for the floor. Use a red metallic material for the light shades. Use a white porcelain material on the cups. Use a beach painting for the wall canvas. Use white material on the seats. Use light wood texture on the seat supports. Do not turn on the lights.

The negative prompt:
Lamp lights.

First camera output:

Everything looks as expected!

Second camera output:

I get almost the same scene. But the color of the areas circled in green changed :(

Third camera output:

Everything looks fine expect that the right shade color specified in the prompt is not respected.

My guess is that to get deterministic results i need to tell stable diffusion what it needs to do for the part of the scenes!
Tell me if i am wrong :)

For instance by adding " The light shade interiors must remain red" to the prompt the third test passes:

Here the second test input images:|

[wbruna]

First, what you want is 'consistent', not 'deterministic' outputs. (I'm not being pedantic: searching the web for 'deterministic' will direct you to different things, like seeds and samplers).

As you noticed, if you leave the model free to guess, it will; and if it has no context to work with, each guess will be different. One possibility is specifying elements using additional reference images, including several different views, so the model doesn't have to guess. A quick example (Klein 4b isn't great at this, and I don't have time to look up the exact syntax it prefers, but hopefully you get the idea):

A concept character sheet showing Hanna, a redhead female, wearing glasses, a wristwatch, a grey blouse and dark blue pants.	Change the first image, placing Hanna, the female from the second image, sitting at the table.

Notice that, even if it didn't generate a terribly consistent reference in the first place, details like the outfit, glasses and wristwatch were kept. So, in your case, you could render elements like the light shade first, then compose the final scene passing it as an additional reference.

Passing a first render as an additional reference for the second one may also work, but it may demand a tighter prompt to keep the model from getting confused between the references.

[StudenteChamp2]

First thank for taking the time to answer me. It is highly appreciated!

[StudenteChamp2]

I am building a rendering software for architects. i cannot ask them to renderer several view before rendering a final image.
Tell me if i am wrong but my only option is to tell them to use tight prompts

[wbruna]

But what forces you to render everything in a single inference? You could first render a scene with one of each different objects, then use that scene as a reference to the final image: since the model will now have an example of each object, it'll likely keep them consistent across renders.

Another possibility is to let the user choose how to render each element separately, then build the final scene according to those choices.

[Green-Sky]

To get consistent results, you should try adding the previews angle also as a reference and engineer the prompt around that.

Related but for video that popped up a few days ago: https://huggingface.co/fal/LTX-2.3-3DREAL-LoRA

[wbruna]

That's what I did in that example: I've passed your scene as the first reference, and the character as the second one.

According to the official docs, Klein supports up to 4 references, designated as "Image 1", "Image 2", etc: https://docs.bfl.ml/guides/prompting_editing_overview

[StudenteChamp2]

@wbruna @Green-Sky
There nothing additional to do? Like in prompt say something like "Use images 2 and 3 to render image one"
I am going to use this one as main render:

And these two as additional references:

[Green-Sky]

I dont think it will do much if you add the unfinished render as extra reference. You need to include the finished image. You called that "camera one".

[StudenteChamp2]

TY! And what should be the prompt?

[wbruna]

"Use images 2 and 3 to render image one"

As I've explained on the other issue, you need to be specific: if you want something from the reference images, tell that to the model, don't leave it guessing.

• "Change Image 1 into a high quality photograph. Preserve composition, colors and shapes, changing only style and textures."
• "Render the scene from Image 1 as a high quality photograph. Follow the style and object details from Image 2."
• "Create a new view of the room shown on Image 1, Image 2 and Image 3. Follow the style and textures from Image 2. Use the couch from Image 1, and the chicken from Image 3."

And so on. The official BFL docs I've linked to have several other examples.

[wbruna]

Following the discussion from #1730, since it is not really about any issues with the sd.cpp engine itself.

To avoid having the model just copying the style reference image, I've made an attempt by repeating the 3d scene (passing the same image as 3 references), and flipping the style render, instructing the model to only get colors and textures from it. The best result so far was this mess:

But this may give us clues about why is it failing. To me, this looks like the model is identifying the elements, but not their identity: the objects on the second scene may have the same "style", but they are not the same objects. The model may simply be too dumb to understand this change.

Image-wise, there are also other factors that make its job harder: the table color, for instance, is completely different. To make this work, it would need to understand the green crude render and the light wood texture are meant for the same object - again, having a strong notion of object identity.

Unfortunately, I do not have ready answers for this. But if I was writing this software, I would try to lower the demand from the model, breaking the problem into smaller, simpler parts, and/or giving it additional clues. For instance:

• labeling the objects: e.g. placing a text "chair1" pointing to the chairs in both the 3d render and the reference images, so the model has a chance to understand they are copies of the same object (Klein is good at removing unwanted text from the final image afterwards)
• following general colors when applying textures: if you want a light wood table, present to the model (not necessarily to the user!) a render in a similar color than the final image
• in both cases, an image reference containing a table matching object names, plain colors and textures could help anchor the change
• another approach would be presenting each object to the model to get the textures first, applying the textures using the 3d render engine, then instruct the model to just give a polish to the final scene

[StudenteChamp2]

Thanks for taking the time to help me

[StudenteChamp2]

@wbruna What do you man by flipping the style render, instructing the model to only get colors and textures from it., From what?
And you said "I've made an attempt by repeating the 3d scene " Which scene the raw render or the reference?

OR

If this works i am saved!

[wbruna]

What do you man by flipping the style render,

Oh, flipping the previously rendered image upside down. Notice the generated image in my test has some table and chair parts on the ceiling.

Which scene the raw render or the reference

The raw render; the idea was to make the model pay more attention to it. It was something like this: -r raw_render.png -r raw_render.png -r raw_render.png -r previous_output_upside_down.png -p 'Turn Image 1 into a realistic scene. Use objects from Image 2. Follow composition from Image 3. Use colors and textures from Image 4.'. Didn't really work that well, as you saw.

[StudenteChamp2]

Made another test with the Sponza atrium
Main image:

Reference:

Doesnt work :/ Reference is returned instead of the primary view ouput

[StudenteChamp2]

And something strange. If i feed stable diffusion 2 raw renders. The second enhanced view is unfortunately returned.

Returns

[StudenteChamp2]

@wbruna So i am stuck there :( :( :(. I will try your suggestion:

1. pointing to the chairs in both the 3d render and the reference images, so the model has a chance to understand they are copies of the same object -> I cannot ask my users to do so

2. following general colors when applying textures: if you want a light wood table, present to the model (not necessarily to the user!) a render in a similar color than the final image --> Will do!

3. in both cases, an image reference containing a table matching object names, plain colors and textures could help anchor the change
   --> Same as (1)

4. another approach would be presenting each object to the model to get the textures first, applying the textures using the 3d render engine, then instruct the model to just give a polish to the final scene --> Will give i a try