Add the option to convert a file to another filetype.

[Yomafil]

Add the option to convert a file to another filetype in Right Click File > Manage.
Example: in Right Click File > Manage > Convert Filetype select the new filetype Webp > PNG.
Maybe overwrite the old file, or create a new one while preserving the old.

[roachcord3]

Maybe overwrite the old file

Incompatible with how Hydrus works (storing images based on shasum.)

create a new one while preserving the old.

You can do this by exporting the file(s) you want to convert, along with sidecar info like their tags. Then, run the images through the conversion program of your choosing, with the settings you like. (There is no one-size-fits-all, which is why I am telling you how to do it this way; implicitly, this is not a feature request that's gonna happen.) Finally, import the converted files, using the sidecars to ensure that they get the tags and such that you wanted associated with them.

If you want to delete the old files, you can use duplicate filtering for that purpose. With a lossless conversion like WebP -> PNG, you can use the option to filter only pixel-perfect duplicates, and with auto-filtering, you can even set up rules that will make the job easy (preferring PNG over WebP, for example.)

[216598762]

I was going to make a post for this, to add, can hydrus not store the origin hash alongside the converted file? For reference, being able to automatically convert all images into webp (or all videos into h264/5) format would greatly reduce filesize. Could have an automatic processing job, and options for what level of compression you want.

Considering ffmpeg is already bundled, I don't see why this is a bad idea.

[roachcord3]

I'm not the dev, but I want to answer your questions with likely approximates of the real reasons, and also give you actual concrete tips on how to go about doing conversion jobs of your own. I don't think dismissing these questions with a short quip or just ignoring them is productive, because you'd just be left confused as to why something you think is a good idea isn't getting implemented... so I will first start with the workaround tips, then do a break, after which comes some commentary/explanation.

being able to automatically convert all images into webp (or all videos into h264/5) format would greatly reduce filesize. Could have an automatic processing job

IIRC, WebP can be lossless, but h264/5 are lossy, so the demand for this would be low. People aren't clamoring for their PNGs to be converted to JPEGs. I can understand a lossless WebP conversion being desirable, but lossy compression is kind of tough for me to believe is desirable... but you're not dumb or wrong for wanting it, it's your media after all. Though I will say that even the lossless WebP conversion would be contentious, given that the format is still not well-supported on all OSes and all devices. Even the stuff you think is low-hanging fruit, is not.

So here are some tips to help you do what you want with your media regardless of how niche your use case is: if you set up an import and export folder, you can supplement that with a script that converts your files. It'd actually be pretty similar to what hyextract does.

And if you want to do this with stuff you download off the internet automatically before it even gets imported to Hydrus, I believe sufficient tweaking of gallery-dl options might make this possible if you use hydownloader. After all, it has a post-processing routine to turn ugoiras into animated images, so I bet there's a way to automate the conversion before even importing.

Even though this has already crossed well into the realm of power user level advice, where it makes me wonder if there is even a reason to share it... I have another tip. If you script up some automation of your own, you can also do the deduplication manually with the API. You'd have your own records of pre- and post-conversion file hashes, so you could easily set the pair relationships and delete the pre-conversion file. The way you'd do this would be idiosyncratic to your preferences, which is why it can't really be baked into Hydrus (an opinionated version of it could be, but that would be not useful for many users and would generate complaints and fix requests for something that is not bugged, but working as intended.)

---

Now, as to why something that seems obvious hasn't already been implemented...

Can hydrus not store the origin hash alongside the converted file?

Yeah it probably could, but have you thought of the upshot of this and compared it to the suggested export -> convert -> import -> deduplicate flow? What is the upshot of having the database have the concept of a file that used to be another file? Why go out of the way to track that? The answer–the upshot–is that you intend to destroy the original data; you would not have need to keep track of something like that if you were planning to keep both copies or to engage with the duplicates filtering system.

Avoiding data destruction is something that Hydrus consistently goes out of its way to do. It's the reason why the feature requests to not copy files into a database directory and to just track them in their existing locations get shot down. It's the reason why there are ALL CAPS RED TEXT WARNINGS next to any options that let you delete stuff after importing or exporting it. It's the reason why there is a trash system instead of just deleting the files. It's the reason why pixel-for-pixel duplicates do not already get automatically filtered, but rather you have to set up those rules/do that filtering yourself.

and options for what level of compression you want.

Considering ffmpeg is already bundled, I don't see why this is a bad idea.

Since the number of potential conversions is so high even if you don't do naive combinatorics and just consider common use cases, it would be a lot to implement. You'd be better off just exposing ffmpeg options directly at that point, which merely makes Hydrus a glorified wrapper around templating the input argument for ffmpeg. And at that point you're back to saying "why not just leave it as-is?" because the value added by Hydrus in that case is minimal.

In my mind, that's the end of the matter, but I imagine that being told "oh it's too complicated, you might as well not try, just make people use ffmpeg directly" when you don't think it's too complicated isn't highly convincing, so... since we're dealing with people's media here, we have to be careful and considerate. There's much to consider, such as metadata preservation and lossy compression. Do not underestimate a user's ability to shoot themselves in the foot; even hiding this stuff behind the "advanced" mode will still lead to plenty of people permanently defacing their collections because they don't make regular backups. What if, out of all the prebuilt compression options, it turns out the user misclicked and selected the wrong one? Or they misunderstood what the options were? Or they simply had no idea what to anticipate from the options?

So the only good solution would have to have enough friction to not be worth it except for users with a serious need (so that only people who are willing to understand the entire process are engaging with it,) enough guardrails to not get used by non-power users accidentally (so that curious but inept users don't destroy their entire collections,) and enough options/customizability to handle every potential use case (including ones that the developer doesn't have himself and would have to amass a bunch of test fixtures for.)

Is there such a potential built-in solution that meets those criteria but somehow is better than the export -> convert -> import -> deduplicate flow? Maybe, but due to the development and maintenance costs of such a feature, I would be stunned if the dev ever implemented anything even remotely in the realm of this kind of thing. It's too fraught with potential peril, the demand is actually pretty niche anyway, and more-than-good-enough workarounds exist. If you waited for it to get built instead of tried building your own workflow, you'd almost certainly be waiting forever. Hopefully my post both helps you get to those workarounds faster and also gives you some level of solace about the features not existing in Hydrus itself.

[216598762]

Sorry for the late response, I don't spend much of my time on github in general. I agree with almost all of your points and don't expect a response to this I'm just going to lay out some final thoughts. I do seriously appreciate the thorough response, as I wouldn't quite consider myself a power user I think I'm far enough above the average, as well I would assume most who use hydrus are. I've been using hydrus for give or take 7 years now? At one point I was active in the discord, frequented the thread when I had time to put aside to it, etc. That's just for some background.

I do generally agree the scope of a h264/h265 conversion pipeline is probably way out of scope for the project, and I would say that asking for something like webp conversions is on the edge of teetering there.

I would however disagree that this is niche, perhaps a decade ago webp was niche, but now it is the go to standard for file size on the internet. Anyone in webdev isn't asking whether or not you should use webp, they just are. Naturally your response would be "Well yeah, they're loading web pages and bandwith is limited", and I think that's a great point for the existence of the webp support. We are talking excessively large collections of files, with excessively large losses in space due to duplicates, poor file encoding, and just sheer volume of data.

I think that's exactly why it fits the scope of hydrus. There's a reason deduplicating has been streamlined as of recently to move towards a more automated approach. If anything, being able to automate conversions into webp lossless, gives users more wiggle room to keep files without having to be concerned with whether or not they're dealing with a large amount of uncaught duplicates, or one artist uploading 10 copies of the exact same image with one slight alteration.

I don't need to remind anyone that's a datahoarder what harddrives are costing now.

As for optimization, I know the wonders hydrus manages given the already insanely large asks. I know databases, and hydrus databases are .. complicated. I'm no expert, but I imagine when doing a conversion the original hash could be used as the hash looked at when searching for files. Rather than having two hashes you're searching against, after all, your hash is practically a brand new webp one you made.

I also agree something like this would be controversial if it was forced, which is why if it was added I would say it should be optional.

As for what hydev decides, that's his call, I know in the past he hasn't been too keen to outside help so a large part of that is more of an undertaking for him. I don't think it's an idea that should be thrown out though, I think it could serve some value.

Thanks for the ideas on automating it, if it comes to it I may decide to do so, but it's hard to find the time for managing most of my collection, and power user stuff I have to be very careful as a nitwit going in with a jackhammer to some fine china.

[usernv40x]

I have never consulted with the dev or other members within the community so I don't know what their thoughts or opinions on this are, but I discovered a work around to this problem and I have converted my library to JXL images. It works on v670 as far as I can tell, but I can't promise it won't cause problems in the future.

I did this because I wanted to have "visually lossless" images at a fraction of the file size, but I also wanted to keep the "hash" associated with the original file so I could fetch tags from the "PTR" and sort files by the original image "dates".

I do somewhat regret doing this as once you convert your images you can never re-import them. Hydrus will assume the images are a "new" file and create a duplicate. For example I recently created a workflow in ComfyUI that uses AI to create image sidecars. I could fix a lot the tag gaps in my library, but I have to import the "original image" with the sidecar files. I can't use the converted file in the library because it has a different "hash".

Anyway here is how you can convert your images to webp or whatever. Before you try this on you main library I suggest creating a test library using a portable installation of hydrus.

NOTE: Sometimes the files you download with hydrus are "corrupted" or contain bad metadata. This can cause conversion errors. The image converter below will skip these files and display the errors at the end of the conversion. This is not a big deal and their are no consequences except that some of your files won't be converted. The ffmpeg script however can be dangerous as it will output and save a file with errors. I have not noticed any problems with this exact script, but I had a previous version that conversion converted audio to "acc". There was a problem with my original files the and audio ffmpeg created was corrupt. The script below just passes the original audio thru to the new file, but I can't guarantee that it won't cause problems. You'll have to independently verify your converted videos or alter the script to accommodate your needs.

Method 1 (xl-converter)

Search for the files you want to convert in hydrus and select them. Then drag and drop them into your preferred image converter. When the files are in the queue close hydrus. I used xl-converter it supports webp, but I don't know if I supports "lossless webp". Once you select your settings make sure the output > save to > source folder and output > after conversion > delete original > permanently options are selected. This will write over your original files within the hydrus database and leave the converted files in there place. They will have the same name, but a different extension.

When the file conversion is finished open hydrus and select the files you converted and right click file > manage > maintenance > regenerate file metadata. This will tell hydrus to check the file extension, file size, etc., but other information like times, urls, tags, etc. will be preserved and if your syncing with the PTR hydrus with think the "converted image" is the "original image".

Method 2 (ffmpeg on windows 11)

Search for the files you want to convert in hydrus and select them. Drag and drop them into a folder on your computer or export the files with the filename "{md5}". Convert the files in a way that preserves the filename. For example I converted my videos with ffmpeg to av1 with audio pass-thru using the below script.

for %%a in ("C:\videos\*.*") do ffmpeg -i "%%a" ^
  -map 0 ^
  -c:v libsvtav1 -pix_fmt yuv420p10le -crf 20 -preset 4 ^
  -c:a aac -b:a 0 ^
  -c:s copy ^
  -map_chapters 0 ^
  "C:\videos 2\%%~na.mkv"

This script can be saved as a .bat file in notepad. Paste the text into notepad and save the file with the .bat extension. It will automatically run when you click on it. Create the appropriate folders first. You can also change the directory if you like. This script takes video files in "C:\videos" converts them and saves the converted files to "C:\videos 2" with the same filename. If you are using another conversion method THE FILE NAMES HAVE TO BE IDENTICAL.

The next step is to sort these files into subfolders so they can replace the original files in the hydrus database. Here is another script that can be saved as a .bat file in notepad.

import os
import shutil

source_dir = r"C:\videos 2"
target_dir = r"C:\videos 3"

os.makedirs(target_dir, exist_ok=True)

for filename in os.listdir(source_dir):
    src_path = os.path.join(source_dir, filename)

    if not os.path.isfile(src_path):
        continue

    name_only = os.path.splitext(filename)[0]
    if len(name_only) < 2:
        continue

    subfolder = "f" + name_only[:2].lower()
    subfolder_path = os.path.join(target_dir, subfolder)
    os.makedirs(subfolder_path, exist_ok=True)

    dst_path = os.path.join(subfolder_path, filename)
    shutil.move(src_path, dst_path)

print("File sorting complete.")

This script takes the converted videos in "C:\videos 2" and moves them to "C:\videos 3" while placing them in to subfolders that match the hydrus database.

Now you have to prepare hydrus to accept the imported files. The file extensions in both databases have to match. If your converted videos went from .mp4 to .mkv or in your case the images went from .png to .webp you have to force the file type in hydrus to match the new extension. Highlight the files you want to change and right click file > manage > force filetype. Once hydrus has processed the request close the program and using the file explorer drag the new files and subfolders into the location of the hydrus database. Windows should ask you if you want to "replace the files in the destination", select this option.

When you finish merging the files reopen hydrus. Select the files you converted and right click file > manage > maintenance > regenerate file metadata.