Search API: doesn't return image_url for newly created dataset

[ekraffmiller]

After creating a dataset with an image file in the SPA, the search API does not return an image_url for the dataset.
It seems that an extra step is needed first, which happens when the logo is requested. This issue occurs even for a published unrestricted dataset, so it isn't related to permissions.

What steps does it take to reproduce the issue?

1. Create a dataset with an image file in the SPA.
2. Publish the dataset
3. Go to the Collections page. The search API call for the Collections page returns the datasetPreview without a image_url property.
4. Go to the JSF and view the Collections page. This triggers a call to datasets/{datasetId}/logo
5. Go back to the Collections page in the SPA - now the image_url is returned in the datasetPreview for the page

• When does this issue occur?
  With a Dataset that is created in the SPA

• Which page(s) does it occurs on?
  Collections Page (search API)

• What happens?
  image_url is not returned

• To whom does it occur (all users, curators, superusers)?
  all users

• What did you expect to happen?
  image_url should be returned for a thumbnail of the image file in the dataset

Screenshots:

SPA - search results with missing image_url

View the Page in JSF

Go back to SPA - now image_url appears in search results

[qqmyers]

Perhaps the image_url should be /api/datasets/{id}/logo ? If the user hasn't selected one, there's a process looking through the datafiles to find one (that has an image, isn't restricted, isn't embargoed, etc.) - probably not things we should do if no one is going to use the image_url.

[ekraffmiller]

@qqmyers yes, that makes sense....if there is no image available, what will api/datasets/{id}/logo return? Can we have logic that only returns image_url if there is something available to display?

[ekraffmiller]

Another thing I noticed is that there doesn't seem to be any permissions check in api/datasets/{id}/logo and it's not part of the Dataverse API documentation.

[qqmyers]

I'm not sure. The current logo api returns null if there isn't one and the display should use a standard icon (triggered by adding class='icon-dataset' on the span). As the standard icon is svg from within the https://demo.dataverse.org/resources/fontcustom/fontcustom_2d8a223b23af3c2ffc668cd1b506f1fa.svg (for that icon-dataset class) I'm not sure if there's an easy way to provide a non-null value to get you the standard icon.

I'd expect it wasn't documented because it's only been used in the UI.
As for permission, I'm not sure it needs any - it can only return public info.

[ekraffmiller]

Ok, we don't need the standard icon to be returned, I was just hoping that rather than calling the url API and checking for a 404, the search results could omit the image_url field if there is no image. But if not, that's ok.

FYI, I tested this more, and calling the url (api/datasets/{id}/logo) on it's own doesn't trigger the creation of the thumbnail. The url is returning a 404 error until I go to the JSF page, and then it returns the image. So I think there is some additional logic happening in the JSF Collections page.

[qqmyers]

Fun! Looks like a bug - the UI has extra/different logic to create a logo for draft datasets and it's smart enough to then record which file is used in the db (associated with the dataset, not a datasetversion).

The api call (which as you noted doesn't have access control) won't generate a logo using a draft file, but, if the jsf code has recorded a file to use, it sends that without checking for draft/restricted (so a bug that after the jsf has run, the api will give you a logo from a draft file without checking your permissions. On it's own, the api is~correct since it would never generate a logo from a draft file itself - my guess is if you created a dataset via api and published it, the /logo api will give you a logo for the published dataset, without ever using the jsf, as it should.).

This bug is also a leak in the jsf - if you add a file to a draft dataset that becomes the logo for the draft (i.e. when the published version didn't have one), the logo appears for the published version too, even though the file isn't in it.

So minimally, we should probably update the API (and JSF!) to use access control, and to check whether the file being selected is not released (in which case you need permissions) before returning a value (regardless of whether the api is finding a new logo or using leveraging the thumbnailfile_id currently only set by jsf). It probably also makes sense to have the API set the thumbnailfile_id for the dataset so each call doesn't do the scan through the files to find one (which it would do now until the jsf is used).

I'm not sure how easy it might be to not return a 404/not include an image_url to start - seems like we'd have to run through the process to see if any file qualifies to do that. (If we update the api to cache a value once found, it helps, but with a dataset that has 1K no image files, I think the code would still scan every time). Pagination would also help limit the cost, but there's at least one use case - @PaulBoon 's geospatial map of results where he's getting 1K+ hits (not paginating) to show many small dots on the map - where scanning would be bad. External search might cause problems as well. One possibility would be to remove the logic that scans for a file from the /logo call and make sure that the scan is done earlier, e.g. when the dataset is created/new files are added, so there's never any scanning when you're just asking for what exists. If we do that, then it wouldn't be too bad to do a custom query to see if a logo exists to decide whether the image_url gets added (we'd want to avoid getting the dataset object in memory - we know from the query_entities discussion that that can be 10x slower than the solr query itself.)

It sounded so simple to start :-)