[QUESTION]: Handling of multi-word keywords in Scopus query construction

[WilmerGaspar]

Your Question

While reviewing the metadata collection workflow, I noticed that multi-word keywords appear to be inserted directly into the Scopus query URL. I have not yet verified whether this causes incorrect behavior in the Scopus API because I currently do not have access to a Scopus API key.

Is this the expected behavior, or should multi-word keywords be URL encoded or quoted before being passed to Scopus?

What I've Already Tried

• Checked the documentation: Yes
• Searched existing issues: Yes
• Installed ComProScanner locally and traced the metadata collection workflow.
• Followed the query path:

main_fetch() → _fetch_paginated_data() → fetch_and_process_data() → _construct_url()

• Generated a test URL using the keyword:

thermal conductivity

Generated URL:

https://api.elsevier.com/content/search/scopus?query=PUBYEAR+%3D+2024+thermal conductivity&count=200&cursor=*

I noticed that the space between "thermal" and "conductivity" is preserved in the generated URL and I could not find explicit URL encoding in this code path.

I currently do not have access to a Scopus API key, so I have not yet verified whether this causes incorrect behavior or whether Scopus handles it internally.

Context

I'm trying to understand whether this could be related to the multi-word keyword issue previously discussed in Slack.

My goal is to help validate and improve the metadata collection workflow before testing with Scopus access.

System & Environment

• OS: Windows 10
• Processor: Intel Core i5 (adjust if needed)
• Installed RAM: 8 GB (adjust if needed)
• GPU Enabled: No
• Terminal: Windows Command Prompt (CMD)
• Python environment: Python 3.12.10

Additional context

I can provide additional testing results once I obtain access to a Scopus API key.

[aritraroy24]

Hi @WilmerGaspar, thanks for the detailed report! At first glance this looks like a bug, though it won't completely break the workflow. Here's what may be affected:

Metadata Collection

• By default, spaces in the query are replaced with %20 by the Scopus backend. While this prevents the workflow from breaking, it reduces search accuracy — e.g., passing thermal conductivity may also return results for electrical conductivity and thermal expansion, which costs unnecessary API calls.

Article Processing

• The main keyword is used for automated file handling, so a space in the input would create filenames like thermal conductivity_metadata.csv. Most OSes handle this fine, but replacing spaces with _ is better practice for scripting.
• If is_sql_db=True (False by default), a space in the table name could cause issues. That said, SQLAlchemy wraps space-containing names in quotes automatically, so it should still work.

Data Extraction, Cleaning, Evaluation, and Visualisation

• Nothing should break.

✅ Recommended fix

• Replace spaces with _ in the main keyword before the metadata collection step.
• Strip the _ before any regex matching in article processing and the data extraction flow.

If you'd like to contribute a fix, I'm happy to assign this to you!

P.S.: For getting the Scopus API, please refer to the API key guide (provided your institution has a subscription to the Scopus database)

[WilmerGaspar]

Thanks a lot for checking this, Aritra. I’m glad the report was useful.

Yes, I’d be happy to work on a small fix for this issue. I understand that the workflow may not completely break, but improving the handling of multi-word keywords could help with query accuracy and file/table naming consistency.

I currently don’t have institutional Scopus API access, so I’ll focus only on the local keyword preprocessing and query-construction behavior, without making claims about live API validation.

I can start by following your recommended fix: replacing spaces with _ before metadata collection and normalizing or stripping _ before regex matching in the later processing steps.

If this sounds good, please assign the issue to me and I’ll start reviewing the relevant code path.

[aritraroy24]

Hi @WilmerGaspar, thanks for showing interest in working on the fix. I'll assign this task to you. Please refer to the CONTRIBUTING.md before submitting the pull request.

Regarding the Scopus API key, I ran some access tests with an API key generated from a personal Gmail-based Elsevier developer account. The results are as follows:

• You will have full access (cursor=* compatibility for multiple-page crawl) if you are on an institutional IP which has the Elsevier subscription.

• If you are not on institutional IP, you will still get some valid results. But you can't use cursor=* in the request URL (& parameter should also be removed as the next header parameter is being removed). So, the request URLs will be (properly space-encoded):

  • through the GET request and API key as a header (recommended):
    https://api.elsevier.com/content/search/scopus?query=PUBYEAR+%3D+2024+thermal%20AND%20conductivity
  • through a direct URL access (e.g., browser [not recommended]):
    https://api.elsevier.com/content/search/scopus?query=PUBYEAR+%3D+2024+thermal%20AND%20conductivity&apiKey=your-scopus-api-key