Unable to create local storage pool – DataStore.getId() NPE (store is null)

[akoskuczi-bw]

problem

When attempting to create a local primary storage pool using cmk, the CloudStack API returns an internal error:
Cannot invoke "org.apache.cloudstack.engine.subsystem.api.storage.DataStore.getId()" because "store" is null

Command used:

_cmk create storagepool zoneid=9020ea93-2aa7-412d-8fe1-19481d7fd545  name=test provider=DefaultPrimary podid=c257a367-ca06-4edd-babf-7872c2d13479 clusterid=6f433ac4-d3c4-423a-a88b-82fa306a2253 url=file://172.16.135.61/var/lib/libvirt/images2 -d_
🙈 Error: (HTTP 530, error code 9999) Cannot invoke "org.apache.cloudstack.engine.subsystem.api.storage.DataStore.getId()" because "store" is null

root@csm-1-1:/home/akos# cmk -d create storagepool zoneid=9020ea93-2aa7-412d-8fe1-19481d7fd545  name=test provider=DefaultPrimary podid=c257a367-ca06-4edd-babf-7872c2d13479 clusterid=6f433ac4-d3c4-423a-a88b-82fa306a2253 url=file://172.16.135.61/var/lib/libvirt/images2 
[debug] Trying to read API cache from:/root/.cmk/profiles/localcloud.cache
[debug] cmdline args:cmk, -d, create, storagepool, zoneid=9020ea93-2aa7-412d-8fe1-19481d7fd545, name=test, provider=DefaultPrimary, podid=c257a367-ca06-4edd-babf-7872c2d13479, clusterid=6f433ac4-d3c4-423a-a88b-82fa306a2253, url=file://172.16.135.61/var/lib/libvirt/images2
[debug] ExecCmd args: create, storagepool, zoneid=9020ea93-2aa7-412d-8fe1-19481d7fd545, name=test, provider=DefaultPrimary, podid=c257a367-ca06-4edd-babf-7872c2d13479, clusterid=6f433ac4-d3c4-423a-a88b-82fa306a2253, url=file://172.16.135.61/var/lib/libvirt/images2
[debug] Login POST URL:http://localhost:8080/client/apimap[command:[login] domain:[/] password:[password] response:[json] username:[admin]]
[debug] Login POST response status code:200
[debug] Login response body:{"loginresponse":{"username":"admin","userid":"d7cea51b-930c-11f0-a295-66656d5a9b03","domainid":"aa16083b-930c-11f0-a295-66656d5a9b03","timeout":1800,"account":"admin","firstname":"admin","lastname":"cloud","type":"1","timezone":"UTC","timezoneoffset":"0.0","registered":"false","sessionkey":"HmXTHKFUTEkJyzhR_MJx-i9MH6Y","is2faenabled":"false","is2faverified":"true","issuerfor2fa":"CloudStack","managementserverid":"6cd281d6-a401-45d4-929a-90c8f1b913a3"}}
[debug] Login sessionkey:HmXTHKFUTEkJyzhR_MJx-i9MH6Y
[debug] NewAPIRequest API request URL:http://localhost:8080/client/api?clusterid=6f433ac4-d3c4-423a-a88b-82fa306a2253&command=createStoragePool&expires=2025-11-18T20%3A50%3A24Z&name=test&podid=c257a367-ca06-4edd-babf-7872c2d13479&provider=DefaultPrimary&response=json&sessionkey=HmXTHKFUTEkJyzhR_MJx-i9MH6Y&signatureversion=3&url=file%3A%2F%2F172.16.135.61%2Fvar%2Flib%2Flibvirt%2Fimages2&zoneid=9020ea93-2aa7-412d-8fe1-19481d7fd545
[debug] Using HTTP POST for the request: http://localhost:8080/client/api
[debug] NewAPIRequest response status code:530
[debug] NewAPIRequest response body:{"createstoragepoolresponse":{"uuidList":[],"errorcode":530,"cserrorcode":9999,"errortext":"Cannot invoke \"org.apache.cloudstack.engine.subsystem.api.storage.DataStore.getId()\" because \"store\" is null"}}
🙈 Error: (HTTP 530, error code 9999) Cannot invoke "org.apache.cloudstack.engine.subsystem.api.storage.DataStore.getId()" because "store" is null

On kvm host:

mount | grep images
/dev/sdb on /var/lib/libvirt/images2 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Same result on gui.

versions

Version of ACS mgmt and agent: 4.21.0.0
Hypervisor host: KVM on Ubuntu 24.04.3 LTS

The steps to reproduce the bug

...

What to do about it?

No response

[weizhouapache]

Any exception in management log ?

[akoskuczi-bw]

@weizhouapache Only the same error shows up there:
zcat /var/log/cloudstack/management/management-server.log.2025-11-18.gz | grep "1f26d0fc"
2025-11-18 20:32:01,237 DEBUG [c.c.a.ApiServlet] (qtp1438988851-25:[ctx-61b9fe0c]) (logid:1f26d0fc) ===START=== 127.0.0.1 -- POST createStoragePool
2025-11-18 20:32:01,237 DEBUG [c.c.a.ApiServlet] (qtp1438988851-25:[ctx-61b9fe0c]) (logid:1f26d0fc) Two factor authentication is already verified for the user 2, so skipping
2025-11-18 20:32:01,249 DEBUG [c.c.a.ApiServer] (qtp1438988851-25:[ctx-61b9fe0c, ctx-5ac4aca1]) (logid:1f26d0fc) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"d7cdd13e-930c-11f0-a295-66656d5a9b03"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-11-18 20:32:01,256 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1438988851-25:[ctx-61b9fe0c, ctx-5ac4aca1]) (logid:1f26d0fc) Account for user id d7cea51b-930c-11f0-a295-66656d5a9b03 is Root Admin or Domain Admin, all APIs are allowed.
2025-11-18 20:32:01,256 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1438988851-25:[ctx-61b9fe0c, ctx-5ac4aca1]) (logid:1f26d0fc) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-11-18 20:32:01,256 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1438988851-25:[ctx-61b9fe0c, ctx-5ac4aca1]) (logid:1f26d0fc) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-11-18 20:32:01,281 INFO [c.c.a.ApiServer] (qtp1438988851-25:[ctx-61b9fe0c, ctx-5ac4aca1]) (logid:1f26d0fc) Cannot invoke "org.apache.cloudstack.engine.subsystem.api.storage.DataStore.getId()" because "store" is null
2025-11-18 20:32:01,281 DEBUG [c.c.a.ApiServlet] (qtp1438988851-25:[ctx-61b9fe0c, ctx-5ac4aca1]) (logid:1f26d0fc) ===END=== 127.0.0.1 -- POST createStoragePool

[DaanHoogland]

reproduced, need to investigate and find relevant stacktraces/errorlogs

[weizhouapache]

reproduced, need to investigate and find relevant stacktraces/errorlogs

which ACS version @DaanHoogland

[DaanHoogland]

will investigate further on 4.22 @weizhouapache (reproduced on 4.21 as reported on)