Skip to content

Encode scope in secondary keys for cross-scope isolation #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
heifner merged 3 commits into from
Apr 7, 2026
Merged

Encode scope in secondary keys for cross-scope isolation #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8674ba3
Encode scope in secondary keys for cross-scope isolation
heifner Apr 4, 2026
e591907
Make pack_size constexpr, fix broken static_assert in secondary_index…
heifner Apr 4, 2026
e83b311
Fix missing ABI type for scopetbl row struct
heifner Apr 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/kv-intrinsics-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ void kv_idx_store(uint64_t payer, uint64_t table, uint32_t index_id,
| `payer` | `uint64_t` | Account to bill for RAM (0 = receiver) |
| `table` | `uint64_t` | Logical table name (e.g., `"accounts"_n.value`) |
| `index_id` | `uint32_t` | Index identifier (0-255, corresponding to the Nth `indexed_by`) |
| `pri_key` | `const void*` | Primary key bytes (`[scope:8B][pk:8B]` = 16 bytes in CDT) |
| `pri_key` | `const void*` | Primary key bytes (`[pk:8B]` = 8 bytes in CDT) |
| `pri_key_size` | `uint32_t` | Primary key size (max 256) |
| `sec_key` | `const void*` | Secondary key bytes (big-endian encoded) |
| `sec_key_size` | `uint32_t` | Secondary key size (max 256) |
Expand Down Expand Up @@ -621,8 +621,8 @@ int32_t kv_idx_primary_key(uint32_t handle, uint32_t offset,
**Returns:** `int32_t` -- Status code (0=OK).

**Behavior:**
- In the CDT multi\_index implementation, the stored primary key is `[scope:8B][pk:8B]` = 16 bytes
- The caller extracts the 8-byte primary key from offset 8
- In the CDT multi\_index implementation, the stored primary key is `[pk:8B]` = 8 bytes
- The caller reads the 8-byte primary key directly from offset 0

---

Expand Down
3 changes: 2 additions & 1 deletion docs/kv-storage-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ Properties:
- Fixed 24 bytes
- Integer fast-path: 8-byte big-endian keys compared via single `bswap64`
- SHiP compatible: reversible to legacy `contract_row` format (table, scope, primary\_key)
- Secondary index keys include scope: `[scope:8B][pk:8B]` = 16 bytes
- Secondary index keys prepend scope to the secondary value: `[scope:8B BE][secondary_value]`
- Primary keys stored in secondary index entries are `[pk:8B]` = 8 bytes (scope is in the secondary key)

### Format=0 (indexed\_table, raw\_table)

Expand Down
142 changes: 108 additions & 34 deletions libraries/sysiolib/contracts/sysio/kv_multi_index.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,17 +236,15 @@ class kv_multi_index {
return p;
}

// 16-byte scoped primary key for secondary index: [scope:8B][pk:8B]
// Scope must be included so that rows with the same primary key in
// different scopes produce distinct secondary-index entries.
// 8-byte primary key for secondary index: [pk:8B]
// Scope is encoded in the sec_key prefix instead (see store_all).
struct pk_bytes_buf {
char data[16];
char data[8];
};

pk_bytes_buf pk_to_bytes(uint64_t pk) const {
pk_bytes_buf b;
_kv_multi_index_detail::encode_be64(b.data, _scope);
_kv_multi_index_detail::encode_be64(b.data + 8, pk);
_kv_multi_index_detail::encode_be64(b.data, pk);
return b;
}

Expand Down Expand Up @@ -296,16 +294,63 @@ class kv_multi_index {
return raw;
}

// Encode [scope:8B BE][secondary_value] using the correct sort-preserving
// encoder for each type. The chain's kv_idx_* intrinsics have no scope
// parameter; encoding scope into sec_key provides equivalent isolation to
// the legacy db_idx*_find_secondary(code, scope, ...) API.
//
// Generic version: delegates to _kv_multi_index_detail::encode_secondary()
// which has specializations for uint64_t, uint128_t, double, and long double.
template<typename SecVal>
std::vector<char> encode_scoped_secondary(const SecVal& val) const {
auto raw = _kv_multi_index_detail::encode_secondary(val);
std::vector<char> buf(8 + raw.size());
_kv_multi_index_detail::encode_be64(buf.data(), _scope);
memcpy(buf.data() + 8, raw.data(), raw.size());
return buf;
}

// Single-allocation fast path for uint64_t.
std::vector<char> encode_scoped_secondary(const uint64_t& val) const {
std::vector<char> buf(16);
_kv_multi_index_detail::encode_be64(buf.data(), _scope);
_kv_multi_index_detail::encode_be64(buf.data() + 8, val);
return buf;
}

// Single-allocation fast path for uint128_t.
std::vector<char> encode_scoped_secondary(const uint128_t& val) const {
std::vector<char> buf(24);
_kv_multi_index_detail::encode_be64(buf.data(), _scope);
_kv_multi_index_detail::encode_be64(buf.data() + 8, static_cast<uint64_t>(val >> 64));
_kv_multi_index_detail::encode_be64(buf.data() + 16, static_cast<uint64_t>(val));
return buf;
}

// Single-allocation fast path for double (sort-preserving transform + scope).
std::vector<char> encode_scoped_secondary(const double& val) const {
uint64_t bits;
memcpy(&bits, &val, 8);
if (bits & (uint64_t(1) << 63))
bits = ~bits;
else
bits ^= (uint64_t(1) << 63);
std::vector<char> buf(16);
_kv_multi_index_detail::encode_be64(buf.data(), _scope);
_kv_multi_index_detail::encode_be64(buf.data() + 8, bits);
return buf;
}

// --- Secondary index helpers ---
template<size_t N, typename Index, typename... Rest>
struct secondary_ops {
static void store_all(uint64_t payer, const kv_multi_index& idx, const T& obj) {
using extractor_t = typename Index::secondary_extractor_type;
extractor_t ext;
auto sec_key = _kv_multi_index_detail::encode_secondary(ext(obj));
auto sec_key = idx.encode_scoped_secondary(ext(obj));
auto pri_key = idx.pk_to_bytes(obj.primary_key());
::kv_idx_store(payer, static_cast<uint64_t>(TableName), N,
pri_key.data, 16,
pri_key.data, 8,
sec_key.data(), sec_key.size());
if constexpr (sizeof...(Rest) > 0) {
secondary_ops<N+1, Rest...>::store_all(payer, idx, obj);
Expand All @@ -315,10 +360,10 @@ class kv_multi_index {
static void remove_all(const kv_multi_index& idx, const T& obj) {
using extractor_t = typename Index::secondary_extractor_type;
extractor_t ext;
auto sec_key = _kv_multi_index_detail::encode_secondary(ext(obj));
auto sec_key = idx.encode_scoped_secondary(ext(obj));
auto pri_key = idx.pk_to_bytes(obj.primary_key());
::kv_idx_remove(static_cast<uint64_t>(TableName), N,
pri_key.data, 16,
pri_key.data, 8,
sec_key.data(), sec_key.size());
if constexpr (sizeof...(Rest) > 0) {
secondary_ops<N+1, Rest...>::remove_all(idx, obj);
Expand All @@ -328,12 +373,12 @@ class kv_multi_index {
static void update_all(uint64_t payer, const kv_multi_index& idx, const T& old_obj, const T& new_obj) {
using extractor_t = typename Index::secondary_extractor_type;
extractor_t ext;
auto old_sec = _kv_multi_index_detail::encode_secondary(ext(old_obj));
auto new_sec = _kv_multi_index_detail::encode_secondary(ext(new_obj));
auto old_sec = idx.encode_scoped_secondary(ext(old_obj));
auto new_sec = idx.encode_scoped_secondary(ext(new_obj));
auto pri_key = idx.pk_to_bytes(old_obj.primary_key());
if (old_sec != new_sec) {
::kv_idx_update(payer, static_cast<uint64_t>(TableName), N,
pri_key.data, 16,
pri_key.data, 8,
old_sec.data(), old_sec.size(),
new_sec.data(), new_sec.size());
}
Expand Down Expand Up @@ -776,18 +821,22 @@ class kv_multi_index {
using index_type = typename std::tuple_element<index_number, std::tuple<Indices...>>::type;
using secondary_extractor_type = typename index_type::secondary_extractor_type;
using secondary_key_type = std::decay_t<typename secondary_extractor_type::result_type>;
// Secondary key encoding uses sizeof(secondary_key_type) for stack buffer sizing.
// Trivially copyable types guarantee sizeof == packed size (no varint prefixes).
static_assert(std::is_trivially_copyable<secondary_key_type>::value,
"secondary key type must be trivially copyable for fixed-size encoding");

const kv_multi_index* _mi;
secondary_index_view(const kv_multi_index& mi) : _mi(&mi) {}

// Helper: read scoped primary key from secondary iterator handle.
// The stored pri_key is [scope:8B][pk:8B] = 16 bytes; we extract the pk.
// Helper: read primary key from secondary iterator handle.
// The stored pri_key is [pk:8B].
static bool read_primary_key(uint32_t handle, uint64_t& pk) {
char pri_buf[16];
char pri_buf[8];
uint32_t actual = 0;
int32_t status = ::kv_idx_primary_key(handle, 0, pri_buf, 16, &actual);
if (status != 0 || actual != 16) return false;
pk = _kv_multi_index_detail::decode_be64(pri_buf + 8);
int32_t status = ::kv_idx_primary_key(handle, 0, pri_buf, 8, &actual);
if (status != 0 || actual != 8) return false;
pk = _kv_multi_index_detail::decode_be64(pri_buf);
return true;
}

Expand All @@ -807,7 +856,7 @@ class kv_multi_index {
const_iterator& operator++() {
check(_has_obj, "cannot increment end iterator");
if (!_has_obj || _handle < 0) return *this;
if (::kv_idx_next(_handle) == 0) { load_current(); }
if (::kv_idx_next(_handle) == 0 && check_scope()) { load_current(); }
else { _has_obj = false; }
return *this;
}
Expand All @@ -817,19 +866,22 @@ class kv_multi_index {

const_iterator& operator--() {
if (_handle < 0) {
// End sentinel: create real iterator at last entry.
// Use lower_bound with a maximal key (all 0xFF) to position
// past all entries, then prev to land on the last one.
char max_sec[kv::kv_key_max_bytes];
memset(max_sec, 0xFF, sizeof(max_sec));
// End sentinel: create real iterator at last entry in THIS scope.
// Build a maximal sec_key for this scope: [scope:8B][0xFF...]
// so lower_bound positions past this scope's entries, then prev
// lands on the last entry in the scope.
constexpr size_t sec_val_size = sizeof(secondary_key_type);
char max_sec[8 + sec_val_size];
_kv_multi_index_detail::encode_be64(max_sec, _mi->_scope);
memset(max_sec + 8, 0xFF, sec_val_size);
_handle = ::kv_idx_lower_bound(
_mi->_code.value, static_cast<uint64_t>(TableName), index_number,
max_sec, sizeof(max_sec));
if (_handle < 0) { _has_obj = false; }
else if (::kv_idx_prev(_handle) == 0) { _has_obj = true; load_current(); }
else if (::kv_idx_prev(_handle) == 0 && check_scope()) { _has_obj = true; load_current(); }
else { _has_obj = false; }
} else {
if (::kv_idx_prev(_handle) == 0) { _has_obj = true; load_current(); }
if (::kv_idx_prev(_handle) == 0 && check_scope()) { _has_obj = true; load_current(); }
else { check(false, "cannot decrement iterator at beginning of index"); }
}
return *this;
Expand Down Expand Up @@ -870,7 +922,7 @@ class kv_multi_index {
if (o._handle >= 0 && _mi && _has_obj) {
using extractor_t = typename std::tuple_element<index_number, std::tuple<typename Indices::secondary_extractor_type...>>::type;
extractor_t ext;
auto sec_bytes = _kv_multi_index_detail::encode_secondary(ext(_obj));
auto sec_bytes = _mi->encode_scoped_secondary(ext(_obj));
_handle = ::kv_idx_find_secondary(
_mi->_code.value, static_cast<uint64_t>(TableName), index_number,
sec_bytes.data(), sec_bytes.size());
Expand Down Expand Up @@ -912,6 +964,20 @@ class kv_multi_index {
if (ptr) { _obj = *ptr; _has_obj = true; }
else { _has_obj = false; }
}

// Verify the current sec_key still starts with [scope:8B].
// After kv_idx_next/prev, the iterator may have crossed into
// another scope — treat that as end-of-range.
bool check_scope() const {
if (_handle < 0 || !_mi) return false;
char scope_buf[8];
uint32_t actual = 0;
int32_t status = ::kv_idx_key(_handle, 0, scope_buf, 8, &actual);
if (status != 0 || actual < 8) return false;
char expected[8];
_kv_multi_index_detail::encode_be64(expected, _mi->_scope);
return memcmp(scope_buf, expected, 8) == 0;
}
};

const_iterator end() const { return const_iterator::make_end(_mi); }
Expand All @@ -925,16 +991,22 @@ class kv_multi_index {
const_reverse_iterator crend() const { return rend(); }

const_iterator begin() const {
// Lower bound with empty key = first entry
// Lower bound with scope prefix = first entry in this scope
char scope_prefix[8];
_kv_multi_index_detail::encode_be64(scope_prefix, _mi->_scope);
int32_t handle = ::kv_idx_lower_bound(
_mi->_code.value, static_cast<uint64_t>(TableName), index_number, nullptr, 0);
_mi->_code.value, static_cast<uint64_t>(TableName), index_number,
scope_prefix, 8);
if (handle < 0) return end();
return const_iterator(_mi, handle, true);
// Verify we landed in the right scope (may be past it if scope is empty)
const_iterator it(_mi, handle, true);
if (it._has_obj && !it.check_scope()) return end();
return it;
}

template<typename SecKey>
const_iterator find(const SecKey& sec_key) const {
auto sec_bytes = _kv_multi_index_detail::encode_secondary(secondary_key_type(sec_key));
auto sec_bytes = _mi->encode_scoped_secondary(secondary_key_type(sec_key));
int32_t handle = ::kv_idx_find_secondary(
_mi->_code.value, static_cast<uint64_t>(TableName), index_number,
sec_bytes.data(), sec_bytes.size());
Expand All @@ -944,12 +1016,14 @@ class kv_multi_index {

template<typename SecKey>
const_iterator lower_bound(const SecKey& sec_key) const {
auto sec_bytes = _kv_multi_index_detail::encode_secondary(secondary_key_type(sec_key));
auto sec_bytes = _mi->encode_scoped_secondary(secondary_key_type(sec_key));
int32_t handle = ::kv_idx_lower_bound(
_mi->_code.value, static_cast<uint64_t>(TableName), index_number,
sec_bytes.data(), sec_bytes.size());
if (handle < 0) return end();
return const_iterator(_mi, handle, true);
const_iterator it(_mi, handle, true);
if (it._has_obj && !it.check_scope()) return end();
return it;
}

template<typename SecKey>
Expand Down
24 changes: 15 additions & 9 deletions libraries/sysiolib/core/sysio/datastream.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,39 +197,39 @@ class datastream<size_t> {
*
* @param init_size - The initial size
*/
datastream( size_t init_size = 0):_size(init_size){}
constexpr datastream( size_t init_size = 0):_size(init_size){}

/**
* Increment the size by s. This behaves the same as write( const char* ,size_t s ).
*
* @param s - The amount of size to increase
* @return true
*/
inline bool skip( size_t s ) { _size += s; return true; }
constexpr bool skip( size_t s ) { _size += s; return true; }

/**
* Increment the size by s. This behaves the same as skip( size_t s )
*
* @param s - The amount of size to increase
* @return true
*/
inline bool write( const char* ,size_t s ) { _size += s; return true; }
constexpr bool write( const char* ,size_t s ) { _size += s; return true; }

/**
* Increment the size by s. This behaves the same as skip( size_t s )
*
* @param s - The amount of size to increase
* @return true
*/
inline bool write( char ) { _size++; return true; }
constexpr bool write( char ) { _size++; return true; }

/**
* Increment the size by s. This behaves the same as skip( size_t s )
*
* @param s - The amount of size to increase
* @return true
*/
inline bool write( const void* ,size_t s ) { _size += s; return true; }
constexpr bool write( const void* ,size_t s ) { _size += s; return true; }

/**
* Increment the size by one
Expand Down Expand Up @@ -258,7 +258,7 @@ class datastream<size_t> {
*
* @return size_t - The size
*/
inline size_t tellp()const { return _size; }
constexpr size_t tellp()const { return _size; }

/**
* Always returns 0
Expand Down Expand Up @@ -997,8 +997,14 @@ DataStream& operator>>( DataStream& ds, T& v ) {
* @return datastream<Stream>& - Reference to the datastream
*/
template<typename Stream, typename T, std::enable_if_t<_datastream_detail::is_primitive<T>()>* = nullptr>
datastream<Stream>& operator<<( datastream<Stream>& ds, const T& v ) {
ds.write( (const char*)&v, sizeof(T) );
constexpr datastream<Stream>& operator<<( datastream<Stream>& ds, const T& v ) {
// datastream<size_t> is a size-counting stream that ignores the pointer;
// use skip() to avoid the reinterpret_cast which is illegal in constexpr.
if constexpr (std::is_same_v<Stream, size_t>) {
ds.skip( sizeof(T) );
} else {
ds.write( (const char*)&v, sizeof(T) );
}
return ds;
}

Expand Down Expand Up @@ -1072,7 +1078,7 @@ T unpack( const std::vector<char>& bytes ) {
* @return size_t - Size of the packed data
*/
template<typename T>
size_t pack_size( const T& value ) {
constexpr size_t pack_size( const T& value ) {
datastream<size_t> ps;
ps << value;
return ps.tellp();
Expand Down
3 changes: 3 additions & 0 deletions tests/integration/contracts.hpp.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ namespace sysio::testing {
static std::vector<uint8_t> test_multi_index_wasm() { return read_wasm("${CMAKE_BINARY_DIR}/../unit/test_contracts/test_multi_index.wasm"); }
static std::vector<char> test_multi_index_abi() { return read_abi("${CMAKE_BINARY_DIR}/../unit/test_contracts/test_multi_index.abi"); }

static std::vector<uint8_t> mi_scope_tests_wasm() { return read_wasm("${CMAKE_BINARY_DIR}/../unit/test_contracts/mi_scope_tests.wasm"); }
static std::vector<char> mi_scope_tests_abi() { return read_abi("${CMAKE_BINARY_DIR}/../unit/test_contracts/mi_scope_tests.abi"); }

static std::vector<uint8_t> kv_table_tests_wasm() { return read_wasm("${CMAKE_BINARY_DIR}/../unit/test_contracts/kv_table_tests.wasm"); }
static std::vector<char> kv_table_tests_abi() { return read_abi("${CMAKE_BINARY_DIR}/../unit/test_contracts/kv_table_tests.abi"); }

Expand Down
Loading
Loading