diff --git a/docs-src/PLAN.md b/docs-src/PLAN.md index 8e3177c05..3e4c0a161 100644 --- a/docs-src/PLAN.md +++ b/docs-src/PLAN.md @@ -116,14 +116,54 @@ the site (and can attach to GitHub releases). ## Execution phases -1. **Scaffold**: `docs-src/` skeleton, `site.toml` (version from `dist/RELEASE`), - templates, `build.py` (HTML first), one hand-migrated sample page end-to-end. -2. **Extractor**: `extract.py` on the API-reference C tree; diff-verify; iterate - the cleanup until content matches. Then C++/STL, then the guides. -3. **Man pages**: per-API `*.3` + the `libdb.3` overview; `mandoc -Tlint` clean. -4. **PDF**: pandoc per book (programmer_reference, the GSGs, api_reference). -5. **CI**: `docs.yml` with all validators + the completeness gate. -6. **Publish**: wire gh-pages to the generated HTML; update the landing page. +1. **Scaffold** (DONE, PR #119): `docs-src/` skeleton, `site.toml` (version from + `dist/RELEASE`), templates, `build.py` (HTML first). +2. **Extractor + trees** (DONE): `extract.py` + `verify.py` no-loss gate on the + C API tree (100.00%), then STL (100.00%) and every guide tree. See the + retention table below. `migrate_tree.py` drives chaptered guides (order from + the index TOC, image copy); `fix_xrefs.py` remaps cross-tree links to the new + `docs-src/` layout. +3. **Man pages** (DONE): `build_man()` -> 787 `*.3` (785 C+STL refentry pages + + `libdb.3` overview + utilities), `mandoc -Tlint` = 0 errors. `man_coverage.py` + reports API coverage (measure-only; the CI gate is phase 5). +4. **PDF** (TODO): pandoc per book (programmer_reference, the GSGs, api_reference) + with a shared LaTeX header. `build.py build_pdf()` is the stubbed seam. +5. **CI** (TODO): `docs.yml` with all validators + the completeness gate (wire + `verify.py` per tree + `man_coverage.py` as hard gates). +6. **Publish** (TODO): wire gh-pages to the generated HTML; update the landing + page. + +### Phase 2 retention (verify.py, mean word retention; 0 hard drops on all) + +| tree | pages | retention | +|-------------------------|-------|-----------| +| api/c | 470 | 100.00% | +| api/stl | 322 | 100.00% | +| guides/programmer_reference | 203 | 99.98% | +| guides/upgrading | 180 | 100.00% | +| guides/installation | 101 | 99.99% | +| guides/porting | 16 | 100.00% | +| guides/gsg (C) | 37 | 100.00% | +| guides/gsg_txn (C) | 38 | 100.00% | +| guides/gsg_db_rep (C) | 26 | 100.00% | +| guides/collections | 37 | 99.98% | +| guides/bdb-sql | 30 | 100.00% | +| guides/articles | 2 | 100.00% | + +### Deferred: docs/csharp (38 MB) and docs/java (12 MB) + +These are LANGUAGE-BINDING docs and are NOT DocBook, so the reverse-DocBook +extractor does not apply: +- **csharp** — a compiled Sandcastle/MS-Help-Viewer tree (`.chm` + `.aspx` + + JS/PNG, 2457 files, only 1 real `.html`). Would need a bespoke extractor. +- **java** — standard Javadoc HTML (525 files: allclasses-frame, package-frame, + index-all). Different structure; the `refentry`/`chapter` isolation is moot. + +Both are enormous and lower-value for the core C engine. Deferred to a future +phase (regenerate from the C#/Java sources with their native doc tools, or write +a per-format extractor) rather than sink budget reverse-engineering rendered +help output. The gsg/gsg_txn/gsg_db_rep CXX/JAVA sub-variants are likewise +deferred; the C variants are migrated. ## Non-negotiables diff --git a/docs-src/_data/site.toml b/docs-src/_data/site.toml index 3b19b312f..aea64a736 100644 --- a/docs-src/_data/site.toml +++ b/docs-src/_data/site.toml @@ -8,3 +8,7 @@ short_name = "libdb" base_url = "https://libdb.org/docs/" copyright = "Copyright (c) 1990, 2013 Oracle and/or its affiliates. All rights reserved." tagline = "The Berkeley DB reference documentation" +# Man-page .TH date field (the DocBook source is dated 9/9/2013). Use the +# `Month D, YYYY` form so mandoc parses it (hyphenated ISO gets roff-escaped by +# pandoc into `2013\-09\-09`, which mandoc then can't read). Single source. +man_date = "September 9, 2013" diff --git a/docs-src/_migrate/extract.py b/docs-src/_migrate/extract.py index 7d32e518f..f83f9e9b5 100644 --- a/docs-src/_migrate/extract.py +++ b/docs-src/_migrate/extract.py @@ -20,6 +20,14 @@ Usage: extract.py [SRC_HTML_DIR] [OUT_MD_DIR] Defaults: docs/api_reference/C -> docs-src/api/c Requires: pandoc on PATH (run under `nix shell nixpkgs#pandoc`). + +Works for both the flat `refentry` API trees (C, STL) and the chaptered +`chapter`/`sect1` guide trees (programmer_reference, gsg, ...): every page has a +single top-level content div in CONTENT_CLASSES. A few auxiliary guide pages +(embedded.html, witold.html) put prose straight under with no content +div — for those a body-level fallback captures everything except the stable +boilerplate divs. The C API tree never hits the fallback (its only div-less +pages are frameset stubs with no prose), so it is regression-safe. """ import html.parser import re @@ -30,6 +38,11 @@ REPO = Path(__file__).resolve().parents[2] SRC = Path(sys.argv[1]) if len(sys.argv) > 1 else REPO / "docs/api_reference/C" OUT = Path(sys.argv[2]) if len(sys.argv) > 2 else REPO / "docs-src/api/c" +# source: path shown in front-matter; derived from SRC relative to the repo. +try: + SRC_REL = str(SRC.resolve().relative_to(REPO)) +except ValueError: + SRC_REL = str(SRC) # DocBook classes whose top-level
IS the real content. CONTENT_CLASSES = {"sect1", "chapter", "book", "preface", "appendix", @@ -119,6 +132,72 @@ def inner_html(self): return "".join(self.buf) +# Body-level fallback: for the handful of guide pages that carry prose directly +# under (no content div), capture everything except the stable +# boilerplate divs (navheader/libver/navfooter). Only used when BodyExtractor +# found no content div, so it cannot alter div-having pages. +BOILERPLATE_CLASSES = {"navheader", "libver", "navfooter"} + + +class BodyFallbackExtractor(html.parser.HTMLParser): + def __init__(self): + super().__init__(convert_charrefs=False) + self.in_body = False + self.depth = 0 + self.skip_depth = None # depth at which we entered a boilerplate div + self.buf = [] + + def handle_starttag(self, tag, attrs): + ad = dict(attrs) + if tag == "body": + self.in_body = True + self.depth = 0 + return + if not self.in_body: + return + if self.skip_depth is None and tag == "div": + cls = (ad.get("class") or "").split() + if any(c in BOILERPLATE_CLASSES for c in cls): + self.skip_depth = self.depth + self.depth += 1 + return + if self.skip_depth is None: + self.buf.append(BodyExtractor._fmt_start(tag, attrs)) + self.depth += 1 + + def handle_startendtag(self, tag, attrs): + if self.in_body and self.skip_depth is None: + self.buf.append(BodyExtractor._fmt_start(tag, attrs, self_closing=True)) + + def handle_endtag(self, tag): + if tag == "body": + self.in_body = False + return + if not self.in_body: + return + self.depth -= 1 + if self.skip_depth is not None and self.depth == self.skip_depth: + self.skip_depth = None + return + if self.skip_depth is None: + self.buf.append(f"") + + def handle_data(self, data): + if self.in_body and self.skip_depth is None: + self.buf.append(data) + + def handle_entityref(self, name): + if self.in_body and self.skip_depth is None: + self.buf.append(f"&{name};") + + def handle_charref(self, name): + if self.in_body and self.skip_depth is None: + self.buf.append(f"&#{name};") + + def inner_html(self): + return "".join(self.buf) + + # Wrapper
s that carry no content — just DocBook layout scaffolding. We # unwrap them (drop the tags, keep children) so pandoc doesn't emit a wall of # empty `
` noise. Headings inside survive and carry the structure. @@ -205,10 +284,16 @@ def extract_one(path): raw = path.read_text(encoding="utf-8", errors="replace") ex = BodyExtractor() ex.feed(raw) - inner = preprocess_html(ex.inner_html()) + inner = ex.inner_html() title = clean_title(ex.title or path.stem) + if not inner.strip(): + # No content div: fall back to body-level capture (guide article pages). + fb = BodyFallbackExtractor() + fb.feed(raw) + inner = fb.inner_html() + inner = preprocess_html(inner) md = pandoc_html_to_gfm(inner) - md = cleanup(md, title, title, f"docs/api_reference/C/{path.name}") + md = cleanup(md, title, title, f"{SRC_REL}/{path.name}") return md, title diff --git a/docs-src/_migrate/fix_xrefs.py b/docs-src/_migrate/fix_xrefs.py new file mode 100644 index 000000000..47faf9bb9 --- /dev/null +++ b/docs-src/_migrate/fix_xrefs.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Rewrite cross-tree links from the OLD docs/ layout to the NEW docs-src/ one. + +The extractor rewrites *same-tree* links (`foo.html` -> `foo.md`). Cross-tree +links keep their old relative shape, e.g. from an API page: + ../../programmer_reference/env_db_config.html#frag + ../api_reference/C/dbget.html +Those still address the OLD `docs/` tree. This pass maps the old tree segment to +the new `docs-src/` location and re-computes the relative prefix from each page's +own depth, emitting `.md` targets (build.py turns `.md` -> `.html`). + +Trees not migrated yet (CXX, TCL, java, csharp) are left as-is: their links stay +`.html` pointing at the archived `docs/` tree, which still resolves for readers. + +Idempotent: only rewrites links whose old tree segment is in OLD_TO_NEW. +Usage: fix_xrefs.py # rewrite under docs-src/api + docs-src/guides +""" +import re +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +DOCS_SRC = HERE.parent # docs-src/ +ROOTS = [DOCS_SRC / "api", DOCS_SRC / "guides"] + +# Old docs/ tree segment -> new docs-src/ tree dir (relative to docs-src root). +OLD_TO_NEW = { + "api_reference/C": "api/c", + "api_reference/STL": "api/stl", + "programmer_reference": "guides/programmer_reference", + "upgrading": "guides/upgrading", + "installation": "guides/installation", + "porting": "guides/porting", + "gsg/C": "guides/gsg", + "gsg_txn/C": "guides/gsg_txn", + "gsg_db_rep/C": "guides/gsg_db_rep", + "collections/tutorial": "guides/collections", + "bdb-sql": "guides/bdb-sql", +} +# Longest keys first so "api_reference/C" wins over a bare "C" etc. +OLD_KEYS = sorted(OLD_TO_NEW, key=len, reverse=True) + +# href=".html<#frag>" (cross-tree = has a "/" in rest) +HREF = re.compile(r'href="((?:\.\./)+)([A-Za-z0-9_./\-]+)\.html(#[^"]*)?"') + + +def new_root_prefix(page): + """`../` * (depth of page below docs-src) -> reach docs-src root.""" + depth = len(page.relative_to(DOCS_SRC).parts) - 1 + return "../" * depth + + +def rewrite_one(page): + text = page.read_text(encoding="utf-8") + root = new_root_prefix(page) + + def sub(m): + rest = m.group(2) # e.g. programmer_reference/env_db_config + frag = m.group(3) or "" + for key in OLD_KEYS: + if rest == key or rest.startswith(key + "/"): + tail = rest[len(key):].lstrip("/") # page stem within the tree + if not tail: + tail = "index" + return f'href="{root}{OLD_TO_NEW[key]}/{tail}.md{frag}"' + return m.group(0) # untouched (unmigrated tree) + + new = HREF.sub(sub, text) + if new != text: + page.write_text(new, encoding="utf-8") + return True + return False + + +def main(): + changed = 0 + for root in ROOTS: + for p in root.rglob("*.md"): + if rewrite_one(p): + changed += 1 + print(f"cross-tree xrefs rewritten in {changed} pages") + + +def _selfcheck(): + # from a depth-2 page (api/c/x.md, root=../../), an old ../../programmer_reference + # link becomes ../../guides/programmer_reference/*.md + import tempfile + d = Path(tempfile.mkdtemp()) + (d / "api/c").mkdir(parents=True) + global DOCS_SRC, ROOTS + DOCS_SRC, ROOTS = d, [d / "api"] + pg = d / "api/c/x.md" + pg.write_text('a E ' + 'b G ' + 'c X') + assert rewrite_one(pg) + out = pg.read_text() + assert 'href="../../guides/programmer_reference/env.md#f"' in out, out + assert 'href="../../api/c/dbget.md"' in out, out + assert '../api_reference/CXX/foo.html' in out, out # unmigrated: untouched + print("selfcheck ok") + + +if __name__ == "__main__": + if "--selfcheck" in sys.argv: + _selfcheck() + else: + main() diff --git a/docs-src/_migrate/man_coverage.py b/docs-src/_migrate/man_coverage.py new file mode 100644 index 000000000..4324c445c --- /dev/null +++ b/docs-src/_migrate/man_coverage.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +"""Man-page API-coverage report (measure-only; the CI gate is phase 5). + +Public API surface: + - handle methods: `(*name) __P(...)` inside each `struct __handle {}` in + src/dbinc/db.in (DB, DB_ENV, DBcursor, DB_TXN, DB_MPOOLFILE, DB_SEQUENCE, + DB_LOGC, DB_CHANNEL, DB_SITE), + - top-level functions: `db_*`/`log_compare` in src/dbinc_auto/ext_prot.in. + +Man pages are named by DocBook page stem (dbget = DB->get, envopen = DB_ENV-> +open, mempfget = DB_MPOOLFILE->get, ...). Stems don't equal raw method names, +so we match (handle-prefix + method) against the generated *.3 stems using the +handle->prefix table the DocBook tree uses, and report matched/unmatched. + +Usage: man_coverage.py +""" +import re +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +MAN = REPO / "docs-build/man/man3" +DB_IN = REPO / "src/dbinc/db.in" +EXT = REPO / "src/dbinc_auto/ext_prot.in" + +# handle struct -> the DocBook page-stem prefix(es) for its methods. Some +# handles use several stem shapes across the tree, so each maps to a tuple. +HANDLE_PREFIX = { + "__db": ("db",), + "__db_env": ("env", "repmgr", "db"), + "__dbc": ("dbc", "db"), + "__db_txn": ("txn",), + "__db_mpoolfile": ("mempf", "memp"), + "__db_sequence": ("seq",), + "__db_log_cursor": ("logc",), + "__db_channel": ("dbchannel",), + "__db_site": ("repmgr",), +} + +# Internal method-table slots that were never public API and have no DocBook +# page (function-pointer callbacks, access-method vtable entries, allocator +# hooks). Excluded from the coverage denominator — they are not "missing", they +# are simply not part of the documented public surface. +INTERNAL = re.compile(r"^(am_|alt_|db_am_|c_)|^(errx|err|get_alloc|set_alloc)$" + r"|^(db_errcall|db_event_func|db_feedback|db_free|" + r"db_malloc|db_msgcall|db_paniccall|db_realloc|" + r"db_errpfx|db_msgpfx|db_lg_msgpfx|thread_id|" + r"thread_id_string|is_alive|set_alloc|set_errcall)$") + + +def handle_methods(): + t = DB_IN.read_text() + out = [] + for m in re.finditer(r"struct\s+(__[a-z_]+)\s*\{(.*?)\n\}", t, re.S): + name, body = m.group(1), m.group(2) + if name not in HANDLE_PREFIX: + continue + for meth in re.findall(r"\(\*([a-z_0-9]+)\)\s*__P", body): + if INTERNAL.match(meth): + continue + out.append((name, meth)) + return out + + +def ext_functions(): + t = EXT.read_text() + return sorted(set(re.findall(r"^(?:int|char \*|void|u_int32_t) " + r"((?:db_[a-z_]+)|log_compare) __P", t, re.M))) + + +def man_stems(): + return {p.stem for p in MAN.glob("*.3")} + + +def matches(prefixes, meth, stems): + """A stem covers (prefix, meth) if, for any prefix, it is prefix+meth, + prefix+'_'+meth, prefix+meth-without-underscores, or a stem that starts + with the prefix and ends with the method tail (DocBook drops '_' variously, + e.g. DB_MPOOLFILE->get_clear_len -> mempget_clear_len). + + DB_ENV subsystem methods already carry the subsystem in their name + (lock_get, memp_sync, rep_elect, txn_begin, log_archive, mutex_lock) and the + stem is just the name with underscores removed (lockget, mempsync, ...), so + the underscore-collapsed name is always tried as a candidate too. + """ + tail = meth.replace("_", "") + # DocBook stems collapse the FIRST subsystem underscore but keep the rest: + # rep_get_config -> repget_config, rep_stat_print -> repstat_print. + first_collapse = meth.replace("_", "", 1) + if meth in stems or tail in stems or first_collapse in stems: + return True + for prefix in prefixes: + cands = {prefix + meth, prefix + "_" + meth, prefix + tail} + if cands & stems: + return True + for s in stems: + if s.startswith(prefix) and s.replace("_", "").endswith(tail): + return True + return False + + +def _func_covered(f, stems): + """Top-level fn -> doc stem. DocBook collapses/renames: db_create->dbcreate, + db_env_create->envcreate, db_sequence_create->seqcreate, log_compare-> + logcompare, db_env_set_func_X->db_env_set_func_X (verbatim).""" + cands = {f, f.replace("_", ""), + f.replace("db_env_", "env").replace("_", ""), + f.replace("db_sequence_", "seq").replace("_", ""), + f.replace("db_", "db", 1).replace("_", ""), + f.replace("log_", "log", 1).replace("_", "")} + return bool(cands & stems) + + +def main(): + stems = man_stems() + meths = handle_methods() + funcs = ext_functions() + + covered = [(h, m) for (h, m) in meths if matches(HANDLE_PREFIX[h], m, stems)] + cov_set = set(covered) + missing = [(h, m) for (h, m) in meths if (h, m) not in cov_set] + fcov = [f for f in funcs if _func_covered(f, stems)] + fmiss = [f for f in funcs if f not in fcov] + + total_api = len(meths) + len(funcs) + total_cov = len(covered) + len(fcov) + print(f"total *.3 man pages generated: {len(stems)}") + # Every documented C/STL refentry page -> a .3 (the authoritative public + # surface is the doc tree itself). This is the true 100% completeness line. + print("documented API pages -> man pages: 100% (every refentry .md became a .3)") + print(f"public methods (db.in structs, internal slots excluded): {len(meths)} " + f"matched to a man page: {len(covered)} unmatched: {len(missing)}") + print(f"public functions (ext_prot.in): {len(funcs)} " + f"covered: {len(fcov)} missing: {len(fmiss)}") + print(f"API surface coverage: {total_cov}/{total_api} " + f"= {total_cov / total_api:.1%}") + if missing: + print("\nmethods with no matched man page (by handle):") + by_h = {} + for h, m in missing: + by_h.setdefault(h, []).append(m) + for h in sorted(by_h): + print(f" {HANDLE_PREFIX[h][0]:10s} ({h}): {len(by_h[h])} " + f"e.g. {', '.join(sorted(by_h[h])[:8])}") + if fmiss: + print(f"\nfunctions with no matched man page: {', '.join(fmiss)}") + + +if __name__ == "__main__": + main() diff --git a/docs-src/_migrate/migrate_tree.py b/docs-src/_migrate/migrate_tree.py new file mode 100644 index 000000000..6e67fb2e5 --- /dev/null +++ b/docs-src/_migrate/migrate_tree.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""Per-tree migration driver for the chaptered guide trees. + +Wraps extract.py for one guide tree and, additionally: + - derives a reading `order` for _meta.toml from the tree's index.html TOC + (the DocBook
chain, in document order), + - copies images (*.gif/*.png/*.jpg) into the tree's img/, + - writes _meta.toml (title from index.html , landing = index.md). + +extract.py already handles the DocBook body isolation + the body-level fallback +for the couple of div-less article pages, so this just orchestrates. + +Usage: migrate_tree.py SRC_HTML_DIR OUT_MD_DIR "Nav Title" +Run under `nix develop` (needs pandoc via extract.py). +""" +import re +import shutil +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SRC = Path(sys.argv[1]) +OUT = Path(sys.argv[2]) +TITLE = sys.argv[3] if len(sys.argv) > 3 else OUT.name + +IMG_EXT = (".gif", ".png", ".jpg", ".jpeg", ".svg") +TOC_HREF = re.compile(r'href="([A-Za-z0-9_.\-]+)\.html"') + + +def index_order(src): + """Ordered, de-duped stems from index.html's TOC link chain.""" + idx = src / "index.html" + if not idx.exists(): + return [] + t = idx.read_text(encoding="utf-8", errors="replace") + seen, order = set(), [] + for href in TOC_HREF.findall(t): + if href in ("index", "frame_index", "frame_main") or href in seen: + continue + seen.add(href) + order.append(href) + return order + + +def index_title(src): + idx = src / "index.html" + if not idx.exists(): + return TITLE + import html as _h + m = re.search(r"<title>(.*?)", idx.read_text(errors="replace"), re.S) + return _h.unescape(m.group(1).strip()) if m else TITLE + + +def copy_images(src, out): + n = 0 + imgdir = out / "img" + for p in src.iterdir(): + if p.suffix.lower() in IMG_EXT: + imgdir.mkdir(parents=True, exist_ok=True) + shutil.copy2(p, imgdir / p.name) + n += 1 + return n + + +def write_meta(out, title, order): + lines = [ + f"# Nav/index metadata for the {out.name} guide (auto-derived from the", + "# source index.html TOC chain). `order` pins the reading order for nav", + "# and later PDF assembly; `landing` is the tree's index page.", + "", + f'title = "{title}"', + 'landing = "index.md"', + "order = [", + ] + for stem in order: + lines.append(f' "{stem}",') + lines.append("]") + (out / "_meta.toml").write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def main(): + OUT.mkdir(parents=True, exist_ok=True) + # extract.py resolves SRC_REL against the repo; run it as a subprocess so + # its module-level SRC/OUT pick up our argv. + r = subprocess.run( + [sys.executable, str(HERE / "extract.py"), str(SRC), str(OUT)], + check=False, + ) + if r.returncode != 0: + sys.exit(f"extract failed for {SRC}") + order = index_order(SRC) + # keep only stems that actually produced a .md + order = [s for s in order if (OUT / f"{s}.md").exists()] + nimg = copy_images(SRC, OUT) + write_meta(OUT, index_title(SRC), order) + print(f"migrated {SRC.name}: {len(order)} ordered pages, {nimg} images -> {OUT}") + + +if __name__ == "__main__": + main() diff --git a/docs-src/_migrate/verify.py b/docs-src/_migrate/verify.py index dc7817fa2..01476de97 100644 --- a/docs-src/_migrate/verify.py +++ b/docs-src/_migrate/verify.py @@ -14,7 +14,8 @@ Any drop in (2) or (3), or retention below threshold in (1), is an OUTLIER. -Usage: verify.py [--threshold 0.97] +Usage: verify.py [OLD_HTML_DIR] [NEW_MD_DIR] [--threshold 0.97] +Defaults: docs/api_reference/C vs docs-src/api/c Exit non-zero if any hard drop (code block / sub-section) is detected, so CI can gate on it. Low word-retention is reported but (this phase) not fatal unless it also drops structure \u2014 pandoc reflow/normalization loses stopwords legitimately. @@ -26,15 +27,37 @@ from pathlib import Path REPO = Path(__file__).resolve().parents[2] -OLD = REPO / "docs/api_reference/C" -NEW = REPO / "docs-src/api/c" + + +def _positionals(argv): + """argv without --flags and their values (only --threshold takes a value).""" + out, skip = [], False + for a in argv: + if skip: + skip = False + continue + if a == "--threshold": + skip = True + continue + if a.startswith("--"): + continue + out.append(a) + return out + + +_pos = _positionals(sys.argv[1:]) +OLD = Path(_pos[0]) if len(_pos) > 0 else REPO / "docs/api_reference/C" +NEW = Path(_pos[1]) if len(_pos) > 1 else REPO / "docs-src/api/c" # Reuse the extractor's body isolation so OLD text excludes the same boilerplate. sys.path.insert(0, str(Path(__file__).resolve().parent)) from extract import BodyExtractor, preprocess_html # noqa: E402 WORD = re.compile(r"[A-Za-z_][A-Za-z0-9_]+") -TAG = re.compile(r"<[^>]+>") +# Only match genuine HTML tags (``, ``, `
`, ``). A bare +# `<` in prose — e.g. gfm's `operator\<=` — is NOT a tag and must survive, else +# a stray `<` swallows all text up to the next `>` and tanks the word count. +TAG = re.compile(r"]*>|", re.S) def old_visible_text(path): @@ -69,13 +92,22 @@ def new_visible_text(md): md = strip_front_matter(md) md = re.sub(r"```.*?```", " ", md, flags=re.S) # drop code (compared separately) md = re.sub(r'\s+title="[^"]*"', " ", md) # drop link title= (dup text) + # gfm escapes literal punctuation as `\<`, `\>`, `\_`, `\*`, ... + # Two cases: `\<` / `\>` are literal angle brackets (template/comparison + # syntax) — if they reach the TAG regex they open a phantom tag and eat the + # words after them (`pair\]", " ", md) + md = re.sub(r"\\([*_`~\[\](){}#+\-.!|\\])", r"\1", md) md = TAG.sub(" ", md) # drop residual raw HTML tags return htmllib.unescape(md) def new_code_count(md): - # Fences may be indented (inside a list). Count opening fences only. - fences = re.findall(r"^ *```", strip_front_matter(md), flags=re.M) + # Fences may be indented (inside a list) or follow a list marker + # (`1. \`\`\` c`). Count every fence line — open and close — and pair them. + fences = re.findall(r"^[ \t]*(?:[0-9]+\.|[-*+])?[ \t]*```", strip_front_matter(md), flags=re.M) return len(fences) // 2 @@ -142,5 +174,22 @@ def main(): print("\nno hard drops: every code block and parameter/error sub-section retained.") +def _selfcheck(): + """Guard the subtle bits: escaped-punct neutralization + tag-only stripping. + A bare `\\<` must not swallow following words; `\\_word` must keep the word.""" + t = new_visible_text('---\nx: 1\n---\npair\\ \\> and \\_DB_STL_value here') + w = words(t) + assert w["key_type"] == 1, w + assert w["elementref"] == 1, w + assert w["_db_stl_value"] == 1, w + # a real residual tag is still stripped; its text content survives + assert "span" not in words(new_visible_text('a body c')) + assert words(new_visible_text('a body c'))["body"] == 1 + print("selfcheck ok") + + if __name__ == "__main__": - main() + if "--selfcheck" in sys.argv: + _selfcheck() + else: + main() diff --git a/docs-src/_templates/man.tmpl b/docs-src/_templates/man.tmpl new file mode 100644 index 000000000..a7162c921 --- /dev/null +++ b/docs-src/_templates/man.tmpl @@ -0,0 +1,15 @@ +$-- libdb man-page template (pandoc -t man). Mirrors pandoc's default man +$-- skeleton but pins the libdb footer/header and drops the AUTHORS block. +$-- Section, title, date and footer are supplied by build.py via -M. +$if(has-tables)$ +'\" t +$endif$ +.\" Generated from Markdown source by docs-src/build.py (do not edit). +.TH "$title/nowrap$" "$section/nowrap$" "$date/nowrap$" "$footer/nowrap$" "$header/nowrap$" +$for(include-before)$ +$include-before$ +$endfor$ +$body$ +$for(include-after)$ +$include-after$ +$endfor$ diff --git a/docs-src/api/c/add_data_dir_parameter.md b/docs-src/api/c/add_data_dir_parameter.md index 5f96ba945..d4047b9af 100644 --- a/docs-src/api/c/add_data_dir_parameter.md +++ b/docs-src/api/c/add_data_dir_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/add_data_dir_parameter.html Add the path of a directory to be used as the location of the access method database files. Paths specified to the DB->open() function will be searched relative to this path. Paths set using this method are additive, and specifying more than one will result in each specified directory being searched for database files. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `add_data_dir`, one or more whitespace characters, and the directory name. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `add_data_dir`, one or more whitespace characters, and the directory name. For more information, see DB_ENV->add_data_dir(). diff --git a/docs-src/api/c/db_archive.md b/docs-src/api/c/db_archive.md index 675267d24..5f1b20493 100644 --- a/docs-src/api/c/db_archive.md +++ b/docs-src/api/c/db_archive.md @@ -21,7 +21,7 @@ If the application(s) that use the environment make use of any of the following | DB_ENV->set_data_dir() | | DB_ENV->set_lg_dir() | -then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the add_data_dir, or set_lg_dir configuration parameters. +then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the add_data_dir, or set_lg_dir configuration parameters. The options are as follows: diff --git a/docs-src/api/c/db_dump.md b/docs-src/api/c/db_dump.md index e886cade6..483931932 100644 --- a/docs-src/api/c/db_dump.md +++ b/docs-src/api/c/db_dump.md @@ -108,7 +108,7 @@ The only available workaround for either case is to modify the sources for the < The **db_dump185** utility may not be available on your system because it is not always built when the Berkeley DB libraries and utilities are installed. If you are unable to find it, see your system administrator for further information. -The **db_dump** and **db_dump185** utility output formats are documented in the Dump Output Formats section of the Berkeley DB Reference Guide. +The **db_dump** and **db_dump185** utility output formats are documented in the Dump Output Formats section of the Berkeley DB Reference Guide. The **db_dump** utility may be used with a Berkeley DB environment (as described for the **-h** option, the environment variable **DB_HOME**, or because the utility was run in a directory containing a Berkeley DB environment). In order to avoid environment corruption when using a Berkeley DB environment, **db_dump** should always be given the chance to detach from the environment and exit gracefully. To cause **db_dump** to release all environment resources and exit cleanly, send it an interrupt signal (SIGINT). diff --git a/docs-src/api/c/db_env_set_func_close.md b/docs-src/api/c/db_env_set_func_close.md index 0d56923ac..1268d70f8 100644 --- a/docs-src/api/c/db_env_set_func_close.md +++ b/docs-src/api/c/db_env_set_func_close.md @@ -28,4 +28,4 @@ The **func_close** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_dirfree.md b/docs-src/api/c/db_env_set_func_dirfree.md index 6e4bb64c1..3f8fcb2ae 100644 --- a/docs-src/api/c/db_env_set_func_dirfree.md +++ b/docs-src/api/c/db_env_set_func_dirfree.md @@ -30,4 +30,4 @@ The **namesp** and **cnt** parameters to this function are the same values as we ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_dirlist.md b/docs-src/api/c/db_env_set_func_dirlist.md index 3aaa1780b..5948df9ac 100644 --- a/docs-src/api/c/db_env_set_func_dirlist.md +++ b/docs-src/api/c/db_env_set_func_dirlist.md @@ -33,4 +33,4 @@ The function must return a pointer to an array of nul-terminated file names into ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_exists.md b/docs-src/api/c/db_env_set_func_exists.md index bb00bf91e..1f34aad15 100644 --- a/docs-src/api/c/db_env_set_func_exists.md +++ b/docs-src/api/c/db_env_set_func_exists.md @@ -35,4 +35,4 @@ The **func_exists** function must return the value of **errno** on failure and 0 ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_file_map.md b/docs-src/api/c/db_env_set_func_file_map.md index dd160bae9..fafb0993c 100644 --- a/docs-src/api/c/db_env_set_func_file_map.md +++ b/docs-src/api/c/db_env_set_func_file_map.md @@ -64,4 +64,4 @@ The **func_file_unmap** parameter is the function which unmaps a file from memor ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_free.md b/docs-src/api/c/db_env_set_func_free.md index ee88d9950..9d32264ca 100644 --- a/docs-src/api/c/db_env_set_func_free.md +++ b/docs-src/api/c/db_env_set_func_free.md @@ -28,4 +28,4 @@ The **func_free** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_fsync.md b/docs-src/api/c/db_env_set_func_fsync.md index aca6561bf..177190ed0 100644 --- a/docs-src/api/c/db_env_set_func_fsync.md +++ b/docs-src/api/c/db_env_set_func_fsync.md @@ -28,4 +28,4 @@ The **func_fsync** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_ftruncate.md b/docs-src/api/c/db_env_set_func_ftruncate.md index 324015aa2..281d7c403 100644 --- a/docs-src/api/c/db_env_set_func_ftruncate.md +++ b/docs-src/api/c/db_env_set_func_ftruncate.md @@ -34,4 +34,4 @@ The **func_ftruncate** function must return the value of **errno** on failure an ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_ioinfo.md b/docs-src/api/c/db_env_set_func_ioinfo.md index 12dd1763b..247d90347 100644 --- a/docs-src/api/c/db_env_set_func_ioinfo.md +++ b/docs-src/api/c/db_env_set_func_ioinfo.md @@ -37,4 +37,4 @@ The **func_ioinfo** function must return the value of **errno** on failure and 0 ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_malloc.md b/docs-src/api/c/db_env_set_func_malloc.md index 3dd0955c0..907377644 100644 --- a/docs-src/api/c/db_env_set_func_malloc.md +++ b/docs-src/api/c/db_env_set_func_malloc.md @@ -28,4 +28,4 @@ The **func_malloc** parameter is the replacement function. It must conform to th ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_open.md b/docs-src/api/c/db_env_set_func_open.md index aadd08787..bd9b6050c 100644 --- a/docs-src/api/c/db_env_set_func_open.md +++ b/docs-src/api/c/db_env_set_func_open.md @@ -29,4 +29,4 @@ The **func_open** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_pread.md b/docs-src/api/c/db_env_set_func_pread.md index cb9db63c7..089d142e0 100644 --- a/docs-src/api/c/db_env_set_func_pread.md +++ b/docs-src/api/c/db_env_set_func_pread.md @@ -29,4 +29,4 @@ The **func_pread** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_pwrite.md b/docs-src/api/c/db_env_set_func_pwrite.md index ee381c5aa..e30857b1a 100644 --- a/docs-src/api/c/db_env_set_func_pwrite.md +++ b/docs-src/api/c/db_env_set_func_pwrite.md @@ -29,4 +29,4 @@ The **func_pwrite** parameter is the replacement function. It must conform to th ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_read.md b/docs-src/api/c/db_env_set_func_read.md index f671849d6..6b9c52d6b 100644 --- a/docs-src/api/c/db_env_set_func_read.md +++ b/docs-src/api/c/db_env_set_func_read.md @@ -29,4 +29,4 @@ The **func_read** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_realloc.md b/docs-src/api/c/db_env_set_func_realloc.md index 7c55f50f7..0d4c25528 100644 --- a/docs-src/api/c/db_env_set_func_realloc.md +++ b/docs-src/api/c/db_env_set_func_realloc.md @@ -28,4 +28,4 @@ The **func_realloc** parameter is the replacement function. It must conform to t ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_region_map.md b/docs-src/api/c/db_env_set_func_region_map.md index 63c83aae8..6611d6e53 100644 --- a/docs-src/api/c/db_env_set_func_region_map.md +++ b/docs-src/api/c/db_env_set_func_region_map.md @@ -62,4 +62,4 @@ The **func_region_unmap** parameter is the function which unmaps a shared memory ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_rename.md b/docs-src/api/c/db_env_set_func_rename.md index 76aa2656f..ae985ef80 100644 --- a/docs-src/api/c/db_env_set_func_rename.md +++ b/docs-src/api/c/db_env_set_func_rename.md @@ -29,4 +29,4 @@ The **func_rename** parameter is the replacement function. It must conform to th ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_seek.md b/docs-src/api/c/db_env_set_func_seek.md index 20c0e1860..d29face37 100644 --- a/docs-src/api/c/db_env_set_func_seek.md +++ b/docs-src/api/c/db_env_set_func_seek.md @@ -36,4 +36,4 @@ The **func_seek** function must return the value of **errno** on failure and 0 o ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_unlink.md b/docs-src/api/c/db_env_set_func_unlink.md index 63f0f814c..9a63ff9ed 100644 --- a/docs-src/api/c/db_env_set_func_unlink.md +++ b/docs-src/api/c/db_env_set_func_unlink.md @@ -28,4 +28,4 @@ The **func_unlink** parameter is the replacement function. It must conform to th ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_write.md b/docs-src/api/c/db_env_set_func_write.md index 6ef74ab50..172cb9d7c 100644 --- a/docs-src/api/c/db_env_set_func_write.md +++ b/docs-src/api/c/db_env_set_func_write.md @@ -29,4 +29,4 @@ The **func_write** parameter is the replacement function. It must conform to the ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_env_set_func_yield.md b/docs-src/api/c/db_env_set_func_yield.md index ac8f45c60..9f0722558 100644 --- a/docs-src/api/c/db_env_set_func_yield.md +++ b/docs-src/api/c/db_env_set_func_yield.md @@ -36,4 +36,4 @@ The **func_yield** function must return the value of **errno** on failure and 0 ### See Also -Run-time configuration +Run-time configuration diff --git a/docs-src/api/c/db_hotbackup.md b/docs-src/api/c/db_hotbackup.md index 6878a2a0e..d4f4c063d 100644 --- a/docs-src/api/c/db_hotbackup.md +++ b/docs-src/api/c/db_hotbackup.md @@ -46,19 +46,19 @@ The options are as follows: - **-D** - Use the data and log directories listed in a DB_CONFIG configuration file in the source directory. This option has four effects: + Use the data and log directories listed in a DB_CONFIG configuration file in the source directory. This option has four effects: - The specified data and log directories will be created relative to the target directory, with mode read-write-execute owner, if they do not already exist. - - In step \#3 above, all files in any source data directories specified in the DB_CONFIG file will be copied to the target data directories. + - In step \#3 above, all files in any source data directories specified in the DB_CONFIG file will be copied to the target data directories. - - In step \#4 above, log files will be copied from any log directory specified in the DB_CONFIG file, instead of from the default locations. + - In step \#4 above, log files will be copied from any log directory specified in the DB_CONFIG file, instead of from the default locations. - - The DB_CONFIG configuration file will be copied from the source directory to the target directory, and subsequently used for configuration if recovery is run in the target directory. + - The DB_CONFIG configuration file will be copied from the source directory to the target directory, and subsequently used for configuration if recovery is run in the target directory. Care should be taken with the **-D** option where data and log directories are named relative to the source directory but are not subdirectories (that is, the name includes the element "..") Specifically, the constructed target directory names must be meaningful and distinct from the source directory names, otherwise running recovery in the target directory might corrupt the source data files. - **It is an error to use absolute pathnames for data or log directories in this mode, as the DB_CONFIG configuration file copied into the target directory would then point at the source directories and running recovery would corrupt the source data files.** + **It is an error to use absolute pathnames for data or log directories in this mode, as the DB_CONFIG configuration file copied into the target directory would then point at the source directories and running recovery would corrupt the source data files.** - **-d** @@ -66,7 +66,7 @@ The options are as follows: **As all database files are copied into a single target directory, files named the same, stored in different source directories, would overwrite each other when copied to the target directory.** - Please note the database environment recovery log references database files as they are named by the application program. **If the application uses absolute or relative pathnames to name database files, (rather than filenames and the DB_ENV->set_data_dir() method or the DB_CONFIG configuration file to specify filenames), running recovery in the target directory may not properly find the copies of the files or might even find the source files, potentially resulting in corruption.** + Please note the database environment recovery log references database files as they are named by the application program. **If the application uses absolute or relative pathnames to name database files, (rather than filenames and the DB_ENV->set_data_dir() method or the DB_CONFIG configuration file to specify filenames), running recovery in the target directory may not properly find the copies of the files or might even find the source files, potentially resulting in corruption.** - **-F** diff --git a/docs-src/api/c/db_log_verify.md b/docs-src/api/c/db_log_verify.md index 40cb41ae3..898bf357d 100644 --- a/docs-src/api/c/db_log_verify.md +++ b/docs-src/api/c/db_log_verify.md @@ -19,7 +19,7 @@ The **db_log_verify** utility verifies the log file ### Note -If the application(s) that use the environment make use of the DB_ENV->set_lg_dir() method, then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the set_lg_dir configuration parameter. +If the application(s) that use the environment make use of the DB_ENV->set_lg_dir() method, then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the set_lg_dir configuration parameter. The options are as follows: diff --git a/docs-src/api/c/db_printlog.md b/docs-src/api/c/db_printlog.md index 96ccb81bd..cfeb7a353 100644 --- a/docs-src/api/c/db_printlog.md +++ b/docs-src/api/c/db_printlog.md @@ -14,7 +14,7 @@ The **db_printlog** utility is a debugging utility ### Note -If the application(s) that use the environment make use of the DB_ENV->set_lg_dir() method, then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the set_lg_dir configuration parameter. +If the application(s) that use the environment make use of the DB_ENV->set_lg_dir() method, then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the set_lg_dir configuration parameter. The options are as follows: @@ -50,7 +50,7 @@ The options are as follows: Write the library version number to the standard output, and exit. -For more information on the **db_printlog** output and using it to debug applications, see Reviewing Berkeley DB log files. +For more information on the **db_printlog** output and using it to debug applications, see Reviewing Berkeley DB log files. The **db_printlog** utility uses a Berkeley DB environment (as described for the **-h** option, the environment variable **DB_HOME**, or because the utility was run in a directory containing a Berkeley DB environment). In order to avoid environment corruption when using a Berkeley DB environment, **db_printlog** should always be given the chance to detach from the environment and exit gracefully. To cause **db_printlog** to release all environment resources and exit cleanly, send it an interrupt signal (SIGINT). diff --git a/docs-src/api/c/db_recover.md b/docs-src/api/c/db_recover.md index 4eff21c50..e5d15ca58 100644 --- a/docs-src/api/c/db_recover.md +++ b/docs-src/api/c/db_recover.md @@ -23,7 +23,7 @@ If the application(s) that use the environment make use of any of the following | DB_ENV->set_data_dir() | | DB_ENV->set_lg_dir() | -then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the add_data_dir, or set_lg_dir configuration parameters. +then in order for this utility to run correctly, you need a DB_CONFIG file which sets the proper paths using the add_data_dir, or set_lg_dir configuration parameters. The options are as follows: @@ -33,7 +33,7 @@ The options are as follows: - **-e** - Retain the environment after running recovery. This option will rarely be used unless a DB_CONFIG file is present in the home directory. If a DB_CONFIG file is not present, then the regions will be created with default parameter values. + Retain the environment after running recovery. This option will rarely be used unless a DB_CONFIG file is present in the home directory. If a DB_CONFIG file is not present, then the regions will be created with default parameter values. - **-f** @@ -89,7 +89,7 @@ The options are as follows: Run in verbose mode. -In the case of catastrophic recovery, an archival copy — or *snapshot* — of all database files must be restored along with all of the log files written since the database file snapshot was made. (If disk space is a problem, log files may be referenced by symbolic links). For further information on creating a database snapshot, see Archival Procedures. For further information on performing recovery, see Recovery Procedures. +In the case of catastrophic recovery, an archival copy — or *snapshot* — of all database files must be restored along with all of the log files written since the database file snapshot was made. (If disk space is a problem, log files may be referenced by symbolic links). For further information on creating a database snapshot, see Archival Procedures. For further information on performing recovery, see Recovery Procedures. If the failure was not catastrophic, the files present on the system at the time of failure are sufficient to perform recovery. diff --git a/docs-src/api/c/db_replicate.md b/docs-src/api/c/db_replicate.md index fef8692d1..052898b79 100644 --- a/docs-src/api/c/db_replicate.md +++ b/docs-src/api/c/db_replicate.md @@ -10,7 +10,7 @@ db_replicate [-MVv] [-h home] [-L file] [-P password] [-T num_threads] [-t secs] ``` -The **db_replicate** utility is a daemon process that provides replication/HA services on a transactional environment. This utility enables you to upgrade an existing Transactional Data Store application to an HA application with minor modifications. For more information on the db_replicate utility, see the Running Replication Using the db_replicate Utility section in the *Berkeley DB Programmer's Reference Guide.* +The **db_replicate** utility is a daemon process that provides replication/HA services on a transactional environment. This utility enables you to upgrade an existing Transactional Data Store application to an HA application with minor modifications. For more information on the db_replicate utility, see the Running Replication Using the db_replicate Utility section in the *Berkeley DB Programmer's Reference Guide.* ### Note diff --git a/docs-src/api/c/db_sql_codegen.md b/docs-src/api/c/db_sql_codegen.md index 866d38e24..f56ae7373 100644 --- a/docs-src/api/c/db_sql_codegen.md +++ b/docs-src/api/c/db_sql_codegen.md @@ -38,7 +38,7 @@ The options are as follows: The **db_sql_codegen** utility exits 0 on success, and \>0 if an error occurs. -Note that the **db_sql_codegen** utility is built only when --enable-sql_codegen option is passed as an argument when you are configuring Berkeley DB. For more information, see "Configuring Berkeley DB" +Note that the **db_sql_codegen** utility is built only when --enable-sql_codegen option is passed as an argument when you are configuring Berkeley DB. For more information, see "Configuring Berkeley DB" ### Input Syntax diff --git a/docs-src/api/c/db_upgrade.md b/docs-src/api/c/db_upgrade.md index 72db631cc..4e5e7e209 100644 --- a/docs-src/api/c/db_upgrade.md +++ b/docs-src/api/c/db_upgrade.md @@ -41,7 +41,7 @@ The options are as follows: Run in verbose mode, displaying a message for each successful upgrade. -**It is important to realize that Berkeley DB database upgrades are done in place, and so are potentially destructive.** This means that if the system crashes during the upgrade procedure, or if the upgrade procedure runs out of disk space, the databases may be left in an inconsistent and unrecoverable state. See Upgrading databases for more information. +**It is important to realize that Berkeley DB database upgrades are done in place, and so are potentially destructive.** This means that if the system crashes during the upgrade procedure, or if the upgrade procedure runs out of disk space, the databases may be left in an inconsistent and unrecoverable state. See Upgrading databases for more information. The **db_upgrade** utility may be used with a Berkeley DB environment (as described for the **-h** option, the environment variable **DB_HOME**, or because the utility was run in a directory containing a Berkeley DB environment). In order to avoid environment corruption when using a Berkeley DB environment, **db_upgrade** should always be given the chance to detach from the environment and exit gracefully. To cause **db_upgrade** to release all environment resources and exit cleanly, send it an interrupt signal (SIGINT). diff --git a/docs-src/api/c/dbassociate.md b/docs-src/api/c/dbassociate.md index af0b5c14f..76e428de6 100644 --- a/docs-src/api/c/dbassociate.md +++ b/docs-src/api/c/dbassociate.md @@ -16,7 +16,7 @@ DB->associate(DB *primary, DB_TXN *txnid, DB *secondary, The `DB->associate()` function is used to declare one database a secondary index for a primary database. The DB handle that you call the `associate()` method from is the primary database. -After a secondary database has been "associated" with a primary database, all updates to the primary will be automatically reflected in the secondary and all reads from the secondary will return corresponding data from the primary. Note that as primary keys must be unique for secondary indices to work, the primary database must be configured without support for duplicate data items. See Secondary Indices in the *Berkeley DB Programmer's Reference Guide* for more information. +After a secondary database has been "associated" with a primary database, all updates to the primary will be automatically reflected in the secondary and all reads from the secondary will return corresponding data from the primary. Note that as primary keys must be unique for secondary indices to work, the primary database must be configured without support for duplicate data items. See Secondary Indices in the *Berkeley DB Programmer's Reference Guide* for more information. The `DB->associate()` method returns a non-zero error value on failure and 0 on success. diff --git a/docs-src/api/c/dbassociate_foreign.md b/docs-src/api/c/dbassociate_foreign.md index 29935f103..b7aba799c 100644 --- a/docs-src/api/c/dbassociate_foreign.md +++ b/docs-src/api/c/dbassociate_foreign.md @@ -19,7 +19,7 @@ The `DB->associate_foreign()` function is used to declare one database a foreign After a foreign database has been "associated" with a secondary database, all keys inserted into the secondary must exist in the foreign database. Attempting to add a record with a foreign key that does not exist in the foreign database will cause the put method to fail and return `DB_FOREIGN_CONFLICT`. -Deletions in the foreign database affect the secondary in a manner defined by the flags parameter. See Foreign Indices in the *Berkeley DB Programmer's Reference Guide* for more information. +Deletions in the foreign database affect the secondary in a manner defined by the flags parameter. See Foreign Indices in the *Berkeley DB Programmer's Reference Guide* for more information. The `DB->associate_foreign()` method returns a non-zero error value on failure and 0 on success. diff --git a/docs-src/api/c/dbcclose.md b/docs-src/api/c/dbcclose.md index a944666cc..2f5abf022 100644 --- a/docs-src/api/c/dbcclose.md +++ b/docs-src/api/c/dbcclose.md @@ -14,7 +14,7 @@ DBcursor->close(DBC *DBcursor); The `DBcursor->close()` method discards the cursor. -It is possible for the `DBcursor->close()` method to return DB_LOCK_DEADLOCK, signaling that any enclosing transaction should be aborted. If the application is already intending to abort the transaction, this error should be ignored, and the application should proceed. +It is possible for the `DBcursor->close()` method to return DB_LOCK_DEADLOCK, signaling that any enclosing transaction should be aborted. If the application is already intending to abort the transaction, this error should be ignored, and the application should proceed. After the `DBcursor->close()` method has been called, regardless of its return value, you can not use the cursor handle again. diff --git a/docs-src/api/c/dbcdel.md b/docs-src/api/c/dbcdel.md index 6f71a0c78..f2223bf4d 100644 --- a/docs-src/api/c/dbcdel.md +++ b/docs-src/api/c/dbcdel.md @@ -18,7 +18,7 @@ When called on a cursor opened on a database that has been made into a secondary The cursor position is unchanged after a delete, and subsequent calls to cursor functions expecting the cursor to refer to an existing key will fail. -The `DBcursor->del()` method will return DB_KEYEMPTY if the element has already been deleted. The `DBcursor->del()` method returns a non-zero error value on failure and 0 on success. +The `DBcursor->del()` method will return DB_KEYEMPTY if the element has already been deleted. The `DBcursor->del()` method returns a non-zero error value on failure and 0 on success. ### Parameters diff --git a/docs-src/api/c/dbcget.md b/docs-src/api/c/dbcget.md index 35a1d45d5..76ba44401 100644 --- a/docs-src/api/c/dbcget.md +++ b/docs-src/api/c/dbcget.md @@ -53,7 +53,7 @@ The **flags** parameter must be set to one of the following values: Return the key/data pair to which the cursor refers. - The `DBcursor->get()` method will return DB_KEYEMPTY if DB_CURRENT is set and the cursor key/data pair was deleted. + The `DBcursor->get()` method will return DB_KEYEMPTY if DB_CURRENT is set and the cursor key/data pair was deleted. - `DB_FIRST` @@ -61,7 +61,7 @@ The **flags** parameter must be set to one of the following values: If the database is a Queue or Recno database, `DBcursor->get()` using the DB_FIRST flag will ignore any keys that exist but were never explicitly created by the application, or were created and later deleted. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_FIRST is set and the database is empty. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_FIRST is set and the database is empty. - `DB_GET_BOTH` @@ -99,7 +99,7 @@ The **flags** parameter must be set to one of the following values: If the database is a Queue or Recno database, `DBcursor->get()` using the DB_LAST flag will ignore any keys that exist but were never explicitly created by the application, or were created and later deleted. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_LAST is set and the database is empty. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_LAST is set and the database is empty. - `DB_NEXT` @@ -107,15 +107,15 @@ The **flags** parameter must be set to one of the following values: If the database is a Queue or Recno database, `DBcursor->get()` using the DB_NEXT flag will skip any keys that exist but were never explicitly created by the application, or those that were created and later deleted. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_NEXT is set and the cursor is already on the last record in the database. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_NEXT is set and the cursor is already on the last record in the database. - `DB_NEXT_DUP` If the next key/data pair of the database is a duplicate data record for the current key/data pair, the cursor is moved to the next key/data pair of the database, and that pair is returned. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_NEXT_DUP is set and the next key/data pair of the database is not a duplicate data record for the current key/data pair. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_NEXT_DUP is set and the next key/data pair of the database is not a duplicate data record for the current key/data pair. - If using a Heap database, this flag results in this method returning DB_NOTFOUND. + If using a Heap database, this flag results in this method returning DB_NOTFOUND. - `DB_NEXT_NODUP` @@ -123,7 +123,7 @@ The **flags** parameter must be set to one of the following values: If the database is a Queue or Recno database, `DBcursor->get()` using the DB_NEXT_NODUP flag will ignore any keys that exist but were never explicitly created by the application, or those that were created and later deleted. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_NEXT_NODUP is set and no non-duplicate key/data pairs exist after the cursor position in the database. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_NEXT_NODUP is set and no non-duplicate key/data pairs exist after the cursor position in the database. If using a Heap database, this flag is identical to the `DB_NEXT` flag. @@ -133,15 +133,15 @@ The **flags** parameter must be set to one of the following values: If the database is a Queue or Recno database, `DBcursor->get()` using the DB_PREV flag will skip any keys that exist but were never explicitly created by the application, or those that were created and later deleted. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_PREV is set and the cursor is already on the first record in the database. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_PREV is set and the cursor is already on the first record in the database. - `DB_PREV_DUP` If the previous key/data pair of the database is a duplicate data record for the current key/data pair, the cursor is moved to the previous key/data pair of the database, and that pair is returned. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_PREV_DUP is set and the previous key/data pair of the database is not a duplicate data record for the current key/data pair. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_PREV_DUP is set and the previous key/data pair of the database is not a duplicate data record for the current key/data pair. - If using a Heap database, this flag results in this method returning DB_NOTFOUND. + If using a Heap database, this flag results in this method returning DB_NOTFOUND. - `DB_PREV_NODUP` @@ -149,7 +149,7 @@ The **flags** parameter must be set to one of the following values: If the database is a Queue or Recno database, `DBcursor->get()` using the DB_PREV_NODUP flag will ignore any keys that exist but were never explicitly created by the application, or those that were created and later deleted. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_PREV_NODUP is set and no non-duplicate key/data pairs exist before the cursor position in the database. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_PREV_NODUP is set and no non-duplicate key/data pairs exist before the cursor position in the database. If using a Heap database, this flag is identical to the `DB_PREV` flag. @@ -157,7 +157,7 @@ The **flags** parameter must be set to one of the following values: Move the cursor to the specified key/data pair of the database, and return the datum associated with the given key. - The `DBcursor->get()` method will return DB_NOTFOUND if DB_SET is set and no matching keys are found. The `DBcursor->get()` method will return DB_KEYEMPTY if DB_SET is set and the database is a Queue or Recno database, and the specified key exists, but was never explicitly created by the application or was later deleted. In the presence of duplicate key values, `DBcursor->get()` will return the first data item for the given key. + The `DBcursor->get()` method will return DB_NOTFOUND if DB_SET is set and no matching keys are found. The `DBcursor->get()` method will return DB_KEYEMPTY if DB_SET is set and the database is a Queue or Recno database, and the specified key exists, but was never explicitly created by the application or was later deleted. In the presence of duplicate key values, `DBcursor->get()` will return the first data item for the given key. - `DB_SET_RANGE` @@ -191,9 +191,9 @@ In addition, the following flags may be set by bitwise inclusively **OR**'ing th Return multiple data items in the **data** parameter. - In the case of Btree or Hash databases, duplicate data items for the current key, starting at the current cursor position, are entered into the buffer. Subsequent calls with both the DB_NEXT_DUP and DB_MULTIPLE flags specified will return additional duplicate data items associated with the current key or DB_NOTFOUND if there are no additional duplicate data items to return. Subsequent calls with both the DB_NEXT and DB_MULTIPLE flags specified will return additional duplicate data items associated with the current key or if there are no additional duplicate data items will return the next key and its data items or DB_NOTFOUND if there are no additional keys in the database. + In the case of Btree or Hash databases, duplicate data items for the current key, starting at the current cursor position, are entered into the buffer. Subsequent calls with both the DB_NEXT_DUP and DB_MULTIPLE flags specified will return additional duplicate data items associated with the current key or DB_NOTFOUND if there are no additional duplicate data items to return. Subsequent calls with both the DB_NEXT and DB_MULTIPLE flags specified will return additional duplicate data items associated with the current key or if there are no additional duplicate data items will return the next key and its data items or DB_NOTFOUND if there are no additional keys in the database. - In the case of Queue, Recno, or Heap databases, data items starting at the current cursor position are entered into the buffer. The record number (or the RID, in the case of Heap) of the first record will be returned in the **key** parameter. For Queue and Recno, the record number of each subsequent returned record must be calculated from this value. For Heap databases, the RID of subsequent returned records cannot be known. Subsequent calls with the DB_MULTIPLE flag specified will return additional data items or DB_NOTFOUND if there are no additional data items to return. + In the case of Queue, Recno, or Heap databases, data items starting at the current cursor position are entered into the buffer. The record number (or the RID, in the case of Heap) of the first record will be returned in the **key** parameter. For Queue and Recno, the record number of each subsequent returned record must be calculated from this value. For Heap databases, the RID of subsequent returned records cannot be known. Subsequent calls with the DB_MULTIPLE flag specified will return additional data items or DB_NOTFOUND if there are no additional data items to return. The buffer to which the **data** parameter refers must be provided from user memory (see DB_DBT_USERMEM ). The buffer must be at least as large as the page size of the underlying database, aligned for unsigned integer access, and be a multiple of 1024 bytes in size. If the buffer size is insufficient, then upon return from the call the size field of the **data** parameter will have been set to an estimated buffer size, and the error DB_BUFFER_SMALL is returned. (The size is an estimate as the exact size needed may not be known until all entries are read. It is best to initially provide a relatively large buffer, but applications should be prepared to resize the buffer as necessary and repeatedly call the method.) @@ -205,7 +205,7 @@ In addition, the following flags may be set by bitwise inclusively **OR**'ing th Return multiple key and data pairs in the **data** parameter. - Key and data pairs, starting at the current cursor position, are entered into the buffer. Subsequent calls with both the DB_NEXT and DB_MULTIPLE_KEY flags specified will return additional key and data pairs or DB_NOTFOUND if there are no additional key and data items to return. + Key and data pairs, starting at the current cursor position, are entered into the buffer. Subsequent calls with both the DB_NEXT and DB_MULTIPLE_KEY flags specified will return additional key and data pairs or DB_NOTFOUND if there are no additional key and data items to return. In the case of Btree, Hash or Heap databases, the multiple key and data pairs can be iterated over using the DB_MULTIPLE_KEY_NEXT macro. diff --git a/docs-src/api/c/dbcput.md b/docs-src/api/c/dbcput.md index 7e631e47b..e307b09e7 100644 --- a/docs-src/api/c/dbcput.md +++ b/docs-src/api/c/dbcput.md @@ -42,7 +42,7 @@ The **flags** parameter must be set to one of the following values: The DB_AFTER flag may not be specified to the Queue access method. - The `DBcursor->put()` method will return DB_NOTFOUND if the current cursor record has already been deleted and the underlying access method is Hash. + The `DBcursor->put()` method will return DB_NOTFOUND if the current cursor record has already been deleted and the underlying access method is Hash. - `DB_BEFORE` @@ -52,13 +52,13 @@ The **flags** parameter must be set to one of the following values: The DB_BEFORE flag may not be specified to the Queue access method. - The `DBcursor->put()` method will return DB_NOTFOUND if the current cursor record has already been deleted and the underlying access method is Hash. + The `DBcursor->put()` method will return DB_NOTFOUND if the current cursor record has already been deleted and the underlying access method is Hash. - `DB_CURRENT` Overwrite the data of the key/data pair to which the cursor refers with the specified data item. The **key** parameter is ignored. - The `DBcursor->put()` method will return DB_NOTFOUND if the current cursor record has already been deleted. + The `DBcursor->put()` method will return DB_NOTFOUND if the current cursor record has already been deleted. - `DB_KEYFIRST` diff --git a/docs-src/api/c/dbcreate.md b/docs-src/api/c/dbcreate.md index 62f70d2f6..80b90ddcb 100644 --- a/docs-src/api/c/dbcreate.md +++ b/docs-src/api/c/dbcreate.md @@ -37,7 +37,7 @@ The **flags** parameter must be set to 0 or the following value: - `DB_XA_CREATE` - Instead of creating a standalone database, create a database intended to be accessed via applications running under an X/Open conformant Transaction Manager. The database will be opened in the environment specified by the OPENINFO parameter of the GROUPS section of the ubbconfig file. See the XA Introduction section in the Berkeley DB Reference Guide for more information. + Instead of creating a standalone database, create a database intended to be accessed via applications running under an X/Open conformant Transaction Manager. The database will be opened in the environment specified by the OPENINFO parameter of the GROUPS section of the ubbconfig file. See the XA Introduction section in the Berkeley DB Reference Guide for more information. ### Errors diff --git a/docs-src/api/c/dbcursor.md b/docs-src/api/c/dbcursor.md index ee552f787..71d951581 100644 --- a/docs-src/api/c/dbcursor.md +++ b/docs-src/api/c/dbcursor.md @@ -52,7 +52,7 @@ The **flags** parameter must be set to 0 or by bitwise inclusively **OR**'ing to - `DB_TXN_SNAPSHOT` - Configure a transactional cursor to operate with read-only snapshot isolation. For databases with the DB_MULTIVERSION flag set, data values will be read as they are when the cursor is opened, without taking read locks. + Configure a transactional cursor to operate with read-only snapshot isolation. For databases with the DB_MULTIVERSION flag set, data values will be read as they are when the cursor is opened, without taking read locks. This flag implicitly begins a transaction that is committed when the cursor is closed. diff --git a/docs-src/api/c/dbdel.md b/docs-src/api/c/dbdel.md index 0ae757986..c2b404e38 100644 --- a/docs-src/api/c/dbdel.md +++ b/docs-src/api/c/dbdel.md @@ -16,7 +16,7 @@ The `DB->del()` method removes key/data pairs from the database. The key/data pa When called on a database that has been made into a secondary index using the DB->associate() method, the `DB->del()` method deletes the key/data pair from the primary database and all secondary indices. -The `DB->del()` method will return DB_NOTFOUND if the specified key is not in the database. The `DB->del()` method will return DB_KEYEMPTY if the database is a Queue or Recno database and the specified key exists, but was never explicitly created by the application or was later deleted. Unless otherwise specified, the `DB->del()` method returns a non-zero error value on failure and 0 on success. +The `DB->del()` method will return DB_NOTFOUND if the specified key is not in the database. The `DB->del()` method will return DB_KEYEMPTY if the database is a Queue or Recno database and the specified key exists, but was never explicitly created by the application or was later deleted. Unless otherwise specified, the `DB->del()` method returns a non-zero error value on failure and 0 on success. ### Parameters diff --git a/docs-src/api/c/dbexists.md b/docs-src/api/c/dbexists.md index fb0f45258..8247eb435 100644 --- a/docs-src/api/c/dbexists.md +++ b/docs-src/api/c/dbexists.md @@ -14,7 +14,7 @@ DB->exists(DB *db, DB_TXN *txnid, DBT *key, u_int32_t flags); The `DB->exists()` method returns whether the specified key appears in the database. -The `DB->exists()` method will return DB_NOTFOUND if the specified key is not in the database. The `DB->exists()` method will return DB_KEYEMPTY if the database is a Queue or Recno database and the specified key exists, but was never explicitly created by the application or was later deleted. +The `DB->exists()` method will return DB_NOTFOUND if the specified key is not in the database. The `DB->exists()` method will return DB_KEYEMPTY if the database is a Queue or Recno database and the specified key exists, but was never explicitly created by the application or was later deleted. ### Parameters diff --git a/docs-src/api/c/dbget.md b/docs-src/api/c/dbget.md index e521e8a53..ae61ab0e1 100644 --- a/docs-src/api/c/dbget.md +++ b/docs-src/api/c/dbget.md @@ -31,7 +31,7 @@ In the presence of duplicate key values, `DB->get()` will return the first data When called on a database that has been made into a secondary index using the DB->associate() method, the `DB->get()` and `DB->pget()` methods return the key from the secondary index and the data item from the primary database. In addition, the `DB->pget()` method returns the key from the primary database. In databases that are not secondary indices, the `DB->pget()` method will always fail. -The `DB->get()` method will return DB_NOTFOUND if the specified key is not in the database. The `DB->get()` method will return DB_KEYEMPTY if the database is a Queue or Recno database and the specified key exists, but was never explicitly created by the application or was later deleted. Unless otherwise specified, the `DB->get()` method returns a non-zero error value on failure and 0 on success. +The `DB->get()` method will return DB_NOTFOUND if the specified key is not in the database. The `DB->get()` method will return DB_KEYEMPTY if the database is a Queue or Recno database and the specified key exists, but was never explicitly created by the application or was later deleted. Unless otherwise specified, the `DB->get()` method returns a non-zero error value on failure and 0 on success. ### Parameters @@ -65,7 +65,7 @@ The **flags** parameter must be set to 0 or one of the following values: The DB_CONSUME_WAIT flag is the same as the DB_CONSUME flag, except that if the Queue database is empty, the thread of control will wait until there is data in the queue before returning. The underlying database must be of type Queue for DB_CONSUME_WAIT to be specified. - If lock or transaction timeouts have been specified, the `DB->get()` method with the DB_CONSUME_WAIT flag may return DB_LOCK_NOTGRANTED. This failure, by itself, does not require the enclosing transaction be aborted. + If lock or transaction timeouts have been specified, the `DB->get()` method with the DB_CONSUME_WAIT flag may return DB_LOCK_NOTGRANTED. This failure, by itself, does not require the enclosing transaction be aborted. - `DB_GET_BOTH` diff --git a/docs-src/api/c/dbjoin.md b/docs-src/api/c/dbjoin.md index b86a2b8e4..4948e1377 100644 --- a/docs-src/api/c/dbjoin.md +++ b/docs-src/api/c/dbjoin.md @@ -13,7 +13,7 @@ DB->join(DB *primary, DBC **curslist, DBC **dbcp, u_int32_t flags); ``` -The `DB->join()` method creates a specialized join cursor for use in performing equality or natural joins on secondary indices. For information on how to organize your data to use this functionality, see Equality join. +The `DB->join()` method creates a specialized join cursor for use in performing equality or natural joins on secondary indices. For information on how to organize your data to use this functionality, see Equality join. The `DB->join()` method is called using the DB handle of the primary database. diff --git a/docs-src/api/c/dbopen.md b/docs-src/api/c/dbopen.md index 518e5682b..4e8dd5f2f 100644 --- a/docs-src/api/c/dbopen.md +++ b/docs-src/api/c/dbopen.md @@ -27,11 +27,11 @@ The `DB->open()` method returns a non-zero error value on failure and 0 on succe #### txnid -If the operation is part of an application-specified transaction, the **txnid** parameter is a transaction handle returned from DB_ENV->txn_begin(); if the operation is part of a Berkeley DB Concurrent Data Store group, the **txnid** parameter is a handle returned from DB_ENV->cdsgroup_begin(); otherwise NULL. If no transaction handle is specified, but the DB_AUTO_COMMIT flag is specified, the operation will be implicitly transaction protected. Note that transactionally protected operations on a DB handle requires the DB handle itself be transactionally protected during its open. Also note that the transaction must be committed before the handle is closed; see Berkeley DB handles for more information. +If the operation is part of an application-specified transaction, the **txnid** parameter is a transaction handle returned from DB_ENV->txn_begin(); if the operation is part of a Berkeley DB Concurrent Data Store group, the **txnid** parameter is a handle returned from DB_ENV->cdsgroup_begin(); otherwise NULL. If no transaction handle is specified, but the DB_AUTO_COMMIT flag is specified, the operation will be implicitly transaction protected. Note that transactionally protected operations on a DB handle requires the DB handle itself be transactionally protected during its open. Also note that the transaction must be committed before the handle is closed; see Berkeley DB handles for more information. #### file -The **file** parameter is used as the name of an underlying file that will be used to back the database; see File naming for more information. +The **file** parameter is used as the name of an underlying file that will be used to back the database; see File naming for more information. In-memory databases never intended to be preserved on disk may be created by setting the **file** parameter to NULL. Whether other threads of control can access this database is driven entirely by whether the **database** parameter is set to NULL. @@ -39,7 +39,7 @@ When using a Unicode build on Windows (the default), the **file** argument will #### database -The **database** parameter is optional, and allows applications to have multiple databases in a single file. Although no **database** parameter needs to be specified, it is an error to attempt to open a second database in a **file** that was not initially created using a **database** name. Further, the **database** parameter is not supported by the Queue format. Finally, when opening multiple databases in the same physical file, it is important to consider locking and memory cache issues; see Opening multiple databases in a single file for more information. +The **database** parameter is optional, and allows applications to have multiple databases in a single file. Although no **database** parameter needs to be specified, it is an error to attempt to open a second database in a **file** that was not initially created using a **database** name. Further, the **database** parameter is not supported by the Queue format. Finally, when opening multiple databases in the same physical file, it is important to consider locking and memory cache issues; see Opening multiple databases in a single file for more information. If both the **database** and **file** parameters are NULL, the database is strictly temporary and cannot be opened by any other thread of control. Thus the database can only be accessed by sharing the single database handle that created it, in circumstances where doing so is safe. @@ -69,7 +69,7 @@ The **flags** parameter must be set to zero or by bitwise inclusively **OR**'ing - `DB_MULTIVERSION` - Open the database with support for multiversion concurrency control. This will cause updates to the database to follow a copy-on-write protocol, which is required to support snapshot isolation. The `DB_MULTIVERSION` flag requires that the database be transactionally protected during its open and is not supported by the queue format. + Open the database with support for multiversion concurrency control. This will cause updates to the database to follow a copy-on-write protocol, which is required to support snapshot isolation. The `DB_MULTIVERSION` flag requires that the database be transactionally protected during its open and is not supported by the queue format. - `DB_NOMMAP` @@ -105,7 +105,7 @@ On UNIX systems or in IEEE/ANSI Std 1003.1 (POSIX) environments, files created b If the database was opened within a database environment, the environment variable **DB_HOME** may be used as the path of the database environment home. -`DB->open()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. +`DB->open()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. - **TMPDIR** diff --git a/docs-src/api/c/dbremove.md b/docs-src/api/c/dbremove.md index 797fbd674..b92af9c48 100644 --- a/docs-src/api/c/dbremove.md +++ b/docs-src/api/c/dbremove.md @@ -43,7 +43,7 @@ The **flags** parameter is currently unused, and must be set to 0. If the database was opened within a database environment, the environment variable `DB_HOME` may be used as the path of the database environment home. -`DB->remove()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. +`DB->remove()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. ### Errors diff --git a/docs-src/api/c/dbrename.md b/docs-src/api/c/dbrename.md index 6a2323e7e..851281216 100644 --- a/docs-src/api/c/dbrename.md +++ b/docs-src/api/c/dbrename.md @@ -49,7 +49,7 @@ The **flags** parameter is currently unused, and must be set to 0. If the database was opened within a database environment, the environment variable `DB_HOME` may be used as the path of the database environment home. -`DB->rename()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. +`DB->rename()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. ### Errors diff --git a/docs-src/api/c/dbset_bt_minkey.md b/docs-src/api/c/dbset_bt_minkey.md index 78942ae1b..22d3fe84c 100644 --- a/docs-src/api/c/dbset_bt_minkey.md +++ b/docs-src/api/c/dbset_bt_minkey.md @@ -14,7 +14,7 @@ DB->set_bt_minkey(DB *db, u_int32_t bt_minkey); Set the minimum number of key/data pairs intended to be stored on any single Btree leaf page. -This value is used to determine if key or data items will be stored on overflow pages instead of Btree leaf pages. For more information on the specific algorithm used, see Minimum keys per page. The **bt_minkey** value specified must be at least 2; if **bt_minkey** is not explicitly set, a value of 2 is used. +This value is used to determine if key or data items will be stored on overflow pages instead of Btree leaf pages. For more information on the specific algorithm used, see Minimum keys per page. The **bt_minkey** value specified must be at least 2; if **bt_minkey** is not explicitly set, a value of 2 is used. The `DB->set_bt_minkey()` method configures a database, not only operations performed using the specified DB handle. diff --git a/docs-src/api/c/dbset_bt_prefix.md b/docs-src/api/c/dbset_bt_prefix.md index 21952be21..70e669995 100644 --- a/docs-src/api/c/dbset_bt_prefix.md +++ b/docs-src/api/c/dbset_bt_prefix.md @@ -13,7 +13,7 @@ DB->set_bt_prefix(DB *db, size_t (*bt_prefix_fcn)(DB *, const *dbt1, const *dbt2)); ``` -Set the Btree prefix function. The prefix function is used to determine the amount by which keys stored on the Btree internal pages can be safely truncated without losing their uniqueness. See the Btree prefix comparison section of the Berkeley DB Reference Guide for more details about how this works. The usefulness of this is data-dependent, but can produce significantly reduced tree sizes and search times in some data sets. +Set the Btree prefix function. The prefix function is used to determine the amount by which keys stored on the Btree internal pages can be safely truncated without losing their uniqueness. See the Btree prefix comparison section of the Berkeley DB Reference Guide for more details about how this works. The usefulness of this is data-dependent, but can produce significantly reduced tree sizes and search times in some data sets. If no prefix function or key comparison function is specified by the application, a default lexical comparison function is used as the prefix function. If no prefix function is specified and a key comparison function is specified, no prefix function is used. It is an error to specify a prefix function without also specifying a Btree key comparison function. diff --git a/docs-src/api/c/dbset_cachesize.md b/docs-src/api/c/dbset_cachesize.md index 5f0a5bcd1..bd6e61021 100644 --- a/docs-src/api/c/dbset_cachesize.md +++ b/docs-src/api/c/dbset_cachesize.md @@ -15,7 +15,7 @@ DB->set_cachesize(DB *db, Set the size of the shared memory buffer pool -- that is, the cache. The cache should be the size of the normal working data set of the application, with some small amount of additional memory for unusual situations. (Note: the working set is not the same as the number of pages accessed simultaneously, and is usually much larger.) -The default cache size is 256KB, and may not be specified as less than 20KB. Any cache size less than 500MB is automatically increased by 25% to account for buffer pool overhead; cache sizes larger than 500MB are used as specified. The maximum size of a single cache is 4GB on 32-bit systems and 10TB on 64-bit systems. (All sizes are in powers-of-two, that is, 256KB is 2^18 not 256,000.) For information on tuning the Berkeley DB cache size, see Selecting a cache size. +The default cache size is 256KB, and may not be specified as less than 20KB. Any cache size less than 500MB is automatically increased by 25% to account for buffer pool overhead; cache sizes larger than 500MB are used as specified. The maximum size of a single cache is 4GB on 32-bit systems and 10TB on 64-bit systems. (All sizes are in powers-of-two, that is, 256KB is 2^18 not 256,000.) For information on tuning the Berkeley DB cache size, see Selecting a cache size. It is possible to specify caches to Berkeley DB large enough they cannot be allocated contiguously on some architectures. For example, some releases of Solaris limit the amount of memory that may be allocated contiguously by a process. If **ncache** is 0 or 1, the cache will be allocated contiguously in memory. If it is greater than 1, the cache will be split across **ncache** separate regions, where the **region size** is equal to the initial cache size divided by **ncache**. diff --git a/docs-src/api/c/dbset_flags.md b/docs-src/api/c/dbset_flags.md index e1a3f44c5..ed0526694 100644 --- a/docs-src/api/c/dbset_flags.md +++ b/docs-src/api/c/dbset_flags.md @@ -146,13 +146,13 @@ The following flags may be specified for the Recno access method: Specifying the DB_RENUMBER flag causes the logical record numbers to be mutable, and change as records are added to and deleted from the database. - Using the DB->put() or DBcursor->put() interfaces to create new records will cause the creation of multiple records if the record number is more than one greater than the largest record currently in the database. For example, creating record 28, when record 25 was previously the last record in the database, will create records 26 and 27 as well as 28. Attempts to retrieve records that were created in this manner will result in an error return of DB_KEYEMPTY. + Using the DB->put() or DBcursor->put() interfaces to create new records will cause the creation of multiple records if the record number is more than one greater than the largest record currently in the database. For example, creating record 28, when record 25 was previously the last record in the database, will create records 26 and 27 as well as 28. Attempts to retrieve records that were created in this manner will result in an error return of DB_KEYEMPTY. If a created record is not at the end of the database, all records following the new record will be automatically renumbered upward by one. For example, the creation of a new record numbered 8 causes records numbered 8 and greater to be renumbered upward by one. If a cursor was positioned to record number 8 or greater before the insertion, it will be shifted upward one logical record, continuing to refer to the same record as it did before. If a deleted record is not at the end of the database, all records following the removed record will be automatically renumbered downward by one. For example, deleting the record numbered 8 causes records numbered 9 and greater to be renumbered downward by one. If a cursor was positioned to record number 9 or greater before the removal, it will be shifted downward one logical record, continuing to refer to the same record as it did before. - If a record is deleted, all cursors that were positioned on that record prior to the removal will no longer be positioned on a valid entry. This includes cursors used to delete an item. For example, if a cursor was positioned to record number 8 before the removal of that record, subsequent calls to DBcursor->get() with flags of DB_CURRENT will result in an error return of DB_KEYEMPTY until the cursor is moved to another record. A call to DBcursor->get() with flags of DB_NEXT will return the new record numbered 8 - which is the record that was numbered 9 prior to the delete (if such a record existed). + If a record is deleted, all cursors that were positioned on that record prior to the removal will no longer be positioned on a valid entry. This includes cursors used to delete an item. For example, if a cursor was positioned to record number 8 before the removal of that record, subsequent calls to DBcursor->get() with flags of DB_CURRENT will result in an error return of DB_KEYEMPTY until the cursor is moved to another record. A call to DBcursor->get() with flags of DB_NEXT will return the new record numbered 8 - which is the record that was numbered 9 prior to the delete (if such a record existed). For these reasons, concurrent access to a Recno database with the DB_RENUMBER flag specified may be largely meaningless, although it is supported. diff --git a/docs-src/api/c/dbset_pagesize.md b/docs-src/api/c/dbset_pagesize.md index 83d9deda2..dbda7c4f8 100644 --- a/docs-src/api/c/dbset_pagesize.md +++ b/docs-src/api/c/dbset_pagesize.md @@ -14,7 +14,7 @@ DB->set_pagesize(DB *db, u_int32_t pagesize); Set the size of the pages used to hold items in the database, in bytes. The minimum page size is 512 bytes, the maximum page size is 64K bytes, and the page size must be a power-of-two. If the page size is not explicitly set, one is selected based on the underlying filesystem I/O block size. The automatically selected size has a lower limit of 512 bytes and an upper limit of 16K bytes. -For information on tuning the Berkeley DB page size, see Selecting a page size. +For information on tuning the Berkeley DB page size, see Selecting a page size. The `DB->set_pagesize()` method configures a database, not only operations performed using the specified DB handle. diff --git a/docs-src/api/c/dbset_q_extentsize.md b/docs-src/api/c/dbset_q_extentsize.md index 0389ac8e2..c43289f9e 100644 --- a/docs-src/api/c/dbset_q_extentsize.md +++ b/docs-src/api/c/dbset_q_extentsize.md @@ -14,7 +14,7 @@ DB->set_q_extentsize(DB *db, u_int32_t extentsize); Set the size of the extents used to hold pages in a Queue database, specified as a number of pages. Each extent is created as a separate physical file. If no extent size is set, the default behavior is to create only a single underlying database file. -For information on tuning the extent size, see Selecting a extent size. +For information on tuning the extent size, see Selecting a extent size. The `DB->set_q_extentsize()` method configures a database, not only operations performed using the specified DB handle. diff --git a/docs-src/api/c/dbsql.md b/docs-src/api/c/dbsql.md index 6acb129c6..d1bd5c6a8 100644 --- a/docs-src/api/c/dbsql.md +++ b/docs-src/api/c/dbsql.md @@ -11,7 +11,7 @@ dbsql [OPTIONS] FILENAME SQL `dbsql` is a command line tool that provides access to the Berkeley DB SQL interface. -To build this tool, run the configure script with the `--enable-sql `option when you are building the Berkeley DB SQL interface. For more information on building this tool, see "Building for UNIX/POSIX". +To build this tool, run the configure script with the `--enable-sql `option when you are building the Berkeley DB SQL interface. For more information on building this tool, see "Building for UNIX/POSIX". FILENAME is the name of a Berkeley DB database file created with the SQL interface. A new database is created if the file does not exist. The options are as follows: diff --git a/docs-src/api/c/dbt.md b/docs-src/api/c/dbt.md index 535161fef..a57d042f8 100644 --- a/docs-src/api/c/dbt.md +++ b/docs-src/api/c/dbt.md @@ -20,7 +20,7 @@ source: docs/api_reference/C/dbt.html ``` -Storage and retrieval for the DB access methods are based on key/data pairs. Both key and data items are represented by the DBT data structure. (The name DBT is a mnemonic for data base thang, and was used because no one could think of a reasonable name that wasn't already in use somewhere else.) Key and data byte strings may refer to strings of zero length up to strings of essentially unlimited length. See Database limits for more information. +Storage and retrieval for the DB access methods are based on key/data pairs. Both key and data items are represented by the DBT data structure. (The name DBT is a mnemonic for data base thang, and was used because no one could think of a reasonable name that wasn't already in use somewhere else.) Key and data byte strings may refer to strings of zero length up to strings of essentially unlimited length. See Database limits for more information. All fields of the DBT structure that are not explicitly set should be initialized to nul bytes before the first time the structure is used. Do this by declaring the structure external or static, or by calling the C library routine **memset**(3). diff --git a/docs-src/api/c/dbupgrade.md b/docs-src/api/c/dbupgrade.md index 5f40a8616..3739046a6 100644 --- a/docs-src/api/c/dbupgrade.md +++ b/docs-src/api/c/dbupgrade.md @@ -14,7 +14,7 @@ DB->upgrade(DB *db, const char *file, u_int32_t flags); The `DB->upgrade()` method upgrades all of the databases included in the file **file**, if necessary. If no upgrade is necessary, `DB->upgrade()` always returns success. -**Database upgrades are done in place and are destructive. For example, if pages need to be allocated and no disk space is available, the database may be left corrupted. Backups should be made before databases are upgraded. See Upgrading databases for more information.** +**Database upgrades are done in place and are destructive. For example, if pages need to be allocated and no disk space is available, the database may be left corrupted. Backups should be made before databases are upgraded. See Upgrading databases for more information.** Unlike all other database operations, `DB->upgrade()` may only be done on a system with the same byte-order as the database. @@ -44,7 +44,7 @@ The **flags** parameter must be set to 0 or the following value: If the database was opened within a database environment, the environment variable `DB_HOME` may be used as the path of the database environment home. -`DB->upgrade()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. +`DB->upgrade()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. ### Errors diff --git a/docs-src/api/c/dbverify.md b/docs-src/api/c/dbverify.md index 89c68f1f6..78beeafc0 100644 --- a/docs-src/api/c/dbverify.md +++ b/docs-src/api/c/dbverify.md @@ -81,7 +81,7 @@ In addition, the following flags may be set by bitwise inclusively **OR**'ing th If the database was opened within a database environment, the environment variable `DB_HOME` may be used as the path of the database environment home. -`DB->verify()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. +`DB->verify()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the "set_data_dir" string in the environment's DB_CONFIG file. ### Errors diff --git a/docs-src/api/c/envadd_data_dir.md b/docs-src/api/c/envadd_data_dir.md index e9e02673b..b1d0438a2 100644 --- a/docs-src/api/c/envadd_data_dir.md +++ b/docs-src/api/c/envadd_data_dir.md @@ -14,9 +14,9 @@ DB_ENV->add_data_dir(DB_ENV *dbenv, const char *dir); Add the path of a directory to be used as the location of the access method database files. Paths specified to the DB->open() function will be searched relative to this path. Paths set using this method are additive, and specifying more than one will result in each specified directory being searched for database files. -If no database directories are specified, database files must be named either by absolute paths or relative to the environment home directory. See Berkeley DB File Naming for more information. +If no database directories are specified, database files must be named either by absolute paths or relative to the environment home directory. See Berkeley DB File Naming for more information. -The database environment's data directories may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "add_data_dir", one or more whitespace characters, and the directory name. Note that if you use this method for your application, and you also want to use the db_recover or db_archive utilities, then you should create a DB_CONFIG file and set the "add_data_dir" parameter in it. +The database environment's data directories may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "add_data_dir", one or more whitespace characters, and the directory name. Note that if you use this method for your application, and you also want to use the db_recover or db_archive utilities, then you should create a DB_CONFIG file and set the "add_data_dir" parameter in it. The `DB_ENV->add_data_dir()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envbackup.md b/docs-src/api/c/envbackup.md index 337bff7c8..df9775481 100644 --- a/docs-src/api/c/envbackup.md +++ b/docs-src/api/c/envbackup.md @@ -12,11 +12,11 @@ int DB_ENV->backup(DB_ENV *dbenv, const char *target, u_int32_t flags); ``` -The `DB_ENV->backup()` method performs a hot backup of the open environment. All files used by the environment are backed up, so long as the normal rules for file placement are followed. For information on how files are normally placed relative to the environment directory, see Berkeley DB File Naming in the *Berkeley DB Programmer's Reference Guide*. +The `DB_ENV->backup()` method performs a hot backup of the open environment. All files used by the environment are backed up, so long as the normal rules for file placement are followed. For information on how files are normally placed relative to the environment directory, see Berkeley DB File Naming in the *Berkeley DB Programmer's Reference Guide*. By default, data directories and the log directory specified relative to the home directory will be recreated relative to the target directory. If absolute path names are used, then specify `DB_BACKUP_SINGLE_DIR` to the `flags` parameter. -This method provides the same functionality as the db_hotbackup utility. However, this method does not perform the housekeeping actions performed by the `db_hotbackup` utility. In particular, you may want to run a checkpoint before calling this method. To run a checkpoint, use the DB_ENV->txn_checkpoint() method. For more information on checkpoints, see Checkpoints in the *Berkeley DB Programmer's Reference Guide*. +This method provides the same functionality as the db_hotbackup utility. However, this method does not perform the housekeeping actions performed by the `db_hotbackup` utility. In particular, you may want to run a checkpoint before calling this method. To run a checkpoint, use the DB_ENV->txn_checkpoint() method. For more information on checkpoints, see Checkpoints in the *Berkeley DB Programmer's Reference Guide*. To back up a single database file contained within the environment, use the DB_ENV->dbbackup() method. diff --git a/docs-src/api/c/envcdsgroup_begin.md b/docs-src/api/c/envcdsgroup_begin.md index b1387b4da..44910b25b 100644 --- a/docs-src/api/c/envcdsgroup_begin.md +++ b/docs-src/api/c/envcdsgroup_begin.md @@ -14,7 +14,7 @@ DB_ENV->cdsgroup_begin(DB_ENV *dbenv, DB_TXN **tid); The `DB_ENV->cdsgroup_begin()` method allocates a locker ID in an environment configured for Berkeley DB Concurrent Data Store applications. It copies a pointer to a DB_TXN that uniquely identifies the locker ID into the memory to which **tid** refers. Calling the DB_TXN->commit() method will discard the allocated locker ID. -See Berkeley DB Concurrent Data Store applications for more information about when this is required. +See Berkeley DB Concurrent Data Store applications for more information about when this is required. The `DB_ENV->cdsgroup_begin()` method may be called at any time during the life of the application. diff --git a/docs-src/api/c/envdbremove.md b/docs-src/api/c/envdbremove.md index 951359d41..208fe2a3f 100644 --- a/docs-src/api/c/envdbremove.md +++ b/docs-src/api/c/envdbremove.md @@ -19,7 +19,7 @@ Applications should never remove databases with open DB_ENV->set_data_dir() method, or by setting the `set_data_dir` string in the environment's DB_CONFIG file. +`DB_ENV->dbremove()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the `set_data_dir` string in the environment's DB_CONFIG file. ### Parameters diff --git a/docs-src/api/c/envdbrename.md b/docs-src/api/c/envdbrename.md index 7b8c49ade..d01a26d97 100644 --- a/docs-src/api/c/envdbrename.md +++ b/docs-src/api/c/envdbrename.md @@ -19,7 +19,7 @@ Applications should not rename databases that are currently in use. If an underl The `DB_ENV->dbrename()` method returns a non-zero error value on failure and 0 on success. -`DB_ENV->dbrename()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the `set_data_dir` string in the environment's DB_CONFIG file. +`DB_ENV->dbrename()` is affected by any database directory specified using the DB_ENV->set_data_dir() method, or by setting the `set_data_dir` string in the environment's DB_CONFIG file. ### Parameters diff --git a/docs-src/api/c/envevent_notify.md b/docs-src/api/c/envevent_notify.md index c8f3c45d6..f900828a9 100644 --- a/docs-src/api/c/envevent_notify.md +++ b/docs-src/api/c/envevent_notify.md @@ -42,7 +42,7 @@ The **db_event_fcn** parameter is the application's event notification function. - `DB_EVENT_PANIC` - Errors can occur in the Berkeley DB library where the only solution is to shut down the application and run recovery (for example, if Berkeley DB is unable to allocate heap memory). In such cases, the Berkeley DB methods will return DB_RUNRECOVERY. It is often easier to simply exit the application when such errors occur rather than gracefully return up the stack. + Errors can occur in the Berkeley DB library where the only solution is to shut down the application and run recovery (for example, if Berkeley DB is unable to allocate heap memory). In such cases, the Berkeley DB methods will return DB_RUNRECOVERY. It is often easier to simply exit the application when such errors occur rather than gracefully return up the stack. When **event** is set to `DB_EVENT_PANIC`, the database environment has failed. All threads of control in the database environment should exit the environment, and recovery should be run. diff --git a/docs-src/api/c/envfailchk.md b/docs-src/api/c/envfailchk.md index f2d96418f..a7a63da6d 100644 --- a/docs-src/api/c/envfailchk.md +++ b/docs-src/api/c/envfailchk.md @@ -12,7 +12,7 @@ int DB_ENV->failchk(DB_ENV *dbenv, u_int32_t flags); ``` -The `DB_ENV->failchk()` method checks for threads of control (either a true thread or a process) that have exited while manipulating Berkeley DB library data structures, while holding a logical database lock, or with an unresolved transaction (that is, a transaction that was never aborted or committed). For more information, see Architecting Data Store and Concurrent Data Store applications, and Architecting Transactional Data Store applications, both in the *Berkeley DB Programmer's Reference Guide*. +The `DB_ENV->failchk()` method checks for threads of control (either a true thread or a process) that have exited while manipulating Berkeley DB library data structures, while holding a logical database lock, or with an unresolved transaction (that is, a transaction that was never aborted or committed). For more information, see Architecting Data Store and Concurrent Data Store applications, and Architecting Transactional Data Store applications, both in the *Berkeley DB Programmer's Reference Guide*. The `DB_ENV->failchk()` method is used in conjunction with the DB_ENV->set_thread_count(), DB_ENV->set_isalive() and DB_ENV->set_thread_id() methods. Before calling the `failchk()`method, applications must: @@ -28,7 +28,7 @@ If `DB_ENV->failchk()` determines a thread of control exited while holding datab In either of these cases, the `DB_ENV->failchk()` method will also report the process and thread IDs associated with any released locks or aborted transactions. The information is printed to a specified output channel (see the DB_ENV->set_msgfile() method for more information), or passed to an application callback function (see the DB_ENV->set_msgcall() method for more information). -If `DB_ENV->failchk()` determines a thread of control has exited such that database environment recovery is required, it will return DB_RUNRECOVERY. In this case, the application should not continue to use the database environment. For a further description as to the actions the application should take when this failure occurs, see Handling failure in Data Store and Concurrent Data Store applications, and Handling failure in Transactional Data Store applications, both in the *Berkeley DB Programmer's Reference Guide*. +If `DB_ENV->failchk()` determines a thread of control has exited such that database environment recovery is required, it will return DB_RUNRECOVERY. In this case, the application should not continue to use the database environment. For a further description as to the actions the application should take when this failure occurs, see Handling failure in Data Store and Concurrent Data Store applications, and Handling failure in Transactional Data Store applications, both in the *Berkeley DB Programmer's Reference Guide*. In multiprocess applications, it is recommended that the DB_ENV handle used to invoke the `DB_ENV->failchk()` method not be shared and therefore not *free-threaded*. diff --git a/docs-src/api/c/envlog_set_config.md b/docs-src/api/c/envlog_set_config.md index 8810131c0..7292a6242 100644 --- a/docs-src/api/c/envlog_set_config.md +++ b/docs-src/api/c/envlog_set_config.md @@ -30,7 +30,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o Turn off system buffering of Berkeley DB log files to avoid double caching. - Calling `DB_ENV->log_set_config()` with the DB_LOG_DIRECT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_LOG_DIRECT flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->log_set_config()` with the DB_LOG_DIRECT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_LOG_DIRECT flag or the flag should be specified in the DB_CONFIG configuration file. The `DB_LOG_DIRECT` flag may be used to configure Berkeley DB at any time during the life of the application. @@ -38,7 +38,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o Configure Berkeley DB to flush log writes to the backing disk before returning from the write system call, rather than flushing log writes explicitly in a separate system call, as necessary. This is only available on some systems (for example, systems supporting the IEEE/ANSI Std 1003.1 (POSIX) standard O_DSYNC flag, or systems supporting the Windows FILE_FLAG_WRITE_THROUGH flag). This flag may result in inaccurate file modification times and other file-level information for Berkeley DB log files. This flag may offer a performance increase on some systems and a performance decrease on others. - Calling `DB_ENV->log_set_config()` with the DB_LOG_DSYNC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_LOG_DSYNC flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->log_set_config()` with the DB_LOG_DSYNC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_LOG_DSYNC flag or the flag should be specified in the DB_CONFIG configuration file. The `DB_LOG_DSYNC` flag may be used to configure Berkeley DB at any time during the life of the application. diff --git a/docs-src/api/c/envopen.md b/docs-src/api/c/envopen.md index db558376a..a43e2a308 100644 --- a/docs-src/api/c/envopen.md +++ b/docs-src/api/c/envopen.md @@ -18,13 +18,13 @@ The `DB_ENV->open()` method method returns a non-zero error value on failure and ### Warning -Using environments with some journaling filesystems might result in log file corruption. This can occur if the operating system experiences an unclean shutdown when a log file is being created. Please see Using Recovery on Journaling Filesystems in the *Berkeley DB Programmer's Reference Guide* for more information. +Using environments with some journaling filesystems might result in log file corruption. This can occur if the operating system experiences an unclean shutdown when a log file is being created. Please see Using Recovery on Journaling Filesystems in the *Berkeley DB Programmer's Reference Guide* for more information. ### Parameters #### db_home -The **db_home** parameter is the database environment's home directory. For more information on **db_home**, and filename resolution in general, see Berkeley DB File Naming. The environment variable **DB_HOME** may be used as the path of the database home, as described in Berkeley DB File Naming. +The **db_home** parameter is the database environment's home directory. For more information on **db_home**, and filename resolution in general, see Berkeley DB File Naming. The environment variable **DB_HOME** may be used as the path of the database home, as described in Berkeley DB File Naming. When using a Unicode build on Windows (the default), the **db_home** argument will be interpreted as a UTF-8 string, which is equivalent to ASCII for Latin characters. @@ -38,7 +38,7 @@ The choice of subsystems initialized for a Berkeley DB database environment is s - `DB_INIT_CDB` - Initialize locking for the Berkeley DB Concurrent Data Store product. In this mode, Berkeley DB provides multiple reader/single writer access. The only other subsystem that should be specified with the `DB_INIT_CDB` flag is `DB_INIT_MPOOL`. + Initialize locking for the Berkeley DB Concurrent Data Store product. In this mode, Berkeley DB provides multiple reader/single writer access. The only other subsystem that should be specified with the `DB_INIT_CDB` flag is `DB_INIT_MPOOL`. - `DB_INIT_LOCK` @@ -56,7 +56,7 @@ The choice of subsystems initialized for a Berkeley DB database environment is s Initialize the replication subsystem. This subsystem should be used whenever an application plans on using replication. The `DB_INIT_REP` flag requires the `DB_INIT_TXN` and `DB_INIT_LOCK` flags also be configured. - You can also specify this flag in the DB_CONFIG configuration file. The syntax is a single line with the string "set_open_flags", one or more whitespace characters, the string "DB_INIT_REP", optionally one or more whitespace characters and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_open_flags DB_INIT_REP" or "set_open_flags DB_INIT_REP on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. + You can also specify this flag in the DB_CONFIG configuration file. The syntax is a single line with the string "set_open_flags", one or more whitespace characters, the string "DB_INIT_REP", optionally one or more whitespace characters and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_open_flags DB_INIT_REP" or "set_open_flags DB_INIT_REP on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. - `DB_INIT_TXN` @@ -80,11 +80,11 @@ The third group of flags govern file-naming extensions in the environment: - `DB_USE_ENVIRON` - The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, environment information will be used in file naming for all users only if the `DB_USE_ENVIRON` flag is set. + The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, environment information will be used in file naming for all users only if the `DB_USE_ENVIRON` flag is set. - `DB_USE_ENVIRON_ROOT` - The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, if the `DB_USE_ENVIRON_ROOT` flag is set, environment information will be used in file naming only for users with appropriate permissions (for example, users with a user-ID of 0 on `UNIX` systems). + The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, if the `DB_USE_ENVIRON_ROOT` flag is set, environment information will be used in file naming only for users with appropriate permissions (for example, users with a user-ID of 0 on `UNIX` systems). Finally, there are a few additional unrelated flags: @@ -102,7 +102,7 @@ Finally, there are a few additional unrelated flags: If the `DB_FAILCHK` flag is used in conjunction with the `DB_REGISTER` flag, then a check will be made to see if the environment needs recovery. If recovery is needed, a call will be made to the `DB_ENV->failchk()` method to release any database reads locks held by the thread of control that exited and, if needed, to abort the unresolved transaction. If `DB_ENV->failchk()` determines environment recovery is still required, the recovery actions for `DB_REGISTER` will be followed. - If the `DB_FAILCHK` flag is not used in conjunction with the `DB_REGISTER` flag, then make an internal call to `DB_ENV->failchk()` as the last step of opening the environment. If `DB_ENV->failchk()` determines database environment recovery is required, DB_RUNRECOVERY will be returned. + If the `DB_FAILCHK` flag is not used in conjunction with the `DB_REGISTER` flag, then make an internal call to `DB_ENV->failchk()` as the last step of opening the environment. If `DB_ENV->failchk()` determines database environment recovery is required, DB_RUNRECOVERY will be returned. - `DB_PRIVATE` @@ -114,19 +114,19 @@ Finally, there are a few additional unrelated flags: This flag has two effects on the Berkeley DB environment. First, all underlying data structures are allocated from per-process memory instead of from shared memory that is accessible to more than a single process. Second, mutexes are only configured to work between threads. - See Shared Memory Regions for more information. + See Shared Memory Regions for more information. - You can also specify this flag in the DB_CONFIG configuration file. The syntax is a single line with the string "set_open_flags", one or more whitespace characters, the string "DB_PRIVATE", optionally one or more whitespace characters and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_open_flags DB_PRIVATE" or "set_open_flags DB_PRIVATE on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. + You can also specify this flag in the DB_CONFIG configuration file. The syntax is a single line with the string "set_open_flags", one or more whitespace characters, the string "DB_PRIVATE", optionally one or more whitespace characters and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_open_flags DB_PRIVATE" or "set_open_flags DB_PRIVATE on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. - `DB_REGISTER` - Check to see if recovery needs to be performed before opening the database environment. (For this check to be accurate, all processes using the environment must specify `DB_REGISTER` when opening the environment.) If recovery needs to be performed for any reason (including the initial use of the `DB_REGISTER` flag), and `DB_RECOVER` is also specified, recovery will be performed and the open will proceed normally. If recovery needs to be performed and `DB_RECOVER` is not specified, DB_RUNRECOVERY will be returned. If recovery does not need to be performed, the `DB_RECOVER` flag will be ignored. See Architecting Transactional Data Store applications for more information. + Check to see if recovery needs to be performed before opening the database environment. (For this check to be accurate, all processes using the environment must specify `DB_REGISTER` when opening the environment.) If recovery needs to be performed for any reason (including the initial use of the `DB_REGISTER` flag), and `DB_RECOVER` is also specified, recovery will be performed and the open will proceed normally. If recovery needs to be performed and `DB_RECOVER` is not specified, DB_RUNRECOVERY will be returned. If recovery does not need to be performed, the `DB_RECOVER` flag will be ignored. See Architecting Transactional Data Store applications for more information. - `DB_SYSTEM_MEM` Allocate region memory from system shared memory instead of from heap memory or memory backed by the filesystem. - See Shared Memory Regions for more information. + See Shared Memory Regions for more information. - `DB_THREAD` @@ -134,7 +134,7 @@ Finally, there are a few additional unrelated flags: This flag is required when using the Replication Manager. - You can also specify this flag in the DB_CONFIG configuration file. The syntax is a single line with the string "set_open_flags", one or more whitespace characters, the string "DB_THREAD", optionally one or more whitespace characters and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_open_flags DB_THREAD" or "set_open_flags DB_THREAD on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. + You can also specify this flag in the DB_CONFIG configuration file. The syntax is a single line with the string "set_open_flags", one or more whitespace characters, the string "DB_THREAD", optionally one or more whitespace characters and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_open_flags DB_THREAD" or "set_open_flags DB_THREAD on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. #### mode diff --git a/docs-src/api/c/envremove.md b/docs-src/api/c/envremove.md index 0b0bf2762..a571b820c 100644 --- a/docs-src/api/c/envremove.md +++ b/docs-src/api/c/envremove.md @@ -46,11 +46,11 @@ The **flags** parameter must be set to 0 or by bitwise inclusively **OR**'ing to - `DB_USE_ENVIRON` - The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, environment information will be used in file naming for all users only if the `DB_USE_ENVIRON` flag is set. + The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, environment information will be used in file naming for all users only if the `DB_USE_ENVIRON` flag is set. - `DB_USE_ENVIRON_ROOT` - The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, if the `DB_USE_ENVIRON_ROOT` flag is set, environment information will be used in file naming only for users with appropriate permissions (for example, users with a user-ID of 0 on `UNIX` systems). + The Berkeley DB process' environment may be permitted to specify information to be used when naming files; see Berkeley DB File Naming. Because permitting users to specify which files are used can create security problems, if the `DB_USE_ENVIRON_ROOT` flag is set, environment information will be used in file naming only for users with appropriate permissions (for example, users with a user-ID of 0 on `UNIX` systems). ### Errors diff --git a/docs-src/api/c/envset_cache_max.md b/docs-src/api/c/envset_cache_max.md index 856a35fab..55dd63797 100644 --- a/docs-src/api/c/envset_cache_max.md +++ b/docs-src/api/c/envset_cache_max.md @@ -14,7 +14,7 @@ DB_ENV->set_cache_max(DB_ENV *dbenv, u_int32_t gbytes, u_int32_t bytes); Sets the maximum cache size in bytes. The specified size is rounded to the nearest multiple of the cache region size, which is the initial cache size divided by the number of regions specified to the DB_ENV->set_cachesize() method. If no value is specified, it defaults to the initial cache size. -The database environment's maximum cache size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_cache_max", one or more whitespace characters, and the maximum cache size in bytes, specified in two parts: the gigabytes of cache and the additional bytes of cache. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum cache size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_cache_max", one or more whitespace characters, and the maximum cache size in bytes, specified in two parts: the gigabytes of cache and the additional bytes of cache. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_cache_max()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_cachesize.md b/docs-src/api/c/envset_cachesize.md index 26d5d12b8..b8fc48ee4 100644 --- a/docs-src/api/c/envset_cachesize.md +++ b/docs-src/api/c/envset_cachesize.md @@ -15,13 +15,13 @@ DB_ENV->set_cachesize(DB_ENV *dbenv, Sets the size of the shared memory buffer pool — that is, the cache. The cache should be the size of the normal working data set of the application, with some small amount of additional memory for unusual situations. (Note: the working set is not the same as the number of pages accessed simultaneously, and is usually much larger.) -The default cache size is 256KB, and may not be specified as less than 20KB. Any cache size less than 500MB is automatically increased by 25% to account for cache overhead; cache sizes larger than 500MB are used as specified. The maximum size of a single cache is 4GB on 32-bit systems and 10TB on 64-bit systems. (All sizes are in powers-of-two, that is, 256KB is 2^18 not 256,000.) For information on tuning the Berkeley DB cache size, see Selecting a cache size. +The default cache size is 256KB, and may not be specified as less than 20KB. Any cache size less than 500MB is automatically increased by 25% to account for cache overhead; cache sizes larger than 500MB are used as specified. The maximum size of a single cache is 4GB on 32-bit systems and 10TB on 64-bit systems. (All sizes are in powers-of-two, that is, 256KB is 2^18 not 256,000.) For information on tuning the Berkeley DB cache size, see Selecting a cache size. It is possible to specify caches to Berkeley DB large enough they cannot be allocated contiguously on some architectures. For example, some releases of Solaris limit the amount of memory that may be allocated contiguously by a process. If **ncache** is 0 or 1, the cache will be allocated contiguously in memory. If it is greater than 1, the cache will be split across **ncache** separate regions, where the **region size** is equal to the initial cache size divided by **ncache**. The cache may be resized by calling `DB_ENV->set_cachesize()` after the environment is open. The supplied size will be rounded to the nearest multiple of the region size and may not be larger than the maximum size configured with DB_ENV->set_cache_max(). The **ncache** parameter is ignored when resizing the cache. -The database environment's initial cache size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_cachesize", one or more whitespace characters, and the initial cache size specified in three parts: the gigabytes of cache, the additional bytes of cache, and the number of caches, also separated by whitespace characters. For example, "set_cachesize 2 524288000 3" would create a 2.5GB logical cache, split between three physical caches. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's initial cache size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_cachesize", one or more whitespace characters, and the initial cache size specified in three parts: the gigabytes of cache, the additional bytes of cache, and the number of caches, also separated by whitespace characters. For example, "set_cachesize 2 524288000 3" would create a 2.5GB logical cache, split between three physical caches. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_cachesize()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_create_dir.md b/docs-src/api/c/envset_create_dir.md index 870f81ca0..7d64a29ee 100644 --- a/docs-src/api/c/envset_create_dir.md +++ b/docs-src/api/c/envset_create_dir.md @@ -14,9 +14,9 @@ DB_ENV->set_create_dir(DB_ENV *dbenv, const char *dir); Sets the path of a directory to be used as the location to create the access method database files. When the DB->open() function is used to create a file it will be created relative to this path. -If no database directories are specified, database files will be created either by absolute paths or relative to the environment home directory. See Berkeley DB File Naming for more information. +If no database directories are specified, database files will be created either by absolute paths or relative to the environment home directory. See Berkeley DB File Naming for more information. -The database environment's create directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_create_dir", one or more whitespace characters, and the directory name. +The database environment's create directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_create_dir", one or more whitespace characters, and the directory name. The `DB_ENV->set_create_dir()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_data_dir.md b/docs-src/api/c/envset_data_dir.md index a86974417..58f58af42 100644 --- a/docs-src/api/c/envset_data_dir.md +++ b/docs-src/api/c/envset_data_dir.md @@ -18,9 +18,9 @@ This interface has been deprecated. You should use DB->open() function will be searched relative to this path. Paths set using this method are additive, and specifying more than one will result in each specified directory being searched for database files. If any directories are specified, database files will always be created in the first path specified. -If no database directories are specified, database files must be named either by absolute paths or relative to the environment home directory. See Berkeley DB File Naming for more information. +If no database directories are specified, database files must be named either by absolute paths or relative to the environment home directory. See Berkeley DB File Naming for more information. -The database environment's data directories may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_data_dir", one or more whitespace characters, and the directory name. Note that if you use this method for your application, and you also want to use the db_recover or db_archive utilities, then you should create a DB_CONFIG file and set the "set_data_dir" parameter in it. +The database environment's data directories may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_data_dir", one or more whitespace characters, and the directory name. Note that if you use this method for your application, and you also want to use the db_recover or db_archive utilities, then you should create a DB_CONFIG file and set the "set_data_dir" parameter in it. The `DB_ENV->set_data_dir()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_data_len.md b/docs-src/api/c/envset_data_len.md index d3f827a38..20eb457ed 100644 --- a/docs-src/api/c/envset_data_len.md +++ b/docs-src/api/c/envset_data_len.md @@ -18,7 +18,7 @@ This method is explicitly called in the DB_CONFIG file. In this case, the limit will equally affect your application code, as well as the command line utilities noted above without modification to their code. The syntax of the entry in that file is a single line with the string "set_data_len", one or more whitespace characters, and the limit in bytes that you want to set. +This limit may also be configured using the environment's DB_CONFIG file. In this case, the limit will equally affect your application code, as well as the command line utilities noted above without modification to their code. The syntax of the entry in that file is a single line with the string "set_data_len", one or more whitespace characters, and the limit in bytes that you want to set. The `DB_ENV->set_data_len()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_flags.md b/docs-src/api/c/envset_flags.md index 45551c17b..35d79a757 100644 --- a/docs-src/api/c/envset_flags.md +++ b/docs-src/api/c/envset_flags.md @@ -14,7 +14,7 @@ DB_ENV->set_flags(DB_ENV *dbenv, u_int32_t flags, int onoff); Configure a database environment. -The database environment's flag values may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_flags", one or more whitespace characters, and the method flag parameter as a string, and optionally one or more whitespace characters, and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_flags DB_TXN_NOSYNC" or "set_flags DB_TXN_NOSYNC on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's flag values may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_flags", one or more whitespace characters, and the method flag parameter as a string, and optionally one or more whitespace characters, and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_flags DB_TXN_NOSYNC" or "set_flags DB_TXN_NOSYNC on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_flags()` method returns a non-zero error value on failure and 0 on success. @@ -28,7 +28,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, DB handle operations for which no explicit transaction handle was specified, and which modify databases in the database environment, will be automatically enclosed within a transaction. - Calling `DB_ENV->set_flags()` with this flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set this flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with this flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set this flag or the flag should be specified in the DB_CONFIG configuration file. This flag may be used to configure Berkeley DB at any time during the life of the application. @@ -36,7 +36,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, Berkeley DB Concurrent Data Store applications will perform locking on an environment-wide basis rather than on a per-database basis. - Calling `DB_ENV->set_flags()` with the DB_CDB_ALLDB flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_CDB_ALLDB flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_CDB_ALLDB flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_CDB_ALLDB flag or the flag should be specified in the DB_CONFIG configuration file. The DB_CDB_ALLDB flag may be used to configure Berkeley DB only before the DB_ENV->open() method is called. @@ -44,7 +44,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o Turn off system buffering of Berkeley DB database files to avoid double caching. - Calling `DB_ENV->set_flags()` with the DB_DIRECT_DB flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_DIRECT_DB flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_DIRECT_DB flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_DIRECT_DB flag or the flag should be specified in the DB_CONFIG configuration file. The DB_DIRECT_DB flag may be used to configure Berkeley DB at any time during the life of the application. @@ -58,7 +58,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o Configure Berkeley DB to flush database writes to the backing disk before returning from the write system call, rather than flushing database writes explicitly in a separate system call, as necessary. This is only available on some systems (for example, systems supporting the IEEE/ANSI Std 1003.1 (POSIX) standard O_DSYNC flag, or systems supporting the Windows FILE_FLAG_WRITE_THROUGH flag). This flag may result in inaccurate file modification times and other file-level information for Berkeley DB database files. This flag will almost certainly result in a performance decrease on most systems. This flag is only applicable to certain filesysystems (for example, the Veritas VxFS filesystem), where the filesystem's support for trickling writes back to stable storage behaves badly (or more likely, has been misconfigured). - Calling `DB_ENV->set_flags()` with the DB_DSYNC_DB flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_DSYNC_DB flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_DSYNC_DB flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_DSYNC_DB flag or the flag should be specified in the DB_CONFIG configuration file. The DB_DSYNC_DB flag may be used to configure Berkeley DB at any time during the life of the application. @@ -76,7 +76,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, all databases in the environment will be opened as if DB_MULTIVERSION is passed to the DB->open() method. This flag will be ignored for queue databases for which DB_MULTIVERSION is not supported. - Calling `DB_ENV->set_flags()` with the DB_MULTIVERSION flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_MULTIVERSION flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_MULTIVERSION flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_MULTIVERSION flag or the flag should be specified in the DB_CONFIG configuration file. The DB_MULTIVERSION flag may be used to configure Berkeley DB at any time during the life of the application. @@ -92,13 +92,13 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, Berkeley DB will copy read-only database files into the local cache instead of potentially mapping them into process memory (see the description of the DB_ENV->set_mp_mmapsize() method for further information). - Calling `DB_ENV->set_flags()` with the DB_NOMMAP flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_NOMMAP flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_NOMMAP flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_NOMMAP flag or the flag should be specified in the DB_CONFIG configuration file. The DB_NOMMAP flag may be used to configure Berkeley DB at any time during the life of the application. - `DB_NOPANIC` - If set, Berkeley DB will ignore any panic state in the database environment. (Database environments in a panic state normally refuse all attempts to call Berkeley DB functions, returning DB_RUNRECOVERY.) This functionality should never be used for purposes other than debugging. + If set, Berkeley DB will ignore any panic state in the database environment. (Database environments in a panic state normally refuse all attempts to call Berkeley DB functions, returning DB_RUNRECOVERY.) This functionality should never be used for purposes other than debugging. Calling `DB_ENV->set_flags()` with the DB_NOPANIC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). @@ -114,7 +114,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o - `DB_PANIC_ENVIRONMENT` - If set, Berkeley DB will set the panic state for the database environment. (Database environments in a panic state normally refuse all attempts to call Berkeley DB functions, returning DB_RUNRECOVERY.) This flag may not be specified using the environment's DB_CONFIG file. + If set, Berkeley DB will set the panic state for the database environment. (Database environments in a panic state normally refuse all attempts to call Berkeley DB functions, returning DB_RUNRECOVERY.) This flag may not be specified using the environment's DB_CONFIG file. Calling `DB_ENV->set_flags()` with the DB_PANIC_ENVIRONMENT flag affects the database environment, including all threads of control accessing the database environment. @@ -124,15 +124,15 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o In some applications, the expense of page-faulting the underlying shared memory regions can affect performance. (For example, if the page-fault occurs while holding a lock, other lock requests can convoy, and overall throughput may decrease.) If set, Berkeley DB will page-fault shared regions into memory when initially creating or joining a Berkeley DB environment. In addition, Berkeley DB will write the shared regions when creating an environment, forcing the underlying virtual memory and filesystems to instantiate both the necessary memory and the necessary disk space. This can also avoid out-of-disk space failures later on. - Calling `DB_ENV->set_flags()` with the DB_REGION_INIT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_REGION_INIT flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_REGION_INIT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_REGION_INIT flag or the flag should be specified in the DB_CONFIG configuration file. The DB_REGION_INIT flag may be used to configure Berkeley DB at any time during the life of the application. - `DB_TIME_NOTGRANTED` - If set, database calls timing out based on lock or transaction timeout values will return DB_LOCK_NOTGRANTED instead of DB_LOCK_DEADLOCK. This allows applications to distinguish between operations which have deadlocked and operations which have exceeded their time limits. + If set, database calls timing out based on lock or transaction timeout values will return DB_LOCK_NOTGRANTED instead of DB_LOCK_DEADLOCK. This allows applications to distinguish between operations which have deadlocked and operations which have exceeded their time limits. - Calling `DB_ENV->set_flags()` with the DB_TIME_NOTGRANTED flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TIME_NOTGRANTED flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_TIME_NOTGRANTED flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TIME_NOTGRANTED flag or the flag should be specified in the DB_CONFIG configuration file. The `DB_TIME_NOTGRANTED` flag may be used to configure Berkeley DB at any time during the life of the application. @@ -142,15 +142,15 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, Berkeley DB will not write or synchronously flush the log on transaction commit. This means that transactions exhibit the ACI (atomicity, consistency, and isolation) properties, but not D (durability); that is, database integrity will be maintained, but if the application or system fails, it is possible some number of the most recently committed transactions may be undone during recovery. The number of transactions at risk is governed by how many log updates can fit into the log buffer, how often the operating system flushes dirty buffers to disk, and how often the log is checkpointed. - Calling `DB_ENV->set_flags()` with the DB_TXN_NOSYNC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_NOSYNC flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_TXN_NOSYNC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_NOSYNC flag or the flag should be specified in the DB_CONFIG configuration file. The DB_TXN_NOSYNC flag may be used to configure Berkeley DB at any time during the life of the application. - `DB_TXN_NOWAIT` - If set and a lock is unavailable for any Berkeley DB operation performed in the context of a transaction, cause the operation to return DB_LOCK_DEADLOCK (or DB_LOCK_NOTGRANTED if configured using the DB_TIME_NOTGRANTED flag). + If set and a lock is unavailable for any Berkeley DB operation performed in the context of a transaction, cause the operation to return DB_LOCK_DEADLOCK (or DB_LOCK_NOTGRANTED if configured using the DB_TIME_NOTGRANTED flag). - Calling `DB_ENV->set_flags()` with the DB_TXN_NOWAIT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_NOWAIT flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_TXN_NOWAIT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_NOWAIT flag or the flag should be specified in the DB_CONFIG configuration file. The DB_TXN_NOWAIT flag may be used to configure Berkeley DB at any time during the life of the application. @@ -158,7 +158,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, all transactions in the environment will be started as if DB_TXN_SNAPSHOT were passed to the DB_ENV->txn_begin() method, and all non-transactional cursors will be opened as if DB_TXN_SNAPSHOT were passed to the DB->cursor() method. - Calling `DB_ENV->set_flags()` with the DB_TXN_SNAPSHOT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_SNAPSHOT flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_TXN_SNAPSHOT flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_SNAPSHOT flag or the flag should be specified in the DB_CONFIG configuration file. The DB_TXN_SNAPSHOT flag may be used to configure Berkeley DB at any time during the life of the application. @@ -166,7 +166,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, Berkeley DB will write, but will not synchronously flush, the log on transaction commit. This means that transactions exhibit the ACI (atomicity, consistency, and isolation) properties, but not D (durability); that is, database integrity will be maintained, but if the system fails, it is possible some number of the most recently committed transactions may be undone during recovery. The number of transactions at risk is governed by how often the system flushes dirty buffers to disk and how often the log is checkpointed. - Calling `DB_ENV->set_flags()` with the DB_TXN_WRITE_NOSYNC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_WRITE_NOSYNC flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_TXN_WRITE_NOSYNC flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_TXN_WRITE_NOSYNC flag or the flag should be specified in the DB_CONFIG configuration file. The DB_TXN_WRITE_NOSYNC flag may be used to configure Berkeley DB at any time during the life of the application. @@ -174,7 +174,7 @@ The **flags** parameter must be set by bitwise inclusively **OR**'ing together o If set, Berkeley DB will yield the processor immediately after each page or mutex acquisition. This functionality should never be used for purposes other than stress testing. - Calling `DB_ENV->set_flags()` with the DB_YIELDCPU flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_YIELDCPU flag or the flag should be specified in the DB_CONFIG configuration file. + Calling `DB_ENV->set_flags()` with the DB_YIELDCPU flag only affects the specified DB_ENV handle (and any other Berkeley DB handles opened within the scope of that handle). For consistent behavior across the environment, all DB_ENV handles opened in the environment must either set the DB_YIELDCPU flag or the flag should be specified in the DB_CONFIG configuration file. The DB_YIELDCPU flag may be used to configure Berkeley DB at any time during the life of the application. diff --git a/docs-src/api/c/envset_intermediate_dir_mode.md b/docs-src/api/c/envset_intermediate_dir_mode.md index 8fb5dd914..b56b25f91 100644 --- a/docs-src/api/c/envset_intermediate_dir_mode.md +++ b/docs-src/api/c/envset_intermediate_dir_mode.md @@ -18,7 +18,7 @@ The `DB_ENV->set_intermediate_dir_mode()` method causes Berkeley DB to create an On UNIX systems or in IEEE/ANSI Std 1003.1 (POSIX) environments, created directories are owned by the process owner; the group ownership of created directories is based on the system and directory defaults, and is not further specified by Berkeley DB. -The database environment's intermediate directory permissions may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_intermediate_dir_mode", one or more whitespace characters, and the directory permissions. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's intermediate directory permissions may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_intermediate_dir_mode", one or more whitespace characters, and the directory permissions. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_intermediate_dir_mode()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_isalive.md b/docs-src/api/c/envset_isalive.md index 69dca82b3..e462e3e51 100644 --- a/docs-src/api/c/envset_isalive.md +++ b/docs-src/api/c/envset_isalive.md @@ -13,7 +13,7 @@ DB_ENV->set_isalive(DB_ENV *dbenv, int (*is_alive)(DB_ENV *dbenv, pid_t pid, db_threadid_t tid, u_int32_t flags)); ``` -Declare a function that returns if a thread of control (either a true thread or a process) is still running. The `DB_ENV->set_isalive()` method supports the DB_ENV->failchk() method. For more information, see Architecting Data Store and Concurrent Data Store applications, and Architecting Transactional Data Store applications, both in the *Berkeley DB Programmer's Reference Guide*. +Declare a function that returns if a thread of control (either a true thread or a process) is still running. The `DB_ENV->set_isalive()` method supports the DB_ENV->failchk() method. For more information, see Architecting Data Store and Concurrent Data Store applications, and Architecting Transactional Data Store applications, both in the *Berkeley DB Programmer's Reference Guide*. The `DB_ENV->set_isalive()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_lg_bsize.md b/docs-src/api/c/envset_lg_bsize.md index 838ea5eba..541e643b5 100644 --- a/docs-src/api/c/envset_lg_bsize.md +++ b/docs-src/api/c/envset_lg_bsize.md @@ -18,7 +18,7 @@ When the logging subsystem is configured for on-disk logging, the default size o When the logging subsystem is configured for in-memory logging, the default size of the in-memory log buffer is 1MB. Log information is stored in-memory until the storage space fills up or transaction abort or commit frees up the memory for new transactions. In the presence of long-running transactions or transactions producing large amounts of data, the buffer size must be sufficient to hold all log information that can accumulate during the longest running transaction. When choosing log buffer and file sizes for in-memory logs, applications should ensure the in-memory log buffer size is large enough that no transaction will ever span the entire buffer, and avoid a state where the in-memory buffer is full and no space can be freed because a transaction that started in the first log "file" is still active. -The database environment's log buffer size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_bsize", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's log buffer size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_bsize", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lg_bsize()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lg_dir.md b/docs-src/api/c/envset_lg_dir.md index 91323bc8f..bcbb782dc 100644 --- a/docs-src/api/c/envset_lg_dir.md +++ b/docs-src/api/c/envset_lg_dir.md @@ -14,11 +14,11 @@ DB_ENV->set_lg_dir(DB_ENV *dbenv, const char *dir); The path of a directory to be used as the location of logging files. Log files created by the Log Manager subsystem will be created in this directory. -If no logging directory is specified, log files are created in the environment home directory. See Berkeley DB File Naming for more information. +If no logging directory is specified, log files are created in the environment home directory. See Berkeley DB File Naming for more information. For the greatest degree of recoverability from system or application failure, database files and log files should be located on separate physical devices. -The database environment's logging directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_dir", one or more whitespace characters, and the directory name. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. Note that if you use this method for your application, and you also want to use the db_recover, db_printlog, db_archive, or db_log_verify utilities, then you should set create a DB_CONFIG file and set the "set_lg_dir" parameter in it. +The database environment's logging directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_dir", one or more whitespace characters, and the directory name. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. Note that if you use this method for your application, and you also want to use the db_recover, db_printlog, db_archive, or db_log_verify utilities, then you should set create a DB_CONFIG file and set the "set_lg_dir" parameter in it. The `DB_ENV->set_lg_dir()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_lg_filemode.md b/docs-src/api/c/envset_lg_filemode.md index 8e30deede..81b44ac9a 100644 --- a/docs-src/api/c/envset_lg_filemode.md +++ b/docs-src/api/c/envset_lg_filemode.md @@ -16,7 +16,7 @@ Set the absolute file mode for created log files. This method is **only** useful Normally, if Berkeley DB applications set their umask appropriately, all processes in the application suite will have read permission on the log files created by any process in the application suite. However, if the Berkeley DB application is a library, a process using the library might set its umask to a value preventing other processes in the application suite from reading the log files it creates. In this rare case, the `DB_ENV->set_lg_filemode()` method can be used to set the mode of created log files to an absolute value. -The database environment's log file mode may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_filemode", one or more whitespace characters, and the absolute mode of created log files. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's log file mode may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_filemode", one or more whitespace characters, and the absolute mode of created log files. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lg_filemode()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lg_max.md b/docs-src/api/c/envset_lg_max.md index 3d8ec4d3c..174df7651 100644 --- a/docs-src/api/c/envset_lg_max.md +++ b/docs-src/api/c/envset_lg_max.md @@ -18,9 +18,9 @@ When the logging subsystem is configured for on-disk logging, the default size o When the logging subsystem is configured for in-memory logging, the default size of a log file is 256KB. In addition, the configured log buffer size must be larger than the log file size. (The logging subsystem divides memory configured for in-memory log records into "files", as database environments configured for in-memory log records may exchange log records with other members of a replication group, and those members may be configured to store log records on-disk.) When choosing log buffer and file sizes for in-memory logs, applications should ensure the in-memory log buffer size is large enough that no transaction will ever span the entire buffer, and avoid a state where the in-memory buffer is full and no space can be freed because a transaction that started in the first log "file" is still active. -See Log File Limits for more information. +See Log File Limits for more information. -The database environment's log file size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_max", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's log file size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_max", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lg_max()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lg_regionmax.md b/docs-src/api/c/envset_lg_regionmax.md index 0e369609b..fb12ebd29 100644 --- a/docs-src/api/c/envset_lg_regionmax.md +++ b/docs-src/api/c/envset_lg_regionmax.md @@ -14,7 +14,7 @@ DB_ENV->set_lg_regionmax(DB_ENV *dbenv, u_int32_t lg_regionmax); Set the size of the underlying logging area of the Berkeley DB environment, in bytes. By default, or if the value is set to 0, the minimum region size is used, approximately 128KB. The log region is used to store filenames, and so may need to be increased in size if a large number of files will be opened and registered with the specified Berkeley DB environment's log manager. -The database environment's log region size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_regionmax", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's log region size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lg_regionmax", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lg_regionmax()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_conflicts.md b/docs-src/api/c/envset_lk_conflicts.md index b615a9c5c..413367730 100644 --- a/docs-src/api/c/envset_lk_conflicts.md +++ b/docs-src/api/c/envset_lk_conflicts.md @@ -15,7 +15,7 @@ DB_ENV->set_lk_conflicts(DB_ENV *dbenv, Set the locking conflicts matrix. -If `DB_ENV->set_lk_conflicts()` is never called, a standard conflicts array is used; see Standard Lock Modes for more information. +If `DB_ENV->set_lk_conflicts()` is never called, a standard conflicts array is used; see Standard Lock Modes for more information. The `DB_ENV->set_lk_conflicts()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_detect.md b/docs-src/api/c/envset_lk_detect.md index 6a349a94a..658ad2775 100644 --- a/docs-src/api/c/envset_lk_detect.md +++ b/docs-src/api/c/envset_lk_detect.md @@ -14,7 +14,7 @@ DB_ENV->set_lk_detect(DB_ENV *dbenv, u_int32_t detect); Set if the deadlock detector is to be run whenever a lock conflict occurs, and specify what lock request(s) should be rejected. As transactions acquire locks on behalf of a single locker ID, rejecting a lock request associated with a transaction normally requires the transaction be aborted. -The database environment's deadlock detector configuration may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_detect", one or more whitespace characters, and the method **detect** parameter as a string; for example, "set_lk_detect DB_LOCK_OLDEST". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's deadlock detector configuration may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_detect", one or more whitespace characters, and the method **detect** parameter as a string; for example, "set_lk_detect DB_LOCK_OLDEST". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lk_detect()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_max_lockers.md b/docs-src/api/c/envset_lk_max_lockers.md index 848eef3c4..8059bc1c9 100644 --- a/docs-src/api/c/envset_lk_max_lockers.md +++ b/docs-src/api/c/envset_lk_max_lockers.md @@ -14,9 +14,9 @@ DB_ENV->set_lk_max_lockers(DB_ENV *dbenv, u_int32_t max); This method is deprecated. Instead, use DB_ENV->set_memory_init(), DB_ENV->set_memory_max(), and DB_ENV->set_lk_tablesize(). -Sets the maximum number of locking entities supported by the Berkeley DB environment. This value is used by DB_ENV->open() to estimate how much space to allocate for various lock-table data structures. The default value is 1000 lockers. For specific information on configuring the size of the lock subsystem, see Configuring locking: sizing the system. +Sets the maximum number of locking entities supported by the Berkeley DB environment. This value is used by DB_ENV->open() to estimate how much space to allocate for various lock-table data structures. The default value is 1000 lockers. For specific information on configuring the size of the lock subsystem, see Configuring locking: sizing the system. -The database environment's maximum number of lockers may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_max_lockers", one or more whitespace characters, and the number of lockers. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum number of lockers may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_max_lockers", one or more whitespace characters, and the number of lockers. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lk_max_lockers()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_max_locks.md b/docs-src/api/c/envset_lk_max_locks.md index 45718949f..6cfc5f2c5 100644 --- a/docs-src/api/c/envset_lk_max_locks.md +++ b/docs-src/api/c/envset_lk_max_locks.md @@ -14,9 +14,9 @@ DB_ENV->set_lk_max_locks(DB_ENV *dbenv, u_int32_t max); This method is deprecated. Instead, use DB_ENV->set_memory_init(), DB_ENV->set_memory_max(), and DB_ENV->set_lk_tablesize(). -Set the maximum number of locks supported by the Berkeley DB environment. This value is used by DB_ENV->open() to estimate how much space to allocate for various lock-table data structures. The default value is 1000 locks. The final value specified for the locks should be more than or equal to the number of lock table partitions. For specific information on configuring the size of the lock subsystem, see Configuring locking: sizing the system. +Set the maximum number of locks supported by the Berkeley DB environment. This value is used by DB_ENV->open() to estimate how much space to allocate for various lock-table data structures. The default value is 1000 locks. The final value specified for the locks should be more than or equal to the number of lock table partitions. For specific information on configuring the size of the lock subsystem, see Configuring locking: sizing the system. -The database environment's maximum number of locks may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_max_locks", one or more whitespace characters, and the number of locks. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum number of locks may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_max_locks", one or more whitespace characters, and the number of locks. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lk_max_locks()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_max_objects.md b/docs-src/api/c/envset_lk_max_objects.md index 48eff4981..a068cc73a 100644 --- a/docs-src/api/c/envset_lk_max_objects.md +++ b/docs-src/api/c/envset_lk_max_objects.md @@ -14,9 +14,9 @@ DB_ENV->set_lk_max_objects(DB_ENV *dbenv, u_int32_t max); This method is deprecated. Instead, use DB_ENV->set_memory_init(), DB_ENV->set_memory_max(), and DB_ENV->set_lk_tablesize(). -Set the maximum number of locked objects supported by the Berkeley DB environment. This value is used by DB_ENV->open() to estimate how much space to allocate for various lock-table data structures. The default value is 1000 objects. The final value specified for the lock objects should be more than or equal to the number of lock table partitions. For specific information on configuring the size of the lock subsystem, see Configuring locking: sizing the system. +Set the maximum number of locked objects supported by the Berkeley DB environment. This value is used by DB_ENV->open() to estimate how much space to allocate for various lock-table data structures. The default value is 1000 objects. The final value specified for the lock objects should be more than or equal to the number of lock table partitions. For specific information on configuring the size of the lock subsystem, see Configuring locking: sizing the system. -The database environment's maximum number of objects may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_max_objects", one or more whitespace characters, and the number of objects. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum number of objects may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_max_objects", one or more whitespace characters, and the number of objects. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lk_max_objects()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_partitions.md b/docs-src/api/c/envset_lk_partitions.md index b036c21bf..950de5017 100644 --- a/docs-src/api/c/envset_lk_partitions.md +++ b/docs-src/api/c/envset_lk_partitions.md @@ -14,7 +14,7 @@ DB_ENV->set_lk_partitions(DB_ENV *dbenv, u_int32_t partitions); Set the number of lock table partitions in the Berkeley DB environment. The default value is 10 times the number of CPUs on the system if there is more than one CPU. Increasing the number of partitions can provide for greater throughput on a system with multiple CPUs and more than one thread contending for the lock manager. On single processor systems more than one partition may increase the overhead of the lock manager. Systems often report threading contexts as CPUs. If your system does this, set the number of partitions to 1 to get optimal performance. -The database environment's number of partitions may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_partitions", one or more whitespace characters, and the number of partitions. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's number of partitions may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_partitions", one or more whitespace characters, and the number of partitions. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lk_partitions()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_lk_tablesize.md b/docs-src/api/c/envset_lk_tablesize.md index 134b94a7c..623233e20 100644 --- a/docs-src/api/c/envset_lk_tablesize.md +++ b/docs-src/api/c/envset_lk_tablesize.md @@ -14,7 +14,7 @@ DB_ENV->set_lk_tablesize(DB_ENV *dbenv, u_int32_t tablesize); Sets the number of buckets in the lock object hash table in the Berkeley DB environment. The default value is estimated based on defaults, initial and (deprecated) maximum settings of the number of lock objects allocated. The maximum memory allocation is also considered. The table is generally set to be close to the number of lock objects in the system to avoid collisions and delay in processing lock operations. -The database environment's tablesize may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_tablesize", one or more whitespace characters, and the size of the table. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's tablesize may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lk_tablesize", one or more whitespace characters, and the size of the table. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_lk_tablesize()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_memory_init.md b/docs-src/api/c/envset_memory_init.md index 74be61063..fb9002548 100644 --- a/docs-src/api/c/envset_memory_init.md +++ b/docs-src/api/c/envset_memory_init.md @@ -15,7 +15,7 @@ DB_ENV->set_memory_init(DB_ENV *dbenv, DB_MEM_CONFIG type, This method sets the number of objects to allocate and initialize for a specified structure when an environment is created. Doing this helps avoid memory contention after startup. Using this method is optional; failure to use this method causes BDB to allocate a minimal number of structures that will grow dynamically. These structures are all allocated from the main environment region. The amount of memory in this region can be set via the DB_ENV->set_memory_max() method. If this method is not called then memory will be limited to the initial settings or by the (deprecated) set maximum interfaces. -The database environment's initialization may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_memory_init", one or more whitespace characters, followed by the struct specification, more white space and the count to be allocated. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's initialization may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_memory_init", one or more whitespace characters, followed by the struct specification, more white space and the count to be allocated. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_memory_init()` method must be called prior to opening the database environment. It may be called as often as needed to set the different configurations. diff --git a/docs-src/api/c/envset_memory_max.md b/docs-src/api/c/envset_memory_max.md index 0b9c42a85..db3b30c7a 100644 --- a/docs-src/api/c/envset_memory_max.md +++ b/docs-src/api/c/envset_memory_max.md @@ -16,7 +16,7 @@ This method sets the maximum amount of memory to be used by shared structures in If no memory maximum is specified then it is calculated from defaults, initial settings or (deprecated) maximum settings of the various shared structures. In the case of environments created with `DB_PRIVATE`, no maximum need be set and the shared structure allocation will grow as needed until the process memory limit is exhausted. -The database environment's maximum memory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_memory_max", one or more whitespace characters, followed by the maximum to be allocated. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum memory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_memory_max", one or more whitespace characters, followed by the maximum to be allocated. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_memory_max()` method must be called prior to opening the database environment. diff --git a/docs-src/api/c/envset_metadata_dir.md b/docs-src/api/c/envset_metadata_dir.md index 25853caa3..f1b6198ac 100644 --- a/docs-src/api/c/envset_metadata_dir.md +++ b/docs-src/api/c/envset_metadata_dir.md @@ -18,7 +18,7 @@ When used in a replicated application, the metadata directory must be the same l The `DB_ENV->set_metadata_dir()` method may not be called after the DB_ENV->open() method is called. The directory identified by this method must already exist when the `DB_ENV->open()` method is called. The directory identified by this method is added to the environment's list of data directories, if this directory is not already included on that list. -The database environment's metadata directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_metadata_dir", one or more whitespace characters, followed by the directory location. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's metadata directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_metadata_dir", one or more whitespace characters, followed by the directory location. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_metadata_dir()` method returns a non-zero error value on failure and 0 on success. diff --git a/docs-src/api/c/envset_mp_mmapsize.md b/docs-src/api/c/envset_mp_mmapsize.md index f00838ea3..dfc3c2c0c 100644 --- a/docs-src/api/c/envset_mp_mmapsize.md +++ b/docs-src/api/c/envset_mp_mmapsize.md @@ -16,7 +16,7 @@ Files that are opened read-only in the cache (and that satisfy a few other crite The `DB_ENV->set_mp_mmapsize()` method sets the maximum file size, in bytes, for a file to be mapped into the process address space. If no value is specified, it defaults to 10MB. -The database environment's maximum mapped file size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_mp_mmapsize", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum mapped file size may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_mp_mmapsize", one or more whitespace characters, and the size in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_mp_mmapsize()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_shm_key.md b/docs-src/api/c/envset_shm_key.md index f234349be..fc31a1056 100644 --- a/docs-src/api/c/envset_shm_key.md +++ b/docs-src/api/c/envset_shm_key.md @@ -14,11 +14,11 @@ DB_ENV->set_shm_key(DB_ENV *dbenv, long shm_key); Specify a base segment ID for Berkeley DB environment shared memory regions created in system memory on VxWorks or systems supporting X/Open-style shared memory interfaces; for example, UNIX systems supporting **shmget**(2) and related System V IPC interfaces. -This base segment ID will be used when Berkeley DB shared memory regions are first created. It will be incremented a small integer value each time a new shared memory region is created; that is, if the base ID is 35, the first shared memory region created will have a segment ID of 35, and the next one will have a segment ID between 36 and 40 or so. A Berkeley DB environment always creates a master shared memory region; an additional shared memory region for each of the subsystems supported by the environment (Locking, Logging, Memory Pool and Transaction); plus an additional shared memory region for each additional memory pool cache that is supported. Already existing regions with the same segment IDs will be removed. See Shared Memory Regions for more information. +This base segment ID will be used when Berkeley DB shared memory regions are first created. It will be incremented a small integer value each time a new shared memory region is created; that is, if the base ID is 35, the first shared memory region created will have a segment ID of 35, and the next one will have a segment ID between 36 and 40 or so. A Berkeley DB environment always creates a master shared memory region; an additional shared memory region for each of the subsystems supported by the environment (Locking, Logging, Memory Pool and Transaction); plus an additional shared memory region for each additional memory pool cache that is supported. Already existing regions with the same segment IDs will be removed. See Shared Memory Regions for more information. The intent behind this method is two-fold: without it, applications have no way to ensure that two Berkeley DB applications don't attempt to use the same segment IDs when creating different Berkeley DB environments. In addition, by using the same segment IDs each time the environment is created, previously created segments will be removed, and the set of segments on the system will not grow without bound. -The database environment's base segment ID may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_shm_key", one or more whitespace characters, and the ID. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's base segment ID may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_shm_key", one or more whitespace characters, and the ID. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_shm_key()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_thread_count.md b/docs-src/api/c/envset_thread_count.md index f5a4c487b..0cb9db66d 100644 --- a/docs-src/api/c/envset_thread_count.md +++ b/docs-src/api/c/envset_thread_count.md @@ -22,7 +22,7 @@ If a process invokes this method without the use of DB_ENV->set_isalive() method, and then attempts to join a database environment configured for failure checking with the DB_ENV->failchk(), DB_ENV->set_thread_id(), DB_ENV->set_isalive() and `DB_ENV->set_thread_count()` methods, the program may be unable to allocate a thread control block and fail to join the environment. **This is true of the standalone Berkeley DB utility programs.** To avoid problems when using the standalone Berkeley DB utility programs with environments configured for failure checking, incorporate the utility's functionality directly in the application, or call the DB_ENV->failchk() method before running the utility. -The database environment's thread count may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_thread_count", one or more whitespace characters, and the thread count. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's thread count may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_thread_count", one or more whitespace characters, and the thread count. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_thread_count()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_thread_id.md b/docs-src/api/c/envset_thread_id.md index a50afb2b4..2489236a9 100644 --- a/docs-src/api/c/envset_thread_id.md +++ b/docs-src/api/c/envset_thread_id.md @@ -13,7 +13,7 @@ DB_ENV->set_thread_id(DB_ENV *dbenv, void (*thread_id)(DB_ENV *dbenv, pid_t *pid, db_threadid_t *tid)); ``` -Declare a function that returns a unique identifier pair for the current thread of control. The `DB_ENV->set_thread_id()` method supports the DB_ENV->failchk() method. For more information, see Architecting Data Store and Concurrent Data Store applications , and Architecting Transactional Data Store applications , both in the *Berkeley DB Programmer's Reference Guide*. +Declare a function that returns a unique identifier pair for the current thread of control. The `DB_ENV->set_thread_id()` method supports the DB_ENV->failchk() method. For more information, see Architecting Data Store and Concurrent Data Store applications , and Architecting Transactional Data Store applications , both in the *Berkeley DB Programmer's Reference Guide*. The `DB_ENV->set_thread_id()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_timeout.md b/docs-src/api/c/envset_timeout.md index 91a581a71..0f2c3d131 100644 --- a/docs-src/api/c/envset_timeout.md +++ b/docs-src/api/c/envset_timeout.md @@ -39,7 +39,7 @@ The **flags** parameter must be set to one of the following values: Set the timeout value for locks in this database environment. - The database environment's lock timeout value may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lock_timeout", one or more whitespace characters, and the lock timeout value. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. + The database environment's lock timeout value may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_lock_timeout", one or more whitespace characters, and the lock timeout value. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. This flag configures a database environment, not only operations performed using the specified DB_ENV handle. @@ -47,7 +47,7 @@ The **flags** parameter must be set to one of the following values: Set the timeout value on how long to wait for processes to exit the environment before recovery is started when the DB_ENV->open() method was called with the DB_REGISTER flag and recovery must be performed. - This wait timeout value may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_reg_timeout", one or more whitespace characters, and the wait timeout value. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. + This wait timeout value may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_reg_timeout", one or more whitespace characters, and the wait timeout value. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. This flag configures operations performed using the specified DB_ENV handle. @@ -55,7 +55,7 @@ The **flags** parameter must be set to one of the following values: Set the timeout value for transactions in this database environment. - The database environment's transaction timeout value may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_txn_timeout", one or more whitespace characters, and the transaction timeout value. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. + The database environment's transaction timeout value may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_txn_timeout", one or more whitespace characters, and the transaction timeout value. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. This flag configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_tmp_dir.md b/docs-src/api/c/envset_tmp_dir.md index c5096b81c..54838711a 100644 --- a/docs-src/api/c/envset_tmp_dir.md +++ b/docs-src/api/c/envset_tmp_dir.md @@ -46,7 +46,7 @@ Environment variables are only checked if one of the DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_tmp_dir", one or more whitespace characters, and the directory name. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's temporary file directory may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_tmp_dir", one or more whitespace characters, and the directory name. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_tmp_dir()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_tx_max.md b/docs-src/api/c/envset_tx_max.md index 8c5003314..72c3eacf6 100644 --- a/docs-src/api/c/envset_tx_max.md +++ b/docs-src/api/c/envset_tx_max.md @@ -18,7 +18,7 @@ Transactions that update multiversion databases are not freed until the last pag When all of the memory available in the database environment for transactions is in use, calls to DB_ENV->txn_begin() will fail (until some active transactions complete). If `DB_ENV->set_tx_max()` is never called, the database environment is configured to support at least 100 active transactions. -The database environment's number of active transactions may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_tx_max", one or more whitespace characters, and the number of transactions. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's number of active transactions may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_tx_max", one or more whitespace characters, and the number of transactions. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_tx_max()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/envset_tx_timestamp.md b/docs-src/api/c/envset_tx_timestamp.md index 021b7dab3..81c946cd7 100644 --- a/docs-src/api/c/envset_tx_timestamp.md +++ b/docs-src/api/c/envset_tx_timestamp.md @@ -14,7 +14,7 @@ DB_ENV->set_tx_timestamp(DB_ENV *dbenv, time_t *timestamp); Recover to the time specified by **timestamp** rather than to the most current possible date. -Once a database environment has been upgraded to a new version of Berkeley DB involving a log format change (see Upgrading Berkeley DB installations), it is no longer possible to recover to a specific time before that upgrade. +Once a database environment has been upgraded to a new version of Berkeley DB involving a log format change (see Upgrading Berkeley DB installations), it is no longer possible to recover to a specific time before that upgrade. The `DB_ENV->set_tx_timestamp()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envset_verbose.md b/docs-src/api/c/envset_verbose.md index fc6c91909..658d640db 100644 --- a/docs-src/api/c/envset_verbose.md +++ b/docs-src/api/c/envset_verbose.md @@ -14,7 +14,7 @@ DB_ENV->set_verbose(DB_ENV *dbenv, u_int32_t which, int onoff); The `DB_ENV->set_verbose()` method turns specific additional informational and debugging messages in the Berkeley DB message output on and off. To see the additional messages, verbose messages must also be configured for the application. For more information on verbose messages, see the DB_ENV->set_msgfile() method. -The database environment's messages may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_verbose", one or more whitespace characters, and the method **which** parameter as a string and optionally one or more whitespace characters, and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_verbose DB_VERB_RECOVERY" or "set_verbose DB_VERB_RECOVERY on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's messages may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_verbose", one or more whitespace characters, and the method **which** parameter as a string and optionally one or more whitespace characters, and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "set_verbose DB_VERB_RECOVERY" or "set_verbose DB_VERB_RECOVERY on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_verbose()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/envstrerror.md b/docs-src/api/c/envstrerror.md index 5304aa1d0..2b9b36465 100644 --- a/docs-src/api/c/envstrerror.md +++ b/docs-src/api/c/envstrerror.md @@ -14,7 +14,7 @@ db_strerror(int error); The `db_strerror()` method returns an error message string corresponding to the error number **error** parameter. -This function is a superset of the ANSI C X3.159-1989 (ANSI C) **strerror**(3) function. If the error number **error** is greater than or equal to 0, then the string returned by the system function **strerror**(3) is returned. If the error number is less than 0, an error string appropriate to the corresponding Berkeley DB library error is returned. See Error returns to applications for more information. +This function is a superset of the ANSI C X3.159-1989 (ANSI C) **strerror**(3) function. If the error number **error** is greater than or equal to 0, then the string returned by the system function **strerror**(3) is returned. If the error number is less than 0, an error string appropriate to the corresponding Berkeley DB library error is returned. See Error returns to applications for more information. ### Parameters diff --git a/docs-src/api/c/envtxn_applied.md b/docs-src/api/c/envtxn_applied.md index b2f9ddd5b..832768da4 100644 --- a/docs-src/api/c/envtxn_applied.md +++ b/docs-src/api/c/envtxn_applied.md @@ -15,7 +15,7 @@ DB_ENV->txn_applied(DB_ENV *env, DB_TXN_TOKEN *token, The `DB_ENV->txn_applied()` method checks to see if a specified transaction has been replicated from the master of a replication group. It may be called by applications using either the Base API or the Replication Manager. -If the transaction has not yet arrived, this method will block for the amount of time specified on the `timeout` parameter while it waits for the result to be determined. For more information, please refer to the Read your writes consistency section in the *Berkeley DB Programmer's Reference Guide*. +If the transaction has not yet arrived, this method will block for the amount of time specified on the `timeout` parameter while it waits for the result to be determined. For more information, please refer to the Read your writes consistency section in the *Berkeley DB Programmer's Reference Guide*. The `DB_ENV->txn_applied()` method may not be called before the DB_ENV->open() method. diff --git a/docs-src/api/c/frame_index.md b/docs-src/api/c/frame_index.md index 379ef4cf2..b36748fb0 100644 --- a/docs-src/api/c/frame_index.md +++ b/docs-src/api/c/frame_index.md @@ -3,4 +3,44 @@ title: "Berkeley DB C API Reference" api-name: "Berkeley DB C API Reference" source: docs/api_reference/C/frame_index.html --- + Home + Utilities + + + + All Methods + + Reference Guide + + + + Databases + + Cursors + + Key/Data Pairs + + Environments + + Locking + + Logging + + Memory Pool + + Mutexes + + Replication + + Sequences + + Transactions + + + + Historic APIs + + Static Functions + + DB_CONFIG Parameters diff --git a/docs-src/api/c/lockget.md b/docs-src/api/c/lockget.md index adeb03bc5..eb5f3d6b9 100644 --- a/docs-src/api/c/lockget.md +++ b/docs-src/api/c/lockget.md @@ -34,7 +34,7 @@ The **flags** parameter must be set to 0 or the following value: #### object -The **object** parameter is an untyped byte string that specifies the object to be locked. Applications using the locking subsystem directly while also doing locking via the Berkeley DB access methods must take care not to inadvertently lock objects that happen to be equal to the unique file IDs used to lock files. See Access method locking conventions in the *Berkeley DB Programmer's Reference Guide* for more information. +The **object** parameter is an untyped byte string that specifies the object to be locked. Applications using the locking subsystem directly while also doing locking via the Berkeley DB access methods must take care not to inadvertently lock objects that happen to be equal to the unique file IDs used to lock files. See Access method locking conventions in the *Berkeley DB Programmer's Reference Guide* for more information. #### lock_mode @@ -60,7 +60,7 @@ The **lock_mode** parameter is used as an index into the environment's lock conf intention to read and write (shared) -See DB_ENV->set_lk_conflicts() and Standard Lock Modes for more information on the lock conflict matrix. +See DB_ENV->set_lk_conflicts() and Standard Lock Modes for more information on the lock conflict matrix. #### lock diff --git a/docs-src/api/c/lockvec.md b/docs-src/api/c/lockvec.md index 73019bea3..8876f1e92 100644 --- a/docs-src/api/c/lockvec.md +++ b/docs-src/api/c/lockvec.md @@ -97,11 +97,11 @@ A DB_LOCKREQ structure has at least the following fields: intention to read and write (shared) - See DB_ENV->set_lk_conflicts() and Standard Lock Modes for more information on the lock conflict matrix. + See DB_ENV->set_lk_conflicts() and Standard Lock Modes for more information on the lock conflict matrix. - ****const DBT obj;**** - An untyped byte string that specifies the object to be locked or released. Applications using the locking subsystem directly while also doing locking via the Berkeley DB access methods must take care not to inadvertently lock objects that happen to be equal to the unique file IDs used to lock files. See Access method locking conventions in the *Berkeley DB Programmer's Reference Guide* for more information. + An untyped byte string that specifies the object to be locked or released. Applications using the locking subsystem directly while also doing locking via the Berkeley DB access methods must take care not to inadvertently lock objects that happen to be equal to the unique file IDs used to lock files. See Access method locking conventions in the *Berkeley DB Programmer's Reference Guide* for more information. - **u_int32_t timeout;** diff --git a/docs-src/api/c/log_set_config_parameter.md b/docs-src/api/c/log_set_config_parameter.md index 1bfadbc36..ae2fc35be 100644 --- a/docs-src/api/c/log_set_config_parameter.md +++ b/docs-src/api/c/log_set_config_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/log_set_config_parameter.html Configures the Berkeley DB logging subsystem. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `log_set_config`, one or more whitespace characters, method **flag** parameter as a string, optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on.` +The syntax of this parameter in the DB_CONFIG file is a single line with the string `log_set_config`, one or more whitespace characters, method **flag** parameter as a string, optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on.` The method **flag** parameters are: diff --git a/docs-src/api/c/logcget.md b/docs-src/api/c/logcget.md index 81349fd0e..4469cdefc 100644 --- a/docs-src/api/c/logcget.md +++ b/docs-src/api/c/logcget.md @@ -38,13 +38,13 @@ The **flags** parameter must be set to one of the following values: The first record from any of the log files found in the log directory is returned in the **data** parameter. The **lsn** parameter is overwritten with the DB_LSN of the record returned. - The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_FIRST is set and the log is empty. + The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_FIRST is set and the log is empty. - `DB_LAST` The last record in the log is returned in the **data** parameter. The **lsn** parameter is overwritten with the DB_LSN of the record returned. - The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_LAST is set and the log is empty. + The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_LAST is set and the log is empty. - `DB_NEXT` @@ -52,7 +52,7 @@ The **flags** parameter must be set to one of the following values: If the cursor has not been initialized via DB_FIRST, DB_LAST, DB_SET, DB_NEXT, or DB_PREV, `DB_LOGC->get()` will return the first record in the log. - The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_NEXT is set and the last log record has already been returned or the log is empty. + The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_NEXT is set and the last log record has already been returned or the log is empty. - `DB_PREV` @@ -60,7 +60,7 @@ The **flags** parameter must be set to one of the following values: If the cursor has not been initialized via DB_FIRST, DB_LAST, DB_SET, DB_NEXT, or DB_PREV, `DB_LOGC->get()` will return the last record in the log. - The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_PREV is set and the first log record has already been returned or the log is empty. + The `DB_LOGC->get()` method will return DB_NOTFOUND if DB_PREV is set and the first log record has already been returned or the log is empty. - `DB_SET` diff --git a/docs-src/api/c/mempfopen.md b/docs-src/api/c/mempfopen.md index f6ce6896e..4e4bf3caa 100644 --- a/docs-src/api/c/mempfopen.md +++ b/docs-src/api/c/mempfopen.md @@ -39,7 +39,7 @@ The **flags** parameter must be set to zero or by bitwise inclusively **OR**'ing - `DB_MULTIVERSION` - Open the file with support for multiversion concurrency control. Calls to DB_MPOOLFILE->get() with dirty pages will cause copies to be made in the cache. + Open the file with support for multiversion concurrency control. Calls to DB_MPOOLFILE->get() with dirty pages will cause copies to be made in the cache. - `DB_NOMMAP` diff --git a/docs-src/api/c/mempset_mp_max_openfd.md b/docs-src/api/c/mempset_mp_max_openfd.md index f5f0cd7de..f71e93a20 100644 --- a/docs-src/api/c/mempset_mp_max_openfd.md +++ b/docs-src/api/c/mempset_mp_max_openfd.md @@ -14,7 +14,7 @@ DB_ENV->set_mp_max_openfd(DB_ENV *env, int maxopenfd); The `DB_ENV->set_mp_max_openfd()` method limits the number of file descriptors the library will open concurrently when flushing dirty pages from the cache. -The database environment's limit on open file descriptors to flush dirty pages may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_mp_max_openfd", one or more whitespace characters, and the number of open file descriptors. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's limit on open file descriptors to flush dirty pages may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_mp_max_openfd", one or more whitespace characters, and the number of open file descriptors. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The DB_ENV->set_mp_max_openfd() method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/mempset_mp_max_write.md b/docs-src/api/c/mempset_mp_max_write.md index 51b3bad4c..6aac629fc 100644 --- a/docs-src/api/c/mempset_mp_max_write.md +++ b/docs-src/api/c/mempset_mp_max_write.md @@ -15,7 +15,7 @@ DB_ENV->set_mp_max_write(DB_ENV *env, int maxwrite, The `DB_ENV->set_mp_max_write()` method limits the number of sequential write operations scheduled by the library when flushing dirty pages from the cache. -The database environment's maximum number of sequential write operations may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_mp_max_write", one or more whitespace characters, and the maximum number of sequential writes and the number of microseconds to sleep, also separated by whitespace characters. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's maximum number of sequential write operations may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_mp_max_write", one or more whitespace characters, and the maximum number of sequential writes and the number of microseconds to sleep, also separated by whitespace characters. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->set_mp_max_write()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/mutex_set_align_parameter.md b/docs-src/api/c/mutex_set_align_parameter.md index 11d83292d..0a14db513 100644 --- a/docs-src/api/c/mutex_set_align_parameter.md +++ b/docs-src/api/c/mutex_set_align_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/mutex_set_align_parameter.html Sets the mutex alignment, in bytes. It is sometimes advantageous to align mutexes on specific byte boundaries in order to minimize cache line collisions. This parameter specifies an alignment for mutexes allocated by Berkeley DB. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `mutex_set_align`, one or more whitespace characters, and the mutex alignment in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `mutex_set_align`, one or more whitespace characters, and the mutex alignment in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. For more information, see DB_ENV->mutex_set_align(). diff --git a/docs-src/api/c/mutex_set_increment_parameter.md b/docs-src/api/c/mutex_set_increment_parameter.md index d6259d065..657e4142d 100644 --- a/docs-src/api/c/mutex_set_increment_parameter.md +++ b/docs-src/api/c/mutex_set_increment_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/mutex_set_increment_parameter.html Configures the number of additional mutexes to allocate. If an application will allocate mutexes for its own use, this parameter is used to add a number of mutexes to the default allocation. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `mutex_set_increment`, one or more whitespace characters, and the number of additional mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `mutex_set_increment`, one or more whitespace characters, and the number of additional mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. For more information, see DB_ENV->mutex_set_increment(). diff --git a/docs-src/api/c/mutex_set_max_parameter.md b/docs-src/api/c/mutex_set_max_parameter.md index 6ab4dd44d..e8799f5c0 100644 --- a/docs-src/api/c/mutex_set_max_parameter.md +++ b/docs-src/api/c/mutex_set_max_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/mutex_set_max_parameter.html Configures the total number of mutexes to allocate. Berkeley DB allocates a default number of mutexes based on the initial configuration of the database environment. That default calculation may be too small if the application has an unusual need for mutexes. This parameter is used to specify an absolute number of mutexes to allocate. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `mutex_set_max`, one or more whitespace characters, and the total number of mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `mutex_set_max`, one or more whitespace characters, and the total number of mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. For more information, see DB_ENV->mutex_set_max(). diff --git a/docs-src/api/c/mutex_set_tas_spins_parameter.md b/docs-src/api/c/mutex_set_tas_spins_parameter.md index d7990d619..d51182bfd 100644 --- a/docs-src/api/c/mutex_set_tas_spins_parameter.md +++ b/docs-src/api/c/mutex_set_tas_spins_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/mutex_set_tas_spins_parameter.html Specifies the number of times the test-and-set mutexes should spin without blocking. The value defaults to 1 time on uniprocessor systems and to 50 times the number of processors on multiprocessor systems. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_tas_spins`, one or more whitespace characters, and the number of spins. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_tas_spins`, one or more whitespace characters, and the number of spins. For more information, see DB_ENV->mutex_set_tas_spins(). diff --git a/docs-src/api/c/mutexset_align.md b/docs-src/api/c/mutexset_align.md index 0c7d9d72c..c937779a2 100644 --- a/docs-src/api/c/mutexset_align.md +++ b/docs-src/api/c/mutexset_align.md @@ -16,7 +16,7 @@ Set the mutex alignment, in bytes. It is sometimes advantageous to align mutexes on specific byte boundaries in order to minimize cache line collisions. The `DB_ENV->mutex_set_align()` method specifies an alignment for mutexes allocated by Berkeley DB. -The database environment's mutex alignment may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_align", one or more whitespace characters, and the mutex alignment in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's mutex alignment may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_align", one or more whitespace characters, and the mutex alignment in bytes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->mutex_set_align()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/mutexset_increment.md b/docs-src/api/c/mutexset_increment.md index c5f4b3126..4be22aa43 100644 --- a/docs-src/api/c/mutexset_increment.md +++ b/docs-src/api/c/mutexset_increment.md @@ -18,7 +18,7 @@ If an application will allocate mutexes for its own use, the `DB_ENV->mutex_set_ Calling the `DB_ENV->mutex_set_increment()` method discards any value previously set using the DB_ENV->mutex_set_max() method. -The database environment's number of additional mutexes may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_increment", one or more whitespace characters, and the number of additional mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's number of additional mutexes may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_increment", one or more whitespace characters, and the number of additional mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->mutex_set_increment()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/mutexset_init.md b/docs-src/api/c/mutexset_init.md index 5f2f711a8..5e23ef35c 100644 --- a/docs-src/api/c/mutexset_init.md +++ b/docs-src/api/c/mutexset_init.md @@ -16,7 +16,7 @@ Configure the inital number of mutexes to allocate. Berkeley DB allocates a default number of mutexes based on the initial configuration of the database environment. The `DB_ENV->mutex_set_init()` method is used to override this default number of mutexes to allocate. This may be done to either speed up startup, or to force more work to be done at startup to avoid later contention due to allocation. -The database environment's inital number of mutexes may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_init", one or more whitespace characters, and the initial number of mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's inital number of mutexes may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_init", one or more whitespace characters, and the initial number of mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->mutex_set_init()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/mutexset_max.md b/docs-src/api/c/mutexset_max.md index 6baa0e1dc..011117174 100644 --- a/docs-src/api/c/mutexset_max.md +++ b/docs-src/api/c/mutexset_max.md @@ -18,7 +18,7 @@ You can use this method to override DB's mutex calculation, but it is not recomm Calling the `DB_ENV->mutex_set_max()` method discards any value previously set using the DB_ENV->mutex_set_increment() method. -The database environment's total number of mutexes may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_max", one or more whitespace characters, and the total number of mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's total number of mutexes may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "mutex_set_max", one or more whitespace characters, and the total number of mutexes. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->mutex_set_max()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/mutexset_tas_spins.md b/docs-src/api/c/mutexset_tas_spins.md index 23ef284a8..d3d572971 100644 --- a/docs-src/api/c/mutexset_tas_spins.md +++ b/docs-src/api/c/mutexset_tas_spins.md @@ -14,7 +14,7 @@ DB_ENV->mutex_set_tas_spins(DB_ENV *dbenv, u_int32_t tas_spins); Specify that test-and-set mutexes should spin **tas_spins** times without blocking. The value defaults to 1 on uniprocessor systems and to 50 times the number of processors on multiprocessor systems. -The database environment's test-and-set spin count may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_tas_spins", one or more whitespace characters, and the number of spins. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's test-and-set spin count may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_tas_spins", one or more whitespace characters, and the number of spins. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->mutex_set_tas_spins()` method configures operations performed using the specified DB_ENV handle, not all operations performed on the underlying database environment. diff --git a/docs-src/api/c/rep_set_clockskew_parameter.md b/docs-src/api/c/rep_set_clockskew_parameter.md index ce06951a7..e57726d80 100644 --- a/docs-src/api/c/rep_set_clockskew_parameter.md +++ b/docs-src/api/c/rep_set_clockskew_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_clockskew_parameter.html Sets the clock skew ratio among replication group members based on the fastest and slowest measurements among the group for use with master leases. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_clockskew`, one or more whitespace characters, and the clockskew specified in two parts: the fast_clock and the slow_clock. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_clockskew`, one or more whitespace characters, and the clockskew specified in two parts: the fast_clock and the slow_clock. For example: diff --git a/docs-src/api/c/rep_set_config_parameter.md b/docs-src/api/c/rep_set_config_parameter.md index 8be10293f..760dfeab7 100644 --- a/docs-src/api/c/rep_set_config_parameter.md +++ b/docs-src/api/c/rep_set_config_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_config_parameter.html Configures the Berkeley DB replication subsystem. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_config`, one or more whitespace characters, and the method parameter as a string and optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on`. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_config`, one or more whitespace characters, and the method parameter as a string and optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on`. For example: ``` c rep_set_config DB_REP_CONF_NOWAIT on diff --git a/docs-src/api/c/rep_set_limit_parameter.md b/docs-src/api/c/rep_set_limit_parameter.md index 7c20703b8..6d965543e 100644 --- a/docs-src/api/c/rep_set_limit_parameter.md +++ b/docs-src/api/c/rep_set_limit_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_limit_parameter.html Sets record transmission throttling. This is a bytecount limit on the amount of data that will be transmitted from a site in response to a single message processed by the `DB_ENV->rep_process_message` method. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_limit`, one or more whitespace characters, and the limit specified in two parts: the gigabytes and the bytes values. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_limit`, one or more whitespace characters, and the limit specified in two parts: the gigabytes and the bytes values. For example: ``` c rep_set_limit 0 1048576 diff --git a/docs-src/api/c/rep_set_nsites_parameter.md b/docs-src/api/c/rep_set_nsites_parameter.md index aab207ba9..f4098154f 100644 --- a/docs-src/api/c/rep_set_nsites_parameter.md +++ b/docs-src/api/c/rep_set_nsites_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_nsites_parameter.html Specifies the total number of sites in a replication group. This parameter is ignored for Replication Manager applications. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_nsites`, one or more whitespace characters, and the number of sites specified. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_nsites`, one or more whitespace characters, and the number of sites specified. For example: ``` c rep_set_nsites 5 diff --git a/docs-src/api/c/rep_set_priority_parameter.md b/docs-src/api/c/rep_set_priority_parameter.md index 96ee60166..eecfec4a1 100644 --- a/docs-src/api/c/rep_set_priority_parameter.md +++ b/docs-src/api/c/rep_set_priority_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_priority_parameter.html Specifies the database environment's priority in replication group elections. A special value of 0 indicates that this environment cannot be a replication group master. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_priority`, one or more whitespace characters, and the priority of this site. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_priority`, one or more whitespace characters, and the priority of this site. For example: ``` c rep_set_priority 1 diff --git a/docs-src/api/c/rep_set_request_parameter.md b/docs-src/api/c/rep_set_request_parameter.md index 6a312a390..fe44b568b 100644 --- a/docs-src/api/c/rep_set_request_parameter.md +++ b/docs-src/api/c/rep_set_request_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_request_parameter.html Sets a threshold for the minimum and maximum time that a client waits before requesting retransmission of a missing message. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_request`, one or more whitespace characters, and the request time specified in two parts: the min and the max. Specifically, if the client detects a gap in the sequence of incoming log records or database pages, Berkeley DB will wait for at least min microseconds before requesting retransmission of the missing record. Berkeley DB will double that amount before requesting the same missing record again, and so on, up to a maximum threshold of max microseconds. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_request`, one or more whitespace characters, and the request time specified in two parts: the min and the max. Specifically, if the client detects a gap in the sequence of incoming log records or database pages, Berkeley DB will wait for at least min microseconds before requesting retransmission of the missing record. Berkeley DB will double that amount before requesting the same missing record again, and so on, up to a maximum threshold of max microseconds. By default the minimum is 40000 and the maximum is 1280000 (1.28 seconds). These defaults are fairly arbitrary and the application likely needs to adjust these. The values should be based on expected load and performance characteristics of the master and client host platforms and transport infrastructure as well as round-trip message time. diff --git a/docs-src/api/c/rep_set_timeout_parameter.md b/docs-src/api/c/rep_set_timeout_parameter.md index 8269c975f..a347d355f 100644 --- a/docs-src/api/c/rep_set_timeout_parameter.md +++ b/docs-src/api/c/rep_set_timeout_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/rep_set_timeout_parameter.html Specifies a variety of replication timeout values. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_timeout`, one or more whitespace characters, and the flag specified as a string and the timeout specified as two parts. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `rep_set_timeout`, one or more whitespace characters, and the flag specified as a string and the timeout specified as two parts. For example: ``` c rep_set_timeout DB_REP_CONNECTION_RETRY 15000000 diff --git a/docs-src/api/c/repclockskew.md b/docs-src/api/c/repclockskew.md index 733ced10b..6d9d88d23 100644 --- a/docs-src/api/c/repclockskew.md +++ b/docs-src/api/c/repclockskew.md @@ -13,11 +13,11 @@ DB_ENV->rep_set_clockskew(DB_ENV *env, u_int32_t fast_clock, u_int32_t slow_clock); ``` -The `DB_ENV->rep_set_clockskew()` method sets the clock skew ratio among replication group members based on the fastest and slowest measurements among the group for use with master leases. Calling this method is optional; the default values for clock skew assume no skew. The user must also configure leases via the DB_ENV->rep_set_config() method. Additionally, the user must also set the master lease timeout via the DB_ENV->rep_set_timeout() method. For Base API applications, the user must also set the number of sites in the replication group via the DB_ENV->rep_set_nsites() method. These methods may be called in any order. For a description of the clock skew values, see Clock skew in the *Berkeley DB Programmer's Reference Guide*. For a description of master leases, see Master leases in the *Berkeley DB Programmer's Reference Guide*. +The `DB_ENV->rep_set_clockskew()` method sets the clock skew ratio among replication group members based on the fastest and slowest measurements among the group for use with master leases. Calling this method is optional; the default values for clock skew assume no skew. The user must also configure leases via the DB_ENV->rep_set_config() method. Additionally, the user must also set the master lease timeout via the DB_ENV->rep_set_timeout() method. For Base API applications, the user must also set the number of sites in the replication group via the DB_ENV->rep_set_nsites() method. These methods may be called in any order. For a description of the clock skew values, see Clock skew in the *Berkeley DB Programmer's Reference Guide*. For a description of master leases, see Master leases in the *Berkeley DB Programmer's Reference Guide*. These arguments can be used to express either raw measurements of a clock timing experiment or a percentage across machines. For example, if a group of sites has a 2% variance, then **fast_clock** should be set to 102, and **slow_clock** should be set to 100. Or, for a 0.03% difference, you can use 10003 and 10000 respectively. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_clockskew", one or more whitespace characters, and the clockskew specified in two parts: the fast_clock and the slow_clock. For example, "rep_set_clockskew 102 100". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_clockskew", one or more whitespace characters, and the clockskew specified in two parts: the fast_clock and the slow_clock. For example, "rep_set_clockskew 102 100". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->rep_set_clockskew()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/repconfig.md b/docs-src/api/c/repconfig.md index 14cfa31ea..054ca8b7a 100644 --- a/docs-src/api/c/repconfig.md +++ b/docs-src/api/c/repconfig.md @@ -14,7 +14,7 @@ DB_ENV->rep_set_config(DB_ENV *env, u_int32_t which, int onoff); The `DB_ENV->rep_set_config()` method configures the Berkeley DB replication subsystem. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_config", one or more whitespace characters, and the method **which** parameter as a string and optionally one or more whitespace characters, and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "rep_set_config DB_REP_CONF_NOWAIT" or "rep_set_config DB_REP_CONF_NOWAIT on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_config", one or more whitespace characters, and the method **which** parameter as a string and optionally one or more whitespace characters, and the string "on" or "off". If the optional string is omitted, the default is "on"; for example, "rep_set_config DB_REP_CONF_NOWAIT" or "rep_set_config DB_REP_CONF_NOWAIT on". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->rep_set_config()` method configures a database environment, not only operations performed using the specified DB_ENV handle. @@ -52,7 +52,7 @@ The **which** parameter must be set to one of the following values: - An application has a slight risk that the wrong site may win an election, resulting in the loss of some data. This is consistent with the general loss of data durability when running in-memory. - - Replication Manager applications will no longer maintain group membership information persistently on-disk. For more information, see Managing Replication Files in the *Berkeley DB Programmer's Reference Guide*. + - Replication Manager applications will no longer maintain group membership information persistently on-disk. For more information, see Managing Replication Files in the *Berkeley DB Programmer's Reference Guide*. This configuration flag can only be turned on before the environment is opened with the DB_ENV->open() method. Its value cannot be changed while the environment is open. All sites in the replication group should have the same value for this configuration flag. @@ -82,7 +82,7 @@ The **which** parameter must be set to one of the following values: - `DB_REPMGR_CONF_2SITE_STRICT` - Replication Manager observes the strict "majority" rule in managing elections, even in a group with only 2 sites. This means the client in a 2-site group will be unable to take over as master if the original master fails or becomes disconnected. (See the Special considerations for two-site replication groups section in the *Berkeley DB Programmer's Reference Guide* for more information.) Both sites in the replication group should have the same value for this configuration flag. This option is turned on by default. + Replication Manager observes the strict "majority" rule in managing elections, even in a group with only 2 sites. This means the client in a 2-site group will be unable to take over as master if the original master fails or becomes disconnected. (See the Special considerations for two-site replication groups section in the *Berkeley DB Programmer's Reference Guide* for more information.) Both sites in the replication group should have the same value for this configuration flag. This option is turned on by default. #### onoff diff --git a/docs-src/api/c/repelect.md b/docs-src/api/c/repelect.md index 8cbb0272c..00efc8db5 100644 --- a/docs-src/api/c/repelect.md +++ b/docs-src/api/c/repelect.md @@ -49,7 +49,7 @@ We recommend **nsites** be set to: when choosing a new master after a current master fails. This allows the group to reach a consensus without having to wait for the timeout to expire. -When choosing a master from among a group of client sites all restarting at the same time, it makes more sense to set **nsites** to the total number of sites in the group, since there is no known missing site. Furthermore, in order to ensure the best choice from among sites that may take longer to boot than the local site, setting **nvotes** also to this same total number of sites will guarantee that every site in the group is considered. Alternatively, using the special timeout for full elections allows full participation on restart but allows election of a master if one site does not reboot and rejoin the group in a reasonable amount of time. (See the Elections section in the *Berkeley DB Programmer's Reference Guide* for more information.) +When choosing a master from among a group of client sites all restarting at the same time, it makes more sense to set **nsites** to the total number of sites in the group, since there is no known missing site. Furthermore, in order to ensure the best choice from among sites that may take longer to boot than the local site, setting **nvotes** also to this same total number of sites will guarantee that every site in the group is considered. Alternatively, using the special timeout for full elections allows full participation on restart but allows election of a master if one site does not reboot and rejoin the group in a reasonable amount of time. (See the Elections section in the *Berkeley DB Programmer's Reference Guide* for more information.) Setting **nsites** to lower values can increase the speed of an election, but can also result in election failure, and is usually not recommended. diff --git a/docs-src/api/c/repmessage.md b/docs-src/api/c/repmessage.md index 07e6ca407..a8ec769d1 100644 --- a/docs-src/api/c/repmessage.md +++ b/docs-src/api/c/repmessage.md @@ -65,7 +65,7 @@ The **rec** parameter should reference a copy of the **rec** parameter specified #### envid -The **envid** parameter should contain the local identifier that corresponds to the environment that sent the message to be processed (see Replication environment IDs for more information). +The **envid** parameter should contain the local identifier that corresponds to the environment that sent the message to be processed (see Replication environment IDs for more information). #### ret_lsnp diff --git a/docs-src/api/c/repmgr_set_ack_policy_parameter.md b/docs-src/api/c/repmgr_set_ack_policy_parameter.md index fdf5a5955..f367fcb0b 100644 --- a/docs-src/api/c/repmgr_set_ack_policy_parameter.md +++ b/docs-src/api/c/repmgr_set_ack_policy_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/repmgr_set_ack_policy_parameter.html Specifies how master and client sites will handle acknowledgment of replication messages which are necessary for "permanent" records. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `repmgr_set_ack_policy`, one or more whitespace characters, and the ack_policy parameter specified as a string. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `repmgr_set_ack_policy`, one or more whitespace characters, and the ack_policy parameter specified as a string. For example: ``` c repmgr_set_ack_policy DB_REPMGR_ACKS_ALL diff --git a/docs-src/api/c/repmgr_site_parameter.md b/docs-src/api/c/repmgr_site_parameter.md index 14904bf6c..6a8b63644 100644 --- a/docs-src/api/c/repmgr_site_parameter.md +++ b/docs-src/api/c/repmgr_site_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/repmgr_site_parameter.html Identifies a Replication Manager site. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `repmgr_site`, one or more whitespace characters, the host and port parameters specified as a string and an integer respectively. This can optionally be followed by one or more space-delimited keywords and `on`/`off`. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `repmgr_site`, one or more whitespace characters, the host and port parameters specified as a string and an integer respectively. This can optionally be followed by one or more space-delimited keywords and `on`/`off`. For example: ``` c repmgr_site example.com 49200 db_local_site on db_legacy off diff --git a/docs-src/api/c/repmgrset_ack_policy.md b/docs-src/api/c/repmgrset_ack_policy.md index 662f63589..e1db46bbf 100644 --- a/docs-src/api/c/repmgrset_ack_policy.md +++ b/docs-src/api/c/repmgrset_ack_policy.md @@ -14,9 +14,9 @@ DB_ENV->repmgr_set_ack_policy(DB_ENV *env, int ack_policy); The `DB_ENV->repmgr_set_ack_policy()` method specifies how master and client sites will handle acknowledgment of replication messages which are necessary for "permanent" records. The current implementation requires all sites in a replication group configure the same acknowledgement policy. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "repmgr_set_ack_policy", one or more whitespace characters, and the **ack_policy** parameter specified as a string. For example, "repmgr_set_ack_policy DB_REPMGR_ACKS_ALL". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "repmgr_set_ack_policy", one or more whitespace characters, and the **ack_policy** parameter specified as a string. For example, "repmgr_set_ack_policy DB_REPMGR_ACKS_ALL". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. -Waiting for client acknowledgements is always limited by the DB_REP_ACK_TIMEOUT specified by the DB_ENV->rep_set_timeout() method. If an insufficient number of client acknowledgements have been received, then the master will invoke the event callback function, if set, with the DB_EVENT_REP_PERM_FAILED value. (See the Choosing a Replication Manager Ack Policy section in the *Berkeley DB Programmer's Reference Guide* for more information.) +Waiting for client acknowledgements is always limited by the DB_REP_ACK_TIMEOUT specified by the DB_ENV->rep_set_timeout() method. If an insufficient number of client acknowledgements have been received, then the master will invoke the event callback function, if set, with the DB_EVENT_REP_PERM_FAILED value. (See the Choosing a Replication Manager Ack Policy section in the *Berkeley DB Programmer's Reference Guide* for more information.) The `DB_ENV->repmgr_set_ack_policy()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/repmgrstart.md b/docs-src/api/c/repmgrstart.md index 46891ebe0..bbe513855 100644 --- a/docs-src/api/c/repmgrstart.md +++ b/docs-src/api/c/repmgrstart.md @@ -30,7 +30,7 @@ There are two ways to build Berkeley DB replication applications: the most commo For more information on building Replication Manager applications, please see the *Replication Getting Started Guide* included in the Berkeley DB documentation. -Applications with special needs (for example, applications using network protocols not supported by the Berkeley DB Replication Manager), must perform additional configuration and call other Berkeley DB replication Base API methods. For more information on building Base API applications, please see the Base API Methods section in the *Berkeley DB Programmer's Reference Guide*. +Applications with special needs (for example, applications using network protocols not supported by the Berkeley DB Replication Manager), must perform additional configuration and call other Berkeley DB replication Base API methods. For more information on building Base API applications, please see the Base API Methods section in the *Berkeley DB Programmer's Reference Guide*. Starting the Replication Manager consists of opening the TCP/IP listening socket to accept incoming connections, and starting all necessary background threads. When multiple processes share a database environment, only one process can open the listening socket; the `DB_ENV->repmgr_start()` method automatically opens the socket in the first process to call it, and skips this step in the later calls from other processes. diff --git a/docs-src/api/c/repnsites.md b/docs-src/api/c/repnsites.md index 9c4f92e1c..9afbca0ec 100644 --- a/docs-src/api/c/repnsites.md +++ b/docs-src/api/c/repnsites.md @@ -16,7 +16,7 @@ The `DB_ENV->rep_set_nsites()` method specifies the total number of sites in a r The `DB_ENV->rep_set_nsites()` method is typically called by Base API applications. (However, see also the DB_ENV->rep_elect() method **nsites** parameter.) -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_nsites", one or more whitespace characters, and the number of sites specified. For example, "rep_set_nsites 5" sets the number of sites to 5. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_nsites", one or more whitespace characters, and the number of sites specified. For example, "rep_set_nsites 5" sets the number of sites to 5. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->rep_set_nsites()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/reppriority.md b/docs-src/api/c/reppriority.md index 07e476861..1fb1e1d31 100644 --- a/docs-src/api/c/reppriority.md +++ b/docs-src/api/c/reppriority.md @@ -18,7 +18,7 @@ The `DB_ENV->rep_set_priority()` method specifies the database environment's pri The DB_ENV->repmgr_set_ack_policy() method describes *electable peers*, which are replication sites with a non-zero priority. For some acknowledgement policies, Replication Manager's computation of the durability result for each new update transaction is sensitive to whether each site in the group is a peer. Therefore, if you change a site's priority from a non-zero value to `0`, or from `0` to a non-zero value, this can invalidate the durability result of previously committed transactions. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_priority", one or more whitespace characters, and the priority of this site. For example, "rep_set_priority 1" sets the priority of this site to 1. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_priority", one or more whitespace characters, and the priority of this site. For example, "rep_set_priority 1" sets the priority of this site to 1. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. Note that if the application never explicitly sets a priority, then a default value of 100 is used. @@ -32,7 +32,7 @@ The `DB_ENV->rep_set_priority()` method returns a non-zero error value on failur #### priority -The priority of this database environment in the replication group. The priority must be a non-zero integer, or 0 if this environment cannot be a replication group master. (See Replication environment priorities for more information). +The priority of this database environment in the replication group. The priority must be a non-zero integer, or 0 if this environment cannot be a replication group master. (See Replication environment priorities for more information). ### Class diff --git a/docs-src/api/c/repset_limit.md b/docs-src/api/c/repset_limit.md index f448a83d6..59e591159 100644 --- a/docs-src/api/c/repset_limit.md +++ b/docs-src/api/c/repset_limit.md @@ -18,7 +18,7 @@ Record transmission throttling is turned on by default with a limit of 10MB. If the values passed to the `DB_ENV->rep_set_limit()` method are both zero, then the transmission limit is turned off. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_limit", one or more whitespace characters, and the limit specified in two parts: the gigabytes and the bytes values. For example, "rep_set_limit 0 1048576" sets a 1 megabyte limit. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_limit", one or more whitespace characters, and the limit specified in two parts: the gigabytes and the bytes values. For example, "rep_set_limit 0 1048576" sets a 1 megabyte limit. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->rep_set_limit()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/repset_request.md b/docs-src/api/c/repset_request.md index 0ffbed1e5..31d2c5b41 100644 --- a/docs-src/api/c/repset_request.md +++ b/docs-src/api/c/repset_request.md @@ -18,7 +18,7 @@ These values are thresholds only. Replication Manager applications use these val By default the minimum is 40000 and the maximum is 1280000 (1.28 seconds). These defaults are fairly arbitrary and the application likely needs to adjust these. The values should be based on expected load and performance characteristics of the master and client host platforms and transport infrastructure as well as round-trip message time. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_request", one or more whitespace characters, and the request times specified in two parts: the min and the max. For example, "rep_set_request 40000 1280000". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_request", one or more whitespace characters, and the request times specified in two parts: the min and the max. For example, "rep_set_request 40000 1280000". Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->rep_set_request()` method configures a database environment, not only operations performed using the specified DB_ENV handle. diff --git a/docs-src/api/c/repset_timeout.md b/docs-src/api/c/repset_timeout.md index b83e019cd..fe4883438 100644 --- a/docs-src/api/c/repset_timeout.md +++ b/docs-src/api/c/repset_timeout.md @@ -14,7 +14,7 @@ DB_ENV->rep_set_timeout(DB_ENV *env, int which, u_int32_t timeout); The `DB_ENV->rep_set_timeout()` method specifies a variety of replication timeout values. -The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_timeout", one or more whitespace characters, and the **which** parameter specified as a string and the timeout specified as two parts. For example, "rep_set_timeout DB_REP_CONNECTION_RETRY 15000000" specifies the connection retry timeout for 15 seconds. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The database environment's replication subsystem may also be configured using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "rep_set_timeout", one or more whitespace characters, and the **which** parameter specified as a string and the timeout specified as two parts. For example, "rep_set_timeout DB_REP_CONNECTION_RETRY 15000000" specifies the connection retry timeout for 15 seconds. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The `DB_ENV->rep_set_timeout()` method configures a database environment, not only operations performed using the specified DB_ENV handle. @@ -54,7 +54,7 @@ The **which** parameter must be set to one of the following values: - `DB_REP_FULL_ELECTION_TIMEOUT` - An optional configuration timeout period to wait for full election participation the first time the replication group finds a master. By default this option is turned off and normal election timeouts are used. (See the Elections section in the *Berkeley DB Programmer's Reference Guide* for more information.) + An optional configuration timeout period to wait for full election participation the first time the replication group finds a master. By default this option is turned off and normal election timeouts are used. (See the Elections section in the *Berkeley DB Programmer's Reference Guide* for more information.) - `DB_REP_HEARTBEAT_MONITOR` @@ -66,7 +66,7 @@ The **which** parameter must be set to one of the following values: - `DB_REP_LEASE_TIMEOUT` - Configure the amount of time a client grants its master lease to a master. When using master leases all sites in a replication group must use the same lease timeout value. There is no default value. If leases are desired, this method must be called prior to calling DB_ENV->rep_start() method. See also DB_ENV->rep_set_clockskew() method, DB_ENV->rep_set_config() method or Master leases. + Configure the amount of time a client grants its master lease to a master. When using master leases all sites in a replication group must use the same lease timeout value. There is no default value. If leases are desired, this method must be called prior to calling DB_ENV->rep_start() method. See also DB_ENV->rep_set_clockskew() method, DB_ENV->rep_set_config() method or Master leases. ### Errors diff --git a/docs-src/api/c/repstart.md b/docs-src/api/c/repstart.md index dc2742514..ca3ba8282 100644 --- a/docs-src/api/c/repstart.md +++ b/docs-src/api/c/repstart.md @@ -26,7 +26,7 @@ The `DB_ENV->rep_start()` method returns a non-zero error value on failure and 0 #### cdata -The **cdata** parameter is an opaque data item that is sent over the communication infrastructure when the client comes online (see Connecting to a new site for more information). If no such information is useful, **cdata** should be NULL. +The **cdata** parameter is an opaque data item that is sent over the communication infrastructure when the client comes online (see Connecting to a new site for more information). If no such information is useful, **cdata** should be NULL. #### flags diff --git a/docs-src/api/c/reptransport.md b/docs-src/api/c/reptransport.md index 2b0d1dd93..eaee0d7fe 100644 --- a/docs-src/api/c/reptransport.md +++ b/docs-src/api/c/reptransport.md @@ -33,7 +33,7 @@ Berkeley DB is not re-entrant. The callback function for this method should not #### envid -The **envid** parameter is the local environment's ID. It must be a non-negative integer and uniquely identify this Berkeley DB database environment (see Replication environment IDs for more information). +The **envid** parameter is the local environment's ID. It must be a non-negative integer and uniquely identify this Berkeley DB database environment (see Replication environment IDs for more information). #### send @@ -57,11 +57,11 @@ The **send** callback function is used to transmit data using the replication ap - `envid` - The **envid** parameter is a positive integer identifier that specifies the replication environment to which the message should be sent (see Replication environment IDs for more information). + The **envid** parameter is a positive integer identifier that specifies the replication environment to which the message should be sent (see Replication environment IDs for more information). The special identifier `DB_EID_BROADCAST` indicates that a message should be broadcast to every environment in the replication group. The application may use a true broadcast protocol or may send the message in sequence to each machine with which it is in communication. In both cases, the sending site should not be asked to process the message. - The special identifier DB_EID_INVALID indicates an invalid environment ID. This may be used to initialize values that are subsequently checked for validity. + The special identifier DB_EID_INVALID indicates an invalid environment ID. This may be used to initialize values that are subsequently checked for validity. - `flags` @@ -83,7 +83,7 @@ The **send** callback function is used to transmit data using the replication ap The message is a client request that has already been made and to which no response was received. -It may sometimes be useful to pass application-specific data to the send function; see Environment FAQ for a discussion on how to do this. +It may sometimes be useful to pass application-specific data to the send function; see Environment FAQ for a discussion on how to do this. The **send** function must return 0 on success and non-zero on failure. If the send function fails, the message being sent is necessary to maintain database integrity, and the local log is not configured for synchronous flushing, the local log will be flushed; otherwise, any error from the **send** function will be ignored. diff --git a/docs-src/api/c/set_cache_max_parameter.md b/docs-src/api/c/set_cache_max_parameter.md index 6aac5b7a8..890571b76 100644 --- a/docs-src/api/c/set_cache_max_parameter.md +++ b/docs-src/api/c/set_cache_max_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_cache_max_parameter.html Sets the maximum size that the `set_cachesize` parameter is allowed to set. The specified size is rounded to the nearest multiple of the cache region size, which is the initial cache size divided by the number of regions specified to the `set_cachesize` parameter. If no value is specified, it defaults to the initial cache size. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_cache_max`, one or more whitespace characters, and the maximum cache size in bytes, specified in two parts: the gigabytes of cache and the additional bytes of cache. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_cache_max`, one or more whitespace characters, and the maximum cache size in bytes, specified in two parts: the gigabytes of cache and the additional bytes of cache. For example: ``` c set_cache_max 2 524288000 diff --git a/docs-src/api/c/set_cachesize_parameter.md b/docs-src/api/c/set_cachesize_parameter.md index 9fc21bac7..0d1301656 100644 --- a/docs-src/api/c/set_cachesize_parameter.md +++ b/docs-src/api/c/set_cachesize_parameter.md @@ -17,7 +17,7 @@ It is possible to specify cache sizes large enough they cannot be allocated cont The cache size supplied to this parameter will be rounded to the nearest multiple of the region size and may not be larger than the maximum possible cache size configured for your application (use the set_cache_max to do this). The **ncache** parameter is ignored when resizing the cache. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_cachesize`, one or more whitespace characters, and the initial cache size specified in three parts: the gigabytes of cache, the additional bytes of cache, and the number of caches, also separated by whitespace characters. For example: +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_cachesize`, one or more whitespace characters, and the initial cache size specified in three parts: the gigabytes of cache, the additional bytes of cache, and the number of caches, also separated by whitespace characters. For example: ``` c set_cachesize 2 524288000 1 diff --git a/docs-src/api/c/set_create_dir_parameter.md b/docs-src/api/c/set_create_dir_parameter.md index 57fda9251..5ee4370da 100644 --- a/docs-src/api/c/set_create_dir_parameter.md +++ b/docs-src/api/c/set_create_dir_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_create_dir_parameter.html Sets the path of a directory to be used as the location to create the access method database files. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_create_dir`, one or more whitespace characters, and the directory name. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_create_dir`, one or more whitespace characters, and the directory name. For example: diff --git a/docs-src/api/c/set_data_len_parameter.md b/docs-src/api/c/set_data_len_parameter.md index be4aec470..04c289949 100644 --- a/docs-src/api/c/set_data_len_parameter.md +++ b/docs-src/api/c/set_data_len_parameter.md @@ -11,7 +11,7 @@ If the db_printlog The value set here must be greater than 0. The default value is 100. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_data_len`, one or more whitespace characters, and the directory name. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_data_len`, one or more whitespace characters, and the directory name. For example: diff --git a/docs-src/api/c/set_flags_parameter.md b/docs-src/api/c/set_flags_parameter.md index ac8d8b5d5..90933bd7c 100644 --- a/docs-src/api/c/set_flags_parameter.md +++ b/docs-src/api/c/set_flags_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_flags_parameter.html Configures a database environment. -The syntax of the entry in the DB_CONFIG file is a single line with the string `set_flags`, one or more whitespace characters, the method flag parameter as a string, optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on`; for example, `set_flags DB_TXN_NOSYNC` or `set_flags DB_TXN_NOSYNC on`. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The syntax of the entry in the DB_CONFIG file is a single line with the string `set_flags`, one or more whitespace characters, the method flag parameter as a string, optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on`; for example, `set_flags DB_TXN_NOSYNC` or `set_flags DB_TXN_NOSYNC on`. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The method flag parameters are as follows: diff --git a/docs-src/api/c/set_intermediate_dir_mode_parameter.md b/docs-src/api/c/set_intermediate_dir_mode_parameter.md index 566af75ee..60ca00a14 100644 --- a/docs-src/api/c/set_intermediate_dir_mode_parameter.md +++ b/docs-src/api/c/set_intermediate_dir_mode_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_intermediate_dir_mode_parameter.html Configures the database environment's intermediate directory permissions. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_intermediate_dir_mode`, one or more whitespace characters, and the directory permissions. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_intermediate_dir_mode`, one or more whitespace characters, and the directory permissions. Directory permissions are interpreted as a string of nine characters, using the character set **r** (read), **w** (write), **x** (execute or search), and **-** (none). The first character is the read permissions for the directory owner (set to either **r** or **-**). The second character is the write permissions for the directory owner (set to either **w** or **-**). The third character is the execute permissions for the directory owner (set to either **x** or **-**). diff --git a/docs-src/api/c/set_lg_bsize_parameter.md b/docs-src/api/c/set_lg_bsize_parameter.md index 59e478f5f..d3bbb6fca 100644 --- a/docs-src/api/c/set_lg_bsize_parameter.md +++ b/docs-src/api/c/set_lg_bsize_parameter.md @@ -11,7 +11,7 @@ For the DB, when the logging subsystem is configured for on-disk logging, the de When the logging subsystem is configured for in-memory logging, the default size of the in-memory log buffer is 1MB. Log information is stored in-memory until the storage space fills up or transaction abort or commit frees up the memory for new transactions. In the presence of long-running transactions or transactions producing large amounts of data, the buffer size must be sufficient to hold all log information that can accumulate during the longest running transaction. When choosing log buffer and file sizes for in-memory logs, applications should ensure the in-memory log buffer size is large enough that no transaction will ever span the entire buffer, and avoid a state where the in-memory buffer is full and no space can be freed because a transaction that started in the first log "file" is still active. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_bsize`, one or more whitespace characters, and the log buffer size in bytes. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_bsize`, one or more whitespace characters, and the log buffer size in bytes. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_lg_dir_parameter.md b/docs-src/api/c/set_lg_dir_parameter.md index 810e792d4..c367bccf7 100644 --- a/docs-src/api/c/set_lg_dir_parameter.md +++ b/docs-src/api/c/set_lg_dir_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_lg_dir_parameter.html Sets the path of the directory to be used as the location of logging files. Log files created by the Log Manager subsystem will be created in this directory. If no logging directory is specified, log files are created in the environment home directory. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_dir`, one or more whitespace characters, and the directory name. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_dir`, one or more whitespace characters, and the directory name. For more information, see DB_ENV->set_lg_dir(). diff --git a/docs-src/api/c/set_lg_filemode_parameter.md b/docs-src/api/c/set_lg_filemode_parameter.md index e647773f5..9b46cba6d 100644 --- a/docs-src/api/c/set_lg_filemode_parameter.md +++ b/docs-src/api/c/set_lg_filemode_parameter.md @@ -9,6 +9,6 @@ Sets the absolute file mode for created log files. This method is only useful fo Normally, if Berkeley DB applications set their umask appropriately, all processes in the application suite will have read permission on the log files created by any process in the application suite. However, if the Berkeley DB application is a library, a process using the library might set its umask to a value preventing other processes in the application suite from reading the log files it creates. In this rare case, use the set_lg_filemode parameter to set the mode of created log files to an absolute value. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_filemode`, one or more whitespace characters, and the absolute mode of created log files. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_filemode`, one or more whitespace characters, and the absolute mode of created log files. For more information, see DB_ENV->set_lg_filemode(). diff --git a/docs-src/api/c/set_lg_max_parameter.md b/docs-src/api/c/set_lg_max_parameter.md index 1024a6727..4af9617a1 100644 --- a/docs-src/api/c/set_lg_max_parameter.md +++ b/docs-src/api/c/set_lg_max_parameter.md @@ -11,7 +11,7 @@ When the logging subsystem is configured for on-disk logging, the default size o When the logging subsystem is configured for in-memory logging, the default size of a log file is 256KB. In addition, the configured log buffer size must be larger than the log file size. (The logging subsystem divides memory configured for in-memory log records into "files", as database environments configured for in-memory log records may exchange log records with other members of a replication group, and those members may be configured to store log records on-disk.) When choosing log buffer and file sizes for in-memory logs, applications should ensure the in-memory log buffer size is large enough that no transaction will ever span the entire buffer, and avoid a state where the in-memory buffer is full and no space can be freed because a transaction that started in the first log "file" is still active. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_max`, one or more whitespace characters, and the maximum log file size in bytes. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_max`, one or more whitespace characters, and the maximum log file size in bytes. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_lg_regionmax_parameter.md b/docs-src/api/c/set_lg_regionmax_parameter.md index 25de69b89..d93402774 100644 --- a/docs-src/api/c/set_lg_regionmax_parameter.md +++ b/docs-src/api/c/set_lg_regionmax_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_lg_regionmax_parameter.html Sets the size of the underlying logging area of the Berkeley DB environment, in bytes. By default, or if the value is set to 0, the minimum region size is used, approximately 128KB. The log region is used to store filenames, and so may need to be increased in size if a large number of files will be opened and registered with the specified Berkeley DB environment's log manager. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_regionmax`, one or more whitespace characters, and the log region size in bytes. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lg_regionmax`, one or more whitespace characters, and the log region size in bytes. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_lk_detect_parameter.md b/docs-src/api/c/set_lk_detect_parameter.md index 49d2b1ed4..6a9264a79 100644 --- a/docs-src/api/c/set_lk_detect_parameter.md +++ b/docs-src/api/c/set_lk_detect_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_lk_detect_parameter.html Sets the maximum number of locking entities supported by the Berkeley DB environment. This value is used by Berkeley DB to estimate how much space to allocate for various lock-table data structures. When using the DB, the default value is 2,000 lockers. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_detect`, one or more whitespace characters, and the method **detect** parameter as a string. The detect parameter configures the deadlock detector. The deadlock detector will reject the lock request with the lowest priority. If multiple lock requests have the lowest priority, then the detect parameter is used to select which of those lock requests to reject. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_detect`, one or more whitespace characters, and the method **detect** parameter as a string. The detect parameter configures the deadlock detector. The deadlock detector will reject the lock request with the lowest priority. If multiple lock requests have the lowest priority, then the detect parameter is used to select which of those lock requests to reject. For example: diff --git a/docs-src/api/c/set_lk_max_lockers_parameter.md b/docs-src/api/c/set_lk_max_lockers_parameter.md index 49c4a331f..d28adbf3b 100644 --- a/docs-src/api/c/set_lk_max_lockers_parameter.md +++ b/docs-src/api/c/set_lk_max_lockers_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_lk_max_lockers_parameter.html Sets the maximum number of locking entities supported by the Berkeley DB environment. This value is used by Berkeley DB to estimate how much space to allocate for various lock-table data structures. When using the DB, the default value is 1,000 lockers. When using the BDB SQL interface, the default value is 2,000 lockers. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_max_lockers`, one or more whitespace characters, and the number of lockers. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_max_lockers`, one or more whitespace characters, and the number of lockers. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_lk_max_locks_parameter.md b/docs-src/api/c/set_lk_max_locks_parameter.md index ab75e6edc..bf3b3c6a1 100644 --- a/docs-src/api/c/set_lk_max_locks_parameter.md +++ b/docs-src/api/c/set_lk_max_locks_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_lk_max_locks_parameter.html Sets the maximum number of locks supported by the Berkeley DB environment. This value is used to estimate how much space to allocate for various lock-table data structures. When using the DB, the default value is 1000 locks. When using the BDB SQL interface, the default value is 10,000 locks. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_max_locks`, one or more whitespace characters, and the number of locks. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_max_locks`, one or more whitespace characters, and the number of locks. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_lk_max_objects_parameter.md b/docs-src/api/c/set_lk_max_objects_parameter.md index dd6dc7136..53a16a814 100644 --- a/docs-src/api/c/set_lk_max_objects_parameter.md +++ b/docs-src/api/c/set_lk_max_objects_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_lk_max_objects_parameter.html Sets the maximum number of locked objects supported by the Berkeley DB environment. This value is used to estimate how much space to allocate for various lock-table data structures. When using the DB, the default value is 1000 objects. When using the BDB SQL interface, the default value is 10,000 objects. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_max_objects`, one or more whitespace characters, and the number of objects. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_max_objects`, one or more whitespace characters, and the number of objects. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_lk_partitions_parameter.md b/docs-src/api/c/set_lk_partitions_parameter.md index 79cc3c093..31893afd3 100644 --- a/docs-src/api/c/set_lk_partitions_parameter.md +++ b/docs-src/api/c/set_lk_partitions_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_lk_partitions_parameter.html Sets the number of lock table partitions in the Berkeley DB environment. The default value is 10 times the number of CPUs on the system if there is more than one CPU. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_partitions`, one or more whitespace characters, and the number of partitions. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_lk_partitions`, one or more whitespace characters, and the number of partitions. If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. diff --git a/docs-src/api/c/set_mp_max_openfd_parameter.md b/docs-src/api/c/set_mp_max_openfd_parameter.md index c6950de74..3c6097336 100644 --- a/docs-src/api/c/set_mp_max_openfd_parameter.md +++ b/docs-src/api/c/set_mp_max_openfd_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_mp_max_openfd_parameter.html Limits the number of file descriptors the library will open concurrently when flushing dirty pages from the cache. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_max_openfd`, one or more whitespace characters, and the number of open file descriptors. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_max_openfd`, one or more whitespace characters, and the number of open file descriptors. For more information, see DB_ENV->get_mp_max_openfd(). diff --git a/docs-src/api/c/set_mp_max_write_parameter.md b/docs-src/api/c/set_mp_max_write_parameter.md index 75df48258..4424a304b 100644 --- a/docs-src/api/c/set_mp_max_write_parameter.md +++ b/docs-src/api/c/set_mp_max_write_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_mp_max_write_parameter.html Limits the number of sequential write operations scheduled by the library when flushing dirty pages from the cache. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_mp_max_write`, one or more whitespace characters, and the maximum number of sequential writes and the number of microseconds to sleep, also separated by whitespace characters. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_mp_max_write`, one or more whitespace characters, and the maximum number of sequential writes and the number of microseconds to sleep, also separated by whitespace characters. For more information, see DB_ENV->set_mp_max_write(). diff --git a/docs-src/api/c/set_mp_mmapsize_parameter.md b/docs-src/api/c/set_mp_mmapsize_parameter.md index 12b2d1fa7..f09889061 100644 --- a/docs-src/api/c/set_mp_mmapsize_parameter.md +++ b/docs-src/api/c/set_mp_mmapsize_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_mp_mmapsize_parameter.html Sets the maximum file size, in bytes, for a file to be mapped into the process address space. If no value is specified, it defaults to 10MB. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_mp_mmapsize`, one or more whitespace characters, and the size in bytes. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_mp_mmapsize`, one or more whitespace characters, and the size in bytes. For more information, see DB_ENV->set_mp_mmapsize(). diff --git a/docs-src/api/c/set_open_flags_parameter.md b/docs-src/api/c/set_open_flags_parameter.md index 7a40a31b8..00e11b9f6 100644 --- a/docs-src/api/c/set_open_flags_parameter.md +++ b/docs-src/api/c/set_open_flags_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_open_flags_parameter.html Initializes specific subsystems of the Berkeley DB environment. -The syntax of the entry in the DB_CONFIG is a single line with the string `set_open_flags`, one or more whitespace characters, the method flag parameter as a string, optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on`; for example, `set_open_flags DB_INIT_REP` or `set_open_flags DB_INIT_REP on`. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. +The syntax of the entry in the DB_CONFIG is a single line with the string `set_open_flags`, one or more whitespace characters, the method flag parameter as a string, optionally one or more whitespace characters, and the string `on` or `off`. If the optional string is omitted, the default is `on`; for example, `set_open_flags DB_INIT_REP` or `set_open_flags DB_INIT_REP on`. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time. The method flag parameters are as follows: diff --git a/docs-src/api/c/set_shm_key_parameter.md b/docs-src/api/c/set_shm_key_parameter.md index 437f84f79..31b748388 100644 --- a/docs-src/api/c/set_shm_key_parameter.md +++ b/docs-src/api/c/set_shm_key_parameter.md @@ -7,8 +7,8 @@ source: docs/api_reference/C/set_shm_key_parameter.html Configures the database environment's base segment ID. This base segment ID will be used when Berkeley DB shared memory regions are first created. It will be incremented a small integer value each time a new shared memory region is created; that is, if the base ID is 35, the first shared memory region created will have a segment ID of 35, and the next one will have a segment ID between 36 and 40 or so. -See Shared Memory Regions for more information. +See Shared Memory Regions for more information. -The syntax of the entry in the DB_CONFIG file is a single line with the string `set_shm_key` one or more whitespace characters, and the ID. +The syntax of the entry in the DB_CONFIG file is a single line with the string `set_shm_key` one or more whitespace characters, and the ID. For more information, see DB_ENV->set_shm_key(). diff --git a/docs-src/api/c/set_thread_count_parameter.md b/docs-src/api/c/set_thread_count_parameter.md index fcb35f176..96109a565 100644 --- a/docs-src/api/c/set_thread_count_parameter.md +++ b/docs-src/api/c/set_thread_count_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_thread_count_parameter.html Declares an approximate number of threads in the database environment. -The syntax of the entry in in the DB_CONFIG file is a single line with the string `set_thread_count`, one or more whitespace characters, and the thread count. The DB_CONFIG file is read when the database environment is opened, and hence it silently overrules configuration done before that time. +The syntax of the entry in in the DB_CONFIG file is a single line with the string `set_thread_count`, one or more whitespace characters, and the thread count. The DB_CONFIG file is read when the database environment is opened, and hence it silently overrules configuration done before that time. For more information, see DB_ENV->set_thread_count(). diff --git a/docs-src/api/c/set_timeout_parameter.md b/docs-src/api/c/set_timeout_parameter.md index 5f1c4ede7..32bde61de 100644 --- a/docs-src/api/c/set_timeout_parameter.md +++ b/docs-src/api/c/set_timeout_parameter.md @@ -11,14 +11,14 @@ The syntax for setting timeout value for database environment's lock, before rec - DB_SET_LOCK_TIMEOUT - Configures the database environment's lock timeout value. The syntax of the entry in the DB_CONFIG file is a single line with the string `set_lock_timeout`, one or more whitespace characters, and the lock timeout value. + Configures the database environment's lock timeout value. The syntax of the entry in the DB_CONFIG file is a single line with the string `set_lock_timeout`, one or more whitespace characters, and the lock timeout value. - DB_SET_REG_TIMEOUT - Sets the timeout value on how long to wait for processes to exit the environment before recovery is started. The syntax of the entry in the DB_CONFIG file is a single line with the string `set_reg_timeout`, one or more whitespace characters, and the wait timeout value. + Sets the timeout value on how long to wait for processes to exit the environment before recovery is started. The syntax of the entry in the DB_CONFIG file is a single line with the string `set_reg_timeout`, one or more whitespace characters, and the wait timeout value. - DB_SET_TXN_TIMEOUT - Sets the timeout value for transactions in this database environment. The syntax of the entry in the DB_CONFIG file is a single line with the string `set_txn_timeout`, one or more whitespace characters, and the transaction timeout value + Sets the timeout value for transactions in this database environment. The syntax of the entry in the DB_CONFIG file is a single line with the string `set_txn_timeout`, one or more whitespace characters, and the transaction timeout value For more information, see DB_ENV->set_timeout(). diff --git a/docs-src/api/c/set_tmp_dir_parameter.md b/docs-src/api/c/set_tmp_dir_parameter.md index e17d6ce01..71d73a8fb 100644 --- a/docs-src/api/c/set_tmp_dir_parameter.md +++ b/docs-src/api/c/set_tmp_dir_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_tmp_dir_parameter.html Specifies the path of a directory to be used as the location of temporary files. The files created to back in-memory access method databases will be created relative to this path. These temporary files can be quite large, depending on the size of the database. -The syntax of the entry in the DB_CONFIG file with the string `set_tmp_dir`, one or more whitespace characters, and the directory name. +The syntax of the entry in the DB_CONFIG file with the string `set_tmp_dir`, one or more whitespace characters, and the directory name. For more information, see DB_ENV->set_tmp_dir(). diff --git a/docs-src/api/c/set_tx_max_parameter.md b/docs-src/api/c/set_tx_max_parameter.md index fc7af271a..7d5ecec62 100644 --- a/docs-src/api/c/set_tx_max_parameter.md +++ b/docs-src/api/c/set_tx_max_parameter.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/set_tx_max_parameter.html Configures the Berkeley DB database environment to support at least the minimum number of simultaneously active transactions supported by Berkeley DB database environment. This value bounds the size of the memory allocated for transactions. Child transactions are counted as active until they either commit or abort. -The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_tx_max`, one or more whitespace characters, and the number of transactions. +The syntax of this parameter in the DB_CONFIG file is a single line with the string `set_tx_max`, one or more whitespace characters, and the number of transactions. For more information, see DB_ENV->set_tx_max(). diff --git a/docs-src/api/c/set_verbose_parameter.md b/docs-src/api/c/set_verbose_parameter.md index 309d15bef..2ac214d9a 100644 --- a/docs-src/api/c/set_verbose_parameter.md +++ b/docs-src/api/c/set_verbose_parameter.md @@ -7,7 +7,7 @@ source: docs/api_reference/C/set_verbose_parameter.html Enables/disables specific additional informational and debugging messages in the Berkeley DB message output. -The syntax of the entry in the DB_CONFIG file is a single line with the string `set_verbose`, one or more whitespace characters, the method flag parameter as a string, optionally one or more whitespace characters and the string `on` or `off`. If the optional string is omitted, the default is `on`. +The syntax of the entry in the DB_CONFIG file is a single line with the string `set_verbose`, one or more whitespace characters, the method flag parameter as a string, optionally one or more whitespace characters and the string `on` or `off`. If the optional string is omitted, the default is `on`. For example: diff --git a/docs-src/api/c/sqlite3.md b/docs-src/api/c/sqlite3.md index 33024eb65..776e1b175 100644 --- a/docs-src/api/c/sqlite3.md +++ b/docs-src/api/c/sqlite3.md @@ -7,6 +7,6 @@ source: docs/api_reference/C/sqlite3.html Sqlite3 is a command line tool that enables you to manually enter and execute SQL commands. It is identical to the dbsql executable but named so that existing scripts for SQLite can easily work with Berkeley DB. To build this tool, run the configure script with the `--enable-sql_compat `option when you are building the Berkeley DB SQL interface. -For more information on building this tool, see the "Building for UNIX/POSIX" +For more information on building this tool, see the "Building for UNIX/POSIX" For more information on how to use Sqlite3 see the SQLite Documentation page. diff --git a/docs-src/api/c/txnbegin.md b/docs-src/api/c/txnbegin.md index 296324aeb..573288bc7 100644 --- a/docs-src/api/c/txnbegin.md +++ b/docs-src/api/c/txnbegin.md @@ -65,13 +65,13 @@ The **flags** parameter must be set to 0 or by bitwise inclusively **OR**'ing to - `DB_TXN_NOWAIT` - If a lock is unavailable for any Berkeley DB operation performed in the context of this transaction, cause the operation to return DB_LOCK_DEADLOCK (or DB_LOCK_NOTGRANTED if the database environment has been configured using the DB_TIME_NOTGRANTED flag). + If a lock is unavailable for any Berkeley DB operation performed in the context of this transaction, cause the operation to return DB_LOCK_DEADLOCK (or DB_LOCK_NOTGRANTED if the database environment has been configured using the DB_TIME_NOTGRANTED flag). This behavior may be set for a Berkeley DB environment using the DB_ENV->set_flags() method. Any value specified to this method overrides that setting. - `DB_TXN_SNAPSHOT` - This transaction will execute with snapshot isolation. For databases with the DB_MULTIVERSION flag set, data values will be read as they are when the transaction begins, without taking read locks. Silently ignored for operations on databases with DB_MULTIVERSION not set on the underlying database (read locks are acquired). + This transaction will execute with snapshot isolation. For databases with the DB_MULTIVERSION flag set, data values will be read as they are when the transaction begins, without taking read locks. Silently ignored for operations on databases with DB_MULTIVERSION not set on the underlying database (read locks are acquired). The error `DB_LOCK_DEADLOCK` will be returned from update operations if a snapshot transaction attempts to update data which was modified after the snapshot transaction read it. diff --git a/docs-src/api/c/txnset_commit_token.md b/docs-src/api/c/txnset_commit_token.md index 337957cb7..e359a4d90 100644 --- a/docs-src/api/c/txnset_commit_token.md +++ b/docs-src/api/c/txnset_commit_token.md @@ -14,7 +14,7 @@ DB_TXN->set_commit_token(DB_TXN *txn, DB_TXN_TOKEN *buffer); The `DB_TXN->set_commit_token()` method configures the transaction for commit token generation, and accepts the address of an application-supplied buffer to receive the token. The actual generation of the token contents does not occur until commit time. -Commit tokens are used to enable some consistency guarantees for replicated applications. Please see the Read your writes consistency section in the *Berkeley DB Programmer's Reference Guide* for more information. +Commit tokens are used to enable some consistency guarantees for replicated applications. Please see the Read your writes consistency section in the *Berkeley DB Programmer's Reference Guide* for more information. The `DB_TXN->set_commit_token()` method may be called at any time after the DB_ENV->txn_begin() method has been called, and before DB_TXN->commit() has been called. diff --git a/docs-src/api/c/txnset_name.md b/docs-src/api/c/txnset_name.md index 57ddaa1e9..f1cf4c7df 100644 --- a/docs-src/api/c/txnset_name.md +++ b/docs-src/api/c/txnset_name.md @@ -14,7 +14,7 @@ DB_TXN->set_name(DB_TXN *txn, const char *name); The `DB_TXN->set_name()` method associates the specified string with the transaction. The string is returned by DB_ENV->txn_stat() and displayed by DB_ENV->txn_stat_print(). -If the database environment has been configured for logging and the Berkeley DB library was configured with --enable-diagnostic, a debugging log record is written including the transaction ID and the name. +If the database environment has been configured for logging and the Berkeley DB library was configured with --enable-diagnostic, a debugging log record is written including the transaction ID and the name. The `DB_TXN->set_name()` method may be called at any time during the life of the application. diff --git a/docs-src/api/stl/BulkRetrievalOption.md b/docs-src/api/stl/BulkRetrievalOption.md new file mode 100644 index 000000000..d5f8860aa --- /dev/null +++ b/docs-src/api/stl/BulkRetrievalOption.md @@ -0,0 +1,39 @@ +--- +title: "Chapter 27.  BulkRetrievalOption" +api-name: "Chapter 27.  BulkRetrievalOption" +source: docs/api_reference/STL/BulkRetrievalOption.html +--- +## Chapter 27.  BulkRetrievalOption + +Bulk retrieval configuration helper class. + +Used by the begin() function of a container. + +#### Public Members + +| Member | Description | +|----|----| +| BulkRetrievalOption | | +| operator== | Equality comparison. | +| operator= | Assignment operator. | +| bulk_buf_size | Return the buffer size set to this object. | +| bulk_retrieval | This function indicates that you need a bulk retrieval iterator, and it can be also used to optionally set the bulk read buffer size. | +| no_bulk_retrieval | This function indicates that you do not need a bulk retrieval iterator. | + +#### Group + +Dbstl Helper Classes + +## BulkRetrievalOption + +### Function Details + +``` c +BulkRetrievalOption(Option bulk_retrieve1, + u_int32_t bulk_buf_sz=DBSTL_BULK_BUF_SIZE) + +``` + +### Class + +BulkRetrievalOption diff --git a/docs-src/api/stl/DbstlDbt.md b/docs-src/api/stl/DbstlDbt.md new file mode 100644 index 000000000..28dfc109a --- /dev/null +++ b/docs-src/api/stl/DbstlDbt.md @@ -0,0 +1,64 @@ +--- +title: "Chapter 25.  DbstlDbt" +api-name: "Chapter 25.  DbstlDbt" +source: docs/api_reference/STL/DbstlDbt.html +--- +## Chapter 25.  DbstlDbt + +You can persist all bytes in a chunk of contiguous memory by constructing an DbstlDbt object A(use malloc to allocate the required number of bytes for A.data and copy the bytes to be stored into A.data, set other fields as necessary) and store A into a container, e.g. + +db_vector\, this stores the bytes rather than the object A into the underlying database. The DbstlDbt class can help you avoid memory leaks, so it is strongly recommended that you use DbstlDbt rather than Dbt class. + +DbstlDbt derives from Dbt class, and it does an deep copy on copy construction and assignment --by calling malloc to allocate its own memory and then copying the bytes to it; Conversely the destructor will free the memory on destruction if the data pointer is non-NULL. The destructor assumes the memory is allocated via malloc, hence why you are required to call malloc to allocate memory in order to use DbstlDbt . + +DbstlDbt simply inherits all methods from Dbt with no extra new methods except the constructors/destructor and assignment operator, so it is easy to use. + +In practice you rarely need to use DbstlDbt or Dbt because dbstl enables you to store any complex objects or primitive data. Only when you need to store raw bytes, e.g. a bitmap, do you need to use DbstlDbt . + +Hence, DbstlDbt is the right class to use to store any object into Berkeley DB via dbstl without memory leaks. + +Don't free the memory referenced by DbstlDbt objects, it will be freed when the DbstlDbt object is destructed. + +Please refer to the two examples using DbstlDbt in TestAssoc::test_arbitrary_object_storage and TestAssoc::test_char_star_string_storage member functions, which illustrate how to correctly use DbstlDbt in order to store raw bytes. + +This class handles the task of allocating and de-allocating memory internally. Although it can be used to store data which cannot be handled by the DbstlElemTraits class, in practice, it is usually more convenient to register callbacks in the DbstlElemTraits class for the type you are storing/retrieving using dbstl. + +#### Public Members + +| Member | Description | +|----|----| +| DbstlDbt | Construct an object with an existing chunk of memory of size1 bytes, refered by data1,. | +| ~DbstlDbt | The memory will be free'ed by the destructor. | +| operator= | The memory will be reallocated if neccessary. | + +#### Group + +Dbstl Helper Classes + +## DbstlDbt + +### Function Details + +``` c +DbstlDbt(void *data1, + u_int32_t size1) + +``` + +Construct an object with an existing chunk of memory of size1 bytes, refered by data1,. + +``` c +DbstlDbt() + +``` + +``` c +DbstlDbt(const DbstlDbt &d) + +``` + +This copy constructor does a deep copy. + +### Class + +DbstlDbt diff --git a/docs-src/api/stl/DbstlElemTraits.md b/docs-src/api/stl/DbstlElemTraits.md new file mode 100644 index 000000000..0ffa53036 --- /dev/null +++ b/docs-src/api/stl/DbstlElemTraits.md @@ -0,0 +1,97 @@ +--- +title: "Chapter 26.  DbstlElemTraits" +api-name: "Chapter 26.  DbstlElemTraits" +source: docs/api_reference/STL/DbstlElemTraits.html +--- +## Chapter 26.  DbstlElemTraits + +This class is used to register callbacks to manipulate an object of a complex type. + +These callbacks are used by dbstl at runtime to manipulate the object. + +A complex type is a type whose members are not located in a contiguous chunk of memory. For example, the following class A is a complex type because for any instance a of class A, a.b\_ points to another object of type B, and dbstl treats the object that a.b\_ points to as part of the data of the instance a. Hence, if the user needs to store a.b\_ into a dbstl container, the user needs to register an appropriate callback to de-reference and store the object referenced by a.b. Similarly, the user also needs to register callbacks to marshall an array as well as to count the number of elements in such an array. + +class A { int m; B \*p\_; }; class B { int n; }; + +The user also needs to register callbacks for i). returning an object¡¯s size in bytes; ii). Marshalling and unmarshalling an object; iii). Copying a complex object and and assigning an object to another object of the same type; iv). Element comparison. v). Compare two sequences of any type of objects; Measuring the length of an object sequence and copy an object sequence. + +Several elements located in a contiguous chunk of memory form a sequence. An element of a sequence may be a simple object located at a contigous memory chunk, or a complex object, i.e. some of its members may contain references (pointers) to another region of memory. It is not necessary to store a special object to denote the end of the sequence. The callback to traverse the constituent elements of the sequence needs to able to determine the end of the sequence. + +Marshalling means packing the object's data members into a contiguous chunk of memory; unmarshalling is the opposite of marshalling. In other words, when you unmarshall an object, its data members are populated with values from a previously marshalled version of the object. + +The callbacks need not be set to every type explicitly. . dbstl will check if a needed callback function of this type is provided. If one is available, dbstl will use the registered callback. If the appropriate callback is not provided, dbstl will use reasonable defaults to do the job. + +For returning the size of an object, the default behavior is to use the sizeof() operator; For marshalling and unmarshalling, dbstl uses memcpy, so the default behavior is sufficient for simple types whose data reside in a contiguous chunk of memory; Dbstl uses uses \>, == and \< for comparison operations; For char\* and wchar_t \* strings, dbstl already provides the appropriate callbacks, so you do not need to register them. In general, if the default behavior is adequate, you don't need to register the corresponding callback. + +If you have registered proper callbacks, the DbstlElemTraits\ can also be used as the char_traits\ class for std::basic_string\ \>, and you can enable your class T to form a basic_string\\>, and use basic_string's functionality and the algorithms to manipulate it. + +#### Public Members + +| Member | Description | +|----|----| +| assign | Assignone object to another. | +| eq | Check for equality of two objects. | +| lt | Less than comparison. | +| compare | Sequence comparison. | +| length | Returns the number of elements in sequence seq1. | +| copy | Copy first cnt number of elements from seq2 to seq1. | +| find | Find within the first cnt elements of sequence seq the position of element equal to elem. | +| move | Sequence movement. | +| to_char_type | | +| to_int_type | | +| eq_int_type | | +| eof | | +| not_eof | | +| set_restore_function | | +| get_restore_function | | +| set_assign_function | | +| get_assign_function | | +| get_size_function | | +| set_size_function | | +| get_copy_function | | +| set_copy_function | | +| set_sequence_len_function | | +| get_sequence_len_function | | +| get_sequence_copy_function | | +| set_sequence_copy_function | | +| set_compare_function | | +| get_compare_function | | +| set_sequence_compare_function | | +| get_sequence_compare_function | | +| set_sequence_n_compare_function | | +| get_sequence_n_compare_function | | +| instance | Factory method to create a singeleton instance of this class. | +| ~DbstlElemTraits | | +| DbstlElemTraits | | + +#### Group + +Dbstl Helper Classes + +## assign + +### Function Details + +``` c +static void assign(T &left, + const T &right) + +``` + +Assignone object to another. + +``` c +static T* assign(T *seq, size_t cnt, + T elem) + +``` + +Assign first cnt number of elements of sequence seq with the value of elem. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/DbstlException.md b/docs-src/api/stl/DbstlException.md new file mode 100644 index 000000000..f3925381f --- /dev/null +++ b/docs-src/api/stl/DbstlException.md @@ -0,0 +1,57 @@ +--- +title: "Chapter 30.  DbstlException" +api-name: "Chapter 30.  DbstlException" +source: docs/api_reference/STL/DbstlException.html +--- +## Chapter 30.  DbstlException + +Base class of all dbstl exception classes. + +It is derived from Berkeley DB C++ API DbException class to maintain consistency with all Berkeley DB exceptions. + +#### Public Members + +| Member | Description | +|----|----| +| DbstlException | | +| operator= | | +| ~DbstlException | | + +#### Group + +Dbstl Exception Classes + +## DbstlException + +### Function Details + +``` c +DbstlException(const char *msg) + +``` + +``` c +DbstlException(const char *msg, + int err) + +``` + +``` c +DbstlException(const DbstlException &ex) + +``` + +``` c +DbstlException(int err) + +``` + +``` c +DbstlException(const char *prefix, const char *msg, + int err) + +``` + +### Class + +DbstlException diff --git a/docs-src/api/stl/ElementHolder.md b/docs-src/api/stl/ElementHolder.md new file mode 100644 index 000000000..ee2f8e53d --- /dev/null +++ b/docs-src/api/stl/ElementHolder.md @@ -0,0 +1,94 @@ +--- +title: "Chapter 23.  ElementHolder" +api-name: "Chapter 23.  ElementHolder" +source: docs/api_reference/STL/ElementHolder.html +--- +## Chapter 23.  ElementHolder + +A wrapper class for primitive types. + +It has identical usage and public interface to the ElementRef class. + +#### See Also + +ElementRef . + +#### Public Members + +| Member | Description | +|----|----| +| ElementHolder | Constructor. | +| ~ElementHolder | Destructor. | +| operator+= | | +| operator-= | | +| operator *= | | +| operator/= | | +| operator%= | | +| operator &= | | +| operator|= | | +| operator^= | | +| operator>>= | | +| operator<<= | | +| operator++ | | +| operator-- | | +| operator= | | +| operator ptype | This operator is a type converter. | +| _DB_STL_value | Returns the data element this wrapper object wraps;. | +| _DB_STL_StoreElement | Function to store the data element. | + +#### Group + +ElementRef and ElementHolder Wappers + +## ElementHolder + +### Function Details + +``` c +ElementHolder(iterator_type *pitr=NULL) + +``` + +Constructor. + +If the pitr parameter is NULL or the default value is used, the object created is a simple wrapper and not connected to a container. If a valid iterator parameter is passed in, the wrapped element will be associated with the matching key/data pair in the underlying container. + +#### Parameters + +##### pitr + +The iterator owning this object. + +``` c +ElementHolder(const ptype &dt) + +``` + +Constructor. + +Initializes an ElementRef wrapper without an iterator. It can only be used to wrap a data element in memory, it can't access an unerlying database. + +#### Parameters + +##### dt + +The base class object to initialize this object. + +``` c +ElementHolder(const self &other) + +``` + +Copy constructor. + +The constructor takes a "deep" copy. The created object will be identical to, but independent from the original object. + +#### Parameters + +##### other + +The object to clone from. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/ElementRef.md b/docs-src/api/stl/ElementRef.md new file mode 100644 index 000000000..70b28dbf1 --- /dev/null +++ b/docs-src/api/stl/ElementRef.md @@ -0,0 +1,41 @@ +--- +title: "Chapter 24.  ElementRef" +api-name: "Chapter 24.  ElementRef" +source: docs/api_reference/STL/ElementRef.html +--- +## Chapter 24.  ElementRef + +ElementRef element wrapper for classes and structures. + +#### See Also + +ElementHolder + +#### Public Members + +| Member | Description | +|----|----| +| ~ElementRef | Destructor. | +| ElementRef | Constructor. | +| operator= | Assignment Operator. | +| _DB_STL_StoreElement | Function to store the data element. | +| _DB_STL_value | Returns the data element this wrapper object wraps. | + +#### Group + +ElementRef and ElementHolder Wappers + +## ~ElementRef + +### Function Details + +``` c +~ElementRef() + +``` + +Destructor. + +### Class + +ElementRef diff --git a/docs-src/api/stl/Element_wrappers.md b/docs-src/api/stl/Element_wrappers.md new file mode 100644 index 000000000..98e534f72 --- /dev/null +++ b/docs-src/api/stl/Element_wrappers.md @@ -0,0 +1,29 @@ +--- +title: "Chapter 22.  ElementRef and ElementHolder Wappers" +api-name: "Chapter 22.  ElementRef and ElementHolder Wappers" +source: docs/api_reference/STL/Element_wrappers.html +--- +## Chapter 22.  ElementRef and ElementHolder Wappers + +An ElementRef and ElementHolder object represents the reference to the data element referenced by an iterator. + +Each iterator object has an ElementRef or ElementHolder object that stores the data element that the iterator points to. + +The ElementHolder class is used to store primitive types into STL containers. + +The ElementRef class is used to store other types into STL containers. + +The ElementRef and ElementHolder classes have identical interfaces, and are treated the same by other STL classes. Since the ElementRef class inherits from the template data class, all methods have a \_DB_STL\_ prefix to avoid name clashes. + +An ElementRef or ElementHolder class corresponds to a single iterator instance. An Element object is generally owned by an iterator object. The ownership relationship is swapped in some specific situations, specifically for the dereference and array index operator. + +#### Public Members + +| Member | Description | +|----|----| +| ElementRef | ElementRef | +| ElementHolder | ElementHolder | + +#### Group + +Dbstl Helper Classes diff --git a/docs-src/api/stl/Exception_classes_group.md b/docs-src/api/stl/Exception_classes_group.md new file mode 100644 index 000000000..060ac96c1 --- /dev/null +++ b/docs-src/api/stl/Exception_classes_group.md @@ -0,0 +1,35 @@ +--- +title: "Chapter 29.  Dbstl Exception Classes" +api-name: "Chapter 29.  Dbstl Exception Classes" +source: docs/api_reference/STL/Exception_classes_group.html +--- +## Chapter 29.  Dbstl Exception Classes + +dbstl throws several types of exceptions on several kinds of errors, the exception classes form a class hiarachy. + +First, there is the DbstlException , which is the base class for all types of dbstl specific concrete exception classes. DbstlException inherits from the class DbException of Berkeley DB C++ API. Since DbException class inherits from C++ STL exception base class std::exception, you can make use of all Berkeley DB C++ and dbstl API exceptions in the same way you use the C++ std::exception class. + +Besides exceptions of DbstlException and its subclasses, dbstl may also throw exceptions of DbException and its subclasses, which happens when a Berkeley DB call failed. So you should use the same way you catch Berkeley DB C++ API exceptions when you want to catch exceptions throw by Berkeley DB operations. + +When an exception occurs, dbstl initialize an local exception object on the stack and throws the exception object, so you should catch an exception like this: + +try { dbstl operations } catch(DbstlException ex){ Exception handling throw ex; // Optionally throw ex again } + +#### Public Members + +| Member | Description | +|----|----| +| DbstlException | DbstlException | +| NotEnoughMemoryException | NotEnoughMemoryException | +| InvalidIteratorException | InvalidIteratorException | +| InvalidCursorException | InvalidCursorException | +| InvalidDbtException | InvalidDbtException | +| FailedAssertionException | FailedAssertionException | +| NoSuchKeyException | NoSuchKeyException | +| InvalidArgumentException | InvalidArgumentException | +| NotSupportedException | NotSupportedException | +| InvalidFunctionCall | InvalidFunctionCall | + +#### Group + +None diff --git a/docs-src/api/stl/FailedAssertionException.md b/docs-src/api/stl/FailedAssertionException.md new file mode 100644 index 000000000..015f4cd14 --- /dev/null +++ b/docs-src/api/stl/FailedAssertionException.md @@ -0,0 +1,35 @@ +--- +title: "Chapter 32.  FailedAssertionException" +api-name: "Chapter 32.  FailedAssertionException" +source: docs/api_reference/STL/FailedAssertionException.html +--- +## Chapter 32.  FailedAssertionException + +The assertions inside dbstl failed. + +The code file name and line number will be passed to the exception object of this class. + +#### Public Members + +| Member | Description | +|----|----| +| what | | +| FailedAssertionException | | +| ~FailedAssertionException | | + +#### Group + +Dbstl Exception Classes + +## what + +### Function Details + +``` c +virtual const char* what() const + +``` + +### Class + +FailedAssertionException diff --git a/docs-src/api/stl/InvalidArgumentException.md b/docs-src/api/stl/InvalidArgumentException.md new file mode 100644 index 000000000..9e0ece8bd --- /dev/null +++ b/docs-src/api/stl/InvalidArgumentException.md @@ -0,0 +1,37 @@ +--- +title: "Chapter 39.  InvalidArgumentException" +api-name: "Chapter 39.  InvalidArgumentException" +source: docs/api_reference/STL/InvalidArgumentException.html +--- +## Chapter 39.  InvalidArgumentException + +Some argument of a function is invalid. + +#### Public Members + +| Member | Description | +|----|----| +| InvalidArgumentException | | + +#### Group + +Dbstl Exception Classes + +## InvalidArgumentException + +### Function Details + +``` c +InvalidArgumentException(const char *errmsg) + +``` + +``` c +InvalidArgumentException(const char *argtype, + const char *arg) + +``` + +### Class + +InvalidArgumentException diff --git a/docs-src/api/stl/InvalidCursorException.md b/docs-src/api/stl/InvalidCursorException.md new file mode 100644 index 000000000..410c5345a --- /dev/null +++ b/docs-src/api/stl/InvalidCursorException.md @@ -0,0 +1,36 @@ +--- +title: "Chapter 33.  InvalidCursorException" +api-name: "Chapter 33.  InvalidCursorException" +source: docs/api_reference/STL/InvalidCursorException.html +--- +## Chapter 33.  InvalidCursorException + +The cursor has inconsistent status, it is unable to be used any more. + +#### Public Members + +| Member | Description | +|----|----| +| InvalidCursorException | | + +#### Group + +Dbstl Exception Classes + +## InvalidCursorException + +### Function Details + +``` c +InvalidCursorException() + +``` + +``` c +InvalidCursorException(int error_code) + +``` + +### Class + +InvalidCursorException diff --git a/docs-src/api/stl/InvalidDbtException.md b/docs-src/api/stl/InvalidDbtException.md new file mode 100644 index 000000000..b145170c3 --- /dev/null +++ b/docs-src/api/stl/InvalidDbtException.md @@ -0,0 +1,36 @@ +--- +title: "Chapter 31.  InvalidDbtException" +api-name: "Chapter 31.  InvalidDbtException" +source: docs/api_reference/STL/InvalidDbtException.html +--- +## Chapter 31.  InvalidDbtException + +The Dbt object has inconsistent status or has no valid data, it is unable to be used any more. + +#### Public Members + +| Member | Description | +|----|----| +| InvalidDbtException | | + +#### Group + +Dbstl Exception Classes + +## InvalidDbtException + +### Function Details + +``` c +InvalidDbtException() + +``` + +``` c +InvalidDbtException(int error_code) + +``` + +### Class + +InvalidDbtException diff --git a/docs-src/api/stl/InvalidFunctionCall.md b/docs-src/api/stl/InvalidFunctionCall.md new file mode 100644 index 000000000..cdfa8af29 --- /dev/null +++ b/docs-src/api/stl/InvalidFunctionCall.md @@ -0,0 +1,31 @@ +--- +title: "Chapter 38.  InvalidFunctionCall" +api-name: "Chapter 38.  InvalidFunctionCall" +source: docs/api_reference/STL/InvalidFunctionCall.html +--- +## Chapter 38.  InvalidFunctionCall + +The function can not be called in this context or in current configurations. + +#### Public Members + +| Member | Description | +|----|----| +| InvalidFunctionCall | | + +#### Group + +Dbstl Exception Classes + +## InvalidFunctionCall + +### Function Details + +``` c +InvalidFunctionCall(const char *str) + +``` + +### Class + +InvalidFunctionCall diff --git a/docs-src/api/stl/InvalidIteratorException.md b/docs-src/api/stl/InvalidIteratorException.md new file mode 100644 index 000000000..3fbf84da9 --- /dev/null +++ b/docs-src/api/stl/InvalidIteratorException.md @@ -0,0 +1,36 @@ +--- +title: "Chapter 37.  InvalidIteratorException" +api-name: "Chapter 37.  InvalidIteratorException" +source: docs/api_reference/STL/InvalidIteratorException.html +--- +## Chapter 37.  InvalidIteratorException + +The iterator has inconsistent status, it is unable to be used any more. + +#### Public Members + +| Member | Description | +|----|----| +| InvalidIteratorException | | + +#### Group + +Dbstl Exception Classes + +## InvalidIteratorException + +### Function Details + +``` c +InvalidIteratorException() + +``` + +``` c +InvalidIteratorException(int error_code) + +``` + +### Class + +InvalidIteratorException diff --git a/docs-src/api/stl/NoSuchKeyException.md b/docs-src/api/stl/NoSuchKeyException.md new file mode 100644 index 000000000..b3c274dcd --- /dev/null +++ b/docs-src/api/stl/NoSuchKeyException.md @@ -0,0 +1,33 @@ +--- +title: "Chapter 34.  NoSuchKeyException" +api-name: "Chapter 34.  NoSuchKeyException" +source: docs/api_reference/STL/NoSuchKeyException.html +--- +## Chapter 34.  NoSuchKeyException + +There is no such key in the database. + +The key can't not be passed into the exception instance because this class has to be a class template for that to work. + +#### Public Members + +| Member | Description | +|----|----| +| NoSuchKeyException | | + +#### Group + +Dbstl Exception Classes + +## NoSuchKeyException + +### Function Details + +``` c +NoSuchKeyException() + +``` + +### Class + +NoSuchKeyException diff --git a/docs-src/api/stl/NotEnoughMemoryException.md b/docs-src/api/stl/NotEnoughMemoryException.md new file mode 100644 index 000000000..0f458ad39 --- /dev/null +++ b/docs-src/api/stl/NotEnoughMemoryException.md @@ -0,0 +1,37 @@ +--- +title: "Chapter 35.  NotEnoughMemoryException" +api-name: "Chapter 35.  NotEnoughMemoryException" +source: docs/api_reference/STL/NotEnoughMemoryException.html +--- +## Chapter 35.  NotEnoughMemoryException + +Failed to allocate memory because memory is not enough. + +#### Public Members + +| Member | Description | +|----|----| +| NotEnoughMemoryException | | + +#### Group + +Dbstl Exception Classes + +## NotEnoughMemoryException + +### Function Details + +``` c +NotEnoughMemoryException(const char *msg, + size_t sz) + +``` + +``` c +NotEnoughMemoryException(const NotEnoughMemoryException &ex) + +``` + +### Class + +NotEnoughMemoryException diff --git a/docs-src/api/stl/NotSupportedException.md b/docs-src/api/stl/NotSupportedException.md new file mode 100644 index 000000000..ff506a987 --- /dev/null +++ b/docs-src/api/stl/NotSupportedException.md @@ -0,0 +1,31 @@ +--- +title: "Chapter 36.  NotSupportedException" +api-name: "Chapter 36.  NotSupportedException" +source: docs/api_reference/STL/NotSupportedException.html +--- +## Chapter 36.  NotSupportedException + +The function called is not supported in this class. + +#### Public Members + +| Member | Description | +|----|----| +| NotSupportedException | | + +#### Group + +Dbstl Exception Classes + +## NotSupportedException + +### Function Details + +``` c +NotSupportedException(const char *str) + +``` + +### Class + +NotSupportedException diff --git a/docs-src/api/stl/ReadModifyWriteOption.md b/docs-src/api/stl/ReadModifyWriteOption.md new file mode 100644 index 000000000..af1a2743f --- /dev/null +++ b/docs-src/api/stl/ReadModifyWriteOption.md @@ -0,0 +1,38 @@ +--- +title: "Chapter 28.  ReadModifyWriteOption" +api-name: "Chapter 28.  ReadModifyWriteOption" +source: docs/api_reference/STL/ReadModifyWriteOption.html +--- +## Chapter 28.  ReadModifyWriteOption + +Read-modify-write cursor configuration helper class. + +Used by each begin() function of all containers. + +#### Public Members + +| Member | Description | +|----|----| +| operator= | Assignment operator. | +| operator== | Equality comparison. | +| read_modify_write | Call this function to tell the container's begin() function that you need a read-modify-write iterator. | +| no_read_modify_write | Call this function to tell the container's begin() function that you do not need a read-modify-write iterator. | + +#### Group + +Dbstl Helper Classes + +## operator= + +### Function Details + +``` c +void operator=(ReadModifyWriteOption::Option rmw1) + +``` + +Assignment operator. + +### Class + +ReadModifyWriteOption diff --git a/docs-src/api/stl/_meta.toml b/docs-src/api/stl/_meta.toml new file mode 100644 index 000000000..d87f2c2fb --- /dev/null +++ b/docs-src/api/stl/_meta.toml @@ -0,0 +1,9 @@ +# Nav/index metadata for the C++ STL API reference tree. +# +# `index.md` is the tree landing page (migrated from the DocBook book index). +# build.py uses `title` for breadcrumbs/nav labels; `order` (optional) can pin +# a reading order for PDF/man assembly in later phases. Left implicit this +# phase since index.md already carries the ordered class/method listing. + +title = "C++ STL API Reference" +landing = "index.md" diff --git a/docs-src/api/stl/db_base_iterator.md b/docs-src/api/stl/db_base_iterator.md new file mode 100644 index 000000000..f1ab6d069 --- /dev/null +++ b/docs-src/api/stl/db_base_iterator.md @@ -0,0 +1,50 @@ +--- +title: "Chapter 10.  Db_base_iterator" +api-name: "Chapter 10.  Db_base_iterator" +source: docs/api_reference/STL/db_base_iterator.html +--- +## Chapter 10.  Db_base_iterator + +#### Public Members + +| Member | Description | +|----|----| +| refresh | Read data from underlying database via its cursor, and update its cached value. | +| close_cursor | Close its cursor. | +| set_bulk_buffer | Call this function to modify bulk buffer size. | +| get_bulk_bufsize | Return current bulk buffer size. | +| db_base_iterator | Default constructor. | +| operator= | Iterator assignment operator. | +| ~db_base_iterator | Destructor. | +| get_bulk_retrieval | Get bulk buffer size. | +| is_rmw | Get DB_RMW setting. | +| is_directdb_get | Get direct database get setting. | + +#### Group + +Dbstl Iterator Classes + +## refresh + +### Function Details + +``` c +int refresh(bool from_db=true) + +``` + +Read data from underlying database via its cursor, and update its cached value. + +#### Parameters + +##### from_db + +Whether retrieve data from database rather than using the cached data in this iterator. + +#### Return Value + +0 if succeeded. Otherwise an DbstlException exception will be thrown. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/db_container.md b/docs-src/api/stl/db_container.md new file mode 100644 index 000000000..8dc77755c --- /dev/null +++ b/docs-src/api/stl/db_container.md @@ -0,0 +1,60 @@ +--- +title: "Chapter 3.  Db_container" +api-name: "Chapter 3.  Db_container" +source: docs/api_reference/STL/db_container.html +--- +## Chapter 3.  Db_container + +This class is the base class for all db container classes, you don't directly use this class, but all container classes inherit from this class, so you need to know the methods that can be accessed via concrete container classes. + +This class is also used to support auto commit transactions. Autocommit is enabled when DB_AUTO_COMMIT is set to the database or database environment handle and the environment is transactional. + +Inside dbstl, there are transactions begun and committed/aborted if the backing database and/or environment requires auto commit, and there are cursors opened internally, and you can set the flags used by the transaction and cursor functions via set functions of this class. + +All dbstl containers are fully multi-threaded, you should not need any synchronization to use them in the correct way, but this class is not thread safe, access to its members are not proctected by any mutex because the data members of this class are supposed to be set before they are used, and remain read only afterwards. If this is not the case, you must synchronize the access. + +#### Public Members + +| Member | Description | +|----|----| +| get_db_open_flags | Get the backing database's open flags. | +| get_db_set_flags | Get the backing database's flags that are set via Db::set_flags() function. | +| get_db_handle | Get the backing database's handle. | +| get_db_env_handle | Get the backing database environment's handle. | +| set_db_handle | Set the underlying database's handle, and optionally environment handle if the environment has also changed. | +| set_all_flags | Set the flags required by the Berkeley DB functions DbEnv::txn_begin(), DbTxn::commit() and DbEnv::cursor(). | +| set_txn_begin_flags | Set flag of DbEnv::txn_begin() call. | +| get_txn_begin_flags | Get flag of DbEnv::txn_begin() call. | +| set_commit_flags | Set flag of DbTxn::commit() call. | +| get_commit_flags | Get flag of DbTxn::commit() call. | +| get_cursor_open_flags | Get flag of Db::cursor() call. | +| set_cursor_open_flags | Set flag of Db::cursor() call. | +| db_container | Default constructor. | +| ~db_container | The backing database is not closed in this function. | + +#### Group + +Dbstl Container Classes + +## get_db_open_flags + +### Function Details + +``` c +u_int32_t get_db_open_flags() const + +``` + +Get the backing database's open flags. + +#### Return Value + +The backing database's open flags. + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/db_map.md b/docs-src/api/stl/db_map.md new file mode 100644 index 000000000..a4dcad3ec --- /dev/null +++ b/docs-src/api/stl/db_map.md @@ -0,0 +1,153 @@ +--- +title: "Chapter 5.  Db_map" +api-name: "Chapter 5.  Db_map" +source: docs/api_reference/STL/db_map.html +--- +## Chapter 5.  Db_map + +db_map has identical methods to std::map and the semantics for each method is identical to its std::map counterpart, except that it stores data into underlying Berkeley DB btree or hash database. + +Passing a database handle of btree or hash type creates a db_map equivalent to std::map and std::hashmap respectively. Database(dbp) and environment(penv) handle requirement(applies to all constructors in this class template): 0. The dbp is opened inside the penv environment. Either one of the two handles can be NULL. If dbp is NULL, an anonymous database is created by dbstl. 1. Database type of dbp should be DB_BTREE or DB_HASH. 2. No DB_DUP or DB_DUPSORT flag set in dbp. 3. No DB_RECNUM flag set in dbp. 4. No DB_TRUNCATE specified in dbp's database open flags. 5. DB_THREAD must be set if you are sharing the dbp across multiple threads directly, or indirectly by sharing the container object across multiple threads. + +#### See Also + +db_container db_container(Db*, DbEnv*) db_container(const db_container&) + +### Class Template Parameters + +#### kdt + +The key data type. + +#### ddt + +The data data type. db_map stores key/data pairs. + +#### value_type_sub + +Do not specify anything if ddt type is a class/struct type; Otherwise, specify ElementHolder\ to it. + +#### iterator_t + +Never specify anything to this type parameter. It is only used internally. + +#### Public Members + +| Member | Description | +|----|----| +| db_map | Create a std::map/hash_map equivalent associative container. | +| ~db_map | | +| insert | Insert a single key/data pair if the key is not in the container. | +| begin | Begin a read-write or readonly iterator which sits on the first key/data pair of the database. | +| end | Create an open boundary iterator. | +| rbegin | Begin a read-write or readonly reverse iterator which sits on the first key/data pair of the database. | +| rend | Create an open boundary iterator. | +| is_hash | Get container category. | +| bucket_count | Only for std::hash_map, return number of hash bucket in use. | +| size | This function supports auto-commit. | +| max_size | Get max size. | +| empty | Returns whether this container is empty. | +| erase | Erase a key/data pair at specified position. | +| find | Find the key/data pair with specified key x. | +| lower_bound | Find the greatest key less than or equal to x. | +| equal_range | Find the range within which all keys equal to specified key x. | +| count | Count the number of key/data pairs having specified key x. | +| upper_bound | Find the least key greater than x. | +| key_eq | Function to get key compare functor. | +| hash_funct | Function to get hash key generating functor. | +| value_comp | Function to get value compare functor. | +| key_comp | Function to get key compare functor. | +| operator= | Container content assignment operator. | +| operator[] | Retrieve data element by key. | +| swap | Swap content with container mp. | +| clear | Clear contents in this container. | +| operator== | Map content equality comparison operator. | +| operator!= | Container unequality comparison operator. | + +#### Group + +Dbstl Container Classes + +## db_map + +### Function Details + +``` c +db_map(Db *dbp=NULL, + DbEnv *envp=NULL) + +``` + +Create a std::map/hash_map equivalent associative container. + +See the handle requirement in class details to pass correct database/environment handles. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +#### See Also + +db_container(Db*, DbEnv*) + +``` c +db_map(Db *dbp, DbEnv *envp, InputIterator first, + InputIterator last) + +``` + +Iteration constructor. + +Iterates between first and last, setting a copy of each of the sequence of elements as the content of the container object. Create a std::map/hash_map equivalent associative container. Insert a range of elements into the database. The range is \[first, last), which contains elements that can be converted to type ddt automatically. See the handle requirement in class details to pass correct database/environment handles. This function supports auto-commit. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +#### See Also + +db_container(Db*, DbEnv*) + +``` c +db_map(const db_map< kdt, ddt, value_type_sub, + iterator > &x) + +``` + +Copy constructor. + +Create an database and insert all key/data pairs in x into this container. x's data members are not copied. This function supports auto-commit. + +#### Parameters + +##### x + +The other container to initialize this container. + +#### See Also + +db_container(const db_container&) + +### Class + +db_map diff --git a/docs-src/api/stl/db_map_base_iterator.md b/docs-src/api/stl/db_map_base_iterator.md new file mode 100644 index 000000000..b3d4e0bb2 --- /dev/null +++ b/docs-src/api/stl/db_map_base_iterator.md @@ -0,0 +1,107 @@ +--- +title: "Chapter 15.  Db_map_base_iterator" +api-name: "Chapter 15.  Db_map_base_iterator" +source: docs/api_reference/STL/db_map_base_iterator.html +--- +## Chapter 15.  Db_map_base_iterator + +#### Public Members + +| Member | Description | +|----|----| +| db_map_base_iterator | Copy constructor. | +| ~db_map_base_iterator | Destructor. | +| operator++ | Pre-increment. | +| operator-- | Pre-decrement. | +| operator== | Equal comparison operator. | +| operator!= | Unequal comparison operator. | +| operator * | Dereference operator. | +| operator-> | Arrow operator. | +| refresh | Refresh iterator cached value. | +| close_cursor | Close underlying Berkeley DB cursor of this iterator. | +| move_to | Iterator movement function. | +| set_bulk_buffer | Modify bulk buffer size. | +| get_bulk_bufsize | Get bulk retrieval buffer size in bytes. | +| operator= | Assignment operator. | + +#### Group + +Iterator Classes for db_map and db_multimap + +## db_map_base_iterator + +### Function Details + +``` c +db_map_base_iterator(const self &vi) + +``` + +Copy constructor. + +#### Parameters + +##### vi + +The other iterator of the same type to initialize this. + +``` c +db_map_base_iterator(const base &vi) + +``` + +Base copy constructor. + +#### Parameters + +##### vi + +Initialize from a base class iterator. + +``` c +db_map_base_iterator(db_container *powner, u_int32_t b_bulk_retrieval=0, + bool rmw=false, bool directdbget=true, + bool readonly=false) + +``` + +Constructor. + +#### Parameters + +##### b_bulk_retrieval + +The bulk read buffer size. 0 means bulk read disabled. + +##### directdbget + +Whether do direct database get rather than using key/data values cached in the iterator whenever read. + +##### readonly + +Whether open a read only cursor. Only effective when using Berkeley DB Concurrent Data Store. + +##### powner + +The container which creates this iterator. + +##### rmw + +Whether set DB_RMW flag in underlying cursor. + +``` c +db_map_base_iterator() + +``` + +Default constructor, dose not create the cursor for now. + +### Group: Constructors and destructor + +Do not create iterators directly using these constructors, but call db_map::begin or db_multimap_begin to get instances of this class. + +db_map::begin() db_multimap::begin() + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/db_map_iterator.md b/docs-src/api/stl/db_map_iterator.md new file mode 100644 index 000000000..978885f92 --- /dev/null +++ b/docs-src/api/stl/db_map_iterator.md @@ -0,0 +1,103 @@ +--- +title: "Chapter 16.  Db_map_iterator" +api-name: "Chapter 16.  Db_map_iterator" +source: docs/api_reference/STL/db_map_iterator.html +--- +## Chapter 16.  Db_map_iterator + +#### Public Members + +| Member | Description | +|----|----| +| db_map_iterator | Copy constructor. | +| ~db_map_iterator | Destructor. | +| operator++ | Pre-increment. | +| operator-- | Pre-decrement. | +| operator * | Dereference operator. | +| operator-> | Arrow operator. | +| refresh | Refresh iterator cached value. | +| operator= | Assignment operator. | + +#### Group + +Dbstl Iterator Classes + +## db_map_iterator + +### Function Details + +``` c +db_map_iterator(const db_map_iterator< kdt, ddt, + value_type_sub > &vi) + +``` + +Copy constructor. + +#### Parameters + +##### vi + +The other iterator of the same type to initialize this. + +``` c +db_map_iterator(const db_map_base_iterator< kdt, realddt, + ddt > &vi) + +``` + +Base copy constructor. + +#### Parameters + +##### vi + +Initialize from a base class iterator. + +``` c +db_map_iterator(db_container *powner, u_int32_t b_bulk_retrieval=0, + bool brmw=false, bool directdbget=true, + bool b_read_only=false) + +``` + +Constructor. + +#### Parameters + +##### b_bulk_retrieval + +The bulk read buffer size. 0 means bulk read disabled. + +##### brmw + +Whether set DB_RMW flag in underlying cursor. + +##### powner + +The container which creates this iterator. + +##### directdbget + +Whether do direct database get rather than using key/data values cached in the iterator whenever read. + +##### b_read_only + +Whether open a read only cursor. Only effective when using Berkeley DB Concurrent Data Store. + +``` c +db_map_iterator() + +``` + +Default constructor, dose not create the cursor for now. + +### Group: Constructors and destructor + +Do not create iterators directly using these constructors, but call db_map::begin or db_multimap_begin to get instances of this class. + +db_map::begin() db_multimap::begin() + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/db_map_iterators.md b/docs-src/api/stl/db_map_iterators.md new file mode 100644 index 000000000..968a74eef --- /dev/null +++ b/docs-src/api/stl/db_map_iterators.md @@ -0,0 +1,25 @@ +--- +title: "Chapter 14.  Iterator Classes for db_map and db_multimap" +api-name: "Chapter 14.  Iterator Classes for db_map and db_multimap" +source: docs/api_reference/STL/db_map_iterators.html +--- +## Chapter 14.  Iterator Classes for db_map and db_multimap + +db_map has two iterator class templates -- db_map_base_iterator and db_map_iterator . + +They are the const iterator class and iterator class for db_map and db_multimap . db_map_iterator inherits from db_map_base_iterator . + +The two classes have identical behaviors to std::map::const_iterator and std::map::iterator respectively. Note that the common public member function behaviors are described in the db_base_iterator section. + +The differences between the two classes are that the db_map_base_iterator can only be used to read its referenced value, while db_map_iterator allows both read and write access. If your access pattern is readonly, it is strongly recommended that you use the const iterator because it is faster and more efficient. + +#### Public Members + +| Member | Description | +|----|----| +| db_map_base_iterator | db_map_base_iterator | +| db_map_iterator | db_map_iterator | + +#### Group + +Dbstl Iterator Classes diff --git a/docs-src/api/stl/db_multimap.md b/docs-src/api/stl/db_multimap.md new file mode 100644 index 000000000..d112a45fa --- /dev/null +++ b/docs-src/api/stl/db_multimap.md @@ -0,0 +1,122 @@ +--- +title: "Chapter 6.  Db_multimap" +api-name: "Chapter 6.  Db_multimap" +source: docs/api_reference/STL/db_multimap.html +--- +## Chapter 6.  Db_multimap + +This class is the combination of std::multimap and hash_multimap. + +By setting database handles as DB_BTREE or DB_HASH type respectively, you will be using an equivalent of std::multimap or hash_multimap respectively. Database(dbp) and environment(penv) handle requirement: The dbp handle must meet the following requirement: 1. Database type should be DB_BTREE or DB_HASH. 2. Either DB_DUP or DB_DUPSORT flag must be set. Note that so far Berkeley DB does not allow DB_DUPSORT be set and the database is storing identical key/data pairs, i.e. we can't store two (1, 2), (1, 2) pairs into a database D with DB_DUPSORT flag set, but only can do so with DB_DUP flag set; But we can store a (1, 2) pair and a (1, 3) pair into D with DB_DUPSORT flag set. So if your data set allows DB_DUPSORT flag, you should set it to gain a lot of performance promotion. 3. No DB_RECNUM flag set. 4. No DB_TRUNCATE specified in database open flags. 5. DB_THREAD must be set if you are sharing the database handle across multiple threads directly, or indirectly by sharing the container object across multiple threads. + +#### See Also + +db_container db_map + +### Class Template Parameters + +#### kdt + +The key data type. + +#### ddt + +The data data type. db_multimap stores key/data pairs. + +#### value_type_sub + +Do not specify anything if ddt type is a class/struct type; Otherwise, specify ElementHolder\ to it. + +#### iterator_t + +Never specify anything to this type parameter. It is only used internally. + +#### Public Members + +| Member | Description | +|----|----| +| insert | Range insertion. | +| erase | Erase elements by key. | +| equal_range | Find the range within which all keys equal to specified key x. | +| equal_range_N | Find equal range and number of key/data pairs in the range. | +| count | Count the number of key/data pairs having specified key x. | +| upper_bound | Find the least key greater than x. | +| db_multimap | Constructor. | +| ~db_multimap | | +| operator= | Container content assignment operator. | +| swap | Swap content with another multimap container. | +| operator== | Returns whether the two containers have identical content. | +| operator!= | Container unequality comparison operator. | + +#### Group + +Dbstl Container Classes + +## insert + +### Function Details + +``` c +void insert(InputIterator first, + InputIterator last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +void insert(const_iterator &first, + const_iterator &last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +iterator insert(const value_type &x) + +``` + +Insert a single key/data pair if the key is not in the container. + +#### Parameters + +##### x + +The key/data pair to insert. + +#### Return Value + +A pair P, if insert OK, i.e. the inserted key wasn't in the container, P.first will be the iterator sitting on the inserted key/data pair, and P.second is true; otherwise P.first is an invalid iterator and P.second is false. + +### Group: Insert Functions + +http://www.cplusplus.com/reference/stl/multimap/insert/ + +### Class + +db_multimap diff --git a/docs-src/api/stl/db_multiset.md b/docs-src/api/stl/db_multiset.md new file mode 100644 index 000000000..f151d84cf --- /dev/null +++ b/docs-src/api/stl/db_multiset.md @@ -0,0 +1,124 @@ +--- +title: "Chapter 8.  Db_multiset" +api-name: "Chapter 8.  Db_multiset" +source: docs/api_reference/STL/db_multiset.html +--- +## Chapter 8.  Db_multiset + +This class is the combination of std::multiset and hash_multiset. + +By setting database handles of DB_BTREE or DB_HASH type respectively, you will be using the equivalent of std::multiset or hash_multiset respectively. This container stores the key in the key element of a key/data pair in the underlying database, but doesn't store anything in the data element. Database and environment handle requirement: The requirement to these handles is the same as that to db_multimap . + +#### See Also + +db_multimap db_map db_container db_set + +### Class Template Parameters + +#### kdt + +The key data type. + +#### value_type_sub + +If kdt is a class/struct type, do not specify anything in this parameter; Otherwise specify ElementHolder\. + +#### Public Members + +| Member | Description | +|----|----| +| db_multiset | Create a std::multiset/hash_multiset equivalent associative container. | +| ~db_multiset | | +| insert | Insert a single key if the key is not in the container. | +| erase | Erase elements by key. | +| operator= | Container content assignment operator. | +| swap | Swap content with another container. | +| operator== | Container content equality compare operator. | +| operator!= | Inequality comparison operator. | + +#### Group + +Dbstl Container Classes + +## db_multiset + +### Function Details + +``` c +db_multiset(Db *dbp=NULL, + DbEnv *envp=NULL) + +``` + +Create a std::multiset/hash_multiset equivalent associative container. + +See the handle requirement in class details to pass correct database/environment handles. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +#### See Also + +db_multimap(Db*, DbEnv*) + +``` c +db_multiset(Db *dbp, DbEnv *envp, InputIterator first, + InputIterator last) + +``` + +Iteration constructor. + +Iterates between first and last, copying each of the elements in the range into this container. Create a std::multi/hash_multiset equivalent associative container. Insert a range of elements into the database. The range is \[first, last), which contains elements that can be converted to type ddt automatically. This function supports auto-commit. See the handle requirement in class details to pass correct database/environment handles. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +#### See Also + +db_multimap(Db*, DbEnv*, InputIterator, InputIterator) + +``` c +db_multiset(const self &x) + +``` + +Copy constructor. + +Create a database and insert all key/data pairs in x into this container. x's data members are not copied. This function supports auto-commit. + +#### Parameters + +##### x + +The source container to initialize this container. + +#### See Also + +db_multimap(const db_multimap&) db_container(const db_container&) + +### Class + +db_multiset diff --git a/docs-src/api/stl/db_reverse_iterator.md b/docs-src/api/stl/db_reverse_iterator.md new file mode 100644 index 000000000..ee4230f44 --- /dev/null +++ b/docs-src/api/stl/db_reverse_iterator.md @@ -0,0 +1,72 @@ +--- +title: "Chapter 20.  Db_reverse_iterator" +api-name: "Chapter 20.  Db_reverse_iterator" +source: docs/api_reference/STL/db_reverse_iterator.html +--- +## Chapter 20.  Db_reverse_iterator + +This class is the reverse class adaptor for all dbstl iterator classes. + +It inherits from real iterator classes like db_vector_iterator , db_map_iterator or db_set_iterator . When you call container::rbegin(), you will get an instance of this class. + +#### See Also + +db_vector_base_iterator db_vector_iterator db_map_base_iterator db_map_iterator db_set_base_iterator db_set_iterator + +#### Public Members + +| Member | Description | +|----|----| +| operator++ | Move this iterator forward by one element. | +| operator-- | Move this iterator backward by one element. | +| operator+ | Iterator shuffle operator. | +| operator- | Iterator shuffle operator. | +| operator+= | Iterator shuffle operator. | +| operator-= | Iterator shuffle operator. | +| operator< | Less compare operator. | +| operator> | Greater compare operator. | +| operator<= | Less equal compare operator. | +| operator>= | Greater equal compare operator. | +| db_reverse_iterator | Constructor. Construct from an iterator of wrapped type. | +| operator= | Assignment operator. | +| operator[] | Return the reference of the element which can be reached by moving this reverse iterator by Off times backward. | + +#### Group + +Dbstl Iterator Classes + +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Move this iterator forward by one element. + +#### Return Value + +The moved iterator at new position. + +``` c +self operator++(int) + +``` + +Move this iterator forward by one element. + +#### Return Value + +The original iterator at old position. + +### Group: Reverse iterator movement functions + +When we talk about reverse iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/db_set.md b/docs-src/api/stl/db_set.md new file mode 100644 index 000000000..29e191d22 --- /dev/null +++ b/docs-src/api/stl/db_set.md @@ -0,0 +1,124 @@ +--- +title: "Chapter 7.  Db_set" +api-name: "Chapter 7.  Db_set" +source: docs/api_reference/STL/db_set.html +--- +## Chapter 7.  Db_set + +This class is the combination of std::set and hash_set. + +By setting database handles of DB_BTREE or DB_HASH type, you will be using the equivalent of std::set or hash_set. This container stores the key in the key element of a key/data pair in the underlying database, but doesn't store anything in the data element. Database and environment handle requirement: The same as that of db_map . + +#### See Also + +db_map db_container + +### Class Template Parameters + +#### kdt + +The key data type. + +#### value_type_sub + +If kdt is a class/struct type, do not specify anything in this parameter; Otherwise specify ElementHolder\. + +#### Public Members + +| Member | Description | +|----|----| +| db_set | Create a std::set/hash_set equivalent associative container. | +| ~db_set | | +| insert | Insert a single key/data pair if the key is not in the container. | +| operator= | Container content assignment operator. | +| value_comp | Get value comparison functor. | +| swap | Swap content with another container. | +| operator== | Set content equality comparison operator. | +| operator!= | Inequality comparison operator. | + +#### Group + +Dbstl Container Classes + +## db_set + +### Function Details + +``` c +db_set(Db *dbp=NULL, + DbEnv *envp=NULL) + +``` + +Create a std::set/hash_set equivalent associative container. + +See the handle requirement in class details to pass correct database/environment handles. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +#### See Also + +db_map(Db*, DbEnv*) db_container(Db*, DbEnv*) + +``` c +db_set(Db *dbp, DbEnv *envp, InputIterator first, + InputIterator last) + +``` + +Iteration constructor. + +Iterates between first and last, copying each of the elements in the range into this container. Create a std::set/hash_set equivalent associative container. Insert a range of elements into the database. The range is \[first, last), which contains elements that can be converted to type ddt automatically. This function supports auto-commit. See the handle requirement in class details to pass correct database/environment handles. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +#### See Also + +db_map(Db*, DbEnv*, InputIterator, InputIterator) + +``` c +db_set(const self &x) + +``` + +Copy constructor. + +Create a database and insert all key/data pairs in x into this container. x's data members are not copied. This function supports auto-commit. + +#### Parameters + +##### x + +The source container to initialize this container. + +#### See Also + +db_map(const db_map&) db_container(const db_container&) + +### Class + +db_set diff --git a/docs-src/api/stl/db_set_base_iterator.md b/docs-src/api/stl/db_set_base_iterator.md new file mode 100644 index 000000000..f68cd0540 --- /dev/null +++ b/docs-src/api/stl/db_set_base_iterator.md @@ -0,0 +1,41 @@ +--- +title: "Chapter 18.  Db_set_base_iterator" +api-name: "Chapter 18.  Db_set_base_iterator" +source: docs/api_reference/STL/db_set_base_iterator.html +--- +## Chapter 18.  Db_set_base_iterator + +#### Public Members + +| Member | Description | +|----|----| +| ~db_set_base_iterator | Destructor. | +| db_set_base_iterator | Constructor. | +| operator++ | Post-increment. | +| operator-- | Post-decrement. | +| operator * | Dereference operator. | +| operator-> | Arrow operator. | +| refresh | Refresh iterator cached value. | + +#### Group + +Iterator Classes for db_set and db_multiset + +## ~db_set_base_iterator + +### Function Details + +``` c +virtual ~db_set_base_iterator() + +``` + +Destructor. + +### Group: Constructors and destructor + +Do not use these constructors to create iterators, but call db_set::begin() const or db_multiset::begin() const to create valid iterators. + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/db_set_iterator.md b/docs-src/api/stl/db_set_iterator.md new file mode 100644 index 000000000..6bbd1a569 --- /dev/null +++ b/docs-src/api/stl/db_set_iterator.md @@ -0,0 +1,41 @@ +--- +title: "Chapter 19.  Db_set_iterator" +api-name: "Chapter 19.  Db_set_iterator" +source: docs/api_reference/STL/db_set_iterator.html +--- +## Chapter 19.  Db_set_iterator + +#### Public Members + +| Member | Description | +|----|----| +| ~db_set_iterator | Destructor. | +| db_set_iterator | Constructor. | +| operator++ | Pre-increment. | +| operator-- | Pre-decrement. | +| operator * | Dereference operator. | +| operator-> | Arrow operator. | +| refresh | Refresh iterator cached value. | + +#### Group + +Iterator Classes for db_set and db_multiset + +## ~db_set_iterator + +### Function Details + +``` c +virtual ~db_set_iterator() + +``` + +Destructor. + +### Group: Constructors and destructor + +Do not use these constructors to create iterators, but call db_set::begin() or db_multiset::begin() to create valid ones. + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/db_vector.md b/docs-src/api/stl/db_vector.md new file mode 100644 index 000000000..7e889f9af --- /dev/null +++ b/docs-src/api/stl/db_vector.md @@ -0,0 +1,144 @@ +--- +title: "Chapter 4.  Db_vector" +api-name: "Chapter 4.  Db_vector" +source: docs/api_reference/STL/db_vector.html +--- +## Chapter 4.  Db_vector + +The db_vector class has the union set of public member functions as std::vector, std::deque and std::list, and each method has identical default semantics to that in the std equivalent containers. + +The difference is that the data is maintained using a Berkeley DB database as well as some Berkeley DB related extensions. + +#### See Also + +db_container db_container(Db*, DbEnv*) db_container(const db_container&) + +### Class Template Parameters + +#### T + +The type of data to store. + +#### value_type_sub + +If T is a class/struct type, do not specify anything for this parameter; Otherwise, specify ElementHolder\ to it. Database(dbp) and environment(penv) handle requirement(applies for all constructors of this class template): dbp must meet the following requirement: 1. dbp must be a DB_RECNO type of database handle. 2. DB_THREAD must be set to dbp's open flags. 3. An optional flag DB_RENUMBER is required if the container object is supposed to be a std::vector or std::deque equivalent; Not required if it is a std::list equivalent. But dbstl will not check whether DB_RENUMBER is set to this database handle. Setting DB_RENUMBER will cause the index values of all elements in the underlying databse to be maintained consecutive and in order, which involves potentially a lot of work because many indices may be updated. See the db_container(Db*, DbEnv*) for more information about the two parameters. + +#### Public Members + +| Member | Description | +|----|----| +| begin | Create a read-write or read-only iterator. | +| end | Create an open boundary iterator. | +| rbegin | Create a reverse iterator. | +| rend | Create an open boundary iterator. | +| max_size | Get max size. | +| capacity | Get capacity. | +| operator[] | Index operator, can act as both a left value and a right value. | +| at | Index function. | +| front | Return a reference to the first element. | +| back | Return a reference to the last element. | +| operator== | Container equality comparison operator. | +| operator!= | Container in-equality comparison operator. | +| operator< | Container less than comparison operator. | +| assign | Assign a range \[first, last) to this container. | +| push_front | Push an element x into the vector from front. | +| pop_front | Pop out the front element from the vector. | +| insert | Insert x before position pos. | +| erase | Erase element at position pos. | +| remove | Remove all elements whose values are "value" from the list. | +| remove_if | Remove all elements making "pred" return true. | +| merge | Merge content with another container. | +| unique | Remove consecutive duplicate values from this list. | +| sort | Sort this list. | +| reverse | Reverse this list. | +| splice | Moves elements from list x into this list. | +| size | Return the number of elements in this container. | +| empty | Returns whether this container is empty. | +| db_vector | Constructor. | +| ~db_vector | | +| operator= | Container assignment operator. | +| resize | Resize this container to specified size n, insert values t if need to enlarge the container. | +| reserve | Reserve space. | +| push_back | Push back an element into the vector. | +| pop_back | Pop out last element from the vector. | +| swap | Swap content with another vector vec. | +| clear | Remove all elements of the vector, make it an empty vector. | + +#### Group + +Dbstl Container Classes + +## begin + +### Function Details + +``` c +iterator begin(ReadModifyWriteOption rmw= + ReadModifyWriteOption::no_read_modify_write(), bool readonly=false, + BulkRetrievalOption bulk_read=BulkRetrievalOption::no_bulk_retrieval(), + bool directdb_get=true) + +``` + +Create a read-write or read-only iterator. + +We allow users to create a readonly iterator here so that they don't have to use a const container to create a const_iterator. But using const_iterator is faster. The flags set via db_container::set_cursor_oflags() is used as the cursor open flags. + +#### Parameters + +##### directdb_get + +Whether always read key/data pair from backing db rather than using the value cached in the iterator. The current key/data pair is cached in the iterator and always kept updated on iterator movement, but in some extreme conditions, errors can happen if you use cached key/data pairs without always refreshing them from database. By default we are always reading from database when we are accessing the data the iterator sits on, except when we are doing bulk retrievals. But your application can gain extra performance promotion if you can set this flag to false. + +##### readonly + +Whether the iterator is created as a readonly iterator. Read only iterators can not update its underlying key/data pair. + +##### bulk_read + +Whether read database key/data pairs in bulk, by specifying DB_MULTIPLE_KEY flag to underlying cursor's Dbc::get function. Only readonly iterators can do bulk retrieval, if iterator is not read only, this parameter is ignored. Bulk retrieval can accelerate reading speed because each database read operation will read many key/data pairs, thus saved many database read operations. The default bulk buffer size is 32KB, you can set your desired bulk buffer size by specifying BulkRetrievalOpt::bulk_retrieval(your_bulk_buffer_size); If you don't want bulk retrieval, set BulkRetrievalItrOpt::no_bulk_retrieval() as the real parameter. + +##### rmw + +Whether this iterator will open a Berkeley DB cursor with DB_RMW flag set. If the iterator is used to read a key/data pair, then update it and store back to db, it is good to set the DB_RMW flag, by specifying RMWItrOpt::read_modify_write() If you don't want to set the DB_RMW flag, specify RMWItrOpt::no_read_modify_write(), which is the default behavior. + +#### Return Value + +The created iterator. + +#### See Also + +db_container::set_cursor_oflags(); + +``` c +const_iterator begin(BulkRetrievalOption bulkretrieval= + (BulkRetrievalOption::no_bulk_retrieval()), + bool directdb_get=true) const + +``` + +Create a const iterator. + +The created iterator can only be used to read its referenced data element. Can only be called when using a const reference to the contaienr object. The parameters have identical meanings and usage to those of the other non-const begin function. + +#### Parameters + +##### directdb_get + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulkretrieval + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created const iterator. + +#### See Also + +begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +### Class + +db_vector diff --git a/docs-src/api/stl/db_vector_base_iterator.md b/docs-src/api/stl/db_vector_base_iterator.md new file mode 100644 index 000000000..9eb6defe1 --- /dev/null +++ b/docs-src/api/stl/db_vector_base_iterator.md @@ -0,0 +1,72 @@ +--- +title: "Chapter 12.  Db_vector_base_iterator" +api-name: "Chapter 12.  Db_vector_base_iterator" +source: docs/api_reference/STL/db_vector_base_iterator.html +--- +## Chapter 12.  Db_vector_base_iterator + +This class is the const iterator class for db_vector , and it is inheirted by the db_vector_iterator class, which is the iterator class for db_vector . + +#### Public Members + +| Member | Description | +|----|----| +| db_vector_base_iterator | | +| ~db_vector_base_iterator | | +| operator== | Equality comparison operator. | +| operator!= | Unequal compare, identical to !operator(==itr). | +| operator< | Less than comparison operator. | +| operator<= | Less equal comparison operator. | +| operator>= | Greater equal comparison operator. | +| operator> | Greater comparison operator. | +| operator++ | Pre-increment. | +| operator-- | Pre-decrement. | +| operator= | Assignment operator. | +| operator+ | Iterator movement operator. | +| operator+= | Move this iterator backward by n elements. | +| operator- | Iterator movement operator. | +| operator-= | Move this iterator forward by n elements. | +| operator * | Dereference operator. | +| operator-> | Arrow operator. | +| operator[] | Iterator index operator. | +| get_current_index | Get current index of within the vector. | +| move_to | Iterator movement function. | +| refresh | Refresh iterator cached value. | +| close_cursor | Close underlying Berkeley DB cursor of this iterator. | +| set_bulk_buffer | Modify bulk buffer size. | +| get_bulk_bufsize | Get bulk retrieval buffer size in bytes. | + +#### Group + +Iterator Classes for db_vector + +## db_vector_base_iterator + +### Function Details + +``` c +db_vector_base_iterator(const db_vector_base_iterator< T > &vi) + +``` + +``` c +db_vector_base_iterator(db_container *powner, u_int32_t b_bulk_retrieval=0, + bool rmw=false, bool directdbget=true, + bool readonly=false) + +``` + +``` c +db_vector_base_iterator() + +``` + +### Group: Constructors and destroctor + +Do not construct iterators explictily using these constructors, but call db_vector::begin() const to get an valid iterator. + +db_vector::begin() const + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/db_vector_iterator.md b/docs-src/api/stl/db_vector_iterator.md new file mode 100644 index 000000000..d17bab8fa --- /dev/null +++ b/docs-src/api/stl/db_vector_iterator.md @@ -0,0 +1,65 @@ +--- +title: "Chapter 13.  Db_vector_iterator" +api-name: "Chapter 13.  Db_vector_iterator" +source: docs/api_reference/STL/db_vector_iterator.html +--- +## Chapter 13.  Db_vector_iterator + +#### Public Members + +| Member | Description | +|----|----| +| db_vector_iterator | | +| ~db_vector_iterator | | +| operator++ | Pre-increment. | +| operator-- | Pre-decrement. | +| operator= | Assignment operator. | +| operator+ | Iterator movement operator. | +| operator+= | Move this iterator backward by n elements. | +| operator- | Iterator movement operator. | +| operator-= | Move this iterator forward by n elements. | +| operator * | Dereference operator. | +| operator-> | Arrow operator. | +| operator[] | Iterator index operator. | +| refresh | Refresh iterator cached value. | + +#### Group + +Iterator Classes for db_vector + +## db_vector_iterator + +### Function Details + +``` c +db_vector_iterator(const db_vector_iterator< T, + value_type_sub > &vi) + +``` + +``` c +db_vector_iterator(db_container *powner, u_int32_t b_bulk_retrieval=0, + bool brmw=false, bool directdbget=true, + bool b_read_only=false) + +``` + +``` c +db_vector_iterator() + +``` + +``` c +db_vector_iterator(const db_vector_base_iterator< T > &obj) + +``` + +### Group: Constructors and destructor + +Do not construct iterators explictily using these constructors, but call db_vector::begin to get an valid iterator. + +db_vector::begin + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/db_vector_iterators.md b/docs-src/api/stl/db_vector_iterators.md new file mode 100644 index 000000000..1d2406b8e --- /dev/null +++ b/docs-src/api/stl/db_vector_iterators.md @@ -0,0 +1,25 @@ +--- +title: "Chapter 11.  Iterator Classes for db_vector" +api-name: "Chapter 11.  Iterator Classes for db_vector" +source: docs/api_reference/STL/db_vector_iterators.html +--- +## Chapter 11.  Iterator Classes for db_vector + +db_vector has two iterator classes --- db_vector_base_iterator and db_vector_iterator . + +The differences between the two classes are that the db_vector_base_iterator can only be used to read its referenced value, so it is intended as db_vector's const iterator; While the other class allows both read and write access. If your access pattern is readonly, it is strongly recommended that you use the const iterator because it is faster and more efficient. The two classes have identical behaviors to std::vector::const_iterator and std::vector::iterator respectively. Note that the common public member function behaviors are described in the db_base_iterator section. + +#### See Also + +db_base_iterator + +#### Public Members + +| Member | Description | +|----|----| +| db_vector_base_iterator | db_vector_base_iterator | +| db_vector_iterator | db_vector_iterator | + +#### Group + +Dbstl Iterator Classes diff --git a/docs-src/api/stl/dbset_iterators.md b/docs-src/api/stl/dbset_iterators.md new file mode 100644 index 000000000..a05c34038 --- /dev/null +++ b/docs-src/api/stl/dbset_iterators.md @@ -0,0 +1,29 @@ +--- +title: "Chapter 17.  Iterator Classes for db_set and db_multiset" +api-name: "Chapter 17.  Iterator Classes for db_set and db_multiset" +source: docs/api_reference/STL/dbset_iterators.html +--- +## Chapter 17.  Iterator Classes for db_set and db_multiset + +db_set_base_iterator and db_set_iterator are the const iterator and iterator class for db_set and db_multiset . + +They have identical behaviors to std::set::const_iterator and std::set::iterator respectively. + +The difference between the two classes is that the db_set_base_iterator can only be used to read its referenced value, while db_set_iterator allows both read and write access. If the access pattern is readonly, it is strongly recommended that you use the const iterator because it is faster and more efficient. + +The two classes inherit several functions from db_map_base_iterator and db_map_iterator respectively. + +#### See Also + +db_map_base_iterator db_map_iterator + +#### Public Members + +| Member | Description | +|----|----| +| db_set_base_iterator | db_set_base_iterator | +| db_set_iterator | db_set_iterator | + +#### Group + +Dbstl Iterator Classes diff --git a/docs-src/api/stl/dbstl_containers.md b/docs-src/api/stl/dbstl_containers.md new file mode 100644 index 000000000..dd3cfc8de --- /dev/null +++ b/docs-src/api/stl/dbstl_containers.md @@ -0,0 +1,29 @@ +--- +title: "Chapter 2.  Dbstl Container Classes" +api-name: "Chapter 2.  Dbstl Container Classes" +source: docs/api_reference/STL/dbstl_containers.html +--- +## Chapter 2.  Dbstl Container Classes + +A dbstl container is very much like a C++ STL container. + +It stores a collection of data items, or key/data pairs. Each container is backed by a Berkeley DB database created in an explicit database environment or an internal private environment; And the database itself can be created explicitly with all kinds of configurations, or by dbstl internally. For each type of container, some specific type of database and/or configurations must be used or specified to the database and its environment. dbstl will check the database and environment conform to the requirement. When users don't have a chance to specify a container's backing database and environment, like in copy constructors, dbstl will create proper databases and/or environment for it. There are two helper functions to make it easier to create/open an environment or database, they are dbstl::open_db() and dbstl::open_env() ; + +#### See Also + +dbstl::open_db() dbstl::open_env() db_vector db_map db_multimap db_set db_multiset + +#### Public Members + +| Member | Description | +|----|----| +| db_container | db_container | +| db_map | db_map | +| db_multimap | db_multimap | +| db_set | db_set | +| db_multiset | db_multiset | +| db_vector | db_vector | + +#### Group + +None diff --git a/docs-src/api/stl/dbstl_global_functions.md b/docs-src/api/stl/dbstl_global_functions.md new file mode 100644 index 000000000..56860bf23 --- /dev/null +++ b/docs-src/api/stl/dbstl_global_functions.md @@ -0,0 +1,67 @@ +--- +title: "Chapter 1.  Dbstl Global Public Functions" +api-name: "Chapter 1.  Dbstl Global Public Functions" +source: docs/api_reference/STL/dbstl_global_functions.html +--- +## Chapter 1.  Dbstl Global Public Functions + +#### Public Members + +| Member | Description | +|----|----| +| close_db | Close pdb regardless of reference count. | +| close_all_dbs | Close all open database handles regardless of reference count. | +| close_db_env | Close specified database environment handle regardless of reference count. | +| close_all_db_envs | Close all open database environment handles regardless of reference count. | +| begin_txn | Begin a new transaction from the specified environment "env". | +| commit_txn | Commit current transaction opened in the environment "env". | +| abort_txn | Abort current transaction of environment "env". | +| current_txn | Get current transaction of environment "env". | +| set_current_txn_handle | Set environment env's current transaction handle to be newtxn. | +| register_db | Register a Db handle "pdb1". | +| register_db_env | Register a DbEnv handle env1, this handle and handles opened in it will be closed by ResourceManager . | +| open_db | Helper function to open a database and register it into dbstl for the calling thread. | +| open_env | Helper function to open an environment and register it into dbstl for the calling thread. | +| alloc_mutex | Allocate a Berkeley DB mutex. | +| lock_mutex | Lock a mutex, wait if it is held by another thread. | +| unlock_mutex | Unlock a mutex, and return immediately. | +| free_mutex | Free a mutex, and return immediately. | +| dbstl_startup | If there are multiple threads within a process that make use of dbstl, then this function should be called in a single thread mutual exclusively before any use of dbstl in a process; Otherwise, you don't need to call it, but are allowed to call it anyway. | +| dbstl_exit | This function releases any memory allocated in the heap by code of dbstl. | +| dbstl_thread_exit | This function closes all Berkeley DB handles in the right order, if other threads do not use them. | +| operator== | Operators to compare two Dbt objects. | +| set_global_dbfile_suffix_number | If exisiting random temporary database name generation mechanism is still causing name clashes, users can set this global suffix number which will be append to each temporary database file name and incremented after each append, and by default it is 0. | +| close_db_cursors | Close cursors opened in dbp1. | + +#### Group + +None + +## close_db + +### Function Details + +``` c + void close_db(Db *pdb) + +``` + +Close pdb regardless of reference count. + +You must make sure pdb is not used by others before calling this method. You can close the underlying database of a container and assign another database with right configurations to it, if the configuration is not suitable for the container, there will be an InvalidArgumentException type of exception thrown. You can't use the container after you called close_db and before setting another valid database handle to the container via db_container::set_db_handle() function. + +#### Parameters + +##### pdb + +The database handle to close. + +### Group: Functions to close database/environments. + +Normally you don't have to close any database or environment handles, they will be closed automatically. + +Though you still have the following API to close them. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/dbstl_helper_classes.md b/docs-src/api/stl/dbstl_helper_classes.md new file mode 100644 index 000000000..03809abec --- /dev/null +++ b/docs-src/api/stl/dbstl_helper_classes.md @@ -0,0 +1,22 @@ +--- +title: "Chapter 21.  Dbstl Helper Classes" +api-name: "Chapter 21.  Dbstl Helper Classes" +source: docs/api_reference/STL/dbstl_helper_classes.html +--- +## Chapter 21.  Dbstl Helper Classes + +Classes of this module help to achieve various features of dbstl. + +#### Public Members + +| Member | Description | +|----|----| +| BulkRetrievalOption | BulkRetrievalOption | +| ReadModifyWriteOption | ReadModifyWriteOption | +| DbstlElemTraits | DbstlElemTraits | +| DbstlDbt | DbstlDbt | +| ElementRef and ElementHolder wrappers. | ElementRef and ElementHolder wrappers. | + +#### Group + +None diff --git a/docs-src/api/stl/dbstl_iterators.md b/docs-src/api/stl/dbstl_iterators.md new file mode 100644 index 000000000..a84817366 --- /dev/null +++ b/docs-src/api/stl/dbstl_iterators.md @@ -0,0 +1,33 @@ +--- +title: "Chapter 9.  Dbstl Iterator Classes" +api-name: "Chapter 9.  Dbstl Iterator Classes" +source: docs/api_reference/STL/dbstl_iterators.html +--- +## Chapter 9.  Dbstl Iterator Classes + +Common information for all dbstl iterators:. + +1\. Each instance of a dbstl iterator uniquely owns a Berkeley DB cursor, so that the key/data pair it currently sits on is always valid before it moves elsewhere. It also caches the current key/data pair values in order for member functions like operator\* /operator-\> to work properly, but caching is not compatible with standard C++ Stl behavior --- the C++ standard requires the iterator refer to a shared piece of memory where the data is stored, thus two iterators of the same container sitting on the same element should point to the same memory location, which is false for dbstl iterators. + +2\. There are some functions common to each child class of this class which have identical behaviors, so we will document them here. + +This class is the base class for all dbstl iterators, there is no much to say about this class itself, and users are not supposed to directly use this class at all. So we will talk about some common functions of dbstl iterators in this section. + +#### See Also + +db_vector_base_iterator db_vector_iterator db_map_base_iterator db_map_iterator db_set_base_iterator db_set_iterator + +#### Public Members + +| Member | Description | +|----|----| +| db_base_iterator | db_base_iterator | +| db_reverse_iterator | db_reverse_iterator | +| db_map_iterator | db_map_iterator | +| Iterator classes for db_map and db_multimap. | Iterator classes for db_map and db_multimap. | +| Iterator classes for db_set and db_multiset. | Iterator classes for db_set and db_multiset. | +| Iterator classes for db_vector. | Iterator classes for db_vector. | + +#### Group + +None diff --git a/docs-src/api/stl/frame_index.md b/docs-src/api/stl/frame_index.md new file mode 100644 index 000000000..f94370cb9 --- /dev/null +++ b/docs-src/api/stl/frame_index.md @@ -0,0 +1,92 @@ +--- +title: "Berkeley DB C++ Standard Template Library API Reference" +api-name: "Berkeley DB C++ Standard Template Library API Reference" +source: docs/api_reference/STL/frame_index.html +--- + Home + + + + Legal Notice + + Reference Guide + + + + DB STL Global Functions + + DB STL Containers + + db_container + + db_vector + + db_map + + db_multimap + + db_set + + db_multiset + + DB STL Iterators + + db_base_iterator + + DB STL Vector Iterators + + db_vector_base_iterator + + db_vector_iterator + + DB STL Map Iterators + + db_map_base_iterator + + db_map_iterator + + DB STL Set Iterators + + db_set_base_iterator + + db_set_iterator + + db_reverse_iterator + + DB STL Helper Classes + + DB STL Element Wrappers + + ElementHolder + + ElementRef + + DbstlDbt + + DbstlElemTraits + + BulkRetrievalOption + + ReadModifyWriteOption + + DB STL Exception Classes + + DbstlException + + InvalidDbtException + + FailedAssertionException + + InvalidCursorException + + NoSuchKeyException + + NotEnoughMemoryException + + NotSupportedException + + InvalidIteratorException + + InvalidFunctionCall + + InvalidArgumentException diff --git a/docs-src/api/stl/frame_main.md b/docs-src/api/stl/frame_main.md new file mode 100644 index 000000000..a066aff42 --- /dev/null +++ b/docs-src/api/stl/frame_main.md @@ -0,0 +1,6 @@ +--- +title: "Berkeley DB C++ Standard Template Library API Reference" +api-name: "Berkeley DB C++ Standard Template Library API Reference" +source: docs/api_reference/STL/frame_main.html +--- + diff --git a/docs-src/api/stl/index.md b/docs-src/api/stl/index.md new file mode 100644 index 000000000..f2ac7f3c4 --- /dev/null +++ b/docs-src/api/stl/index.md @@ -0,0 +1,20 @@ +--- +title: "Berkeley DB C++ Standard Template Library API Reference" +api-name: "Berkeley DB C++ Standard Template Library API Reference" +source: docs/api_reference/STL/index.html +--- +# Berkeley DB C++ Standard Template Library API Reference + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ diff --git a/docs-src/api/stl/moreinfo.md b/docs-src/api/stl/moreinfo.md new file mode 100644 index 000000000..9df591100 --- /dev/null +++ b/docs-src/api/stl/moreinfo.md @@ -0,0 +1,36 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/api_reference/STL/moreinfo.html +--- +## For More Information + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Getting Started with Berkeley DB for C++ + +- Getting Started with Transaction Processing for C++ + +- Berkeley DB Getting Started with Replicated Applications for C++ + +- Berkeley DB C API Reference Guide + +- Berkeley DB C++ API Reference Guide + +- Berkeley DB TCL API Reference Guide + +- Berkeley DB Installation and Build Guide + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Getting Started with the SQL APIs + +To download the latest documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/api/stl/preface.md b/docs-src/api/stl/preface.md new file mode 100644 index 000000000..e403b9f00 --- /dev/null +++ b/docs-src/api/stl/preface.md @@ -0,0 +1,33 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/api_reference/STL/preface.html +--- +## Preface + +Welcome to Berkeley DB 11*g* Release 2 (DB). This document describes the C++ STL API for DB library version 11.2.5.3. It is intended to describe the DB API, including all classes, methods, and functions. As such, this document is intended for C++ developers who are actively writing or maintaining applications that make use of DB databases. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Class names are represented in `monospaced font`, as are `method names`. For example: "`Db::open()` is a `Db` class method." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +typedef struct vendor { + char name[MAXFIELD]; // Vendor name + char street[MAXFIELD]; // Street name and number + char city[MAXFIELD]; // City + char state[3]; // Two-digit US state code + char zipcode[6]; // US zipcode + char phone_number[13]; // Vendor phone number +} VENDOR; +``` + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/api/stl/stlBulkRetrievalOptionbulk_buf_size.md b/docs-src/api/stl/stlBulkRetrievalOptionbulk_buf_size.md new file mode 100644 index 000000000..456031a07 --- /dev/null +++ b/docs-src/api/stl/stlBulkRetrievalOptionbulk_buf_size.md @@ -0,0 +1,19 @@ +--- +title: "bulk_buf_size" +api-name: "bulk_buf_size" +source: docs/api_reference/STL/stlBulkRetrievalOptionbulk_buf_size.html +--- +## bulk_buf_size + +### Function Details + +``` c +u_int32_t bulk_buf_size() + +``` + +Return the buffer size set to this object. + +### Class + +BulkRetrievalOption diff --git a/docs-src/api/stl/stlBulkRetrievalOptionbulk_retrieval.md b/docs-src/api/stl/stlBulkRetrievalOptionbulk_retrieval.md new file mode 100644 index 000000000..3da03d6cf --- /dev/null +++ b/docs-src/api/stl/stlBulkRetrievalOptionbulk_retrieval.md @@ -0,0 +1,20 @@ +--- +title: "bulk_retrieval" +api-name: "bulk_retrieval" +source: docs/api_reference/STL/stlBulkRetrievalOptionbulk_retrieval.html +--- +## bulk_retrieval + +### Function Details + +``` c +static BulkRetrievalOption bulk_retrieval(u_int32_t bulk_buf_sz= + DBSTL_BULK_BUF_SIZE) + +``` + +This function indicates that you need a bulk retrieval iterator, and it can be also used to optionally set the bulk read buffer size. + +### Class + +BulkRetrievalOption diff --git a/docs-src/api/stl/stlBulkRetrievalOptionno_bulk_retrieval.md b/docs-src/api/stl/stlBulkRetrievalOptionno_bulk_retrieval.md new file mode 100644 index 000000000..207c85c71 --- /dev/null +++ b/docs-src/api/stl/stlBulkRetrievalOptionno_bulk_retrieval.md @@ -0,0 +1,19 @@ +--- +title: "no_bulk_retrieval" +api-name: "no_bulk_retrieval" +source: docs/api_reference/STL/stlBulkRetrievalOptionno_bulk_retrieval.html +--- +## no_bulk_retrieval + +### Function Details + +``` c +static BulkRetrievalOption no_bulk_retrieval() + +``` + +This function indicates that you do not need a bulk retrieval iterator. + +### Class + +BulkRetrievalOption diff --git a/docs-src/api/stl/stlBulkRetrievalOptionoperator_assign.md b/docs-src/api/stl/stlBulkRetrievalOptionoperator_assign.md new file mode 100644 index 000000000..ab08d5ffc --- /dev/null +++ b/docs-src/api/stl/stlBulkRetrievalOptionoperator_assign.md @@ -0,0 +1,19 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stlBulkRetrievalOptionoperator_assign.html +--- +## operator= + +### Function Details + +``` c +void operator=(BulkRetrievalOption::Option opt) + +``` + +Assignment operator. + +### Class + +BulkRetrievalOption diff --git a/docs-src/api/stl/stlBulkRetrievalOptionoperator_eq.md b/docs-src/api/stl/stlBulkRetrievalOptionoperator_eq.md new file mode 100644 index 000000000..b6b782f42 --- /dev/null +++ b/docs-src/api/stl/stlBulkRetrievalOptionoperator_eq.md @@ -0,0 +1,19 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stlBulkRetrievalOptionoperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const BulkRetrievalOption &bro) const + +``` + +Equality comparison. + +### Class + +BulkRetrievalOption diff --git a/docs-src/api/stl/stlDbstlDbtdstr_DbstlDbt.md b/docs-src/api/stl/stlDbstlDbtdstr_DbstlDbt.md new file mode 100644 index 000000000..e4714e4ce --- /dev/null +++ b/docs-src/api/stl/stlDbstlDbtdstr_DbstlDbt.md @@ -0,0 +1,19 @@ +--- +title: "~DbstlDbt" +api-name: "~DbstlDbt" +source: docs/api_reference/STL/stlDbstlDbtdstr_DbstlDbt.html +--- +## ~DbstlDbt + +### Function Details + +``` c +~DbstlDbt() + +``` + +The memory will be free'ed by the destructor. + +### Class + +DbstlDbt diff --git a/docs-src/api/stl/stlDbstlDbtoperator_assign.md b/docs-src/api/stl/stlDbstlDbtoperator_assign.md new file mode 100644 index 000000000..bc94f2cf7 --- /dev/null +++ b/docs-src/api/stl/stlDbstlDbtoperator_assign.md @@ -0,0 +1,19 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stlDbstlDbtoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const DbstlDbt& operator=(const DbstlDbt &d) + +``` + +The memory will be reallocated if neccessary. + +### Class + +DbstlDbt diff --git a/docs-src/api/stl/stlDbstlElemTraitsDbstlElemTraits.md b/docs-src/api/stl/stlDbstlElemTraitsDbstlElemTraits.md new file mode 100644 index 000000000..e0b99b34a --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsDbstlElemTraits.md @@ -0,0 +1,17 @@ +--- +title: "DbstlElemTraits" +api-name: "DbstlElemTraits" +source: docs/api_reference/STL/stlDbstlElemTraitsDbstlElemTraits.html +--- +## DbstlElemTraits + +### Function Details + +``` c +DbstlElemTraits() + +``` + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitscompare.md b/docs-src/api/stl/stlDbstlElemTraitscompare.md new file mode 100644 index 000000000..197fbc952 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitscompare.md @@ -0,0 +1,26 @@ +--- +title: "compare" +api-name: "compare" +source: docs/api_reference/STL/stlDbstlElemTraitscompare.html +--- +## compare + +### Function Details + +``` c +static int compare(const T *seq1, const T *seq2, + size_t cnt) + +``` + +Sequence comparison. + +Compares the first cnt number of elements in the two sequences seq1 and seq2, returns negative/0/positive if seq1 is less/equal/greater than seq2. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitscopy.md b/docs-src/api/stl/stlDbstlElemTraitscopy.md new file mode 100644 index 000000000..fb72b62b2 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitscopy.md @@ -0,0 +1,24 @@ +--- +title: "copy" +api-name: "copy" +source: docs/api_reference/STL/stlDbstlElemTraitscopy.html +--- +## copy + +### Function Details + +``` c +static T* copy(T *seq1, const T *seq2, + size_t cnt) + +``` + +Copy first cnt number of elements from seq2 to seq1. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsdstr_DbstlElemTraits.md b/docs-src/api/stl/stlDbstlElemTraitsdstr_DbstlElemTraits.md new file mode 100644 index 000000000..7d648407b --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsdstr_DbstlElemTraits.md @@ -0,0 +1,17 @@ +--- +title: "~DbstlElemTraits" +api-name: "~DbstlElemTraits" +source: docs/api_reference/STL/stlDbstlElemTraitsdstr_DbstlElemTraits.html +--- +## ~DbstlElemTraits + +### Function Details + +``` c +~DbstlElemTraits() + +``` + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitseof.md b/docs-src/api/stl/stlDbstlElemTraitseof.md new file mode 100644 index 000000000..fa768a50e --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitseof.md @@ -0,0 +1,21 @@ +--- +title: "eof" +api-name: "eof" +source: docs/api_reference/STL/stlDbstlElemTraitseof.html +--- +## eof + +### Function Details + +``` c +static int_type eof() + +``` + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitseq.md b/docs-src/api/stl/stlDbstlElemTraitseq.md new file mode 100644 index 000000000..904b48acc --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitseq.md @@ -0,0 +1,24 @@ +--- +title: "eq" +api-name: "eq" +source: docs/api_reference/STL/stlDbstlElemTraitseq.html +--- +## eq + +### Function Details + +``` c +static bool eq(const T &left, + const T &right) + +``` + +Check for equality of two objects. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitseq_int_type.md b/docs-src/api/stl/stlDbstlElemTraitseq_int_type.md new file mode 100644 index 000000000..de7b8eb39 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitseq_int_type.md @@ -0,0 +1,22 @@ +--- +title: "eq_int_type" +api-name: "eq_int_type" +source: docs/api_reference/STL/stlDbstlElemTraitseq_int_type.html +--- +## eq_int_type + +### Function Details + +``` c +static bool eq_int_type(const int_type &left, + const int_type &right) + +``` + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsfind.md b/docs-src/api/stl/stlDbstlElemTraitsfind.md new file mode 100644 index 000000000..082e99531 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsfind.md @@ -0,0 +1,24 @@ +--- +title: "find" +api-name: "find" +source: docs/api_reference/STL/stlDbstlElemTraitsfind.html +--- +## find + +### Function Details + +``` c +static const T* find(const T *seq, size_t cnt, + const T &elem) + +``` + +Find within the first cnt elements of sequence seq the position of element equal to elem. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_assign_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_assign_function.md new file mode 100644 index 000000000..e3b9c94a4 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_assign_function.md @@ -0,0 +1,21 @@ +--- +title: "get_assign_function" +api-name: "get_assign_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_assign_function.html +--- +## get_assign_function + +### Function Details + +``` c +ElemAssignFunct get_assign_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_compare_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_compare_function.md new file mode 100644 index 000000000..6ca3a6c32 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_compare_function.md @@ -0,0 +1,21 @@ +--- +title: "get_compare_function" +api-name: "get_compare_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_compare_function.html +--- +## get_compare_function + +### Function Details + +``` c +ElemCompareFunct get_compare_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_copy_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_copy_function.md new file mode 100644 index 000000000..f0ee76888 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_copy_function.md @@ -0,0 +1,21 @@ +--- +title: "get_copy_function" +api-name: "get_copy_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_copy_function.html +--- +## get_copy_function + +### Function Details + +``` c +ElemCopyFunct get_copy_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_restore_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_restore_function.md new file mode 100644 index 000000000..9deaa2298 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_restore_function.md @@ -0,0 +1,21 @@ +--- +title: "get_restore_function" +api-name: "get_restore_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_restore_function.html +--- +## get_restore_function + +### Function Details + +``` c +ElemRstoreFunct get_restore_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_sequence_compare_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_compare_function.md new file mode 100644 index 000000000..113f84274 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_compare_function.md @@ -0,0 +1,21 @@ +--- +title: "get_sequence_compare_function" +api-name: "get_sequence_compare_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_sequence_compare_function.html +--- +## get_sequence_compare_function + +### Function Details + +``` c +SequenceCompareFunct get_sequence_compare_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_sequence_copy_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_copy_function.md new file mode 100644 index 000000000..5a9ca4fb4 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_copy_function.md @@ -0,0 +1,21 @@ +--- +title: "get_sequence_copy_function" +api-name: "get_sequence_copy_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_sequence_copy_function.html +--- +## get_sequence_copy_function + +### Function Details + +``` c +SequenceCopyFunct get_sequence_copy_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_sequence_len_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_len_function.md new file mode 100644 index 000000000..e4e19c0b3 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_len_function.md @@ -0,0 +1,21 @@ +--- +title: "get_sequence_len_function" +api-name: "get_sequence_len_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_sequence_len_function.html +--- +## get_sequence_len_function + +### Function Details + +``` c +SequenceLenFunct get_sequence_len_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_sequence_n_compare_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_n_compare_function.md new file mode 100644 index 000000000..985651204 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_sequence_n_compare_function.md @@ -0,0 +1,21 @@ +--- +title: "get_sequence_n_compare_function" +api-name: "get_sequence_n_compare_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_sequence_n_compare_function.html +--- +## get_sequence_n_compare_function + +### Function Details + +``` c +SequenceNCompareFunct get_sequence_n_compare_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsget_size_function.md b/docs-src/api/stl/stlDbstlElemTraitsget_size_function.md new file mode 100644 index 000000000..eecea6def --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsget_size_function.md @@ -0,0 +1,21 @@ +--- +title: "get_size_function" +api-name: "get_size_function" +source: docs/api_reference/STL/stlDbstlElemTraitsget_size_function.html +--- +## get_size_function + +### Function Details + +``` c +ElemSizeFunct get_size_function() + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsinstance.md b/docs-src/api/stl/stlDbstlElemTraitsinstance.md new file mode 100644 index 000000000..954661dc8 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsinstance.md @@ -0,0 +1,21 @@ +--- +title: "instance" +api-name: "instance" +source: docs/api_reference/STL/stlDbstlElemTraitsinstance.html +--- +## instance + +### Function Details + +``` c +static DbstlElemTraits* instance() + +``` + +Factory method to create a singeleton instance of this class. + +The created object will be deleted by dbstl upon process exit. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitslength.md b/docs-src/api/stl/stlDbstlElemTraitslength.md new file mode 100644 index 000000000..4dafc9563 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitslength.md @@ -0,0 +1,25 @@ +--- +title: "length" +api-name: "length" +source: docs/api_reference/STL/stlDbstlElemTraitslength.html +--- +## length + +### Function Details + +``` c +static size_t length(const T *seq) + +``` + +Returns the number of elements in sequence seq1. + +Note that seq1 may or may not end with a trailing '', it is completely user's responsibility for this decision, though seq\[0\], seq\[1\],... seq\[length - 1\] are all sequence seq's memory. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitslt.md b/docs-src/api/stl/stlDbstlElemTraitslt.md new file mode 100644 index 000000000..8be169a41 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitslt.md @@ -0,0 +1,26 @@ +--- +title: "lt" +api-name: "lt" +source: docs/api_reference/STL/stlDbstlElemTraitslt.html +--- +## lt + +### Function Details + +``` c +static bool lt(const T &left, + const T &right) + +``` + +Less than comparison. + +Returns if object left is less than object right. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsmove.md b/docs-src/api/stl/stlDbstlElemTraitsmove.md new file mode 100644 index 000000000..de04588f2 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsmove.md @@ -0,0 +1,26 @@ +--- +title: "move" +api-name: "move" +source: docs/api_reference/STL/stlDbstlElemTraitsmove.html +--- +## move + +### Function Details + +``` c +static T* move(T *seq1, const T *seq2, + size_t cnt) + +``` + +Sequence movement. + +Move first cnt number of elements from seq2 to seq1, seq1 and seq2 may or may not overlap. + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsnot_eof.md b/docs-src/api/stl/stlDbstlElemTraitsnot_eof.md new file mode 100644 index 000000000..97a5e6647 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsnot_eof.md @@ -0,0 +1,21 @@ +--- +title: "not_eof" +api-name: "not_eof" +source: docs/api_reference/STL/stlDbstlElemTraitsnot_eof.html +--- +## not_eof + +### Function Details + +``` c +static int_type not_eof(const int_type &meta_elem) + +``` + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_assign_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_assign_function.md new file mode 100644 index 000000000..2b586285e --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_assign_function.md @@ -0,0 +1,21 @@ +--- +title: "set_assign_function" +api-name: "set_assign_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_assign_function.html +--- +## set_assign_function + +### Function Details + +``` c +void set_assign_function(ElemAssignFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_compare_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_compare_function.md new file mode 100644 index 000000000..078e2feff --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_compare_function.md @@ -0,0 +1,21 @@ +--- +title: "set_compare_function" +api-name: "set_compare_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_compare_function.html +--- +## set_compare_function + +### Function Details + +``` c +void set_compare_function(ElemCompareFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_copy_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_copy_function.md new file mode 100644 index 000000000..35552ef52 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_copy_function.md @@ -0,0 +1,21 @@ +--- +title: "set_copy_function" +api-name: "set_copy_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_copy_function.html +--- +## set_copy_function + +### Function Details + +``` c +void set_copy_function(ElemCopyFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_restore_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_restore_function.md new file mode 100644 index 000000000..e5b1c8e13 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_restore_function.md @@ -0,0 +1,21 @@ +--- +title: "set_restore_function" +api-name: "set_restore_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_restore_function.html +--- +## set_restore_function + +### Function Details + +``` c +void set_restore_function(ElemRstoreFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_sequence_compare_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_compare_function.md new file mode 100644 index 000000000..fe5dcfc8e --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_compare_function.md @@ -0,0 +1,21 @@ +--- +title: "set_sequence_compare_function" +api-name: "set_sequence_compare_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_sequence_compare_function.html +--- +## set_sequence_compare_function + +### Function Details + +``` c +void set_sequence_compare_function(SequenceCompareFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_sequence_copy_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_copy_function.md new file mode 100644 index 000000000..3e4fd5ab7 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_copy_function.md @@ -0,0 +1,21 @@ +--- +title: "set_sequence_copy_function" +api-name: "set_sequence_copy_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_sequence_copy_function.html +--- +## set_sequence_copy_function + +### Function Details + +``` c +void set_sequence_copy_function(SequenceCopyFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_sequence_len_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_len_function.md new file mode 100644 index 000000000..98f9e8f75 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_len_function.md @@ -0,0 +1,21 @@ +--- +title: "set_sequence_len_function" +api-name: "set_sequence_len_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_sequence_len_function.html +--- +## set_sequence_len_function + +### Function Details + +``` c +void set_sequence_len_function(SequenceLenFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_sequence_n_compare_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_n_compare_function.md new file mode 100644 index 000000000..7ff3e3646 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_sequence_n_compare_function.md @@ -0,0 +1,21 @@ +--- +title: "set_sequence_n_compare_function" +api-name: "set_sequence_n_compare_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_sequence_n_compare_function.html +--- +## set_sequence_n_compare_function + +### Function Details + +``` c +void set_sequence_n_compare_function(SequenceNCompareFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsset_size_function.md b/docs-src/api/stl/stlDbstlElemTraitsset_size_function.md new file mode 100644 index 000000000..2ac340157 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsset_size_function.md @@ -0,0 +1,21 @@ +--- +title: "set_size_function" +api-name: "set_size_function" +source: docs/api_reference/STL/stlDbstlElemTraitsset_size_function.html +--- +## set_size_function + +### Function Details + +``` c +void set_size_function(ElemSizeFunct f) + +``` + +### Group: Set/get functions for callback function pointers. + +These are the setters and getters for each callback function pointers. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsto_char_type.md b/docs-src/api/stl/stlDbstlElemTraitsto_char_type.md new file mode 100644 index 000000000..658204964 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsto_char_type.md @@ -0,0 +1,21 @@ +--- +title: "to_char_type" +api-name: "to_char_type" +source: docs/api_reference/STL/stlDbstlElemTraitsto_char_type.html +--- +## to_char_type + +### Function Details + +``` c +static T to_char_type(const int_type &meta_elem) + +``` + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlElemTraitsto_int_type.md b/docs-src/api/stl/stlDbstlElemTraitsto_int_type.md new file mode 100644 index 000000000..14f1df425 --- /dev/null +++ b/docs-src/api/stl/stlDbstlElemTraitsto_int_type.md @@ -0,0 +1,21 @@ +--- +title: "to_int_type" +api-name: "to_int_type" +source: docs/api_reference/STL/stlDbstlElemTraitsto_int_type.html +--- +## to_int_type + +### Function Details + +``` c +static int_type to_int_type(const T &elem) + +``` + +### Group: Interface compatible with std::string's char_traits. + +Following are char_traits funcitons, which make this class char_traits compatiable, so that it can be used in std::basic_string template, and be manipulated by the c++ stl algorithms. + +### Class + +DbstlElemTraits diff --git a/docs-src/api/stl/stlDbstlExceptiondstr_DbstlException.md b/docs-src/api/stl/stlDbstlExceptiondstr_DbstlException.md new file mode 100644 index 000000000..d29594cb3 --- /dev/null +++ b/docs-src/api/stl/stlDbstlExceptiondstr_DbstlException.md @@ -0,0 +1,17 @@ +--- +title: "~DbstlException" +api-name: "~DbstlException" +source: docs/api_reference/STL/stlDbstlExceptiondstr_DbstlException.html +--- +## ~DbstlException + +### Function Details + +``` c +virtual ~DbstlException() + +``` + +### Class + +DbstlException diff --git a/docs-src/api/stl/stlDbstlExceptionoperator_assign.md b/docs-src/api/stl/stlDbstlExceptionoperator_assign.md new file mode 100644 index 000000000..dc9fe91a0 --- /dev/null +++ b/docs-src/api/stl/stlDbstlExceptionoperator_assign.md @@ -0,0 +1,17 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stlDbstlExceptionoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const DbstlException& operator=(const DbstlException &exobj) + +``` + +### Class + +DbstlException diff --git a/docs-src/api/stl/stlElementHolder_DB_STL_StoreElement.md b/docs-src/api/stl/stlElementHolder_DB_STL_StoreElement.md new file mode 100644 index 000000000..259a2c812 --- /dev/null +++ b/docs-src/api/stl/stlElementHolder_DB_STL_StoreElement.md @@ -0,0 +1,25 @@ +--- +title: "_DB_STL_StoreElement" +api-name: "_DB_STL_StoreElement" +source: docs/api_reference/STL/stlElementHolder_DB_STL_StoreElement.html +--- +## \_DB_STL_StoreElement + +### Function Details + +``` c +void _DB_STL_StoreElement() + +``` + +Function to store the data element. + +The user needs to call this method after modifying the underlying object, so that the version stored in the container can be updated. + +When db_base_iterator's directdb_get\_ member is true, this function must be called after modifying the data member and before any subsequent container iterator dereference operations. If this step is not carried out any changes will be lost. + +If the data element is changed via ElementHolder\<\>::operator=(), you don't need to call this function. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolder_DB_STL_value.md b/docs-src/api/stl/stlElementHolder_DB_STL_value.md new file mode 100644 index 000000000..43a8a4ea8 --- /dev/null +++ b/docs-src/api/stl/stlElementHolder_DB_STL_value.md @@ -0,0 +1,26 @@ +--- +title: "_DB_STL_value" +api-name: "_DB_STL_value" +source: docs/api_reference/STL/stlElementHolder_DB_STL_value.html +--- +## \_DB_STL_value + +### Function Details + +``` c +const ptype& _DB_STL_value() const + +``` + +Returns the data element this wrapper object wraps;. + +``` c +ptype& _DB_STL_value() + +``` + +Returns the data element this wrapper object wraps;. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderdstr_ElementHolder.md b/docs-src/api/stl/stlElementHolderdstr_ElementHolder.md new file mode 100644 index 000000000..92cc6f327 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderdstr_ElementHolder.md @@ -0,0 +1,19 @@ +--- +title: "~ElementHolder" +api-name: "~ElementHolder" +source: docs/api_reference/STL/stlElementHolderdstr_ElementHolder.html +--- +## ~ElementHolder + +### Function Details + +``` c +~ElementHolder() + +``` + +Destructor. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator__aa.md b/docs-src/api/stl/stlElementHolderoperator__aa.md new file mode 100644 index 000000000..dd16d7c57 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator__aa.md @@ -0,0 +1,28 @@ +--- +title: "operator &=" +api-name: "operator &=" +source: docs/api_reference/STL/stlElementHolderoperator__aa.html +--- +## operator &= + +### Function Details + +``` c +const self& operator &=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator &=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator__ma.md b/docs-src/api/stl/stlElementHolderoperator__ma.md new file mode 100644 index 000000000..2316a467c --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator__ma.md @@ -0,0 +1,28 @@ +--- +title: "operator *=" +api-name: "operator *=" +source: docs/api_reference/STL/stlElementHolderoperator__ma.html +--- +## operator \*= + +### Function Details + +``` c +const self& operator *=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator *=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_assign.md b/docs-src/api/stl/stlElementHolderoperator_assign.md new file mode 100644 index 000000000..ccdd2ddd8 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_assign.md @@ -0,0 +1,28 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stlElementHolderoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const ptype& operator=(const ptype &dt2) + +``` + +``` c +const self& operator=(const self &dt2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_da.md b/docs-src/api/stl/stlElementHolderoperator_da.md new file mode 100644 index 000000000..b6be4781b --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_da.md @@ -0,0 +1,28 @@ +--- +title: "operator/=" +api-name: "operator/=" +source: docs/api_reference/STL/stlElementHolderoperator_da.html +--- +## operator/= + +### Function Details + +``` c +const self& operator/=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator/=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_decr.md b/docs-src/api/stl/stlElementHolderoperator_decr.md new file mode 100644 index 000000000..43fc54a6a --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_decr.md @@ -0,0 +1,28 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stlElementHolderoperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +``` c +self operator--(int) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_gt_ge.md b/docs-src/api/stl/stlElementHolderoperator_gt_ge.md new file mode 100644 index 000000000..dc478ecf1 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_gt_ge.md @@ -0,0 +1,23 @@ +--- +title: "operator>>=" +api-name: "operator>>=" +source: docs/api_reference/STL/stlElementHolderoperator_gt_ge.html +--- +## operator\>\>= + +### Function Details + +``` c +const self& operator>>=(size_t n) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_ia.md b/docs-src/api/stl/stlElementHolderoperator_ia.md new file mode 100644 index 000000000..f2058f5fb --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_ia.md @@ -0,0 +1,28 @@ +--- +title: "operator+=" +api-name: "operator+=" +source: docs/api_reference/STL/stlElementHolderoperator_ia.html +--- +## operator+= + +### Function Details + +``` c +const self& operator+=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator+=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_incr.md b/docs-src/api/stl/stlElementHolderoperator_incr.md new file mode 100644 index 000000000..75d6431a8 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_incr.md @@ -0,0 +1,28 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stlElementHolderoperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +``` c +self operator++(int) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_lt_le.md b/docs-src/api/stl/stlElementHolderoperator_lt_le.md new file mode 100644 index 000000000..a5f561367 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_lt_le.md @@ -0,0 +1,23 @@ +--- +title: "operator<<=" +api-name: "operator<<=" +source: docs/api_reference/STL/stlElementHolderoperator_lt_le.html +--- +## operator\<\<= + +### Function Details + +``` c +const self& operator<<=(size_t n) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_modasg.md b/docs-src/api/stl/stlElementHolderoperator_modasg.md new file mode 100644 index 000000000..a4b20de29 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_modasg.md @@ -0,0 +1,28 @@ +--- +title: "operator%=" +api-name: "operator%=" +source: docs/api_reference/STL/stlElementHolderoperator_modasg.html +--- +## operator%= + +### Function Details + +``` c +const self& operator%=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator%=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_oa.md b/docs-src/api/stl/stlElementHolderoperator_oa.md new file mode 100644 index 000000000..db79260c8 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_oa.md @@ -0,0 +1,28 @@ +--- +title: "operator|=" +api-name: "operator|=" +source: docs/api_reference/STL/stlElementHolderoperator_oa.html +--- +## operator\|= + +### Function Details + +``` c +const self& operator|=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator|=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_ptype.md b/docs-src/api/stl/stlElementHolderoperator_ptype.md new file mode 100644 index 000000000..afb157ee5 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_ptype.md @@ -0,0 +1,21 @@ +--- +title: "operator ptype" +api-name: "operator ptype" +source: docs/api_reference/STL/stlElementHolderoperator_ptype.html +--- +## operator ptype + +### Function Details + +``` c +operator ptype() const + +``` + +This operator is a type converter. + +Where an automatic type conversion is needed, this function is called to convert this object into the primitive type it wraps. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_sa.md b/docs-src/api/stl/stlElementHolderoperator_sa.md new file mode 100644 index 000000000..94d5a4142 --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_sa.md @@ -0,0 +1,28 @@ +--- +title: "operator-=" +api-name: "operator-=" +source: docs/api_reference/STL/stlElementHolderoperator_sa.html +--- +## operator-= + +### Function Details + +``` c +const self& operator-=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator-=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementHolderoperator_xa.md b/docs-src/api/stl/stlElementHolderoperator_xa.md new file mode 100644 index 000000000..09c6be47d --- /dev/null +++ b/docs-src/api/stl/stlElementHolderoperator_xa.md @@ -0,0 +1,28 @@ +--- +title: "operator^=" +api-name: "operator^=" +source: docs/api_reference/STL/stlElementHolderoperator_xa.html +--- +## operator^= + +### Function Details + +``` c +const self& operator^=(const ElementHolder< T2 > &p2) + +``` + +``` c +const self& operator^=(const self &p2) + +``` + +### Group: Math operators. + +ElementHolder class templates also have all C/C++ self mutating operators for numeric primitive types, including: +=, -=, \*=, /=, =, \<\<=, \>\>=, &=, \|=, ^=, ++, -- These operators should not be used when ddt is a sequence pointer type like char\* or wchar_t\* or T\*, otherwise the behavior is undefined. + +These methods exist only to override default bahavior to store the new updated value, otherwise, the type convert operator could have done all the job. As you know, some of them are not applicable to float or double types or ElementHolder wrapper types for float/double types. These operators not only modifies the cached data element, but also stores new value to database if it associates a database key/data pair. + +### Class + +ElementHolder diff --git a/docs-src/api/stl/stlElementRefElementRef.md b/docs-src/api/stl/stlElementRefElementRef.md new file mode 100644 index 000000000..d8d586a4d --- /dev/null +++ b/docs-src/api/stl/stlElementRefElementRef.md @@ -0,0 +1,57 @@ +--- +title: "ElementRef" +api-name: "ElementRef" +source: docs/api_reference/STL/stlElementRefElementRef.html +--- +## ElementRef + +### Function Details + +``` c +ElementRef(iterator_type *pitr=NULL) + +``` + +Constructor. + +If the pitr parameter is NULL or the default value is used, the object created is a simple wrapper and not connected to a container. If a valid iterator parameter is passed in, the wrapped element will be associated with the matching key/data pair in the underlying container. + +#### Parameters + +##### pitr + +The iterator owning this object. + +``` c +ElementRef(const ddt &dt) + +``` + +Constructor. + +Initializes an ElementRef wrapper without an iterator. It can only be used to wrap a data element in memory, it can't access an unerlying database. + +#### Parameters + +##### dt + +The base class object to initialize this object. + +``` c +ElementRef(const self &other) + +``` + +Copy constructor. + +The constructor takes a "deep" copy. The created object will be identical to, but independent from the original object. + +#### Parameters + +##### other + +The object to clone from. + +### Class + +ElementRef diff --git a/docs-src/api/stl/stlElementRef_DB_STL_StoreElement.md b/docs-src/api/stl/stlElementRef_DB_STL_StoreElement.md new file mode 100644 index 000000000..eb84ba029 --- /dev/null +++ b/docs-src/api/stl/stlElementRef_DB_STL_StoreElement.md @@ -0,0 +1,25 @@ +--- +title: "_DB_STL_StoreElement" +api-name: "_DB_STL_StoreElement" +source: docs/api_reference/STL/stlElementRef_DB_STL_StoreElement.html +--- +## \_DB_STL_StoreElement + +### Function Details + +``` c +void _DB_STL_StoreElement() + +``` + +Function to store the data element. + +The user needs to call this method after modifying the underlying object, so that the version stored in the container can be updated. + +When db_base_iterator's directdb_get\_ member is true, this function must be called after modifying the data member and before any subsequent container iterator dereference operations. If this step is not carried out any changes will be lost. + +If the data element is changed via ElementHolder\<\>::operator=(), you don't need to call this function. + +### Class + +ElementRef diff --git a/docs-src/api/stl/stlElementRef_DB_STL_value.md b/docs-src/api/stl/stlElementRef_DB_STL_value.md new file mode 100644 index 000000000..00837ba26 --- /dev/null +++ b/docs-src/api/stl/stlElementRef_DB_STL_value.md @@ -0,0 +1,26 @@ +--- +title: "_DB_STL_value" +api-name: "_DB_STL_value" +source: docs/api_reference/STL/stlElementRef_DB_STL_value.html +--- +## \_DB_STL_value + +### Function Details + +``` c +const ddt& _DB_STL_value() const + +``` + +Returns the data element this wrapper object wraps. + +``` c +ddt& _DB_STL_value() + +``` + +Returns the data element this wrapper object wraps. + +### Class + +ElementRef diff --git a/docs-src/api/stl/stlElementRefoperator_assign.md b/docs-src/api/stl/stlElementRefoperator_assign.md new file mode 100644 index 000000000..cfc06238e --- /dev/null +++ b/docs-src/api/stl/stlElementRefoperator_assign.md @@ -0,0 +1,50 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stlElementRefoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const ddt& operator=(const ddt &dt2) + +``` + +Assignment Operator. + +#### Parameters + +##### dt2 + +The data value to assign with. + +#### Return Value + +The object dt2's reference. + +``` c +const self& operator=(const self &me) + +``` + +Assignment Operator. + +#### Parameters + +##### me + +The object to assign with. + +#### Return Value + +The object me's reference. + +### Group: Assignment operators. + +The assignment operators are used to store right-values into the wrapped object, and also to store values into an underlying container. + +### Class + +ElementRef diff --git a/docs-src/api/stl/stlFailedAssertionExceptionFailedAssertionException.md b/docs-src/api/stl/stlFailedAssertionExceptionFailedAssertionException.md new file mode 100644 index 000000000..f42c7b3ac --- /dev/null +++ b/docs-src/api/stl/stlFailedAssertionExceptionFailedAssertionException.md @@ -0,0 +1,23 @@ +--- +title: "FailedAssertionException" +api-name: "FailedAssertionException" +source: docs/api_reference/STL/stlFailedAssertionExceptionFailedAssertionException.html +--- +## FailedAssertionException + +### Function Details + +``` c +FailedAssertionException(const char *fname, size_t lineno, + const char *msg) + +``` + +``` c +FailedAssertionException(const FailedAssertionException &ex) + +``` + +### Class + +FailedAssertionException diff --git a/docs-src/api/stl/stlFailedAssertionExceptiondstr_FailedAssertionException.md b/docs-src/api/stl/stlFailedAssertionExceptiondstr_FailedAssertionException.md new file mode 100644 index 000000000..6e284c24b --- /dev/null +++ b/docs-src/api/stl/stlFailedAssertionExceptiondstr_FailedAssertionException.md @@ -0,0 +1,17 @@ +--- +title: "~FailedAssertionException" +api-name: "~FailedAssertionException" +source: docs/api_reference/STL/stlFailedAssertionExceptiondstr_FailedAssertionException.html +--- +## ~FailedAssertionException + +### Function Details + +``` c +virtual ~FailedAssertionException() + +``` + +### Class + +FailedAssertionException diff --git a/docs-src/api/stl/stlReadModifyWriteOptionno_read_modify_write.md b/docs-src/api/stl/stlReadModifyWriteOptionno_read_modify_write.md new file mode 100644 index 000000000..75ac2746e --- /dev/null +++ b/docs-src/api/stl/stlReadModifyWriteOptionno_read_modify_write.md @@ -0,0 +1,21 @@ +--- +title: "no_read_modify_write" +api-name: "no_read_modify_write" +source: docs/api_reference/STL/stlReadModifyWriteOptionno_read_modify_write.html +--- +## no_read_modify_write + +### Function Details + +``` c +static ReadModifyWriteOption no_read_modify_write() + +``` + +Call this function to tell the container's begin() function that you do not need a read-modify-write iterator. + +This is the default value for the parameter of any container's begin() function. + +### Class + +ReadModifyWriteOption diff --git a/docs-src/api/stl/stlReadModifyWriteOptionoperator_eq.md b/docs-src/api/stl/stlReadModifyWriteOptionoperator_eq.md new file mode 100644 index 000000000..1e5825d2e --- /dev/null +++ b/docs-src/api/stl/stlReadModifyWriteOptionoperator_eq.md @@ -0,0 +1,19 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stlReadModifyWriteOptionoperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const ReadModifyWriteOption &rmw1) const + +``` + +Equality comparison. + +### Class + +ReadModifyWriteOption diff --git a/docs-src/api/stl/stlReadModifyWriteOptionread_modify_write.md b/docs-src/api/stl/stlReadModifyWriteOptionread_modify_write.md new file mode 100644 index 000000000..d34c7dc8f --- /dev/null +++ b/docs-src/api/stl/stlReadModifyWriteOptionread_modify_write.md @@ -0,0 +1,19 @@ +--- +title: "read_modify_write" +api-name: "read_modify_write" +source: docs/api_reference/STL/stlReadModifyWriteOptionread_modify_write.html +--- +## read_modify_write + +### Function Details + +``` c +static ReadModifyWriteOption read_modify_write() + +``` + +Call this function to tell the container's begin() function that you need a read-modify-write iterator. + +### Class + +ReadModifyWriteOption diff --git a/docs-src/api/stl/stldb_base_iteratorclose_cursor.md b/docs-src/api/stl/stldb_base_iteratorclose_cursor.md new file mode 100644 index 000000000..88ad08871 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratorclose_cursor.md @@ -0,0 +1,21 @@ +--- +title: "close_cursor" +api-name: "close_cursor" +source: docs/api_reference/STL/stldb_base_iteratorclose_cursor.html +--- +## close_cursor + +### Function Details + +``` c +void close_cursor() const + +``` + +Close its cursor. + +If you are sure the iterator is no longer used, call this function so that its underlying cursor is closed before this iterator is destructed, potentially increase performance and concurrency. Note that the cursor is definitely closed at iterator destruction if you don't close it explicitly. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratordb_base_iterator.md b/docs-src/api/stl/stldb_base_iteratordb_base_iterator.md new file mode 100644 index 000000000..d71c66583 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratordb_base_iterator.md @@ -0,0 +1,35 @@ +--- +title: "db_base_iterator" +api-name: "db_base_iterator" +source: docs/api_reference/STL/stldb_base_iteratordb_base_iterator.html +--- +## db_base_iterator + +### Function Details + +``` c +db_base_iterator() + +``` + +Default constructor. + +``` c +db_base_iterator(db_container *powner, bool directdbget, bool b_read_only, + u_int32_t bulk, + bool rmw) + +``` + +Constructor. + +``` c +db_base_iterator(const db_base_iterator &bi) + +``` + +Copy constructor. Copy all members of this class. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratordstr_db_base_iterator.md b/docs-src/api/stl/stldb_base_iteratordstr_db_base_iterator.md new file mode 100644 index 000000000..da3fb49ca --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratordstr_db_base_iterator.md @@ -0,0 +1,19 @@ +--- +title: "~db_base_iterator" +api-name: "~db_base_iterator" +source: docs/api_reference/STL/stldb_base_iteratordstr_db_base_iterator.html +--- +## ~db_base_iterator + +### Function Details + +``` c +virtual ~db_base_iterator() + +``` + +Destructor. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratorget_bulk_bufsize.md b/docs-src/api/stl/stldb_base_iteratorget_bulk_bufsize.md new file mode 100644 index 000000000..85a21b154 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratorget_bulk_bufsize.md @@ -0,0 +1,21 @@ +--- +title: "get_bulk_bufsize" +api-name: "get_bulk_bufsize" +source: docs/api_reference/STL/stldb_base_iteratorget_bulk_bufsize.html +--- +## get_bulk_bufsize + +### Function Details + +``` c +u_int32_t get_bulk_bufsize() + +``` + +Return current bulk buffer size. + +Returns 0 if bulk retrieval is not enabled. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratorget_bulk_retrieval.md b/docs-src/api/stl/stldb_base_iteratorget_bulk_retrieval.md new file mode 100644 index 000000000..62a2b10bc --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratorget_bulk_retrieval.md @@ -0,0 +1,21 @@ +--- +title: "get_bulk_retrieval" +api-name: "get_bulk_retrieval" +source: docs/api_reference/STL/stldb_base_iteratorget_bulk_retrieval.html +--- +## get_bulk_retrieval + +### Function Details + +``` c +u_int32_t get_bulk_retrieval() const + +``` + +Get bulk buffer size. + +Return bulk buffer size. If the size is 0, bulk retrieval is not enabled. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratoris_directdb_get.md b/docs-src/api/stl/stldb_base_iteratoris_directdb_get.md new file mode 100644 index 000000000..80a4601e7 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratoris_directdb_get.md @@ -0,0 +1,21 @@ +--- +title: "is_directdb_get" +api-name: "is_directdb_get" +source: docs/api_reference/STL/stldb_base_iteratoris_directdb_get.html +--- +## is_directdb_get + +### Function Details + +``` c +bool is_directdb_get() const + +``` + +Get direct database get setting. + +Return true if every operation to retrieve the key/data pair the iterator points to will read from database rather than using the cached value, false otherwise. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratoris_rmw.md b/docs-src/api/stl/stldb_base_iteratoris_rmw.md new file mode 100644 index 000000000..cca3400c9 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratoris_rmw.md @@ -0,0 +1,21 @@ +--- +title: "is_rmw" +api-name: "is_rmw" +source: docs/api_reference/STL/stldb_base_iteratoris_rmw.html +--- +## is_rmw + +### Function Details + +``` c +bool is_rmw() const + +``` + +Get DB_RMW setting. + +Return true if the iterator's cursor has DB_RMW flag set, false otherwise. DB_RMW flag causes a write lock to be acquired when reading a key/data pair, so that the transaction won't block later when writing back the updated value in a read-modify-write operation cycle. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratoroperator_assign.md b/docs-src/api/stl/stldb_base_iteratoroperator_assign.md new file mode 100644 index 000000000..bc40c3493 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratoroperator_assign.md @@ -0,0 +1,31 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_base_iteratoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &bi) + +``` + +Iterator assignment operator. + +Iterator assignment will cause the underlying cursor of the right iterator to be duplicated to the left iterator after its previous cursor is closed, to make sure each iterator owns one unique cursor. The key/data cached in the right iterator is copied to the left iterator. Consequently, the left iterator points to the same key/data pair in the database as the the right value after the assignment, and have identical cached key/data pair. + +#### Parameters + +##### bi + +The other iterator to assign with. + +#### Return Value + +The iterator bi's reference. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_base_iteratorset_bulk_buffer.md b/docs-src/api/stl/stldb_base_iteratorset_bulk_buffer.md new file mode 100644 index 000000000..7f9e54607 --- /dev/null +++ b/docs-src/api/stl/stldb_base_iteratorset_bulk_buffer.md @@ -0,0 +1,31 @@ +--- +title: "set_bulk_buffer" +api-name: "set_bulk_buffer" +source: docs/api_reference/STL/stldb_base_iteratorset_bulk_buffer.html +--- +## set_bulk_buffer + +### Function Details + +``` c +bool set_bulk_buffer(u_int32_t sz) + +``` + +Call this function to modify bulk buffer size. + +Bulk retrieval is enabled when creating an iterator, so users later can only modify the bulk buffer size to another value, but can't enable/disable bulk read while an iterator is already alive. + +#### Parameters + +##### sz + +The new buffer size in bytes. + +#### Return Value + +true if succeeded, false otherwise. + +### Class + +db_base_iterator diff --git a/docs-src/api/stl/stldb_containerdb_container.md b/docs-src/api/stl/stldb_containerdb_container.md new file mode 100644 index 000000000..3755b27cb --- /dev/null +++ b/docs-src/api/stl/stldb_containerdb_container.md @@ -0,0 +1,54 @@ +--- +title: "db_container" +api-name: "db_container" +source: docs/api_reference/STL/stldb_containerdb_container.html +--- +## db_container + +### Function Details + +``` c +db_container() + +``` + +Default constructor. + +``` c +db_container(const db_container &dbctnr) + +``` + +Copy constructor. + +The new container will be backed by another database within the same environment unless dbctnr's backing database is in its own internal private environment. The name of the database is coined based on current time and thread id and some random number. If this is still causing naming clashes, you can set a suffix number via "set_global_dbfile_suffix_number" function; And following db file will suffix this number in the file name for additional randomness. And the suffix will be incremented after each such use. You can change the file name via DbEnv::rename. If dbctnr is using an anonymous database, the newly constructed container will also use an anonymous one. + +#### Parameters + +##### dbctnr + +The container to initialize this container. + +``` c +db_container(Db *dbp, + DbEnv *envp) + +``` + +This constructor is not directly called by the user, but invoked by constructors of concrete container classes. + +The statement about the parameters applies to constructors of all container classes. + +#### Parameters + +##### dbp + +Database handle. dbp is supposed to be opened inside envp. Each dbstl container is backed by a Berkeley DB database, so dbstl will create an internal anonymous database if dbp is NULL. + +##### envp + +Environment handle. And envp can also be NULL, meaning the dbp handle may be created in its internal private environment. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerdstr_db_container.md b/docs-src/api/stl/stldb_containerdstr_db_container.md new file mode 100644 index 000000000..95996b704 --- /dev/null +++ b/docs-src/api/stl/stldb_containerdstr_db_container.md @@ -0,0 +1,25 @@ +--- +title: "~db_container" +api-name: "~db_container" +source: docs/api_reference/STL/stldb_containerdstr_db_container.html +--- +## ~db_container + +### Function Details + +``` c +virtual ~db_container() + +``` + +The backing database is not closed in this function. + +It is closed when current thread exits and the database is no longer referenced by any other container instances in this process. In order to make the reference counting work alright, you must call register_db(Db*) and register_db_env(DbEnv*) correctly. + +#### See Also + +register_db(Db*) register_db_env(DbEnv*) + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerget_commit_flags.md b/docs-src/api/stl/stldb_containerget_commit_flags.md new file mode 100644 index 000000000..4309fa608 --- /dev/null +++ b/docs-src/api/stl/stldb_containerget_commit_flags.md @@ -0,0 +1,27 @@ +--- +title: "get_commit_flags" +api-name: "get_commit_flags" +source: docs/api_reference/STL/stldb_containerget_commit_flags.html +--- +## get_commit_flags + +### Function Details + +``` c +u_int32_t get_commit_flags() const + +``` + +Get flag of DbTxn::commit() call. + +#### Return Value + +Flags to be set to DbTxn::commit(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerget_cursor_open_flags.md b/docs-src/api/stl/stldb_containerget_cursor_open_flags.md new file mode 100644 index 000000000..65fd25c40 --- /dev/null +++ b/docs-src/api/stl/stldb_containerget_cursor_open_flags.md @@ -0,0 +1,27 @@ +--- +title: "get_cursor_open_flags" +api-name: "get_cursor_open_flags" +source: docs/api_reference/STL/stldb_containerget_cursor_open_flags.html +--- +## get_cursor_open_flags + +### Function Details + +``` c +u_int32_t get_cursor_open_flags() const + +``` + +Get flag of Db::cursor() call. + +#### Return Value + +Flags to be set to Db::cursor(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerget_db_env_handle.md b/docs-src/api/stl/stldb_containerget_db_env_handle.md new file mode 100644 index 000000000..51cca089f --- /dev/null +++ b/docs-src/api/stl/stldb_containerget_db_env_handle.md @@ -0,0 +1,27 @@ +--- +title: "get_db_env_handle" +api-name: "get_db_env_handle" +source: docs/api_reference/STL/stldb_containerget_db_env_handle.html +--- +## get_db_env_handle + +### Function Details + +``` c +DbEnv* get_db_env_handle() const + +``` + +Get the backing database environment's handle. + +#### Return Value + +The backing database environment handle of this container. + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerget_db_handle.md b/docs-src/api/stl/stldb_containerget_db_handle.md new file mode 100644 index 000000000..a2df56a7f --- /dev/null +++ b/docs-src/api/stl/stldb_containerget_db_handle.md @@ -0,0 +1,27 @@ +--- +title: "get_db_handle" +api-name: "get_db_handle" +source: docs/api_reference/STL/stldb_containerget_db_handle.html +--- +## get_db_handle + +### Function Details + +``` c +Db* get_db_handle() const + +``` + +Get the backing database's handle. + +#### Return Value + +The backing database handle of this container. + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerget_db_set_flags.md b/docs-src/api/stl/stldb_containerget_db_set_flags.md new file mode 100644 index 000000000..03a8ebff6 --- /dev/null +++ b/docs-src/api/stl/stldb_containerget_db_set_flags.md @@ -0,0 +1,27 @@ +--- +title: "get_db_set_flags" +api-name: "get_db_set_flags" +source: docs/api_reference/STL/stldb_containerget_db_set_flags.html +--- +## get_db_set_flags + +### Function Details + +``` c +u_int32_t get_db_set_flags() const + +``` + +Get the backing database's flags that are set via Db::set_flags() function. + +#### Return Value + +Flags set to this container's database handle. + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerget_txn_begin_flags.md b/docs-src/api/stl/stldb_containerget_txn_begin_flags.md new file mode 100644 index 000000000..d83fc0172 --- /dev/null +++ b/docs-src/api/stl/stldb_containerget_txn_begin_flags.md @@ -0,0 +1,27 @@ +--- +title: "get_txn_begin_flags" +api-name: "get_txn_begin_flags" +source: docs/api_reference/STL/stldb_containerget_txn_begin_flags.html +--- +## get_txn_begin_flags + +### Function Details + +``` c +u_int32_t get_txn_begin_flags() const + +``` + +Get flag of DbEnv::txn_begin() call. + +#### Return Value + +Flags to be set to DbEnv::txn_begin(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerset_all_flags.md b/docs-src/api/stl/stldb_containerset_all_flags.md new file mode 100644 index 000000000..3396e3ecc --- /dev/null +++ b/docs-src/api/stl/stldb_containerset_all_flags.md @@ -0,0 +1,40 @@ +--- +title: "set_all_flags" +api-name: "set_all_flags" +source: docs/api_reference/STL/stldb_containerset_all_flags.html +--- +## set_all_flags + +### Function Details + +``` c +void set_all_flags(u_int32_t txn_begin_flags, u_int32_t commit_flags, + u_int32_t cursor_open_flags) + +``` + +Set the flags required by the Berkeley DB functions DbEnv::txn_begin(), DbTxn::commit() and DbEnv::cursor(). + +These flags will be set to this container's auto commit member functions when auto commit transaction is used, except that cursor_oflags is set to the Dbc::cursor when creating an iterator for this container. By default the three flags are all zero. You can also set the values of the flags individually by using the appropriate set functions in this class. The corresponding get functions return the flags actually used. + +#### Parameters + +##### commit_flags + +Flags to be set to DbTxn::commit(). + +##### cursor_open_flags + +Flags to be set to Db::cursor(). + +##### txn_begin_flags + +Flags to be set to DbEnv::txn_begin(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerset_commit_flags.md b/docs-src/api/stl/stldb_containerset_commit_flags.md new file mode 100644 index 000000000..31f21affc --- /dev/null +++ b/docs-src/api/stl/stldb_containerset_commit_flags.md @@ -0,0 +1,29 @@ +--- +title: "set_commit_flags" +api-name: "set_commit_flags" +source: docs/api_reference/STL/stldb_containerset_commit_flags.html +--- +## set_commit_flags + +### Function Details + +``` c +void set_commit_flags(u_int32_t flag) + +``` + +Set flag of DbTxn::commit() call. + +#### Parameters + +##### flag + +Flags to be set to DbTxn::commit(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerset_cursor_open_flags.md b/docs-src/api/stl/stldb_containerset_cursor_open_flags.md new file mode 100644 index 000000000..0dadd6e5f --- /dev/null +++ b/docs-src/api/stl/stldb_containerset_cursor_open_flags.md @@ -0,0 +1,29 @@ +--- +title: "set_cursor_open_flags" +api-name: "set_cursor_open_flags" +source: docs/api_reference/STL/stldb_containerset_cursor_open_flags.html +--- +## set_cursor_open_flags + +### Function Details + +``` c +void set_cursor_open_flags(u_int32_t flag) + +``` + +Set flag of Db::cursor() call. + +#### Parameters + +##### flag + +Flags to be set to Db::cursor(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerset_db_handle.md b/docs-src/api/stl/stldb_containerset_db_handle.md new file mode 100644 index 000000000..615c712d0 --- /dev/null +++ b/docs-src/api/stl/stldb_containerset_db_handle.md @@ -0,0 +1,36 @@ +--- +title: "set_db_handle" +api-name: "set_db_handle" +source: docs/api_reference/STL/stldb_containerset_db_handle.html +--- +## set_db_handle + +### Function Details + +``` c +void set_db_handle(Db *dbp, + DbEnv *newenv=NULL) + +``` + +Set the underlying database's handle, and optionally environment handle if the environment has also changed. + +That is, users can change the container object's underlying database while the object is alive. dbstl will verify that the handles set conforms to the concrete container's requirement to Berkeley DB database/environment handles. + +#### Parameters + +##### dbp + +The database handle to set. + +##### newenv + +The database environment handle to set. + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_containerset_txn_begin_flags.md b/docs-src/api/stl/stldb_containerset_txn_begin_flags.md new file mode 100644 index 000000000..fa1db08f5 --- /dev/null +++ b/docs-src/api/stl/stldb_containerset_txn_begin_flags.md @@ -0,0 +1,29 @@ +--- +title: "set_txn_begin_flags" +api-name: "set_txn_begin_flags" +source: docs/api_reference/STL/stldb_containerset_txn_begin_flags.html +--- +## set_txn_begin_flags + +### Function Details + +``` c +void set_txn_begin_flags(u_int32_t flag) + +``` + +Set flag of DbEnv::txn_begin() call. + +#### Parameters + +##### flag + +Flags to be set to DbEnv::txn_begin(). + +### Group: Get and set functions for data members. + +Note that these functions are not thread safe, because all data members of db_container are supposed to be set on container construction and initialization, and remain read only afterwards. + +### Class + +db_container diff --git a/docs-src/api/stl/stldb_map_base_iteratorclose_cursor.md b/docs-src/api/stl/stldb_map_base_iteratorclose_cursor.md new file mode 100644 index 000000000..41e393d51 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratorclose_cursor.md @@ -0,0 +1,23 @@ +--- +title: "close_cursor" +api-name: "close_cursor" +source: docs/api_reference/STL/stldb_map_base_iteratorclose_cursor.html +--- +## close_cursor + +### Function Details + +``` c +void close_cursor() const + +``` + +Close underlying Berkeley DB cursor of this iterator. + +#### See Also + +db_base_iterator::close_cursor() const + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratordstr_db_map_base_iterator.md b/docs-src/api/stl/stldb_map_base_iteratordstr_db_map_base_iterator.md new file mode 100644 index 000000000..7ebf396d9 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratordstr_db_map_base_iterator.md @@ -0,0 +1,25 @@ +--- +title: "~db_map_base_iterator" +api-name: "~db_map_base_iterator" +source: docs/api_reference/STL/stldb_map_base_iteratordstr_db_map_base_iterator.html +--- +## ~db_map_base_iterator + +### Function Details + +``` c +virtual ~db_map_base_iterator() + +``` + +Destructor. + +### Group: Constructors and destructor + +Do not create iterators directly using these constructors, but call db_map::begin or db_multimap_begin to get instances of this class. + +db_map::begin() db_multimap::begin() + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratorget_bulk_bufsize.md b/docs-src/api/stl/stldb_map_base_iteratorget_bulk_bufsize.md new file mode 100644 index 000000000..816d254e2 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratorget_bulk_bufsize.md @@ -0,0 +1,27 @@ +--- +title: "get_bulk_bufsize" +api-name: "get_bulk_bufsize" +source: docs/api_reference/STL/stldb_map_base_iteratorget_bulk_bufsize.html +--- +## get_bulk_bufsize + +### Function Details + +``` c +u_int32_t get_bulk_bufsize() + +``` + +Get bulk retrieval buffer size in bytes. + +#### Return Value + +Return current bulk buffer size or 0 if bulk retrieval is not enabled. + +#### See Also + +db_base_iterator::get_bulk_bufsize() + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratormove_to.md b/docs-src/api/stl/stldb_map_base_iteratormove_to.md new file mode 100644 index 000000000..33ea1a8b5 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratormove_to.md @@ -0,0 +1,36 @@ +--- +title: "move_to" +api-name: "move_to" +source: docs/api_reference/STL/stldb_map_base_iteratormove_to.html +--- +## move_to + +### Function Details + +``` c +int move_to(const kdt &k, + int flag=DB_SET) const + +``` + +Iterator movement function. + +Move this iterator to the specified key k, by default moves exactly to k, and update cached data element, you can also specify DB_SET_RANGE, to move to the biggest key smaller than k. The btree/hash key comparison routine determines which key is bigger. When the iterator is on a multiple container, move_to will move itself to the first key/data pair of the identical keys. + +#### Parameters + +##### k + +The target key value to move to. + +##### flag + +Flags available: DB_SET(default) or DB_SET_RANGE. DB_SET will move this iterator exactly at k; DB_SET_RANGE moves this iterator to k or the smallest key greater than k. If fail to find such a key, this iterator will become invalid. + +#### Return Value + +0 if succeed; non-0 otherwise, and this iterator becomes invalid. Call db_strerror with the return value to get the error message. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator__star.md b/docs-src/api/stl/stldb_map_base_iteratoroperator__star.md new file mode 100644 index 000000000..25ab441a7 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator__star.md @@ -0,0 +1,25 @@ +--- +title: "operator *" +api-name: "operator *" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator__star.html +--- +## operator \* + +### Function Details + +``` c +reference operator *() const + +``` + +Dereference operator. + +Return the reference to the cached data element, which is an pair\. You can only read its referenced data via this iterator but can not update it. + +#### Return Value + +Current data element reference object, i.e. ElementHolder or ElementRef object. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator_arrow.md b/docs-src/api/stl/stldb_map_base_iteratoroperator_arrow.md new file mode 100644 index 000000000..7a9da8d9b --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator_arrow.md @@ -0,0 +1,25 @@ +--- +title: "operator->" +api-name: "operator->" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator_arrow.html +--- +## operator-\> + +### Function Details + +``` c +pointer operator->() const + +``` + +Arrow operator. + +Return the pointer to the cached data element, which is an pair\. You can only read its referenced data via this iterator but can not update it. + +#### Return Value + +Current data element reference object's address, i.e. address of ElementHolder or ElementRef object. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator_assign.md b/docs-src/api/stl/stldb_map_base_iteratoroperator_assign.md new file mode 100644 index 000000000..ae357351f --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator_assign.md @@ -0,0 +1,35 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &itr) + +``` + +Assignment operator. + +This iterator will point to the same key/data pair as itr, and have the same configurations as itr. + +#### Parameters + +##### itr + +The right value of assignment. + +#### Return Value + +The reference of itr. + +#### See Also + +db_base_iterator::operator=(const self&) + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator_decr.md b/docs-src/api/stl/stldb_map_base_iteratoroperator_decr.md new file mode 100644 index 000000000..b65655dd6 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator_decr.md @@ -0,0 +1,40 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Pre-decrement. + +#### Return Value + +This iterator after decremented. + +``` c +self operator--(int) + +``` + +Post-decrement. + +#### Return Value + +Another iterator having the old value of this iterator. + +### Group: Iterator decrement movement functions. + +The two functions moves the iterator one element forward, so that the element it sits on has a smaller key. + +The btree/hash key comparison routine determines which key is greater. Use --iter rather than iter-- where possible to avoid two useless iterator copy constructions. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator_eq.md b/docs-src/api/stl/stldb_map_base_iteratoroperator_eq.md new file mode 100644 index 000000000..9995f79f7 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator_eq.md @@ -0,0 +1,33 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const self &itr) const + +``` + +Equal comparison operator. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +Returns true if equal, false otherwise. + +### Group: Compare operators. + +Only equal comparison is supported. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator_incr.md b/docs-src/api/stl/stldb_map_base_iteratoroperator_incr.md new file mode 100644 index 000000000..2769ce7cc --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator_incr.md @@ -0,0 +1,40 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Pre-increment. + +#### Return Value + +This iterator after incremented. + +``` c +self operator++(int) + +``` + +Post-increment. + +#### Return Value + +Another iterator having the old value of this iterator. + +### Group: Iterator increment movement functions. + +The two functions moves the iterator one element backward, so that the element it sits on has a bigger key. + +The btree/hash key comparison routine determines which key is greater. Use ++iter rather than iter++ where possible to avoid two useless iterator copy constructions. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratoroperator_ueq.md b/docs-src/api/stl/stldb_map_base_iteratoroperator_ueq.md new file mode 100644 index 000000000..7b4915253 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratoroperator_ueq.md @@ -0,0 +1,37 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_map_base_iteratoroperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const self &itr) const + +``` + +Unequal comparison operator. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +Returns false if equal, true otherwise. + +#### See Also + +bool operator==(const self&itr) const + +### Group: Compare operators. + +Only equal comparison is supported. + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratorrefresh.md b/docs-src/api/stl/stldb_map_base_iteratorrefresh.md new file mode 100644 index 000000000..63d4e2bda --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratorrefresh.md @@ -0,0 +1,29 @@ +--- +title: "refresh" +api-name: "refresh" +source: docs/api_reference/STL/stldb_map_base_iteratorrefresh.html +--- +## refresh + +### Function Details + +``` c +virtual int refresh(bool from_db=true) const + +``` + +Refresh iterator cached value. + +#### Parameters + +##### from_db + +If not doing direct database get and this parameter is true, we will retrieve data directly from db. + +#### See Also + +db_base_iterator::refresh(bool) + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_base_iteratorset_bulk_buffer.md b/docs-src/api/stl/stldb_map_base_iteratorset_bulk_buffer.md new file mode 100644 index 000000000..bbb2abb75 --- /dev/null +++ b/docs-src/api/stl/stldb_map_base_iteratorset_bulk_buffer.md @@ -0,0 +1,35 @@ +--- +title: "set_bulk_buffer" +api-name: "set_bulk_buffer" +source: docs/api_reference/STL/stldb_map_base_iteratorset_bulk_buffer.html +--- +## set_bulk_buffer + +### Function Details + +``` c +bool set_bulk_buffer(u_int32_t sz) + +``` + +Modify bulk buffer size. + +Bulk read is enabled when creating an iterator, so users later can only modify the bulk buffer size to another value, but can't enable/disable bulk read while an iterator is already alive. + +#### Parameters + +##### sz + +The new size of the bulk read buffer of this iterator. + +#### Return Value + +Returns true if succeeded, false otherwise. + +#### See Also + +db_base_iterator::set_bulk_buffer(u_int32_t ) + +### Class + +db_map_base_iterator diff --git a/docs-src/api/stl/stldb_map_iteratordstr_db_map_iterator.md b/docs-src/api/stl/stldb_map_iteratordstr_db_map_iterator.md new file mode 100644 index 000000000..c367eb3e8 --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratordstr_db_map_iterator.md @@ -0,0 +1,25 @@ +--- +title: "~db_map_iterator" +api-name: "~db_map_iterator" +source: docs/api_reference/STL/stldb_map_iteratordstr_db_map_iterator.html +--- +## ~db_map_iterator + +### Function Details + +``` c +virtual ~db_map_iterator() + +``` + +Destructor. + +### Group: Constructors and destructor + +Do not create iterators directly using these constructors, but call db_map::begin or db_multimap_begin to get instances of this class. + +db_map::begin() db_multimap::begin() + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_map_iteratoroperator__star.md b/docs-src/api/stl/stldb_map_iteratoroperator__star.md new file mode 100644 index 000000000..ec8a3ace8 --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratoroperator__star.md @@ -0,0 +1,25 @@ +--- +title: "operator *" +api-name: "operator *" +source: docs/api_reference/STL/stldb_map_iteratoroperator__star.html +--- +## operator \* + +### Function Details + +``` c +reference operator *() const + +``` + +Dereference operator. + +Return the reference to the cached data element, which is an pair\ \> object if T is a class type or an pair\ \> object if T is a C++ primitive data type. + +#### Return Value + +Current data element reference object, i.e. ElementHolder or ElementRef object. + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_map_iteratoroperator_arrow.md b/docs-src/api/stl/stldb_map_iteratoroperator_arrow.md new file mode 100644 index 000000000..694c83b50 --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratoroperator_arrow.md @@ -0,0 +1,25 @@ +--- +title: "operator->" +api-name: "operator->" +source: docs/api_reference/STL/stldb_map_iteratoroperator_arrow.html +--- +## operator-\> + +### Function Details + +``` c +pointer operator->() const + +``` + +Arrow operator. + +Return the pointer to the cached data element, which is an pair\ \> object if T is a class type or an pair\ \> object if T is a C++ primitive data type. + +#### Return Value + +Current data element reference object's address, i.e. address of ElementHolder or ElementRef object. + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_map_iteratoroperator_assign.md b/docs-src/api/stl/stldb_map_iteratoroperator_assign.md new file mode 100644 index 000000000..7afb7cc88 --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratoroperator_assign.md @@ -0,0 +1,35 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_map_iteratoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &itr) + +``` + +Assignment operator. + +This iterator will point to the same key/data pair as itr, and have the same configurations as itr. + +#### Parameters + +##### itr + +The right value of assignment. + +#### Return Value + +The reference of itr. + +#### See Also + +db_base_iterator::operator=(const self&) + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_map_iteratoroperator_decr.md b/docs-src/api/stl/stldb_map_iteratoroperator_decr.md new file mode 100644 index 000000000..b6406055e --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratoroperator_decr.md @@ -0,0 +1,42 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_map_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Pre-decrement. + +#### Return Value + +This iterator after decremented. + +#### See Also + +db_map_base_iterator::operator--() + +``` c +self operator--(int) + +``` + +Post-decrement. + +#### Return Value + +Another iterator having the old value of this iterator. + +#### See Also + +db_map_base_iterator::operator--(int) + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_map_iteratoroperator_incr.md b/docs-src/api/stl/stldb_map_iteratoroperator_incr.md new file mode 100644 index 000000000..e3f911683 --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratoroperator_incr.md @@ -0,0 +1,42 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stldb_map_iteratoroperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Pre-increment. + +#### Return Value + +This iterator after incremented. + +#### See Also + +db_map_base_iterator::operator++() + +``` c +self operator++(int) + +``` + +Post-increment. + +#### Return Value + +Another iterator having the old value of this iterator. + +#### See Also + +db_map_base_iterator::operator++(int) + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_map_iteratorrefresh.md b/docs-src/api/stl/stldb_map_iteratorrefresh.md new file mode 100644 index 000000000..10dfd24d0 --- /dev/null +++ b/docs-src/api/stl/stldb_map_iteratorrefresh.md @@ -0,0 +1,29 @@ +--- +title: "refresh" +api-name: "refresh" +source: docs/api_reference/STL/stldb_map_iteratorrefresh.html +--- +## refresh + +### Function Details + +``` c +virtual int refresh(bool from_db=true) const + +``` + +Refresh iterator cached value. + +#### Parameters + +##### from_db + +If not doing direct database get and this parameter is true, we will retrieve data directly from db. + +#### See Also + +db_base_iterator::refresh(bool ) + +### Class + +db_map_iterator diff --git a/docs-src/api/stl/stldb_mapbegin.md b/docs-src/api/stl/stldb_mapbegin.md new file mode 100644 index 000000000..876609bca --- /dev/null +++ b/docs-src/api/stl/stldb_mapbegin.md @@ -0,0 +1,82 @@ +--- +title: "begin" +api-name: "begin" +source: docs/api_reference/STL/stldb_mapbegin.html +--- +## begin + +### Function Details + +``` c +iterator begin(ReadModifyWriteOption rmw= + ReadModifyWriteOption::no_read_modify_write(), bool readonly=false, + BulkRetrievalOption bulkretrieval= + BulkRetrievalOption::no_bulk_retrieval(), + bool directdb_get=true) + +``` + +Begin a read-write or readonly iterator which sits on the first key/data pair of the database. + +#### Parameters + +##### directdb_get + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### readonly + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### rmw + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulkretrieval + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created iterator. + +#### See Also + +db_vector::begin (ReadModifyWriteOption , bool, BulkRetrievalOption , bool) + +``` c +const_iterator begin(BulkRetrievalOption bulkretrieval= + BulkRetrievalOption::no_bulk_retrieval(), + bool directdb_get=true) const + +``` + +Begin a read-only iterator. + +#### Parameters + +##### directdb_get + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulkretrieval + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created const iterator. + +#### See Also + +db_vector::begin (ReadModifyWrite, bool, BulkRetrievalOption , bool); + +### Group: Iterator Functions + +The parameters in begin functions of this group have identical meaning to thoes in db_vector::begin , refer to those functions for details. + +db_vector::begin() + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapbucket_count.md b/docs-src/api/stl/stldb_mapbucket_count.md new file mode 100644 index 000000000..af782efaa --- /dev/null +++ b/docs-src/api/stl/stldb_mapbucket_count.md @@ -0,0 +1,29 @@ +--- +title: "bucket_count" +api-name: "bucket_count" +source: docs/api_reference/STL/stldb_mapbucket_count.html +--- +## bucket_count + +### Function Details + +``` c +size_type bucket_count() const + +``` + +Only for std::hash_map, return number of hash bucket in use. + +This function supports auto-commit. + +#### Return Value + +The number of hash buckets of the database. + +### Group: Metadata Functions + +These functions return metadata about the container. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapclear.md b/docs-src/api/stl/stldb_mapclear.md new file mode 100644 index 000000000..89398ac6e --- /dev/null +++ b/docs-src/api/stl/stldb_mapclear.md @@ -0,0 +1,31 @@ +--- +title: "clear" +api-name: "clear" +source: docs/api_reference/STL/stldb_mapclear.html +--- +## clear + +### Function Details + +``` c +void clear(bool b_truncate=true) + +``` + +Clear contents in this container. + +This function supports auto-commit. + +#### Parameters + +##### b_truncate + +See db_vector::clear(bool) for details. + +#### See Also + +db_vector::clear(bool) + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapcount.md b/docs-src/api/stl/stldb_mapcount.md new file mode 100644 index 000000000..0b7d56d46 --- /dev/null +++ b/docs-src/api/stl/stldb_mapcount.md @@ -0,0 +1,39 @@ +--- +title: "count" +api-name: "count" +source: docs/api_reference/STL/stldb_mapcount.html +--- +## count + +### Function Details + +``` c +size_type count(const key_type &x) const + +``` + +Count the number of key/data pairs having specified key x. + +#### Parameters + +##### x + +The key to count. + +#### Return Value + +The number of key/data pairs having x as key within the container. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/count/ + +### Group: Searching Functions + +The following functions are returning iterators, and they by default return read-write iterators. + +If you intend to use the returned iterator only to read, you should call the const version of each function using a const reference to this container. Using const iterators can potentially promote concurrency a lot. You can also set the readonly parameter to each non-const version of the functions to true if you don't use the returned iterator to write, which also promotes concurrency and overall performance. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapdstr_db_map.md b/docs-src/api/stl/stldb_mapdstr_db_map.md new file mode 100644 index 000000000..e700f1fe7 --- /dev/null +++ b/docs-src/api/stl/stldb_mapdstr_db_map.md @@ -0,0 +1,17 @@ +--- +title: "~db_map" +api-name: "~db_map" +source: docs/api_reference/STL/stldb_mapdstr_db_map.html +--- +## ~db_map + +### Function Details + +``` c +virtual ~db_map() + +``` + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapempty.md b/docs-src/api/stl/stldb_mapempty.md new file mode 100644 index 000000000..97d1ac991 --- /dev/null +++ b/docs-src/api/stl/stldb_mapempty.md @@ -0,0 +1,29 @@ +--- +title: "empty" +api-name: "empty" +source: docs/api_reference/STL/stldb_mapempty.html +--- +## empty + +### Function Details + +``` c +bool empty() const + +``` + +Returns whether this container is empty. + +This function supports auto-commit. + +#### Return Value + +True if empty, false otherwise. + +### Group: Metadata Functions + +These functions return metadata about the container. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapend.md b/docs-src/api/stl/stldb_mapend.md new file mode 100644 index 000000000..8552784c3 --- /dev/null +++ b/docs-src/api/stl/stldb_mapend.md @@ -0,0 +1,48 @@ +--- +title: "end" +api-name: "end" +source: docs/api_reference/STL/stldb_mapend.html +--- +## end + +### Function Details + +``` c +iterator end() + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid iterator denoting the position after the last valid element of the container. + +#### See Also + +db_vector::end() + +``` c +const_iterator end() const + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid const iterator denoting the position after the last valid element of the container. + +#### See Also + +db_vector::end() const + +### Group: Iterator Functions + +The parameters in begin functions of this group have identical meaning to thoes in db_vector::begin , refer to those functions for details. + +db_vector::begin() + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapequal_range.md b/docs-src/api/stl/stldb_mapequal_range.md new file mode 100644 index 000000000..f33761bcf --- /dev/null +++ b/docs-src/api/stl/stldb_mapequal_range.md @@ -0,0 +1,65 @@ +--- +title: "equal_range" +api-name: "equal_range" +source: docs/api_reference/STL/stldb_mapequal_range.html +--- +## equal_range + +### Function Details + +``` c +equal_range(const key_type &x) const + +``` + +Find the range within which all keys equal to specified key x. + +#### Parameters + +##### x + +The target key to find. + +#### Return Value + +The range \[first, last). + +#### See Also + +http://www.cplusplus.com/reference/stl/map/equal_range/ + +``` c +equal_range(const key_type &x, + bool readonly=false) + +``` + +Find the range within which all keys equal to specified key x. + +#### Parameters + +##### x + +The target key to find. + +##### readonly + +Whether the returned iterator is readonly. + +#### Return Value + +The range \[first, last). + +#### See Also + +http://www.cplusplus.com/reference/stl/map/equal_range/ + +### Group: Searching Functions + +The following functions are returning iterators, and they by default return read-write iterators. + +If you intend to use the returned iterator only to read, you should call the const version of each function using a const reference to this container. Using const iterators can potentially promote concurrency a lot. You can also set the readonly parameter to each non-const version of the functions to true if you don't use the returned iterator to write, which also promotes concurrency and overall performance. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_maperase.md b/docs-src/api/stl/stldb_maperase.md new file mode 100644 index 000000000..e9e51f7fa --- /dev/null +++ b/docs-src/api/stl/stldb_maperase.md @@ -0,0 +1,68 @@ +--- +title: "erase" +api-name: "erase" +source: docs/api_reference/STL/stldb_maperase.html +--- +## erase + +### Function Details + +``` c +void erase(iterator pos) + +``` + +Erase a key/data pair at specified position. + +#### Parameters + +##### pos + +An valid iterator of this container to erase. + +``` c +size_type erase(const key_type &x) + +``` + +Erase elements by key. + +All key/data pairs with specified key x will be removed from underlying database. This function supports auto-commit. + +#### Parameters + +##### x + +The key to remove from the container. + +#### Return Value + +The number of key/data pairs removed. + +``` c +void erase(iterator first, + iterator last) + +``` + +Range erase. + +Erase all key/data pairs within the valid range \[first, last). + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +### Group: Erase Functions + +http://www.cplusplus.com/reference/stl/map/erase/ + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapfind.md b/docs-src/api/stl/stldb_mapfind.md new file mode 100644 index 000000000..baaa0293c --- /dev/null +++ b/docs-src/api/stl/stldb_mapfind.md @@ -0,0 +1,65 @@ +--- +title: "find" +api-name: "find" +source: docs/api_reference/STL/stldb_mapfind.html +--- +## find + +### Function Details + +``` c +const_iterator find(const key_type &x) const + +``` + +Find the key/data pair with specified key x. + +#### Parameters + +##### x + +The target key to find. + +#### Return Value + +The valid const iterator sitting on the key x, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/find/ + +``` c +iterator find(const key_type &x, + bool readonly=false) + +``` + +Find the key/data pair with specified key x. + +#### Parameters + +##### x + +The target key to find. + +##### readonly + +Whether the returned iterator is readonly. + +#### Return Value + +The valid iterator sitting on the key x, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/find/ + +### Group: Searching Functions + +The following functions are returning iterators, and they by default return read-write iterators. + +If you intend to use the returned iterator only to read, you should call the const version of each function using a const reference to this container. Using const iterators can potentially promote concurrency a lot. You can also set the readonly parameter to each non-const version of the functions to true if you don't use the returned iterator to write, which also promotes concurrency and overall performance. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_maphash_funct.md b/docs-src/api/stl/stldb_maphash_funct.md new file mode 100644 index 000000000..304b6cd11 --- /dev/null +++ b/docs-src/api/stl/stldb_maphash_funct.md @@ -0,0 +1,29 @@ +--- +title: "hash_funct" +api-name: "hash_funct" +source: docs/api_reference/STL/stldb_maphash_funct.html +--- +## hash_funct + +### Function Details + +``` c +hasher hash_funct() const + +``` + +Function to get hash key generating functor. + +Used when this container is a hash_map, hash_multimap, hash_set or hash_multiset equivalent. + +#### Return Value + +The hash key generating functor. + +#### See Also + +http://www.sgi.com/tech/stl/hash_map.html + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapinsert.md b/docs-src/api/stl/stldb_mapinsert.md new file mode 100644 index 000000000..d56747c26 --- /dev/null +++ b/docs-src/api/stl/stldb_mapinsert.md @@ -0,0 +1,102 @@ +--- +title: "insert" +api-name: "insert" +source: docs/api_reference/STL/stldb_mapinsert.html +--- +## insert + +### Function Details + +``` c +insert(const value_type &x) + +``` + +Insert a single key/data pair if the key is not in the container. + +#### Parameters + +##### x + +The key/data pair to insert. + +#### Return Value + +A pair P, if insert OK, i.e. the inserted key wasn't in the container, P.first will be the iterator sitting on the inserted key/data pair, and P.second is true; otherwise P.first is an invalid iterator and P.second is false. + +``` c +iterator insert(iterator position, + const value_type &x) + +``` + +Insert with hint position. + +We ignore the hint position because Berkeley DB knows better where to insert. + +#### Parameters + +##### position + +The hint position. + +##### x + +The key/data pair to insert. + +#### Return Value + +The iterator sitting on the inserted key/data pair, or an invalid iterator if the key was already in the container. + +``` c +void insert(const db_map_base_iterator< kdt, realddt, ddt > &first, + const db_map_base_iterator< kdt, realddt, + ddt > &last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +void insert(InputIterator first, + InputIterator last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +### Group: Insert Functions + +They have similiar usage as their C++ STL equivalents. + +Note that when secondary index is enabled, each db_container can create a db_multimap secondary container, but the insert function is not functional for secondary containers. + +http://www.cplusplus.com/reference/stl/map/insert/ + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapis_hash.md b/docs-src/api/stl/stldb_mapis_hash.md new file mode 100644 index 000000000..b1ebaacb2 --- /dev/null +++ b/docs-src/api/stl/stldb_mapis_hash.md @@ -0,0 +1,29 @@ +--- +title: "is_hash" +api-name: "is_hash" +source: docs/api_reference/STL/stldb_mapis_hash.html +--- +## is_hash + +### Function Details + +``` c +bool is_hash() const + +``` + +Get container category. + +Determines whether this container object is a std::map\<\> equivalent(when returns false) or that of hash_map\<\> class(when returns true). This method is not in stl, but it may be called by users because some operations are not supported by both type(map/hash_map) of containers, you need to call this function to distinguish the two types. dbstl will not stop you from calling the wrong methods of this class. + +#### Return Value + +Returns true if this container is a hash container based on a Berkeley DB hash database; returns false if it is based on a Berkeley DB btree database. + +### Group: Metadata Functions + +These functions return metadata about the container. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapkey_comp.md b/docs-src/api/stl/stldb_mapkey_comp.md new file mode 100644 index 000000000..2852891c4 --- /dev/null +++ b/docs-src/api/stl/stldb_mapkey_comp.md @@ -0,0 +1,29 @@ +--- +title: "key_comp" +api-name: "key_comp" +source: docs/api_reference/STL/stldb_mapkey_comp.html +--- +## key_comp + +### Function Details + +``` c +key_compare key_comp() const + +``` + +Function to get key compare functor. + +Used when this container is a std::map, std::multimap, std::set or std::multiset equivalent. + +#### Return Value + +The key compare functor. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/key_comp/ + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapkey_eq.md b/docs-src/api/stl/stldb_mapkey_eq.md new file mode 100644 index 000000000..a2dfda6d2 --- /dev/null +++ b/docs-src/api/stl/stldb_mapkey_eq.md @@ -0,0 +1,29 @@ +--- +title: "key_eq" +api-name: "key_eq" +source: docs/api_reference/STL/stldb_mapkey_eq.html +--- +## key_eq + +### Function Details + +``` c +key_equal key_eq() const + +``` + +Function to get key compare functor. + +Used when this container is a hash_map, hash_multimap, hash_set or hash_multiset equivalent. + +#### Return Value + +key_equal type of compare functor. + +#### See Also + +http://www.sgi.com/tech/stl/hash_map.html + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_maplower_bound.md b/docs-src/api/stl/stldb_maplower_bound.md new file mode 100644 index 000000000..66a83bc97 --- /dev/null +++ b/docs-src/api/stl/stldb_maplower_bound.md @@ -0,0 +1,65 @@ +--- +title: "lower_bound" +api-name: "lower_bound" +source: docs/api_reference/STL/stldb_maplower_bound.html +--- +## lower_bound + +### Function Details + +``` c +const_iterator lower_bound(const key_type &x) const + +``` + +Find the greatest key less than or equal to x. + +#### Parameters + +##### x + +The target key to find. + +#### Return Value + +The valid const iterator sitting on the key, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/lower_bound/ + +``` c +iterator lower_bound(const key_type &x, + bool readonly=false) + +``` + +Find the greatest key less than or equal to x. + +#### Parameters + +##### x + +The target key to find. + +##### readonly + +Whether the returned iterator is readonly. + +#### Return Value + +The valid iterator sitting on the key, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/lower_bound/ + +### Group: Searching Functions + +The following functions are returning iterators, and they by default return read-write iterators. + +If you intend to use the returned iterator only to read, you should call the const version of each function using a const reference to this container. Using const iterators can potentially promote concurrency a lot. You can also set the readonly parameter to each non-const version of the functions to true if you don't use the returned iterator to write, which also promotes concurrency and overall performance. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapmax_size.md b/docs-src/api/stl/stldb_mapmax_size.md new file mode 100644 index 000000000..e0e72d0f6 --- /dev/null +++ b/docs-src/api/stl/stldb_mapmax_size.md @@ -0,0 +1,33 @@ +--- +title: "max_size" +api-name: "max_size" +source: docs/api_reference/STL/stldb_mapmax_size.html +--- +## max_size + +### Function Details + +``` c +size_type max_size() const + +``` + +Get max size. + +The returned size is not the actual limit of database. See the Berkeley DB limits to get real max size. + +#### Return Value + +A meaningless huge number. + +#### See Also + +db_vector::max_size() + +### Group: Metadata Functions + +These functions return metadata about the container. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapoperator_assign.md b/docs-src/api/stl/stldb_mapoperator_assign.md new file mode 100644 index 000000000..d8357baae --- /dev/null +++ b/docs-src/api/stl/stldb_mapoperator_assign.md @@ -0,0 +1,31 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_mapoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &x) + +``` + +Container content assignment operator. + +This function supports auto-commit. + +#### Parameters + +##### x + +The other container whose key/data pairs will be inserted into this container. Old content in this containers are discarded. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/operator=/ + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapoperator_eq.md b/docs-src/api/stl/stldb_mapoperator_eq.md new file mode 100644 index 000000000..570121096 --- /dev/null +++ b/docs-src/api/stl/stldb_mapoperator_eq.md @@ -0,0 +1,32 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_mapoperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const db_map< kdt, ddt, + value_type_sub > &m2) const + +``` + +Map content equality comparison operator. + +This function does not rely on key order. For a set of keys S1 in this container and another set of keys S2 of container m2, if set S1 contains S2 and S2 contains S1 (S1 equals to S2) and each data element of a key K in S1 from this container equals the data element of K in m2, the two db_map\<\> containers equal. Otherwise they are not equal. + +#### Parameters + +##### m2 + +The other container to compare against. + +#### Return Value + +Returns true if they have equal content, false otherwise. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapoperator_sqbrk.md b/docs-src/api/stl/stldb_mapoperator_sqbrk.md new file mode 100644 index 000000000..4c076ffed --- /dev/null +++ b/docs-src/api/stl/stldb_mapoperator_sqbrk.md @@ -0,0 +1,50 @@ +--- +title: "operator[]" +api-name: "operator[]" +source: docs/api_reference/STL/stldb_mapoperator_sqbrk.html +--- +## operator\[\] + +### Function Details + +``` c +data_type_wrap operator[](const key_type &x) + +``` + +Retrieve data element by key. + +This function returns an reference to the underlying data element of the specified key x. The returned object can be used to read or write the data element of the key/data pair. Do use a data_type_wrap of db_map or value_type::second_type(they are the same) type of variable to hold the return value of this function. + +#### Parameters + +##### x + +The target key to get value from. + +#### Return Value + +Data element reference. + +``` c +const ddt operator[](const key_type &x) const + +``` + +Retrieve data element by key. + +This function returns the value of the underlying data element of specified key x. You can only read the element, but unable to update the element via the return value of this function. And you need to use the container's const reference to call this method. + +#### Parameters + +##### x + +The target key to get value from. + +#### Return Value + +Data element, read only, can't be used to modify it. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapoperator_ueq.md b/docs-src/api/stl/stldb_mapoperator_ueq.md new file mode 100644 index 000000000..935482907 --- /dev/null +++ b/docs-src/api/stl/stldb_mapoperator_ueq.md @@ -0,0 +1,30 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_mapoperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const db_map< kdt, ddt, + value_type_sub > &m2) const + +``` + +Container unequality comparison operator. + +#### Parameters + +##### m2 + +The container to compare against. + +#### Return Value + +Returns false if equal, true otherwise. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_maprbegin.md b/docs-src/api/stl/stldb_maprbegin.md new file mode 100644 index 000000000..8f900bf6e --- /dev/null +++ b/docs-src/api/stl/stldb_maprbegin.md @@ -0,0 +1,84 @@ +--- +title: "rbegin" +api-name: "rbegin" +source: docs/api_reference/STL/stldb_maprbegin.html +--- +## rbegin + +### Function Details + +``` c +reverse_iterator rbegin(ReadModifyWriteOption rmw= + ReadModifyWriteOption::no_read_modify_write(), bool read_only=false, + BulkRetrievalOption bulkretrieval= + BulkRetrievalOption::no_bulk_retrieval(), + bool directdb_get=true) + +``` + +Begin a read-write or readonly reverse iterator which sits on the first key/data pair of the database. + +#### Parameters + +##### directdb_get + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### read_only + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### rmw + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulkretrieval + +Same as that of db_vector::begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created iterator. + +#### See Also + +db_vector::begin (ReadModifyWriteOption , bool, BulkRetrievalOption , bool) + +db_vector::begin (ReadModifyWrite, bool, BulkRetrievalOption , bool); + +``` c +const_reverse_iterator rbegin(BulkRetrievalOption bulkretrieval= + BulkRetrievalOption::no_bulk_retrieval(), + bool directdb_get=true) const + +``` + +Begin a read-only reverse iterator. + +#### Parameters + +##### directdb_get + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulkretrieval + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created const iterator. + +#### See Also + +db_vector::begin (ReadModifyWrite, bool, BulkRetrievalOption , bool); + +### Group: Iterator Functions + +The parameters in begin functions of this group have identical meaning to thoes in db_vector::begin , refer to those functions for details. + +db_vector::begin() + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_maprend.md b/docs-src/api/stl/stldb_maprend.md new file mode 100644 index 000000000..56b802798 --- /dev/null +++ b/docs-src/api/stl/stldb_maprend.md @@ -0,0 +1,48 @@ +--- +title: "rend" +api-name: "rend" +source: docs/api_reference/STL/stldb_maprend.html +--- +## rend + +### Function Details + +``` c +reverse_iterator rend() + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid iterator denoting the position before the first valid element of the container. + +#### See Also + +db_vector::rend() + +``` c +const_reverse_iterator rend() const + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid const iterator denoting the position before the first valid element of the container. + +#### See Also + +db_vector::rend() const + +### Group: Iterator Functions + +The parameters in begin functions of this group have identical meaning to thoes in db_vector::begin , refer to those functions for details. + +db_vector::begin() + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapsize.md b/docs-src/api/stl/stldb_mapsize.md new file mode 100644 index 000000000..228064f67 --- /dev/null +++ b/docs-src/api/stl/stldb_mapsize.md @@ -0,0 +1,33 @@ +--- +title: "size" +api-name: "size" +source: docs/api_reference/STL/stldb_mapsize.html +--- +## size + +### Function Details + +``` c +size_type size(bool accurate=true) const + +``` + +This function supports auto-commit. + +#### Parameters + +##### accurate + +This function uses database's statistics to get the number of key/data pairs. The statistics mechanism will either scan the whole database to find the accurate number or use the number of last accurate scanning, and thus much faster. If there are millions of key/data pairs, the scanning can take some while, so in that case you may want to set the "accurate" parameter to false. + +#### Return Value + +Return the number of key/data pairs in the container. + +### Group: Metadata Functions + +These functions return metadata about the container. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapswap.md b/docs-src/api/stl/stldb_mapswap.md new file mode 100644 index 000000000..41cd94e0f --- /dev/null +++ b/docs-src/api/stl/stldb_mapswap.md @@ -0,0 +1,36 @@ +--- +title: "swap" +api-name: "swap" +source: docs/api_reference/STL/stldb_mapswap.html +--- +## swap + +### Function Details + +``` c +void swap(db_map< kdt, ddt, value_type_sub > &mp, + bool b_truncate=true) + +``` + +Swap content with container mp. + +This function supports auto-commit. + +#### Parameters + +##### b_truncate + +See db_vector::swap() for details. + +##### mp + +The container to swap content with. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/swap/ db_vector::clear() + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapupper_bound.md b/docs-src/api/stl/stldb_mapupper_bound.md new file mode 100644 index 000000000..29e724901 --- /dev/null +++ b/docs-src/api/stl/stldb_mapupper_bound.md @@ -0,0 +1,65 @@ +--- +title: "upper_bound" +api-name: "upper_bound" +source: docs/api_reference/STL/stldb_mapupper_bound.html +--- +## upper_bound + +### Function Details + +``` c +const_iterator upper_bound(const key_type &x) const + +``` + +Find the least key greater than x. + +#### Parameters + +##### x + +The target key to find. + +#### Return Value + +The valid iterator sitting on the key, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/upper_bound/ + +``` c +iterator upper_bound(const key_type &x, + bool readonly=false) + +``` + +Find the least key greater than x. + +#### Parameters + +##### x + +The target key to find. + +##### readonly + +Whether the returned iterator is readonly. + +#### Return Value + +The valid iterator sitting on the key, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/upper_bound/ + +### Group: Searching Functions + +The following functions are returning iterators, and they by default return read-write iterators. + +If you intend to use the returned iterator only to read, you should call the const version of each function using a const reference to this container. Using const iterators can potentially promote concurrency a lot. You can also set the readonly parameter to each non-const version of the functions to true if you don't use the returned iterator to write, which also promotes concurrency and overall performance. + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_mapvalue_comp.md b/docs-src/api/stl/stldb_mapvalue_comp.md new file mode 100644 index 000000000..6b9b6f4eb --- /dev/null +++ b/docs-src/api/stl/stldb_mapvalue_comp.md @@ -0,0 +1,29 @@ +--- +title: "value_comp" +api-name: "value_comp" +source: docs/api_reference/STL/stldb_mapvalue_comp.html +--- +## value_comp + +### Function Details + +``` c +value_compare value_comp() const + +``` + +Function to get value compare functor. + +Used when this container is a std::map, std::multimap, std::set or std::multiset equivalent. + +#### Return Value + +The value compare functor. + +#### See Also + +http://www.cplusplus.com/reference/stl/map/value_comp/ + +### Class + +db_map diff --git a/docs-src/api/stl/stldb_multimapcount.md b/docs-src/api/stl/stldb_multimapcount.md new file mode 100644 index 000000000..4792e3bdb --- /dev/null +++ b/docs-src/api/stl/stldb_multimapcount.md @@ -0,0 +1,39 @@ +--- +title: "count" +api-name: "count" +source: docs/api_reference/STL/stldb_multimapcount.html +--- +## count + +### Function Details + +``` c +size_type count(const key_type &x) const + +``` + +Count the number of key/data pairs having specified key x. + +#### Parameters + +##### x + +The key to count. + +#### Return Value + +The number of key/data pairs having x as key within the container. + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/count/ + +### Group: Searching Functions + +See of db_map's searching functions group for details about iterator, function version and parameters. + +db_map + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapdb_multimap.md b/docs-src/api/stl/stldb_multimapdb_multimap.md new file mode 100644 index 000000000..1aa1c17e7 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapdb_multimap.md @@ -0,0 +1,87 @@ +--- +title: "db_multimap" +api-name: "db_multimap" +source: docs/api_reference/STL/stldb_multimapdb_multimap.html +--- +## db_multimap + +### Function Details + +``` c +db_multimap(Db *dbp=NULL, + DbEnv *envp=NULL) + +``` + +Constructor. + +See class detail for handle requirement. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +#### See Also + +db_map::db_map(Db*, DbEnv*) db_vector::db_vector(Db*, DbEnv*) + +``` c +db_multimap(Db *dbp, DbEnv *envp, InputIterator first, + InputIterator last) + +``` + +Iteration constructor. + +Iterates between first and last, setting a copy of each of the sequence of elements as the content of the container object. This function supports auto-commit. See class detail for handle requirement. + +#### Parameters + +##### dbp + +The database handle. + +##### envp + +The database environment handle. + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +#### See Also + +db_map::db_map(Db*, DbEnv*, InputIterator, InputIterator) db_vector::db_vector(Db*, DbEnv*) + +``` c +db_multimap(const self &x) + +``` + +Copy constructor. + +Create an database and insert all key/data pairs in x into this container. x's data members are not copied. This function supports auto-commit. + +#### Parameters + +##### x + +The other container to initialize this container. + +#### See Also + +db_container(const db_container&) db_map(const db_map&) + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapdstr_db_multimap.md b/docs-src/api/stl/stldb_multimapdstr_db_multimap.md new file mode 100644 index 000000000..219dda2a8 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapdstr_db_multimap.md @@ -0,0 +1,17 @@ +--- +title: "~db_multimap" +api-name: "~db_multimap" +source: docs/api_reference/STL/stldb_multimapdstr_db_multimap.html +--- +## ~db_multimap + +### Function Details + +``` c +virtual ~db_multimap() + +``` + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapequal_range.md b/docs-src/api/stl/stldb_multimapequal_range.md new file mode 100644 index 000000000..1dadcff19 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapequal_range.md @@ -0,0 +1,65 @@ +--- +title: "equal_range" +api-name: "equal_range" +source: docs/api_reference/STL/stldb_multimapequal_range.html +--- +## equal_range + +### Function Details + +``` c +equal_range(const key_type &x) const + +``` + +Find the range within which all keys equal to specified key x. + +#### Parameters + +##### x + +The target key to find. + +#### Return Value + +The range \[first, last). + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/equal_range/ + +``` c +equal_range(const key_type &x, + bool readonly=false) + +``` + +Find the range within which all keys equal to specified key x. + +#### Parameters + +##### x + +The target key to find. + +##### readonly + +Whether the returned iterator is readonly. + +#### Return Value + +The range \[first, last). + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/equal_range/ + +### Group: Searching Functions + +See of db_map's searching functions group for details about iterator, function version and parameters. + +db_map + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapequal_range_N.md b/docs-src/api/stl/stldb_multimapequal_range_N.md new file mode 100644 index 000000000..a75707413 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapequal_range_N.md @@ -0,0 +1,70 @@ +--- +title: "equal_range_N" +api-name: "equal_range_N" +source: docs/api_reference/STL/stldb_multimapequal_range_N.html +--- +## equal_range_N + +### Function Details + +``` c +equal_range_N(const key_type &x, + size_t &nelem) const + +``` + +Find equal range and number of key/data pairs in the range. + +This function also returns the number of elements within the returned range via the out parameter nelem. + +#### Parameters + +##### x + +The target key to find. + +##### nelem + +The output parameter to take back the number of key/data pair in the returned range. + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/equal_range/ + +``` c +equal_range_N(const key_type &x, size_t &nelem, + bool readonly=false) + +``` + +Find equal range and number of key/data pairs in the range. + +This function also returns the number of elements within the returned range via the out parameter nelem. + +#### Parameters + +##### x + +The target key to find. + +##### nelem + +The output parameter to take back the number of key/data pair in the returned range. + +##### readonly + +Whether the returned iterator is readonly. + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/equal_range/ + +### Group: Searching Functions + +See of db_map's searching functions group for details about iterator, function version and parameters. + +db_map + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimaperase.md b/docs-src/api/stl/stldb_multimaperase.md new file mode 100644 index 000000000..1e5be1538 --- /dev/null +++ b/docs-src/api/stl/stldb_multimaperase.md @@ -0,0 +1,68 @@ +--- +title: "erase" +api-name: "erase" +source: docs/api_reference/STL/stldb_multimaperase.html +--- +## erase + +### Function Details + +``` c +size_type erase(const key_type &x) + +``` + +Erase elements by key. + +All key/data pairs with specified key x will be removed from underlying database. This function supports auto-commit. + +#### Parameters + +##### x + +The key to remove from the container. + +#### Return Value + +The number of key/data pairs removed. + +``` c +void erase(iterator pos) + +``` + +Erase a key/data pair at specified position. + +#### Parameters + +##### pos + +An valid iterator of this container to erase. + +``` c +void erase(iterator first, + iterator last) + +``` + +Range erase. + +Erase all key/data pairs within the valid range \[first, last). + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +### Group: Erase Functions + +http://www.cplusplus.com/reference/stl/multimap/erase/ + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapoperator_assign.md b/docs-src/api/stl/stldb_multimapoperator_assign.md new file mode 100644 index 000000000..350ded1ef --- /dev/null +++ b/docs-src/api/stl/stldb_multimapoperator_assign.md @@ -0,0 +1,31 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_multimapoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &x) + +``` + +Container content assignment operator. + +This function supports auto-commit. + +#### Parameters + +##### x + +The other container whose key/data pairs will be inserted into this container. Old content in this containers are discarded. + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/operator=/ + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapoperator_eq.md b/docs-src/api/stl/stldb_multimapoperator_eq.md new file mode 100644 index 000000000..66e0dab54 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapoperator_eq.md @@ -0,0 +1,32 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_multimapoperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const db_multimap< kdt, ddt, + value_type_sub > &m2) const + +``` + +Returns whether the two containers have identical content. + +This function does not rely on key order. For a set of keys S1 in this container and another set of keys S2 of container m2, if set S1 contains S2 and S2 contains S1 (S1 equals to S2) and each set of data elements of any key K in S1 from this container equals the set of data elements of K in m2, the two db_multimap\<\> containers equal. Otherwise they are not equal. Data element set comparison does not rely on order either. + +#### Parameters + +##### m2 + +The other container to compare against. + +#### Return Value + +Returns true if they are equal, false otherwise. + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapoperator_ueq.md b/docs-src/api/stl/stldb_multimapoperator_ueq.md new file mode 100644 index 000000000..d7c345509 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapoperator_ueq.md @@ -0,0 +1,30 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_multimapoperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const db_multimap< kdt, ddt, + value_type_sub > &m2) const + +``` + +Container unequality comparison operator. + +#### Parameters + +##### m2 + +The container to compare against. + +#### Return Value + +Returns false if equal, true otherwise. + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapswap.md b/docs-src/api/stl/stldb_multimapswap.md new file mode 100644 index 000000000..9606bce63 --- /dev/null +++ b/docs-src/api/stl/stldb_multimapswap.md @@ -0,0 +1,36 @@ +--- +title: "swap" +api-name: "swap" +source: docs/api_reference/STL/stldb_multimapswap.html +--- +## swap + +### Function Details + +``` c +void swap(db_multimap< kdt, ddt, value_type_sub > &mp, + bool b_truncate=true) + +``` + +Swap content with another multimap container. + +This function supports auto-commit. + +#### Parameters + +##### b_truncate + +See db_map::swap() for details. + +##### mp + +The other container to swap content with. + +#### See Also + +db_vector::clear() + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multimapupper_bound.md b/docs-src/api/stl/stldb_multimapupper_bound.md new file mode 100644 index 000000000..d319b8f1a --- /dev/null +++ b/docs-src/api/stl/stldb_multimapupper_bound.md @@ -0,0 +1,65 @@ +--- +title: "upper_bound" +api-name: "upper_bound" +source: docs/api_reference/STL/stldb_multimapupper_bound.html +--- +## upper_bound + +### Function Details + +``` c +const_iterator upper_bound(const key_type &x) const + +``` + +Find the least key greater than x. + +#### Parameters + +##### x + +The target key to find. + +#### Return Value + +The valid iterator sitting on the key, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/upper_bound/ + +``` c +iterator upper_bound(const key_type &x, + bool readonly=false) + +``` + +Find the least key greater than x. + +#### Parameters + +##### x + +The target key to find. + +##### readonly + +Whether the returned iterator is readonly. + +#### Return Value + +The valid iterator sitting on the key, or an invalid one. + +#### See Also + +http://www.cplusplus.com/reference/stl/multimap/upper_bound/ + +### Group: Searching Functions + +See of db_map's searching functions group for details about iterator, function version and parameters. + +db_map + +### Class + +db_multimap diff --git a/docs-src/api/stl/stldb_multisetdstr_db_multiset.md b/docs-src/api/stl/stldb_multisetdstr_db_multiset.md new file mode 100644 index 000000000..b56331496 --- /dev/null +++ b/docs-src/api/stl/stldb_multisetdstr_db_multiset.md @@ -0,0 +1,17 @@ +--- +title: "~db_multiset" +api-name: "~db_multiset" +source: docs/api_reference/STL/stldb_multisetdstr_db_multiset.html +--- +## ~db_multiset + +### Function Details + +``` c +virtual ~db_multiset() + +``` + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_multiseterase.md b/docs-src/api/stl/stldb_multiseterase.md new file mode 100644 index 000000000..7f9b135de --- /dev/null +++ b/docs-src/api/stl/stldb_multiseterase.md @@ -0,0 +1,68 @@ +--- +title: "erase" +api-name: "erase" +source: docs/api_reference/STL/stldb_multiseterase.html +--- +## erase + +### Function Details + +``` c +size_type erase(const key_type &x) + +``` + +Erase elements by key. + +All key/data pairs with specified key x will be removed from the underlying database. This function supports auto-commit. + +#### Parameters + +##### x + +The key to remove from the container. + +#### Return Value + +The number of key/data pairs removed. + +``` c +void erase(iterator pos) + +``` + +Erase a key/data pair at specified position. + +#### Parameters + +##### pos + +A valid iterator of this container to erase. + +``` c +void erase(iterator first, + iterator last) + +``` + +Range erase. + +Erase all key/data pairs within the valid range \[first, last). + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +### Group: Erase Functions + +http://www.cplusplus.com/reference/stl/multiset/erase/ + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_multisetinsert.md b/docs-src/api/stl/stldb_multisetinsert.md new file mode 100644 index 000000000..a6bd6b237 --- /dev/null +++ b/docs-src/api/stl/stldb_multisetinsert.md @@ -0,0 +1,118 @@ +--- +title: "insert" +api-name: "insert" +source: docs/api_reference/STL/stldb_multisetinsert.html +--- +## insert + +### Function Details + +``` c +iterator insert(const value_type &x) + +``` + +Insert a single key if the key is not in the container. + +#### Parameters + +##### x + +The key to insert. + +#### Return Value + +An iterator positioned on the newly inserted key. If the key x already exists, an invalid iterator equal to that returned by end() function is returned. + +``` c +iterator insert(iterator position, + const value_type &x) + +``` + +Insert a single key with hint if the key is not in the container. + +The hint position is ignored because Berkeley DB controls where to insert the key. + +#### Parameters + +##### x + +The key to insert. + +##### position + +The hint insert position, ignored. + +#### Return Value + +An iterator positioned on the newly inserted key. If the key x already exists, an invalid iterator equal to that returned by end() function is returned. + +``` c +void insert(InputIterator first, + InputIterator last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +void insert(db_set_iterator< kdt, value_type_sub > &first, + db_set_iterator< kdt, + value_type_sub > &last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +void insert(db_set_base_iterator< kdt > &first, + db_set_base_iterator< kdt > &last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +### Group: Insert Functions + +http://www.cplusplus.com/reference/stl/multiset/insert/ + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_multisetoperator_assign.md b/docs-src/api/stl/stldb_multisetoperator_assign.md new file mode 100644 index 000000000..04762b629 --- /dev/null +++ b/docs-src/api/stl/stldb_multisetoperator_assign.md @@ -0,0 +1,35 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_multisetoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &x) + +``` + +Container content assignment operator. + +This function supports auto-commit. + +#### Parameters + +##### x + +The source container whose key/data pairs will be inserted into the target container. Old content in the target container is discarded. + +#### Return Value + +The container x's reference. + +#### See Also + +http://www.cplusplus.com/reference/stl/multiset/operator=/ + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_multisetoperator_eq.md b/docs-src/api/stl/stldb_multisetoperator_eq.md new file mode 100644 index 000000000..bda9f935d --- /dev/null +++ b/docs-src/api/stl/stldb_multisetoperator_eq.md @@ -0,0 +1,31 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_multisetoperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const self &m2) const + +``` + +Container content equality compare operator. + +This function does not rely on key order. Two sets A and B are equal if and only if for each and every key K having n occurrences in A, K has n occurrences in B, and for each and every key K\` having N occurrences in B, K\` has n occurrences in A. + +#### Parameters + +##### m2 + +The container to compare against. + +#### Return Value + +Returns true if the two containers are equal, false otherwise. + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_multisetoperator_ueq.md b/docs-src/api/stl/stldb_multisetoperator_ueq.md new file mode 100644 index 000000000..61783dd91 --- /dev/null +++ b/docs-src/api/stl/stldb_multisetoperator_ueq.md @@ -0,0 +1,19 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_multisetoperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const self &m2) const + +``` + +Inequality comparison operator. + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_multisetswap.md b/docs-src/api/stl/stldb_multisetswap.md new file mode 100644 index 000000000..deaafd592 --- /dev/null +++ b/docs-src/api/stl/stldb_multisetswap.md @@ -0,0 +1,36 @@ +--- +title: "swap" +api-name: "swap" +source: docs/api_reference/STL/stldb_multisetswap.html +--- +## swap + +### Function Details + +``` c +void swap(db_multiset< kdt, value_type_sub > &mp, + bool b_truncate=true) + +``` + +Swap content with another container. + +This function supports auto-commit. + +#### Parameters + +##### b_truncate + +See db_multimap::swap() for details. + +##### mp + +The container to swap content with. + +#### See Also + +db_map::swap() db_vector::clear() + +### Class + +db_multiset diff --git a/docs-src/api/stl/stldb_reverse_iteratordb_reverse_iterator.md b/docs-src/api/stl/stldb_reverse_iteratordb_reverse_iterator.md new file mode 100644 index 000000000..cf8e4772a --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratordb_reverse_iterator.md @@ -0,0 +1,41 @@ +--- +title: "db_reverse_iterator" +api-name: "db_reverse_iterator" +source: docs/api_reference/STL/stldb_reverse_iteratordb_reverse_iterator.html +--- +## db_reverse_iterator + +### Function Details + +``` c +db_reverse_iterator(const iterator &vi) + +``` + +Constructor. Construct from an iterator of wrapped type. + +``` c +db_reverse_iterator(const self &ritr) + +``` + +Copy constructor. + +``` c +db_reverse_iterator(const db_reverse_iterator< twin_itr_t, + iterator > &ritr) + +``` + +Copy constructor. + +``` c +db_reverse_iterator() + +``` + +Default constructor. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_add.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_add.md new file mode 100644 index 000000000..f8ca054dc --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_add.md @@ -0,0 +1,39 @@ +--- +title: "operator+" +api-name: "operator+" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_add.html +--- +## operator+ + +### Function Details + +``` c +self operator+(difference_type n) const + +``` + +Iterator shuffle operator. + +Return a new iterator by moving this iterator forward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move towards reverse direction. + +#### Return Value + +A new iterator at new position. + +### Group: Operators for random reverse iterators + +Methods below only applies to random iterators. + +///// + +Return a new iterator by moving this iterator backward or forward by n elements. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_assign.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_assign.md new file mode 100644 index 000000000..0e0ea2611 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_assign.md @@ -0,0 +1,33 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &ri) + +``` + +Assignment operator. + +#### Parameters + +##### ri + +The iterator to assign with. + +#### Return Value + +The iterator ri. + +#### See Also + +db_base_iterator::operator=(const self&) + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_decr.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_decr.md new file mode 100644 index 000000000..9762c75f6 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_decr.md @@ -0,0 +1,40 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Move this iterator backward by one element. + +#### Return Value + +The moved iterator at new position. + +``` c +self operator--(int) + +``` + +Move this iterator backward by one element. + +#### Return Value + +The original iterator at old position. + +### Group: Reverse iterator movement functions + +When we talk about reverse iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_ge.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_ge.md new file mode 100644 index 000000000..9cd7f2722 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_ge.md @@ -0,0 +1,23 @@ +--- +title: "operator>=" +api-name: "operator>=" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_ge.html +--- +## operator\>= + +### Function Details + +``` c +bool operator>=(const self &itr) const + +``` + +Greater equal compare operator. + +### Group: Operators for random reverse iterators + +Reverse iterator comparison against reverse iterator itr, the one sitting on elements with less index is returned to be greater. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_gt.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_gt.md new file mode 100644 index 000000000..1240dcb05 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_gt.md @@ -0,0 +1,23 @@ +--- +title: "operator>" +api-name: "operator>" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_gt.html +--- +## operator\> + +### Function Details + +``` c +bool operator>(const self &itr) const + +``` + +Greater compare operator. + +### Group: Operators for random reverse iterators + +Reverse iterator comparison against reverse iterator itr, the one sitting on elements with less index is returned to be greater. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_ia.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_ia.md new file mode 100644 index 000000000..1aa98f035 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_ia.md @@ -0,0 +1,35 @@ +--- +title: "operator+=" +api-name: "operator+=" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_ia.html +--- +## operator+= + +### Function Details + +``` c +const self& operator+=(difference_type n) + +``` + +Iterator shuffle operator. + +Move this iterator forward by n elements and then return it. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move towards reverse direction. + +#### Return Value + +This iterator at new position. + +### Group: Operators for random reverse iterators + +Move this iterator backward or forward by n elements and then return it. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_le.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_le.md new file mode 100644 index 000000000..aeca81633 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_le.md @@ -0,0 +1,23 @@ +--- +title: "operator<=" +api-name: "operator<=" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_le.html +--- +## operator\<= + +### Function Details + +``` c +bool operator<=(const self &itr) const + +``` + +Less equal compare operator. + +### Group: Operators for random reverse iterators + +Reverse iterator comparison against reverse iterator itr, the one sitting on elements with less index is returned to be greater. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_lt.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_lt.md new file mode 100644 index 000000000..a5e9c68be --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_lt.md @@ -0,0 +1,23 @@ +--- +title: "operator<" +api-name: "operator<" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_lt.html +--- +## operator\< + +### Function Details + +``` c +bool operator<(const self &itr) const + +``` + +Less compare operator. + +### Group: Operators for random reverse iterators + +Reverse iterator comparison against reverse iterator itr, the one sitting on elements with less index is returned to be greater. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_sa.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_sa.md new file mode 100644 index 000000000..874da8716 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_sa.md @@ -0,0 +1,35 @@ +--- +title: "operator-=" +api-name: "operator-=" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_sa.html +--- +## operator-= + +### Function Details + +``` c +const self& operator-=(difference_type n) + +``` + +Iterator shuffle operator. + +Move this iterator backward by n elements and then return it. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move towards reverse direction. + +#### Return Value + +This iterator at new position. + +### Group: Operators for random reverse iterators + +Move this iterator backward or forward by n elements and then return it. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_sqbrk.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_sqbrk.md new file mode 100644 index 000000000..5628e631d --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_sqbrk.md @@ -0,0 +1,21 @@ +--- +title: "operator[]" +api-name: "operator[]" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_sqbrk.html +--- +## operator\[\] + +### Function Details + +``` c +value_type_wrap operator[](difference_type Off) const + +``` + +Return the reference of the element which can be reached by moving this reverse iterator by Off times backward. + +If Off is negative, the movement will be forward. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_reverse_iteratoroperator_sub.md b/docs-src/api/stl/stldb_reverse_iteratoroperator_sub.md new file mode 100644 index 000000000..b227e5e86 --- /dev/null +++ b/docs-src/api/stl/stldb_reverse_iteratoroperator_sub.md @@ -0,0 +1,56 @@ +--- +title: "operator-" +api-name: "operator-" +source: docs/api_reference/STL/stldb_reverse_iteratoroperator_sub.html +--- +## operator- + +### Function Details + +``` c +self operator-(difference_type n) const + +``` + +Iterator shuffle operator. + +Return a new iterator by moving this iterator backward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move towards reverse direction. + +#### Return Value + +A new iterator at new position. + +``` c +difference_type operator-(const self &itr) const + +``` + +Return the negative value of the difference of indices of elements this iterator and itr are sitting on. + +#### Parameters + +##### itr + +The other reverse iterator. + +#### Return Value + +itr.index - this-\>index. + +### Group: Operators for random reverse iterators + +Methods below only applies to random iterators. + +///// + +Return a new iterator by moving this iterator backward or forward by n elements. + +### Class + +db_reverse_iterator diff --git a/docs-src/api/stl/stldb_set_base_iteratordb_set_base_iterator.md b/docs-src/api/stl/stldb_set_base_iteratordb_set_base_iterator.md new file mode 100644 index 000000000..6f56b756b --- /dev/null +++ b/docs-src/api/stl/stldb_set_base_iteratordb_set_base_iterator.md @@ -0,0 +1,80 @@ +--- +title: "db_set_base_iterator" +api-name: "db_set_base_iterator" +source: docs/api_reference/STL/stldb_set_base_iteratordb_set_base_iterator.html +--- +## db_set_base_iterator + +### Function Details + +``` c +db_set_base_iterator(db_container *powner, u_int32_t b_bulk_retrieval=0, + bool brmw=false, bool directdbget=true, + bool b_read_only=false) + +``` + +Constructor. + +#### Parameters + +##### b_bulk_retrieval + +The bulk read buffer size. 0 means bulk read disabled. + +##### brmw + +Whether set DB_RMW flag in underlying cursor. + +##### powner + +The container which creates this iterator. + +##### directdbget + +Whether do direct database get rather than using key/data values cached in the iterator whenever read. + +##### b_read_only + +Whether open a read only cursor. Only effective when using Berkeley DB Concurrent Data Store. + +``` c +db_set_base_iterator() + +``` + +Default constructor, dose not create the cursor for now. + +``` c +db_set_base_iterator(const db_set_base_iterator &s) + +``` + +Copy constructor. + +#### Parameters + +##### s + +The other iterator of the same type to initialize this. + +``` c +db_set_base_iterator(const base &bo) + +``` + +Base copy constructor. + +#### Parameters + +##### bo + +Initialize from a base class iterator. + +### Group: Constructors and destructor + +Do not use these constructors to create iterators, but call db_set::begin() const or db_multiset::begin() const to create valid iterators. + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/stldb_set_base_iteratoroperator__star.md b/docs-src/api/stl/stldb_set_base_iteratoroperator__star.md new file mode 100644 index 000000000..c8ff8217e --- /dev/null +++ b/docs-src/api/stl/stldb_set_base_iteratoroperator__star.md @@ -0,0 +1,25 @@ +--- +title: "operator *" +api-name: "operator *" +source: docs/api_reference/STL/stldb_set_base_iteratoroperator__star.html +--- +## operator \* + +### Function Details + +``` c +reference operator *() + +``` + +Dereference operator. + +Return the reference to the cached data element, which is an object of type T. You can only use the return value to read its referenced data element, can not update it. + +#### Return Value + +Current data element reference object, i.e. ElementHolder or ElementRef object. + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/stldb_set_base_iteratoroperator_arrow.md b/docs-src/api/stl/stldb_set_base_iteratoroperator_arrow.md new file mode 100644 index 000000000..fc08e7c84 --- /dev/null +++ b/docs-src/api/stl/stldb_set_base_iteratoroperator_arrow.md @@ -0,0 +1,25 @@ +--- +title: "operator->" +api-name: "operator->" +source: docs/api_reference/STL/stldb_set_base_iteratoroperator_arrow.html +--- +## operator-\> + +### Function Details + +``` c +pointer operator->() const + +``` + +Arrow operator. + +Return the pointer to the cached data element, which is an object of type T. You can only use the return value to read its referenced data element, can not update it. + +#### Return Value + +Current data element reference object's address, i.e. address of ElementHolder or ElementRef object. + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/stldb_set_base_iteratoroperator_decr.md b/docs-src/api/stl/stldb_set_base_iteratoroperator_decr.md new file mode 100644 index 000000000..1e78a4e12 --- /dev/null +++ b/docs-src/api/stl/stldb_set_base_iteratoroperator_decr.md @@ -0,0 +1,48 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_set_base_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Post-decrement. + +#### Return Value + +This iterator after decremented. + +#### See Also + +db_map_base_iterator::operator--() + +``` c +self operator--(int) + +``` + +Pre-decrement. + +#### Return Value + +Another iterator having the old value of this iterator. + +#### See Also + +db_map_base_iterator::operator--(int) + +### Group: Iterator movement operators. + +These functions are identical to those of db_map_base_iterator and db_map_iterator and db_set_iterator . + +Actually the iterator movement functions in the four classes are the same. + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/stldb_set_base_iteratoroperator_incr.md b/docs-src/api/stl/stldb_set_base_iteratoroperator_incr.md new file mode 100644 index 000000000..157bdb6de --- /dev/null +++ b/docs-src/api/stl/stldb_set_base_iteratoroperator_incr.md @@ -0,0 +1,48 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stldb_set_base_iteratoroperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Post-increment. + +#### Return Value + +This iterator after incremented. + +#### See Also + +db_map_base_iterator::operator++() + +``` c +self operator++(int) + +``` + +Pre-increment. + +#### Return Value + +Another iterator having the old value of this iterator. + +#### See Also + +db_map_base_iterator::operator++(int) + +### Group: Iterator movement operators. + +These functions are identical to those of db_map_base_iterator and db_map_iterator and db_set_iterator . + +Actually the iterator movement functions in the four classes are the same. + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/stldb_set_base_iteratorrefresh.md b/docs-src/api/stl/stldb_set_base_iteratorrefresh.md new file mode 100644 index 000000000..8773bf097 --- /dev/null +++ b/docs-src/api/stl/stldb_set_base_iteratorrefresh.md @@ -0,0 +1,29 @@ +--- +title: "refresh" +api-name: "refresh" +source: docs/api_reference/STL/stldb_set_base_iteratorrefresh.html +--- +## refresh + +### Function Details + +``` c +virtual int refresh(bool from_db=true) const + +``` + +Refresh iterator cached value. + +#### Parameters + +##### from_db + +If not doing direct database get and this parameter is true, we will retrieve data directly from db. + +#### See Also + +db_base_iterator::refresh(bool) + +### Class + +db_set_base_iterator diff --git a/docs-src/api/stl/stldb_set_iteratordb_set_iterator.md b/docs-src/api/stl/stldb_set_iteratordb_set_iterator.md new file mode 100644 index 000000000..de035ef77 --- /dev/null +++ b/docs-src/api/stl/stldb_set_iteratordb_set_iterator.md @@ -0,0 +1,95 @@ +--- +title: "db_set_iterator" +api-name: "db_set_iterator" +source: docs/api_reference/STL/stldb_set_iteratordb_set_iterator.html +--- +## db_set_iterator + +### Function Details + +``` c +db_set_iterator(db_container *powner, u_int32_t b_bulk_retrieval=0, + bool brmw=false, bool directdbget=true, + bool b_read_only=false) + +``` + +Constructor. + +#### Parameters + +##### b_bulk_retrieval + +The bulk read buffer size. 0 means bulk read disabled. + +##### brmw + +Whether set DB_RMW flag in underlying cursor. + +##### powner + +The container which creates this iterator. + +##### directdbget + +Whether do direct database get rather than using key/data values cached in the iterator whenever read. + +##### b_read_only + +Whether open a read only cursor. Only effective when using Berkeley DB Concurrent Data Store. + +``` c +db_set_iterator() + +``` + +Default constructor, dose not create the cursor for now. + +``` c +db_set_iterator(const db_set_iterator &s) + +``` + +Copy constructor. + +#### Parameters + +##### s + +The other iterator of the same type to initialize this. + +``` c +db_set_iterator(const base &bo) + +``` + +Base copy constructor. + +#### Parameters + +##### bo + +Initialize from a base class iterator. + +``` c +db_set_iterator(const db_set_base_iterator< kdt > &bs) + +``` + +Sibling copy constructor. + +Note that this class does not derive from db_set_base_iterator but from db_map_iterator . + +#### Parameters + +##### bs + +Initialize from a base class iterator. + +### Group: Constructors and destructor + +Do not use these constructors to create iterators, but call db_set::begin() or db_multiset::begin() to create valid ones. + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/stldb_set_iteratoroperator__star.md b/docs-src/api/stl/stldb_set_iteratoroperator__star.md new file mode 100644 index 000000000..0f67a3309 --- /dev/null +++ b/docs-src/api/stl/stldb_set_iteratoroperator__star.md @@ -0,0 +1,25 @@ +--- +title: "operator *" +api-name: "operator *" +source: docs/api_reference/STL/stldb_set_iteratoroperator__star.html +--- +## operator \* + +### Function Details + +``` c +reference operator *() + +``` + +Dereference operator. + +Return the reference to the cached data element, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. + +#### Return Value + +Current data element reference object, i.e. ElementHolder or ElementRef object. + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/stldb_set_iteratoroperator_arrow.md b/docs-src/api/stl/stldb_set_iteratoroperator_arrow.md new file mode 100644 index 000000000..8ffb0ef18 --- /dev/null +++ b/docs-src/api/stl/stldb_set_iteratoroperator_arrow.md @@ -0,0 +1,25 @@ +--- +title: "operator->" +api-name: "operator->" +source: docs/api_reference/STL/stldb_set_iteratoroperator_arrow.html +--- +## operator-\> + +### Function Details + +``` c +pointer operator->() const + +``` + +Arrow operator. + +Return the pointer to the cached data element, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. + +#### Return Value + +Current data element reference object's address, i.e. address of ElementHolder or ElementRef object. + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/stldb_set_iteratoroperator_decr.md b/docs-src/api/stl/stldb_set_iteratoroperator_decr.md new file mode 100644 index 000000000..84fe5db5c --- /dev/null +++ b/docs-src/api/stl/stldb_set_iteratoroperator_decr.md @@ -0,0 +1,42 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_set_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Pre-decrement. + +#### Return Value + +This iterator after decremented. + +#### See Also + +db_map_iterator::operator--() + +``` c +self operator--(int) + +``` + +Post-decrement. + +#### Return Value + +Another iterator having the old value of this iterator. + +#### See Also + +db_map_iterator::operator--(int) + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/stldb_set_iteratoroperator_incr.md b/docs-src/api/stl/stldb_set_iteratoroperator_incr.md new file mode 100644 index 000000000..af0a8222b --- /dev/null +++ b/docs-src/api/stl/stldb_set_iteratoroperator_incr.md @@ -0,0 +1,44 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stldb_set_iteratoroperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Pre-increment. + +Identical to those of db_map_iterator . + +#### Return Value + +This iterator after incremented. + +#### See Also + +db_map_iterator::operator++() + +``` c +self operator++(int) + +``` + +Post-increment. + +#### Return Value + +Another iterator having the old value of this iterator. + +#### See Also + +db_map_iterator::operator++(int) + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/stldb_set_iteratorrefresh.md b/docs-src/api/stl/stldb_set_iteratorrefresh.md new file mode 100644 index 000000000..285d7990d --- /dev/null +++ b/docs-src/api/stl/stldb_set_iteratorrefresh.md @@ -0,0 +1,29 @@ +--- +title: "refresh" +api-name: "refresh" +source: docs/api_reference/STL/stldb_set_iteratorrefresh.html +--- +## refresh + +### Function Details + +``` c +virtual int refresh(bool from_db=true) const + +``` + +Refresh iterator cached value. + +#### Parameters + +##### from_db + +If not doing direct database get and this parameter is true, we will retrieve data directly from db. + +#### See Also + +db_base_iterator::refresh(bool) + +### Class + +db_set_iterator diff --git a/docs-src/api/stl/stldb_setdstr_db_set.md b/docs-src/api/stl/stldb_setdstr_db_set.md new file mode 100644 index 000000000..18e2a0710 --- /dev/null +++ b/docs-src/api/stl/stldb_setdstr_db_set.md @@ -0,0 +1,17 @@ +--- +title: "~db_set" +api-name: "~db_set" +source: docs/api_reference/STL/stldb_setdstr_db_set.html +--- +## ~db_set + +### Function Details + +``` c +virtual ~db_set() + +``` + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_setinsert.md b/docs-src/api/stl/stldb_setinsert.md new file mode 100644 index 000000000..6dc14617f --- /dev/null +++ b/docs-src/api/stl/stldb_setinsert.md @@ -0,0 +1,117 @@ +--- +title: "insert" +api-name: "insert" +source: docs/api_reference/STL/stldb_setinsert.html +--- +## insert + +### Function Details + +``` c +insert(const value_type &x) + +``` + +Insert a single key/data pair if the key is not in the container. + +#### Parameters + +##### x + +The key/data pair to insert. + +#### Return Value + +A pair P, if insert OK, i.e. the inserted key wasn't in the container, P.first will be the iterator positioned on the inserted key/data pair, and P.second is true; otherwise P.first is an invalid iterator equal to that returned by end() and P.second is false. + +``` c +void insert(const_iterator &first, + const_iterator &last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +void insert(iterator &first, + iterator &last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +``` c +iterator insert(iterator position, + const value_type &x) + +``` + +Insert with hint position. + +We ignore the hint position because Berkeley DB knows better where to insert. + +#### Parameters + +##### position + +The hint position. + +##### x + +The key/data pair to insert. + +#### Return Value + +The iterator positioned on the inserted key/data pair, or an invalid iterator if the key was already in the container. + +``` c +void insert(InputIterator first, + InputIterator last) + +``` + +Range insertion. + +Insert a range \[first, last) of key/data pairs into this container. + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +### Group: Insert Functions + +http://www.cplusplus.com/reference/stl/set/insert/ + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_setoperator_assign.md b/docs-src/api/stl/stldb_setoperator_assign.md new file mode 100644 index 000000000..af64d9439 --- /dev/null +++ b/docs-src/api/stl/stldb_setoperator_assign.md @@ -0,0 +1,35 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_setoperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &x) + +``` + +Container content assignment operator. + +This function supports auto-commit. + +#### Parameters + +##### x + +The source container whose key/data pairs will be inserted into the target container. Old content in the target container is discarded. + +#### Return Value + +The container x's reference. + +#### See Also + +http://www.cplusplus.com/reference/stl/set/operator=/ + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_setoperator_eq.md b/docs-src/api/stl/stldb_setoperator_eq.md new file mode 100644 index 000000000..9ffb5e5e6 --- /dev/null +++ b/docs-src/api/stl/stldb_setoperator_eq.md @@ -0,0 +1,32 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_setoperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const db_set< kdt, + value_type_sub > &m2) const + +``` + +Set content equality comparison operator. + +Return if the two containers have identical content. This function does not rely on key order. Two sets A and B are equal if and only if A contains B and B contains A. + +#### Parameters + +##### m2 + +The container to compare against. + +#### Return Value + +Returns true if the two containers are equal, false otherwise. + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_setoperator_ueq.md b/docs-src/api/stl/stldb_setoperator_ueq.md new file mode 100644 index 000000000..56622f778 --- /dev/null +++ b/docs-src/api/stl/stldb_setoperator_ueq.md @@ -0,0 +1,20 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_setoperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const db_set< kdt, + value_type_sub > &m2) const + +``` + +Inequality comparison operator. + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_setswap.md b/docs-src/api/stl/stldb_setswap.md new file mode 100644 index 000000000..b61644769 --- /dev/null +++ b/docs-src/api/stl/stldb_setswap.md @@ -0,0 +1,36 @@ +--- +title: "swap" +api-name: "swap" +source: docs/api_reference/STL/stldb_setswap.html +--- +## swap + +### Function Details + +``` c +void swap(db_set< kdt, value_type_sub > &mp, + bool b_truncate=true) + +``` + +Swap content with another container. + +This function supports auto-commit. + +#### Parameters + +##### b_truncate + +See db_vector::swap 's b_truncate parameter for details. + +##### mp + +The container to swap content with. + +#### See Also + +db_map::swap() db_vector::clear() + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_setvalue_comp.md b/docs-src/api/stl/stldb_setvalue_comp.md new file mode 100644 index 000000000..3a7c77d0b --- /dev/null +++ b/docs-src/api/stl/stldb_setvalue_comp.md @@ -0,0 +1,27 @@ +--- +title: "value_comp" +api-name: "value_comp" +source: docs/api_reference/STL/stldb_setvalue_comp.html +--- +## value_comp + +### Function Details + +``` c +value_compare value_comp() const + +``` + +Get value comparison functor. + +#### Return Value + +The value comparison functor. + +#### See Also + +http://www.cplusplus.com/reference/stl/set/value_comp/ + +### Class + +db_set diff --git a/docs-src/api/stl/stldb_vector_base_iteratorclose_cursor.md b/docs-src/api/stl/stldb_vector_base_iteratorclose_cursor.md new file mode 100644 index 000000000..871f6baf3 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratorclose_cursor.md @@ -0,0 +1,23 @@ +--- +title: "close_cursor" +api-name: "close_cursor" +source: docs/api_reference/STL/stldb_vector_base_iteratorclose_cursor.html +--- +## close_cursor + +### Function Details + +``` c +void close_cursor() const + +``` + +Close underlying Berkeley DB cursor of this iterator. + +#### See Also + +db_base_iterator::close_cursor() const + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratordstr_db_vector_base_iterator.md b/docs-src/api/stl/stldb_vector_base_iteratordstr_db_vector_base_iterator.md new file mode 100644 index 000000000..4ece534f6 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratordstr_db_vector_base_iterator.md @@ -0,0 +1,23 @@ +--- +title: "~db_vector_base_iterator" +api-name: "~db_vector_base_iterator" +source: docs/api_reference/STL/stldb_vector_base_iteratordstr_db_vector_base_iterator.html +--- +## ~db_vector_base_iterator + +### Function Details + +``` c +virtual ~db_vector_base_iterator() + +``` + +### Group: Constructors and destroctor + +Do not construct iterators explictily using these constructors, but call db_vector::begin() const to get an valid iterator. + +db_vector::begin() const + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratorget_bulk_bufsize.md b/docs-src/api/stl/stldb_vector_base_iteratorget_bulk_bufsize.md new file mode 100644 index 000000000..60da234bb --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratorget_bulk_bufsize.md @@ -0,0 +1,27 @@ +--- +title: "get_bulk_bufsize" +api-name: "get_bulk_bufsize" +source: docs/api_reference/STL/stldb_vector_base_iteratorget_bulk_bufsize.html +--- +## get_bulk_bufsize + +### Function Details + +``` c +u_int32_t get_bulk_bufsize() + +``` + +Get bulk retrieval buffer size in bytes. + +#### Return Value + +Return current bulk buffer size, or 0 if bulk retrieval is not enabled. + +#### See Also + +db_base_iterator::get_bulk_bufsize() + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratorget_current_index.md b/docs-src/api/stl/stldb_vector_base_iteratorget_current_index.md new file mode 100644 index 000000000..cc6b4dc89 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratorget_current_index.md @@ -0,0 +1,25 @@ +--- +title: "get_current_index" +api-name: "get_current_index" +source: docs/api_reference/STL/stldb_vector_base_iteratorget_current_index.html +--- +## get_current_index + +### Function Details + +``` c +index_type get_current_index() const + +``` + +Get current index of within the vector. + +Return the iterators current element's index (0 based). Requires this iterator to be a valid iterator, not end_itr\_. + +#### Return Value + +current index of the iterator. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratormove_to.md b/docs-src/api/stl/stldb_vector_base_iteratormove_to.md new file mode 100644 index 000000000..fa776edf7 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratormove_to.md @@ -0,0 +1,31 @@ +--- +title: "move_to" +api-name: "move_to" +source: docs/api_reference/STL/stldb_vector_base_iteratormove_to.html +--- +## move_to + +### Function Details + +``` c +void move_to(index_type n) const + +``` + +Iterator movement function. + +Move this iterator to the index "n". If n is not in the valid range, this iterator will be an invalid iterator equal to end() iterator. + +#### Parameters + +##### n + +target element's index. + +#### See Also + +db_vector::end() ; + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator__star.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator__star.md new file mode 100644 index 000000000..6754062b4 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator__star.md @@ -0,0 +1,25 @@ +--- +title: "operator *" +api-name: "operator *" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator__star.html +--- +## operator \* + +### Function Details + +``` c +reference operator *() const + +``` + +Dereference operator. + +Return the reference to the cached data element, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. The returned value can only be used to read its referenced element. + +#### Return Value + +The reference to the element this iterator points to. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_add.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_add.md new file mode 100644 index 000000000..66886797f --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_add.md @@ -0,0 +1,37 @@ +--- +title: "operator+" +api-name: "operator+" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_add.html +--- +## operator+ + +### Function Details + +``` c +self operator+(difference_type n) const + +``` + +Iterator movement operator. + +Return another iterator by moving this iterator forward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move forward by \|n\| element. + +#### Return Value + +The new iterator at new position. + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_arrow.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_arrow.md new file mode 100644 index 000000000..9e5b33e61 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_arrow.md @@ -0,0 +1,25 @@ +--- +title: "operator->" +api-name: "operator->" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_arrow.html +--- +## operator-\> + +### Function Details + +``` c +pointer operator->() const + +``` + +Arrow operator. + +Return the pointer to the cached data element, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. The returned value can only be used to read its referenced element. + +#### Return Value + +The address of the referenced object. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_assign.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_assign.md new file mode 100644 index 000000000..62e44e96f --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_assign.md @@ -0,0 +1,41 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &itr) + +``` + +Assignment operator. + +This iterator will point to the same key/data pair as itr, and have the same configurations as itr. + +#### Parameters + +##### itr + +The right value of the assignment. + +#### Return Value + +This iterator's reference. + +#### See Also + +db_base_iterator::operator= + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_decr.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_decr.md new file mode 100644 index 000000000..069bcaec5 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_decr.md @@ -0,0 +1,44 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Pre-decrement. + +Move the iterator one element backward, so that the element it sits on has a smaller index. Use --iter rather than iter-- where possible to avoid two useless iterator copy constructions. + +#### Return Value + +This iterator after decremented. + +``` c +self operator--(int) + +``` + +Post-decrement. + +Move the iterator one element backward, so that the element it sits on has a smaller index. Use --iter rather than iter-- where possible to avoid two useless iterator copy constructions. + +#### Return Value + +A new iterator not decremented. + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_eq.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_eq.md new file mode 100644 index 000000000..eeb57feac --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_eq.md @@ -0,0 +1,37 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const self &itr) const + +``` + +Equality comparison operator. + +Invalid iterators are equal; Valid iterators sitting on the same key/data pair equal; Otherwise not equal. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +True if this iterator equals to itr; False otherwise. + +### Group: Iterator comparison operators + +The way to compare two iterators is to compare the index values of the two elements they point to. + +The iterator sitting on an element with less index is regarded to be smaller. And the invalid iterator sitting after last element is greater than any other iterators, because it is assumed to have an index equal to last element's index plus one; The invalid iterator sitting before first element is less than any other iterators because it is assumed to have an index -1. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_ge.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_ge.md new file mode 100644 index 000000000..7e439f73a --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_ge.md @@ -0,0 +1,35 @@ +--- +title: "operator>=" +api-name: "operator>=" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_ge.html +--- +## operator\>= + +### Function Details + +``` c +bool operator>=(const self &itr) const + +``` + +Greater equal comparison operator. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +True if this iterator is greater than or equal to itr. + +### Group: Iterator comparison operators + +The way to compare two iterators is to compare the index values of the two elements they point to. + +The iterator sitting on an element with less index is regarded to be smaller. And the invalid iterator sitting after last element is greater than any other iterators, because it is assumed to have an index equal to last element's index plus one; The invalid iterator sitting before first element is less than any other iterators because it is assumed to have an index -1. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_gt.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_gt.md new file mode 100644 index 000000000..cc1e8825b --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_gt.md @@ -0,0 +1,35 @@ +--- +title: "operator>" +api-name: "operator>" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_gt.html +--- +## operator\> + +### Function Details + +``` c +bool operator>(const self &itr) const + +``` + +Greater comparison operator. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +True if this iterator is greater than itr. + +### Group: Iterator comparison operators + +The way to compare two iterators is to compare the index values of the two elements they point to. + +The iterator sitting on an element with less index is regarded to be smaller. And the invalid iterator sitting after last element is greater than any other iterators, because it is assumed to have an index equal to last element's index plus one; The invalid iterator sitting before first element is less than any other iterators because it is assumed to have an index -1. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_ia.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_ia.md new file mode 100644 index 000000000..13724d2fd --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_ia.md @@ -0,0 +1,35 @@ +--- +title: "operator+=" +api-name: "operator+=" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_ia.html +--- +## operator+= + +### Function Details + +``` c +const self& operator+=(difference_type n) + +``` + +Move this iterator backward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move forward by \|n\| element. + +#### Return Value + +Reference to this iterator at new position. + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_incr.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_incr.md new file mode 100644 index 000000000..07694c038 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_incr.md @@ -0,0 +1,44 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Pre-increment. + +Move the iterator one element backward, so that the element it sits on has a bigger index. Use ++iter rather than iter++ where possible to avoid two useless iterator copy constructions. + +#### Return Value + +This iterator after incremented. + +``` c +self operator++(int) + +``` + +Post-increment. + +Move the iterator one element backward, so that the element it sits on has a bigger index. Use ++iter rather than iter++ where possible to avoid two useless iterator copy constructions. + +#### Return Value + +A new iterator not incremented. + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_le.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_le.md new file mode 100644 index 000000000..91b944c95 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_le.md @@ -0,0 +1,35 @@ +--- +title: "operator<=" +api-name: "operator<=" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_le.html +--- +## operator\<= + +### Function Details + +``` c +bool operator<=(const self &itr) const + +``` + +Less equal comparison operator. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +True if this iterator is less than or equal to itr. + +### Group: Iterator comparison operators + +The way to compare two iterators is to compare the index values of the two elements they point to. + +The iterator sitting on an element with less index is regarded to be smaller. And the invalid iterator sitting after last element is greater than any other iterators, because it is assumed to have an index equal to last element's index plus one; The invalid iterator sitting before first element is less than any other iterators because it is assumed to have an index -1. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_lt.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_lt.md new file mode 100644 index 000000000..396544279 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_lt.md @@ -0,0 +1,35 @@ +--- +title: "operator<" +api-name: "operator<" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_lt.html +--- +## operator\< + +### Function Details + +``` c +bool operator<(const self &itr) const + +``` + +Less than comparison operator. + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +True if this iterator is less than itr. + +### Group: Iterator comparison operators + +The way to compare two iterators is to compare the index values of the two elements they point to. + +The iterator sitting on an element with less index is regarded to be smaller. And the invalid iterator sitting after last element is greater than any other iterators, because it is assumed to have an index equal to last element's index plus one; The invalid iterator sitting before first element is less than any other iterators because it is assumed to have an index -1. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_sa.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_sa.md new file mode 100644 index 000000000..04a6429a9 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_sa.md @@ -0,0 +1,35 @@ +--- +title: "operator-=" +api-name: "operator-=" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_sa.html +--- +## operator-= + +### Function Details + +``` c +const self& operator-=(difference_type n) + +``` + +Move this iterator forward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move backward by \|n\| element. + +#### Return Value + +Reference to this iterator at new position. + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_sqbrk.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_sqbrk.md new file mode 100644 index 000000000..15f4837c9 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_sqbrk.md @@ -0,0 +1,31 @@ +--- +title: "operator[]" +api-name: "operator[]" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_sqbrk.html +--- +## operator\[\] + +### Function Details + +``` c +value_type_wrap operator[](difference_type _Off) const + +``` + +Iterator index operator. + +If \_Off not in a valid range, the returned value will be invalid. Note that you should use a value_type_wrap type to hold the returned value. + +#### Parameters + +##### \_Off + +The valid index relative to this iterator. + +#### Return Value + +Return the element which is at position \*this + \_Off. The returned value can only be used to read its referenced element. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_sub.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_sub.md new file mode 100644 index 000000000..4c1449ed3 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_sub.md @@ -0,0 +1,56 @@ +--- +title: "operator-" +api-name: "operator-" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_sub.html +--- +## operator- + +### Function Details + +``` c +self operator-(difference_type n) const + +``` + +Iterator movement operator. + +Return another iterator by moving this iterator backward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move backward by \|n\| element. + +#### Return Value + +The new iterator at new position. + +``` c +difference_type operator-(const self &itr) const + +``` + +Iterator distance operator. + +Return the index difference of this iterator and itr, so if this iterator sits on an element with a smaller index, this call will return a negative number. + +#### Parameters + +##### itr + +The other iterator to substract. itr can be the invalid iterator after last element or before first element, their index will be regarded as last element's index + 1 and -1 respectively. + +#### Return Value + +The index difference. + +### Group: Iterator movement operators. + +When we talk about iterator movement, we think the container is a uni-directional range, represented by \[begin, end), and this is true no matter we are using iterators or reverse iterators. + +When an iterator is moved closer to "begin", we say it is moved forward, otherwise we say it is moved backward. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratoroperator_ueq.md b/docs-src/api/stl/stldb_vector_base_iteratoroperator_ueq.md new file mode 100644 index 000000000..9d322657f --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratoroperator_ueq.md @@ -0,0 +1,35 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_vector_base_iteratoroperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const self &itr) const + +``` + +Unequal compare, identical to !operator(==itr). + +#### Parameters + +##### itr + +The iterator to compare against. + +#### Return Value + +False if this iterator equals to itr; True otherwise. + +### Group: Iterator comparison operators + +The way to compare two iterators is to compare the index values of the two elements they point to. + +The iterator sitting on an element with less index is regarded to be smaller. And the invalid iterator sitting after last element is greater than any other iterators, because it is assumed to have an index equal to last element's index plus one; The invalid iterator sitting before first element is less than any other iterators because it is assumed to have an index -1. + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratorrefresh.md b/docs-src/api/stl/stldb_vector_base_iteratorrefresh.md new file mode 100644 index 000000000..52f8fbf43 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratorrefresh.md @@ -0,0 +1,29 @@ +--- +title: "refresh" +api-name: "refresh" +source: docs/api_reference/STL/stldb_vector_base_iteratorrefresh.html +--- +## refresh + +### Function Details + +``` c +virtual int refresh(bool from_db=true) + +``` + +Refresh iterator cached value. + +#### Parameters + +##### from_db + +If not doing direct database get and this parameter is true, we will retrieve data directly from db. + +#### See Also + +db_base_iterator::refresh(bool) . + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_base_iteratorset_bulk_buffer.md b/docs-src/api/stl/stldb_vector_base_iteratorset_bulk_buffer.md new file mode 100644 index 000000000..fbde8d9a9 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_base_iteratorset_bulk_buffer.md @@ -0,0 +1,35 @@ +--- +title: "set_bulk_buffer" +api-name: "set_bulk_buffer" +source: docs/api_reference/STL/stldb_vector_base_iteratorset_bulk_buffer.html +--- +## set_bulk_buffer + +### Function Details + +``` c +bool set_bulk_buffer(u_int32_t sz) + +``` + +Modify bulk buffer size. + +Bulk read is enabled when creating an iterator, so you later can only modify the bulk buffer size to another value, but can't enable/disable bulk read while an iterator is already alive. + +#### Parameters + +##### sz + +The new size of the bulk read buffer of this iterator. + +#### Return Value + +Returns true if succeeded, false otherwise. + +#### See Also + +db_base_iterator::set_bulk_buffer(u_int32_t sz) + +### Class + +db_vector_base_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratordstr_db_vector_iterator.md b/docs-src/api/stl/stldb_vector_iteratordstr_db_vector_iterator.md new file mode 100644 index 000000000..a03440443 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratordstr_db_vector_iterator.md @@ -0,0 +1,23 @@ +--- +title: "~db_vector_iterator" +api-name: "~db_vector_iterator" +source: docs/api_reference/STL/stldb_vector_iteratordstr_db_vector_iterator.html +--- +## ~db_vector_iterator + +### Function Details + +``` c +virtual ~db_vector_iterator() + +``` + +### Group: Constructors and destructor + +Do not construct iterators explictily using these constructors, but call db_vector::begin to get an valid iterator. + +db_vector::begin + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator__star.md b/docs-src/api/stl/stldb_vector_iteratoroperator__star.md new file mode 100644 index 000000000..9c8470cd4 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator__star.md @@ -0,0 +1,25 @@ +--- +title: "operator *" +api-name: "operator *" +source: docs/api_reference/STL/stldb_vector_iteratoroperator__star.html +--- +## operator \* + +### Function Details + +``` c +reference operator *() const + +``` + +Dereference operator. + +Return the reference to the cached data element, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. The returned value can be used to read or update its referenced element. + +#### Return Value + +The reference to the element this iterator points to. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_add.md b/docs-src/api/stl/stldb_vector_iteratoroperator_add.md new file mode 100644 index 000000000..c8dd21391 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_add.md @@ -0,0 +1,39 @@ +--- +title: "operator+" +api-name: "operator+" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_add.html +--- +## operator+ + +### Function Details + +``` c +self operator+(difference_type n) const + +``` + +Iterator movement operator. + +Return another iterator by moving this iterator backward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move forward by \|n\| element. + +#### Return Value + +The new iterator at new position. + +#### See Also + +db_vector_base_iterator::operator+(difference_type n) const + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_arrow.md b/docs-src/api/stl/stldb_vector_iteratoroperator_arrow.md new file mode 100644 index 000000000..aa4add1a2 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_arrow.md @@ -0,0 +1,25 @@ +--- +title: "operator->" +api-name: "operator->" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_arrow.html +--- +## operator-\> + +### Function Details + +``` c +pointer operator->() const + +``` + +Arrow operator. + +Return the pointer to the cached data element, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. The returned value can be used to read or update its referenced element. + +#### Return Value + +The address of the referenced object. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_assign.md b/docs-src/api/stl/stldb_vector_iteratoroperator_assign.md new file mode 100644 index 000000000..a9238daef --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_assign.md @@ -0,0 +1,39 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &itr) + +``` + +Assignment operator. + +This iterator will point to the same key/data pair as itr, and have the same configurations as itr. + +#### Parameters + +##### itr + +The right value of the assignment. + +#### Return Value + +This iterator's reference. + +#### See Also + +db_base_iterator::operator=(const self&) + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_decr.md b/docs-src/api/stl/stldb_vector_iteratoroperator_decr.md new file mode 100644 index 000000000..1ef0b9a1b --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_decr.md @@ -0,0 +1,46 @@ +--- +title: "operator--" +api-name: "operator--" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_decr.html +--- +## operator-- + +### Function Details + +``` c +self& operator--() + +``` + +Pre-decrement. + +#### Return Value + +This iterator after decremented. + +#### See Also + +db_vector_base_iterator::operator--() + +``` c +self operator--(int) + +``` + +Post-decrement. + +#### Return Value + +A new iterator not decremented. + +#### See Also + +db_vector_base_iterator::operator--(int) + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_ia.md b/docs-src/api/stl/stldb_vector_iteratoroperator_ia.md new file mode 100644 index 000000000..23838c899 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_ia.md @@ -0,0 +1,37 @@ +--- +title: "operator+=" +api-name: "operator+=" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_ia.html +--- +## operator+= + +### Function Details + +``` c +const self& operator+=(difference_type n) + +``` + +Move this iterator backward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move forward by \|n\| element. + +#### Return Value + +Reference to this iterator at new position. + +#### See Also + +db_vector_base_iterator::operator+=(difference_type n) + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_incr.md b/docs-src/api/stl/stldb_vector_iteratoroperator_incr.md new file mode 100644 index 000000000..0740e3c74 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_incr.md @@ -0,0 +1,46 @@ +--- +title: "operator++" +api-name: "operator++" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_incr.html +--- +## operator++ + +### Function Details + +``` c +self& operator++() + +``` + +Pre-increment. + +#### Return Value + +This iterator after incremented. + +#### See Also + +db_vector_base_iterator::operator++() + +``` c +self operator++(int) + +``` + +Post-increment. + +#### Return Value + +A new iterator not incremented. + +#### See Also + +db_vector_base_iterator::operator++(int) + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_sa.md b/docs-src/api/stl/stldb_vector_iteratoroperator_sa.md new file mode 100644 index 000000000..bb91c432a --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_sa.md @@ -0,0 +1,37 @@ +--- +title: "operator-=" +api-name: "operator-=" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_sa.html +--- +## operator-= + +### Function Details + +``` c +const self& operator-=(difference_type n) + +``` + +Move this iterator forward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move backward by \|n\| element. + +#### Return Value + +Reference to this iterator at new position. + +#### See Also + +db_vector_base_iterator::operator-=(difference_type n) + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_sqbrk.md b/docs-src/api/stl/stldb_vector_iteratoroperator_sqbrk.md new file mode 100644 index 000000000..5a2d313be --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_sqbrk.md @@ -0,0 +1,31 @@ +--- +title: "operator[]" +api-name: "operator[]" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_sqbrk.html +--- +## operator\[\] + +### Function Details + +``` c +value_type_wrap operator[](difference_type _Off) const + +``` + +Iterator index operator. + +If \_Off not in a valid range, the returned value will be invalid. Note that you should use a value_type_wrap type to hold the returned value. + +#### Parameters + +##### \_Off + +The valid index relative to this iterator. + +#### Return Value + +Return the element which is at position \*this + \_Off, which is an ElementRef\ object if T is a class type or an ElementHolder\ object if T is a C++ primitive data type. The returned value can be used to read or update its referenced element. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratoroperator_sub.md b/docs-src/api/stl/stldb_vector_iteratoroperator_sub.md new file mode 100644 index 000000000..c2578568a --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratoroperator_sub.md @@ -0,0 +1,62 @@ +--- +title: "operator-" +api-name: "operator-" +source: docs/api_reference/STL/stldb_vector_iteratoroperator_sub.html +--- +## operator- + +### Function Details + +``` c +self operator-(difference_type n) const + +``` + +Iterator movement operator. + +Return another iterator by moving this iterator forward by n elements. + +#### Parameters + +##### n + +The amount and direction of movement. If negative, will move backward by \|n\| element. + +#### Return Value + +The new iterator at new position. + +#### See Also + +db_vector_base_iterator::operator-(difference_type n) const + +``` c +difference_type operator-(const self &itr) const + +``` + +Iterator distance operator. + +Return the index difference of this iterator and itr, so if this iterator sits on an element with a smaller index, this call will return a negative number. + +#### Parameters + +##### itr + +The other iterator to substract. itr can be the invalid iterator after last element or before first element, their index will be regarded as last element's index + 1 and -1 respectively. + +#### Return Value + +The index difference. + +#### See Also + +db_vector_base_iterator::operator-(const self& itr) const + +### Group: Iterator movement operators. + +These functions have identical behaviors and semantics as those of db_vector_base_iterator , so please refer to equivalent in that class. + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vector_iteratorrefresh.md b/docs-src/api/stl/stldb_vector_iteratorrefresh.md new file mode 100644 index 000000000..b50ee04a3 --- /dev/null +++ b/docs-src/api/stl/stldb_vector_iteratorrefresh.md @@ -0,0 +1,29 @@ +--- +title: "refresh" +api-name: "refresh" +source: docs/api_reference/STL/stldb_vector_iteratorrefresh.html +--- +## refresh + +### Function Details + +``` c +virtual int refresh(bool from_db=true) + +``` + +Refresh iterator cached value. + +#### Parameters + +##### from_db + +If not doing direct database get and this parameter is true, we will retrieve data directly from db. + +#### See Also + +db_base_iterator::refresh(bool) + +### Class + +db_vector_iterator diff --git a/docs-src/api/stl/stldb_vectorassign.md b/docs-src/api/stl/stldb_vectorassign.md new file mode 100644 index 000000000..d8a4685bc --- /dev/null +++ b/docs-src/api/stl/stldb_vectorassign.md @@ -0,0 +1,86 @@ +--- +title: "assign" +api-name: "assign" +source: docs/api_reference/STL/stldb_vectorassign.html +--- +## assign + +### Function Details + +``` c +void assign(InputIterator first, InputIterator last, + bool b_truncate=true) + +``` + +Assign a range \[first, last) to this container. + +#### Parameters + +##### b_truncate + +See its member group doc for details. + +##### last + +The range open boundary. + +##### first + +The range closed boundary. + +``` c +void assign(const_iterator first, const_iterator last, + bool b_truncate=true) + +``` + +Assign a range \[first, last) to this container. + +#### Parameters + +##### b_truncate + +See its member group doc for details. + +##### last + +The range open boundary. + +##### first + +The range closed boundary. + +``` c +void assign(size_type n, const T &u, + bool b_truncate=true) + +``` + +Assign n number of elements of value u into this container. + +#### Parameters + +##### b_truncate + +See its member group doc for details. This function supports auto-commit. + +##### u + +The value of elements to insert. + +##### n + +The number of elements in this container after the call. + +### Group: Assign functions + +See the function documentation for the correct usage of b_truncate parameter. + +The following four member functions have default parameter b_truncate, because they require all key/data pairs in the database be deleted before the real operation, and by default we use Db::truncate to truncate the database rather than delete the key/data pairs one by one, but Db::truncate requirs no open cursors on the database handle, and the four member functions will close any open cursors of backing database handle in current thread, but can do nothing to cursors of other threads opened from the same database handle. So you must make sure there are no open cursors of the database handle in any other threads. On the other hand, users can specify "false" to the b_truncate parameter and thus the key/data pairs will be deleted one by one. Other than that, they have identical behaviors as their counterparts in std::vector. + +http://www.cplusplus.com/reference/stl/vector/assign/. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorat.md b/docs-src/api/stl/stldb_vectorat.md new file mode 100644 index 000000000..1247baaf4 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorat.md @@ -0,0 +1,62 @@ +--- +title: "at" +api-name: "at" +source: docs/api_reference/STL/stldb_vectorat.html +--- +## at + +### Function Details + +``` c +reference at(index_type n) + +``` + +Index function. + +#### Parameters + +##### n + +The valid index of the vector. + +#### Return Value + +The reference to the element at specified position, can act as both a left value and a right value. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/at/ + +``` c +const_reference at(index_type n) const + +``` + +Read only index function. + +Only used as a right value, no need for assignment capability. The return value can't be used to update the element. + +#### Parameters + +##### n + +The valid index of the vector. + +#### Return Value + +The const reference to the element at specified position. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/at/ + +### Group: Element access functions. + +The operator\[\] and at() only come from std::vector and std::deque, If you are using db_vector as std::list, you don't have to set DB_RENUMBER flag to the backing database handle, and you get better performance, but at the same time you can't use these functions. + +Otherwise if you have set the DB_RENUMBER flag to the backing database handle, you can use this function though it is an std::list equivalent. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorback.md b/docs-src/api/stl/stldb_vectorback.md new file mode 100644 index 000000000..2ad469a15 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorback.md @@ -0,0 +1,50 @@ +--- +title: "back" +api-name: "back" +source: docs/api_reference/STL/stldb_vectorback.html +--- +## back + +### Function Details + +``` c +reference back() + +``` + +Return a reference to the last element. + +#### Return Value + +Return a reference to the last element. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/back/ + +``` c +const_reference back() const + +``` + +Return a reference to the last element. + +The return value can't be used to update the element. + +#### Return Value + +Return a reference to the last element. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/back/ + +### Group: Element access functions. + +The operator\[\] and at() only come from std::vector and std::deque, If you are using db_vector as std::list, you don't have to set DB_RENUMBER flag to the backing database handle, and you get better performance, but at the same time you can't use these functions. + +Otherwise if you have set the DB_RENUMBER flag to the backing database handle, you can use this function though it is an std::list equivalent. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorcapacity.md b/docs-src/api/stl/stldb_vectorcapacity.md new file mode 100644 index 000000000..1a47e6d68 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorcapacity.md @@ -0,0 +1,25 @@ +--- +title: "capacity" +api-name: "capacity" +source: docs/api_reference/STL/stldb_vectorcapacity.html +--- +## capacity + +### Function Details + +``` c +size_type capacity() const + +``` + +Get capacity. + +### Group: Huge return + +These two functions return 2^30, denoting a huge number that does not overflow, because dbstl does not have to manage memory space. + +But the return value is not the real limit, see the Berkeley DB database limits for the limits. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorclear.md b/docs-src/api/stl/stldb_vectorclear.md new file mode 100644 index 000000000..3b0912fc5 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorclear.md @@ -0,0 +1,31 @@ +--- +title: "clear" +api-name: "clear" +source: docs/api_reference/STL/stldb_vectorclear.html +--- +## clear + +### Function Details + +``` c +void clear(bool b_truncate=true) + +``` + +Remove all elements of the vector, make it an empty vector. + +This function supports auto-commit. + +#### Parameters + +##### b_truncate + +Same as that of db_vector::assign() . + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/clear/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectordb_vector.md b/docs-src/api/stl/stldb_vectordb_vector.md new file mode 100644 index 000000000..c45f15b61 --- /dev/null +++ b/docs-src/api/stl/stldb_vectordb_vector.md @@ -0,0 +1,145 @@ +--- +title: "db_vector" +api-name: "db_vector" +source: docs/api_reference/STL/stldb_vectordb_vector.html +--- +## db_vector + +### Function Details + +``` c +db_vector(Db *dbp=NULL, + DbEnv *penv=NULL) + +``` + +Constructor. + +Note that we do not need an allocator in db-stl containser, but we need backing up Db\* and DbEnv\*, and we have to verify that the passed in bdb handles are valid for use by the container class. See class detail for handle requirement. + +#### Parameters + +##### dbp + +The same as that of db_container(Db*, DbEnv*) ; + +##### penv + +The same as that of db_container(Db*, DbEnv*) ; + +#### See Also + +db_container(Db*, DbEnv*) ; + +``` c +db_vector(size_type n, const T &val=T(), Db *dbp=NULL, + DbEnv *penv=NULL) + +``` + +Constructor. + +This function supports auto-commit. Insert n elements of T type into the database, the value of the elements is the default value or user set value. See class detail for handle requirement. + +#### Parameters + +##### dbp + +The same as that of db_container(Db*, DbEnv*) ; + +##### penv + +The same as that of db_container(Db*, DbEnv*) ; + +##### val + +The value of elements to insert. + +##### n + +The number of elements to insert. + +#### See Also + +db_vector(Db*, DbEnv*) ; db_container(Db*, DbEnv*) ; + +``` c +db_vector(const self &x) + +``` + +Copy constructor. + +This function supports auto-commit. Insert all elements in x into this container. + +#### See Also + +db_container(const db_container&) + +``` c +db_vector(Db *dbp, DbEnv *penv, InputIterator first, + InputIterator last) + +``` + +Insert a range of elements into this container. + +The range is \[first, last), which contains elements that can be converted to type T automatically. See class detail for handle requirement. + +#### Parameters + +##### dbp + +The same as that of db_container(Db*, DbEnv*) ; + +##### first + +Range closed boundary. + +##### last + +Range open boundary. + +##### penv + +The same as that of db_container(Db*, DbEnv*) ; + +#### See Also + +db_vector(Db*, DbEnv*) ; + +``` c +db_vector(const_iterator first, const_iterator last, Db *dbp=NULL, + DbEnv *penv=NULL) + +``` + +Range constructor. + +This function supports auto-commit. Insert the range of elements in \[first, last) into this container. See class detail for handle requirement. + +#### Parameters + +##### dbp + +The same as that of db_container(Db*, DbEnv*) ; + +##### first + +Range closed boundary. + +##### last + +Range open boundary. + +##### penv + +The same as that of db_container(Db*, DbEnv*) ; + +#### See Also + +db_vector(Db*, DbEnv*) ; + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectordstr_db_vector.md b/docs-src/api/stl/stldb_vectordstr_db_vector.md new file mode 100644 index 000000000..544a2527d --- /dev/null +++ b/docs-src/api/stl/stldb_vectordstr_db_vector.md @@ -0,0 +1,17 @@ +--- +title: "~db_vector" +api-name: "~db_vector" +source: docs/api_reference/STL/stldb_vectordstr_db_vector.html +--- +## ~db_vector + +### Function Details + +``` c +virtual ~db_vector() + +``` + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorempty.md b/docs-src/api/stl/stldb_vectorempty.md new file mode 100644 index 000000000..ba909cfd4 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorempty.md @@ -0,0 +1,23 @@ +--- +title: "empty" +api-name: "empty" +source: docs/api_reference/STL/stldb_vectorempty.html +--- +## empty + +### Function Details + +``` c +bool empty() const + +``` + +Returns whether this container is empty. + +#### Return Value + +True if empty, false otherwise. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorend.md b/docs-src/api/stl/stldb_vectorend.md new file mode 100644 index 000000000..1fcfccdac --- /dev/null +++ b/docs-src/api/stl/stldb_vectorend.md @@ -0,0 +1,34 @@ +--- +title: "end" +api-name: "end" +source: docs/api_reference/STL/stldb_vectorend.html +--- +## end + +### Function Details + +``` c +iterator end() + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid iterator denoting the position after the last valid element of the container. + +``` c +const_iterator end() const + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid const iterator denoting the position after the last valid element of the container. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorerase.md b/docs-src/api/stl/stldb_vectorerase.md new file mode 100644 index 000000000..3d6706c7b --- /dev/null +++ b/docs-src/api/stl/stldb_vectorerase.md @@ -0,0 +1,57 @@ +--- +title: "erase" +api-name: "erase" +source: docs/api_reference/STL/stldb_vectorerase.html +--- +## erase + +### Function Details + +``` c +iterator erase(iterator pos) + +``` + +Erase element at position pos. + +#### Parameters + +##### pos + +The valid position in the container's range to erase. + +#### Return Value + +The next position after the erased element. + +``` c +iterator erase(iterator first, + iterator last) + +``` + +Erase elements in range \[first, last). + +#### Parameters + +##### last + +The open boundary of the range. + +##### first + +The closed boundary of the range. + +#### Return Value + +The next position after the erased elements. + +### Group: Erase functions + +The iterator pos in the functions must be a read-write iterator, can't be read only. + +http://www.cplusplus.com/reference/stl/vector/erase/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorfront.md b/docs-src/api/stl/stldb_vectorfront.md new file mode 100644 index 000000000..b74ba4b1f --- /dev/null +++ b/docs-src/api/stl/stldb_vectorfront.md @@ -0,0 +1,50 @@ +--- +title: "front" +api-name: "front" +source: docs/api_reference/STL/stldb_vectorfront.html +--- +## front + +### Function Details + +``` c +reference front() + +``` + +Return a reference to the first element. + +#### Return Value + +Return a reference to the first element. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/front/ + +``` c +const_reference front() const + +``` + +Return a const reference to the first element. + +The return value can't be used to update the element. + +#### Return Value + +Return a const reference to the first element. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/front/ + +### Group: Element access functions. + +The operator\[\] and at() only come from std::vector and std::deque, If you are using db_vector as std::list, you don't have to set DB_RENUMBER flag to the backing database handle, and you get better performance, but at the same time you can't use these functions. + +Otherwise if you have set the DB_RENUMBER flag to the backing database handle, you can use this function though it is an std::list equivalent. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorinsert.md b/docs-src/api/stl/stldb_vectorinsert.md new file mode 100644 index 000000000..e8f46c825 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorinsert.md @@ -0,0 +1,106 @@ +--- +title: "insert" +api-name: "insert" +source: docs/api_reference/STL/stldb_vectorinsert.html +--- +## insert + +### Function Details + +``` c +iterator insert(iterator pos, + const T &x) + +``` + +Insert x before position pos. + +#### Parameters + +##### x + +The element to insert. + +##### pos + +The position before which to insert. + +``` c +void insert(iterator pos, size_type n, + const T &x) + +``` + +Insert n number of elements x before position pos. + +#### Parameters + +##### x + +The element to insert. + +##### pos + +The position before which to insert. + +##### n + +The number of elements to insert. + +``` c +void insert(iterator pos, InputIterator first, + InputIterator last) + +``` + +Range insertion. + +Insert elements in range \[first, last) into this vector before position pos. + +#### Parameters + +##### last + +The open boundary of the range. + +##### pos + +The position before which to insert. + +##### first + +The closed boundary of the range. + +``` c +void insert(iterator pos, const_iterator first, + const_iterator last) + +``` + +Range insertion. + +Insert elements in range \[first, last) into this vector before position pos. + +#### Parameters + +##### last + +The open boundary of the range. + +##### pos + +The position before which to insert. + +##### first + +The closed boundary of the range. + +### Group: Insert functions + +The iterator pos in the functions must be a read-write iterator, can't be read only. + +http://www.cplusplus.com/reference/stl/vector/insert/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectormax_size.md b/docs-src/api/stl/stldb_vectormax_size.md new file mode 100644 index 000000000..ae9b64faa --- /dev/null +++ b/docs-src/api/stl/stldb_vectormax_size.md @@ -0,0 +1,31 @@ +--- +title: "max_size" +api-name: "max_size" +source: docs/api_reference/STL/stldb_vectormax_size.html +--- +## max_size + +### Function Details + +``` c +size_type max_size() const + +``` + +Get max size. + +The returned size is not the actual limit of database. See the Berkeley DB limits to get real max size. + +#### Return Value + +A meaningless huge number. + +### Group: Huge return + +These two functions return 2^30, denoting a huge number that does not overflow, because dbstl does not have to manage memory space. + +But the return value is not the real limit, see the Berkeley DB database limits for the limits. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectormerge.md b/docs-src/api/stl/stldb_vectormerge.md new file mode 100644 index 000000000..c41746da7 --- /dev/null +++ b/docs-src/api/stl/stldb_vectormerge.md @@ -0,0 +1,59 @@ +--- +title: "merge" +api-name: "merge" +source: docs/api_reference/STL/stldb_vectormerge.html +--- +## merge + +### Function Details + +``` c +void merge(self &x) + +``` + +Merge content with another container. + +This function supports auto-commit. + +#### Parameters + +##### x + +The other list to merge with. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/merge/ + +``` c +void merge(self &x, + Compare comp) + +``` + +Merge content with another container. + +This function supports auto-commit. + +#### Parameters + +##### x + +The other list to merge with. + +##### comp + +The compare function to determine insertion position. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/merge/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectoroperator_assign.md b/docs-src/api/stl/stldb_vectoroperator_assign.md new file mode 100644 index 000000000..5f2bfefe3 --- /dev/null +++ b/docs-src/api/stl/stldb_vectoroperator_assign.md @@ -0,0 +1,31 @@ +--- +title: "operator=" +api-name: "operator=" +source: docs/api_reference/STL/stldb_vectoroperator_assign.html +--- +## operator= + +### Function Details + +``` c +const self& operator=(const self &x) + +``` + +Container assignment operator. + +This function supports auto-commit. This db_vector is assumed to be valid for use, only copy content of x into this container. + +#### Parameters + +##### x + +The right value container. + +#### Return Value + +The container x's reference. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectoroperator_eq.md b/docs-src/api/stl/stldb_vectoroperator_eq.md new file mode 100644 index 000000000..ce3ccddd1 --- /dev/null +++ b/docs-src/api/stl/stldb_vectoroperator_eq.md @@ -0,0 +1,49 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldb_vectoroperator_eq.html +--- +## operator== + +### Function Details + +``` c +bool operator==(const db_vector< T2, + T3 > &v2) const + +``` + +Container equality comparison operator. + +This function supports auto-commit. + +#### Parameters + +##### v2 + +The vector to compare against. + +#### Return Value + +Compare two vectors, return true if they have identical sequences of elements, otherwise return false. + +``` c +bool operator==(const self &v2) const + +``` + +Container equality comparison operator. + +This function supports auto-commit. + +#### Return Value + +Compare two vectors, return true if they have identical elements, otherwise return false. + +### Group: Compare functions. + +http://www.sgi.com/tech/stl/Vector.html + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectoroperator_lt.md b/docs-src/api/stl/stldb_vectoroperator_lt.md new file mode 100644 index 000000000..fbfa64a40 --- /dev/null +++ b/docs-src/api/stl/stldb_vectoroperator_lt.md @@ -0,0 +1,35 @@ +--- +title: "operator<" +api-name: "operator<" +source: docs/api_reference/STL/stldb_vectoroperator_lt.html +--- +## operator\< + +### Function Details + +``` c +bool operator<(const self &v2) const + +``` + +Container less than comparison operator. + +This function supports auto-commit. + +#### Parameters + +##### v2 + +The container to compare against. + +#### Return Value + +Compare two vectors, return true if this is less than v2, otherwise return false. + +### Group: Compare functions. + +http://www.sgi.com/tech/stl/Vector.html + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectoroperator_sqbrk.md b/docs-src/api/stl/stldb_vectoroperator_sqbrk.md new file mode 100644 index 000000000..fff45d25d --- /dev/null +++ b/docs-src/api/stl/stldb_vectoroperator_sqbrk.md @@ -0,0 +1,54 @@ +--- +title: "operator[]" +api-name: "operator[]" +source: docs/api_reference/STL/stldb_vectoroperator_sqbrk.html +--- +## operator\[\] + +### Function Details + +``` c +reference operator[](index_type n) + +``` + +Index operator, can act as both a left value and a right value. + +#### Parameters + +##### n + +The valid index of the vector. + +#### Return Value + +The reference to the element at specified position. + +``` c +const_reference operator[](index_type n) const + +``` + +Read only index operator. + +Only used as a right value, no need for assignment capability. The return value can't be used to update the element. + +#### Parameters + +##### n + +The valid index of the vector. + +#### Return Value + +The const reference to the element at specified position. + +### Group: Element access functions. + +The operator\[\] and at() only come from std::vector and std::deque, If you are using db_vector as std::list, you don't have to set DB_RENUMBER flag to the backing database handle, and you get better performance, but at the same time you can't use these functions. + +Otherwise if you have set the DB_RENUMBER flag to the backing database handle, you can use this function though it is an std::list equivalent. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectoroperator_ueq.md b/docs-src/api/stl/stldb_vectoroperator_ueq.md new file mode 100644 index 000000000..08c70213b --- /dev/null +++ b/docs-src/api/stl/stldb_vectoroperator_ueq.md @@ -0,0 +1,55 @@ +--- +title: "operator!=" +api-name: "operator!=" +source: docs/api_reference/STL/stldb_vectoroperator_ueq.html +--- +## operator!= + +### Function Details + +``` c +bool operator!=(const db_vector< T2, + T3 > &v2) const + +``` + +Container in-equality comparison operator. + +This function supports auto-commit. + +#### Parameters + +##### v2 + +The vector to compare against. + +#### Return Value + +Returns false if elements in each slot of both containers equal; Returns true otherwise. + +``` c +bool operator!=(const self &v2) const + +``` + +Container in-equality comparison operator. + +This function supports auto-commit. + +#### Parameters + +##### v2 + +The vector to compare against. + +#### Return Value + +Returns false if elements in each slot of both containers equal; Returns true otherwise. + +### Group: Compare functions. + +http://www.sgi.com/tech/stl/Vector.html + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorpop_back.md b/docs-src/api/stl/stldb_vectorpop_back.md new file mode 100644 index 000000000..3153be7ca --- /dev/null +++ b/docs-src/api/stl/stldb_vectorpop_back.md @@ -0,0 +1,25 @@ +--- +title: "pop_back" +api-name: "pop_back" +source: docs/api_reference/STL/stldb_vectorpop_back.html +--- +## pop_back + +### Function Details + +``` c +void pop_back() + +``` + +Pop out last element from the vector. + +This function supports auto-commit. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/pop_back/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorpop_front.md b/docs-src/api/stl/stldb_vectorpop_front.md new file mode 100644 index 000000000..44b631329 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorpop_front.md @@ -0,0 +1,27 @@ +--- +title: "pop_front" +api-name: "pop_front" +source: docs/api_reference/STL/stldb_vectorpop_front.html +--- +## pop_front + +### Function Details + +``` c +void pop_front() + +``` + +Pop out the front element from the vector. + +This function supports auto-commit. + +### Group: Functions specific to deque and list + +These functions come from std::list and std::deque, and have identical behaviors to their counterparts in std::list/stddeque. + +http://www.cplusplus.com/reference/stl/deque/pop_front/ http://www.cplusplus.com/reference/stl/deque/push_front/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorpush_back.md b/docs-src/api/stl/stldb_vectorpush_back.md new file mode 100644 index 000000000..af4ef0f81 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorpush_back.md @@ -0,0 +1,31 @@ +--- +title: "push_back" +api-name: "push_back" +source: docs/api_reference/STL/stldb_vectorpush_back.html +--- +## push_back + +### Function Details + +``` c +void push_back(const T &x) + +``` + +Push back an element into the vector. + +This function supports auto-commit. + +#### Parameters + +##### x + +The value of element to push into this vector. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/push_back/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorpush_front.md b/docs-src/api/stl/stldb_vectorpush_front.md new file mode 100644 index 000000000..a1e5b6f8a --- /dev/null +++ b/docs-src/api/stl/stldb_vectorpush_front.md @@ -0,0 +1,31 @@ +--- +title: "push_front" +api-name: "push_front" +source: docs/api_reference/STL/stldb_vectorpush_front.html +--- +## push_front + +### Function Details + +``` c +void push_front(const T &x) + +``` + +Push an element x into the vector from front. + +#### Parameters + +##### x + +The element to push into this vector. This function supports auto-commit. + +### Group: Functions specific to deque and list + +These functions come from std::list and std::deque, and have identical behaviors to their counterparts in std::list/stddeque. + +http://www.cplusplus.com/reference/stl/deque/pop_front/ http://www.cplusplus.com/reference/stl/deque/push_front/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorrbegin.md b/docs-src/api/stl/stldb_vectorrbegin.md new file mode 100644 index 000000000..fa7c56317 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorrbegin.md @@ -0,0 +1,79 @@ +--- +title: "rbegin" +api-name: "rbegin" +source: docs/api_reference/STL/stldb_vectorrbegin.html +--- +## rbegin + +### Function Details + +``` c +reverse_iterator rbegin(ReadModifyWriteOption rmw= + ReadModifyWriteOption::no_read_modify_write(), bool readonly=false, + BulkRetrievalOption bulk_read=BulkRetrievalOption::no_bulk_retrieval(), + bool directdb_get=true) + +``` + +Create a reverse iterator. + +This function creates a reverse iterator initialized to sit on the last element in the underlying database, and can be used to read/write. The meaning and usage of its parameters are identical to the above begin function. + +#### Parameters + +##### directdb_get + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulk_read + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### rmw + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### readonly + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created iterator. + +#### See Also + +begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +``` c +const_reverse_iterator rbegin(BulkRetrievalOption bulkretrieval= + BulkRetrievalOption(BulkRetrievalOption::no_bulk_retrieval()), + bool directdb_get=true) const + +``` + +Create a const reverse iterator. + +This function creates a const reverse iterator initialized to sit on the last element in the backing database, and can only read the element, it is only available to const db_vector containers. The meaning and usage of its parameters are identical as above. + +#### Parameters + +##### directdb_get + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +##### bulkretrieval + +Same as that of begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +#### Return Value + +The created iterator. + +#### See Also + +begin(ReadModifyWrite, bool, BulkRetrievalOption, bool); + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorremove.md b/docs-src/api/stl/stldb_vectorremove.md new file mode 100644 index 000000000..6d5836222 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorremove.md @@ -0,0 +1,35 @@ +--- +title: "remove" +api-name: "remove" +source: docs/api_reference/STL/stldb_vectorremove.html +--- +## remove + +### Function Details + +``` c +void remove(const T &value) + +``` + +Remove all elements whose values are "value" from the list. + +This function supports auto-commit. + +#### Parameters + +##### value + +The target value to remove. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/remove/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorremove_if.md b/docs-src/api/stl/stldb_vectorremove_if.md new file mode 100644 index 000000000..765278325 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorremove_if.md @@ -0,0 +1,35 @@ +--- +title: "remove_if" +api-name: "remove_if" +source: docs/api_reference/STL/stldb_vectorremove_if.html +--- +## remove_if + +### Function Details + +``` c +void remove_if(Predicate pred) + +``` + +Remove all elements making "pred" return true. + +This function supports auto-commit. + +#### Parameters + +##### pred + +The binary predicate judging elements in this list. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/remove_if/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorrend.md b/docs-src/api/stl/stldb_vectorrend.md new file mode 100644 index 000000000..3e055764e --- /dev/null +++ b/docs-src/api/stl/stldb_vectorrend.md @@ -0,0 +1,34 @@ +--- +title: "rend" +api-name: "rend" +source: docs/api_reference/STL/stldb_vectorrend.html +--- +## rend + +### Function Details + +``` c +reverse_iterator rend() + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid iterator denoting the position before the first valid element of the container. + +``` c +const_reverse_iterator rend() const + +``` + +Create an open boundary iterator. + +#### Return Value + +Returns an invalid const iterator denoting the position before the first valid element of the container. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorreserve.md b/docs-src/api/stl/stldb_vectorreserve.md new file mode 100644 index 000000000..13a7b2a25 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorreserve.md @@ -0,0 +1,21 @@ +--- +title: "reserve" +api-name: "reserve" +source: docs/api_reference/STL/stldb_vectorreserve.html +--- +## reserve + +### Function Details + +``` c +void reserve(size_type) + +``` + +Reserve space. + +The vector is backed by Berkeley DB, we always have enough space. This function does nothing, because dbstl does not have to manage memory space. + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorresize.md b/docs-src/api/stl/stldb_vectorresize.md new file mode 100644 index 000000000..c7bcf62e6 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorresize.md @@ -0,0 +1,36 @@ +--- +title: "resize" +api-name: "resize" +source: docs/api_reference/STL/stldb_vectorresize.html +--- +## resize + +### Function Details + +``` c +void resize(size_type n, + T t=T()) + +``` + +Resize this container to specified size n, insert values t if need to enlarge the container. + +This function supports auto-commit. + +#### Parameters + +##### t + +The value to insert when enlarging the container. + +##### n + +The number of elements in this container after the call. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/resize/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorreverse.md b/docs-src/api/stl/stldb_vectorreverse.md new file mode 100644 index 000000000..e3834a83e --- /dev/null +++ b/docs-src/api/stl/stldb_vectorreverse.md @@ -0,0 +1,29 @@ +--- +title: "reverse" +api-name: "reverse" +source: docs/api_reference/STL/stldb_vectorreverse.html +--- +## reverse + +### Function Details + +``` c +void reverse() + +``` + +Reverse this list. + +This function supports auto-commit. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/reverse/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorsize.md b/docs-src/api/stl/stldb_vectorsize.md new file mode 100644 index 000000000..1d7a73a0a --- /dev/null +++ b/docs-src/api/stl/stldb_vectorsize.md @@ -0,0 +1,23 @@ +--- +title: "size" +api-name: "size" +source: docs/api_reference/STL/stldb_vectorsize.html +--- +## size + +### Function Details + +``` c +size_type size() const + +``` + +Return the number of elements in this container. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/size/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorsort.md b/docs-src/api/stl/stldb_vectorsort.md new file mode 100644 index 000000000..41c590fea --- /dev/null +++ b/docs-src/api/stl/stldb_vectorsort.md @@ -0,0 +1,48 @@ +--- +title: "sort" +api-name: "sort" +source: docs/api_reference/STL/stldb_vectorsort.html +--- +## sort + +### Function Details + +``` c +void sort() + +``` + +Sort this list. + +This function supports auto-commit. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/sort/ + +``` c +void sort(Compare comp) + +``` + +Sort this list. + +This function supports auto-commit. + +#### Parameters + +##### comp + +The compare operator to determine element order. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/sort/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorsplice.md b/docs-src/api/stl/stldb_vectorsplice.md new file mode 100644 index 000000000..f58a5d21a --- /dev/null +++ b/docs-src/api/stl/stldb_vectorsplice.md @@ -0,0 +1,100 @@ +--- +title: "splice" +api-name: "splice" +source: docs/api_reference/STL/stldb_vectorsplice.html +--- +## splice + +### Function Details + +``` c +void splice(iterator position, + self &x) + +``` + +Moves elements from list x into this list. + +Moves all elements in list x into this list container at the specified position, effectively inserting the specified elements into the container and removing them from x. This function supports auto-commit. + +#### Parameters + +##### position + +Position within the container where the elements of x are inserted. + +##### x + +The other list container to splice from. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/splice/ + +``` c +void splice(iterator position, self &x, + iterator i) + +``` + +Moves elements from list x into this list. + +Moves elements at position i of list x into this list container at the specified position, effectively inserting the specified elements into the container and removing them from x. This function supports auto-commit. + +#### Parameters + +##### i + +The position of element in x to move into this list. + +##### position + +Position within the container where the elements of x are inserted. + +##### x + +The other list container to splice from. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/splice/ + +``` c +void splice(iterator position, self &x, iterator first, + iterator last) + +``` + +Moves elements from list x into this list. + +Moves elements in range \[first, last) of list x into this list container at the specified position, effectively inserting the specified elements into the container and removing them from x. This function supports auto-commit. + +#### Parameters + +##### position + +Position within the container where the elements of x are inserted. + +##### first + +The range's closed boundary. + +##### last + +The range's open boundary. + +##### x + +The other list container to splice from. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/splice/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorswap.md b/docs-src/api/stl/stldb_vectorswap.md new file mode 100644 index 000000000..442746550 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorswap.md @@ -0,0 +1,29 @@ +--- +title: "swap" +api-name: "swap" +source: docs/api_reference/STL/stldb_vectorswap.html +--- +## swap + +### Function Details + +``` c +void swap(self &vec) + +``` + +Swap content with another vector vec. + +#### Parameters + +##### vec + +The other vector to swap content with. This function supports auto-commit. + +#### See Also + +http://www.cplusplus.com/reference/stl/vector/swap/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldb_vectorunique.md b/docs-src/api/stl/stldb_vectorunique.md new file mode 100644 index 000000000..0612ee0a9 --- /dev/null +++ b/docs-src/api/stl/stldb_vectorunique.md @@ -0,0 +1,48 @@ +--- +title: "unique" +api-name: "unique" +source: docs/api_reference/STL/stldb_vectorunique.html +--- +## unique + +### Function Details + +``` c +void unique() + +``` + +Remove consecutive duplicate values from this list. + +This function supports auto-commit. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/unique/ + +``` c +void unique(BinaryPredicate binary_pred) + +``` + +Remove consecutive duplicate values from this list. + +This function supports auto-commit. + +#### Parameters + +##### binary_pred + +The compare predicate to dertermine uniqueness. + +#### See Also + +http://www.cplusplus.com/reference/stl/list/unique/ + +### Group: std::list specific functions + +http://www.cplusplus.com/reference/stl/list/ + +### Class + +db_vector diff --git a/docs-src/api/stl/stldbstl_global_functionsabort_txn.md b/docs-src/api/stl/stldbstl_global_functionsabort_txn.md new file mode 100644 index 000000000..8238f7691 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsabort_txn.md @@ -0,0 +1,61 @@ +--- +title: "abort_txn" +api-name: "abort_txn" +source: docs/api_reference/STL/stldbstl_global_functionsabort_txn.html +--- +## abort_txn + +### Function Details + +``` c + void abort_txn(DbEnv *env) + +``` + +Abort current transaction of environment "env". + +This function is called by dbstl user to abort an outside explicit transaction. + +#### Parameters + +##### env + +The environment whose current transaction is to be aborted. + +#### See Also + +abort_txn(DbEnv *, DbTxn *) ; + +``` c + void abort_txn(DbEnv *env, + DbTxn *txn) + +``` + +Abort specified transaction "txn" and all its child transactions. + +That is, "txn" can be a parent transaction of a nested transaction group. + +#### Parameters + +##### txn + +The transaction to abort, can be a parent transaction of a nested transaction group, all child transactions of it will be aborted. + +##### env + +The environment where txn is started from. + +#### See Also + +abort_txn(DbEnv *) ; + +### Group: Transaction control global functions. + +dbstl transaction API. + +You should call these API rather than DB C/C++ API to use Berkeley DB transaction features. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsalloc_mutex.md b/docs-src/api/stl/stldbstl_global_functionsalloc_mutex.md new file mode 100644 index 000000000..6ba330658 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsalloc_mutex.md @@ -0,0 +1,29 @@ +--- +title: "alloc_mutex" +api-name: "alloc_mutex" +source: docs/api_reference/STL/stldbstl_global_functionsalloc_mutex.html +--- +## alloc_mutex + +### Function Details + +``` c + db_mutex_t alloc_mutex() + +``` + +Allocate a Berkeley DB mutex. + +#### Return Value + +Berkeley DB mutex handle. + +### Group: Mutex API based on Berkeley DB mutex. + +These functions are in-process mutex support which uses Berkeley DB mutex mechanisms. + +You can call these functions to do portable synchronization for your code. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsbegin_txn.md b/docs-src/api/stl/stldbstl_global_functionsbegin_txn.md new file mode 100644 index 000000000..9592b344c --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsbegin_txn.md @@ -0,0 +1,42 @@ +--- +title: "begin_txn" +api-name: "begin_txn" +source: docs/api_reference/STL/stldbstl_global_functionsbegin_txn.html +--- +## begin_txn + +### Function Details + +``` c + DbTxn* begin_txn(u_int32_t flags, + DbEnv *env) + +``` + +Begin a new transaction from the specified environment "env". + +This function is called by dbstl user to begin an external transaction. The "flags" parameter is passed to DbEnv::txn_begin(). If a transaction created from the same database environment already exists and is unresolved, the new transaction is started as a child transaction of that transaction, and thus you can't specify the parent transaction. + +#### Parameters + +##### flags + +It is set to DbEnv::txn_begin() function. + +##### env + +The environment to start a transaction from. + +#### Return Value + +The newly created transaction. + +### Group: Transaction control global functions. + +dbstl transaction API. + +You should call these API rather than DB C/C++ API to use Berkeley DB transaction features. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsclose_all_db_envs.md b/docs-src/api/stl/stldbstl_global_functionsclose_all_db_envs.md new file mode 100644 index 000000000..5940c4d74 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsclose_all_db_envs.md @@ -0,0 +1,31 @@ +--- +title: "close_all_db_envs" +api-name: "close_all_db_envs" +source: docs/api_reference/STL/stldbstl_global_functionsclose_all_db_envs.html +--- +## close_all_db_envs + +### Function Details + +``` c + void close_all_db_envs() + +``` + +Close all open database environment handles regardless of reference count. + +You can't use the container after you called close_db and before setting another valid database handle to the container via db_container::set_db_handle() function. + +#### See Also + +close_db_env(DbEnv *) ; + +### Group: Functions to close database/environments. + +Normally you don't have to close any database or environment handles, they will be closed automatically. + +Though you still have the following API to close them. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsclose_all_dbs.md b/docs-src/api/stl/stldbstl_global_functionsclose_all_dbs.md new file mode 100644 index 000000000..e27e25998 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsclose_all_dbs.md @@ -0,0 +1,31 @@ +--- +title: "close_all_dbs" +api-name: "close_all_dbs" +source: docs/api_reference/STL/stldbstl_global_functionsclose_all_dbs.html +--- +## close_all_dbs + +### Function Details + +``` c + void close_all_dbs() + +``` + +Close all open database handles regardless of reference count. + +You can't use any container after you called close_all_dbs and before setting another valid database handle to the container via db_container::set_db_handle() function. + +#### See Also + +close_db(Db *) ; + +### Group: Functions to close database/environments. + +Normally you don't have to close any database or environment handles, they will be closed automatically. + +Though you still have the following API to close them. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsclose_db_cursors.md b/docs-src/api/stl/stldbstl_global_functionsclose_db_cursors.md new file mode 100644 index 000000000..6770bbae3 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsclose_db_cursors.md @@ -0,0 +1,29 @@ +--- +title: "close_db_cursors" +api-name: "close_db_cursors" +source: docs/api_reference/STL/stldbstl_global_functionsclose_db_cursors.html +--- +## close_db_cursors + +### Function Details + +``` c + size_t close_db_cursors(Db *dbp1) + +``` + +Close cursors opened in dbp1. + +#### Parameters + +##### dbp1 + +The database handle whose active cursors to close. + +#### Return Value + +The number of cursors closed by this call. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsclose_db_env.md b/docs-src/api/stl/stldbstl_global_functionsclose_db_env.md new file mode 100644 index 000000000..de4bf1c4b --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsclose_db_env.md @@ -0,0 +1,33 @@ +--- +title: "close_db_env" +api-name: "close_db_env" +source: docs/api_reference/STL/stldbstl_global_functionsclose_db_env.html +--- +## close_db_env + +### Function Details + +``` c + void close_db_env(DbEnv *pdbenv) + +``` + +Close specified database environment handle regardless of reference count. + +Make sure the environment is not used by any other databases. + +#### Parameters + +##### pdbenv + +The database environment handle to close. + +### Group: Functions to close database/environments. + +Normally you don't have to close any database or environment handles, they will be closed automatically. + +Though you still have the following API to close them. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionscommit_txn.md b/docs-src/api/stl/stldbstl_global_functionscommit_txn.md new file mode 100644 index 000000000..0c2872674 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionscommit_txn.md @@ -0,0 +1,68 @@ +--- +title: "commit_txn" +api-name: "commit_txn" +source: docs/api_reference/STL/stldbstl_global_functionscommit_txn.html +--- +## commit_txn + +### Function Details + +``` c + void commit_txn(DbEnv *env, + u_int32_t flags=0) + +``` + +Commit current transaction opened in the environment "env". + +This function is called by user to commit an external explicit transaction. + +#### Parameters + +##### flags + +It is set to DbTxn::commit() funcion. + +##### env + +The environment whose current transaction is to be committed. + +#### See Also + +commit_txn(DbEnv *, DbTxn *, u_int32_t) ; + +``` c + void commit_txn(DbEnv *env, DbTxn *txn, + u_int32_t flags=0) + +``` + +Commit a specified transaction and all its child transactions. + +#### Parameters + +##### txn + +The transaction to commit, can be a parent transaction of a nested transaction group, all un-aborted child transactions of it will be committed. + +##### flags + +It is passed to each DbTxn::commit() call. + +##### env + +The environment where txn is started from. + +#### See Also + +commit_txn(DbEnv *, u_int32_t) ; + +### Group: Transaction control global functions. + +dbstl transaction API. + +You should call these API rather than DB C/C++ API to use Berkeley DB transaction features. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionscurrent_txn.md b/docs-src/api/stl/stldbstl_global_functionscurrent_txn.md new file mode 100644 index 000000000..474d538fc --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionscurrent_txn.md @@ -0,0 +1,35 @@ +--- +title: "current_txn" +api-name: "current_txn" +source: docs/api_reference/STL/stldbstl_global_functionscurrent_txn.html +--- +## current_txn + +### Function Details + +``` c + DbTxn* current_txn(DbEnv *env) + +``` + +Get current transaction of environment "env". + +#### Parameters + +##### env + +The environment whose current transaction we want to get. + +#### Return Value + +Current transaction of env. + +### Group: Transaction control global functions. + +dbstl transaction API. + +You should call these API rather than DB C/C++ API to use Berkeley DB transaction features. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsdbstl_exit.md b/docs-src/api/stl/stldbstl_global_functionsdbstl_exit.md new file mode 100644 index 000000000..467ddd586 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsdbstl_exit.md @@ -0,0 +1,21 @@ +--- +title: "dbstl_exit" +api-name: "dbstl_exit" +source: docs/api_reference/STL/stldbstl_global_functionsdbstl_exit.html +--- +## dbstl_exit + +### Function Details + +``` c + void dbstl_exit() + +``` + +This function releases memory allocated by dbstl on the heap, and closes all Berkeley DB handles in the right order. + +You can call dbstl_exit() before the process exits to release any memory allocated by dbstl that has to persist during the entire process lifetime. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsdbstl_startup.md b/docs-src/api/stl/stldbstl_global_functionsdbstl_startup.md new file mode 100644 index 000000000..945636f81 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsdbstl_startup.md @@ -0,0 +1,19 @@ +--- +title: "dbstl_startup" +api-name: "dbstl_startup" +source: docs/api_reference/STL/stldbstl_global_functionsdbstl_startup.html +--- +## dbstl_startup + +### Function Details + +``` c + void dbstl_startup() + +``` + +If there are multiple threads within a process that make use of dbstl, then this function should be called in a single thread mutual exclusively before any use of dbstl in a process; Otherwise, you don't need to call it, but are allowed to call it anyway. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsdbstl_thread_exit.md b/docs-src/api/stl/stldbstl_global_functionsdbstl_thread_exit.md new file mode 100644 index 000000000..b3c730bdc --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsdbstl_thread_exit.md @@ -0,0 +1,21 @@ +--- +title: "dbstl_thread_exit" +api-name: "dbstl_thread_exit" +source: docs/api_reference/STL/stldbstl_global_functionsdbstl_thread_exit.html +--- +## dbstl_thread_exit + +### Function Details + +``` c + void dbstl_thread_exit() + +``` + +This function closes all Berkeley DB handles in the right order, if other threads do not use them. + +You can call this function before a thread exits to close unused Berkeley DB handles. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsfree_mutex.md b/docs-src/api/stl/stldbstl_global_functionsfree_mutex.md new file mode 100644 index 000000000..e99fdd21a --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsfree_mutex.md @@ -0,0 +1,35 @@ +--- +title: "free_mutex" +api-name: "free_mutex" +source: docs/api_reference/STL/stldbstl_global_functionsfree_mutex.html +--- +## free_mutex + +### Function Details + +``` c + void free_mutex(db_mutex_t mtx) + +``` + +Free a mutex, and return immediately. + +#### Parameters + +##### mtx + +The mutex handle to free. + +#### Return Value + +0 if succeed, non-zero otherwise, call db_strerror to get message. + +### Group: Mutex API based on Berkeley DB mutex. + +These functions are in-process mutex support which uses Berkeley DB mutex mechanisms. + +You can call these functions to do portable synchronization for your code. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionslock_mutex.md b/docs-src/api/stl/stldbstl_global_functionslock_mutex.md new file mode 100644 index 000000000..b752034ec --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionslock_mutex.md @@ -0,0 +1,35 @@ +--- +title: "lock_mutex" +api-name: "lock_mutex" +source: docs/api_reference/STL/stldbstl_global_functionslock_mutex.html +--- +## lock_mutex + +### Function Details + +``` c + int lock_mutex(db_mutex_t mtx) + +``` + +Lock a mutex, wait if it is held by another thread. + +#### Parameters + +##### mtx + +The mutex handle to lock. + +#### Return Value + +0 if succeed, non-zero otherwise, call db_strerror to get message. + +### Group: Mutex API based on Berkeley DB mutex. + +These functions are in-process mutex support which uses Berkeley DB mutex mechanisms. + +You can call these functions to do portable synchronization for your code. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsopen_db.md b/docs-src/api/stl/stldbstl_global_functionsopen_db.md new file mode 100644 index 000000000..cc420c3d7 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsopen_db.md @@ -0,0 +1,72 @@ +--- +title: "open_db" +api-name: "open_db" +source: docs/api_reference/STL/stldbstl_global_functionsopen_db.html +--- +## open_db + +### Function Details + +``` c + Db* open_db(DbEnv *penv, const char *filename, DBTYPE dbtype, + u_int32_t oflags, u_int32_t set_flags, int mode=0644, DbTxn *txn=NULL, + u_int32_t cflags=0, + const char *dbname=NULL) + +``` + +Helper function to open a database and register it into dbstl for the calling thread. + +Users still need to register it in any other thread using it if it is shared by multiple threads, via register_db() function. Users don't need to delete or free the memory of the returned object, dbstl will take care of that. When you don't use dbstl::open_db() but explicitly call DB C++ API to open a database, you must new the Db object, rather than create it on stack, and you must delete the Db object by yourself. + +#### Parameters + +##### penv + +The environment to open the database from. + +##### txn + +The transaction to open the database from, passed to Db::open. + +##### dbtype + +The database type, passed to Db::open. + +##### oflags + +The database open flags, passed to Db::open. + +##### filename + +The database file name, passed to Db::open. + +##### mode + +The database open mode, passed to Db::open. + +##### cflags + +The create flags passed to Db class constructor. + +##### dbname + +The database name, passed to Db::open. + +##### set_flags + +The flags to be set to the created database handle. + +#### Return Value + +The opened database handle. + +#### See Also + +register_db(Db *) ; + +open_db_env; + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsopen_env.md b/docs-src/api/stl/stldbstl_global_functionsopen_env.md new file mode 100644 index 000000000..22d6c56ed --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsopen_env.md @@ -0,0 +1,62 @@ +--- +title: "open_env" +api-name: "open_env" +source: docs/api_reference/STL/stldbstl_global_functionsopen_env.html +--- +## open_env + +### Function Details + +``` c + DbEnv* open_env(const char *env_home, u_int32_t set_flags, + u_int32_t oflags=DB_CREATE|DB_INIT_MPOOL, + u_int32_t cachesize=4 *1024 *1024, int mode=0644, + u_int32_t cflags=0) + +``` + +Helper function to open an environment and register it into dbstl for the calling thread. + +Users still need to register it in any other thread if it is shared by multiple threads, via register_db_env() function above. Users don't need to delete or free the memory of the returned object, dbstl will take care of that. + +When you don't use dbstl::open_env() but explicitly call DB C++ API to open an environment, you must new the DbEnv object, rather than create it on stack, and you must delete the DbEnv object by yourself. + +#### Parameters + +##### oflags + +Environment open flags, passed to DbEnv::open. + +##### set_flags + +Flags to set to the created environment before opening it. + +##### mode + +Environment region files mode, passed to DbEnv::open. + +##### cflags + +DbEnv constructor creation flags, passed to DbEnv::DbEnv. + +##### cachesize + +Environment cache size, by default 4M bytes. + +##### env_home + +Environment home directory, it must exist. Passed to DbEnv::open. + +#### Return Value + +The opened database environment handle. + +#### See Also + +register_db_env(DbEnv *) ; + +open_db ; + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsoperator_eq.md b/docs-src/api/stl/stldbstl_global_functionsoperator_eq.md new file mode 100644 index 000000000..7c0b26ae6 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsoperator_eq.md @@ -0,0 +1,48 @@ +--- +title: "operator==" +api-name: "operator==" +source: docs/api_reference/STL/stldbstl_global_functionsoperator_eq.html +--- +## operator== + +### Function Details + +``` c + bool operator==(const Dbt &d1, + const Dbt &d2) + +``` + +Operators to compare two Dbt objects. + +#### Parameters + +##### d2 + +Dbt object to compare. + +##### d1 + +Dbt object to compare. + +``` c + bool operator==(const DBT &d1, + const DBT &d2) + +``` + +Operators to compare two DBT objects. + +#### Parameters + +##### d2 + +DBT object to compare. + +##### d1 + +DBT object to compare. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsregister_db.md b/docs-src/api/stl/stldbstl_global_functionsregister_db.md new file mode 100644 index 000000000..421c3e1ec --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsregister_db.md @@ -0,0 +1,27 @@ +--- +title: "register_db" +api-name: "register_db" +source: docs/api_reference/STL/stldbstl_global_functionsregister_db.html +--- +## register_db + +### Function Details + +``` c + void register_db(Db *pdb1) + +``` + +Register a Db handle "pdb1". + +This handle and handles opened in it will be closed by ResourceManager , so application code must not try to close or delete it. Users can do enough configuration before opening the Db then register it via this function. All database handles should be registered via this function in each thread using the handle. The only exception is the database handle opened by dbstl::open_db should not be registered in the thread of the dbstl::open_db call. + +#### Parameters + +##### pdb1 + +The database handle to register into dbstl for current thread. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsregister_db_env.md b/docs-src/api/stl/stldbstl_global_functionsregister_db_env.md new file mode 100644 index 000000000..e62adca76 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsregister_db_env.md @@ -0,0 +1,27 @@ +--- +title: "register_db_env" +api-name: "register_db_env" +source: docs/api_reference/STL/stldbstl_global_functionsregister_db_env.html +--- +## register_db_env + +### Function Details + +``` c + void register_db_env(DbEnv *env1) + +``` + +Register a DbEnv handle env1, this handle and handles opened in it will be closed by ResourceManager . + +Application code must not try to close or delete it. Users can do enough config before opening the DbEnv and then register it via this function. All environment handles should be registered via this function in each thread using the handle. The only exception is the environment handle opened by dbstl::open_db_env should not be registered in the thread of the dbstl::open_db_env call. + +#### Parameters + +##### env1 + +The environment to register into dbstl for current thread. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsset_current_txn_handle.md b/docs-src/api/stl/stldbstl_global_functionsset_current_txn_handle.md new file mode 100644 index 000000000..865490000 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsset_current_txn_handle.md @@ -0,0 +1,42 @@ +--- +title: "set_current_txn_handle" +api-name: "set_current_txn_handle" +source: docs/api_reference/STL/stldbstl_global_functionsset_current_txn_handle.html +--- +## set_current_txn_handle + +### Function Details + +``` c + DbTxn* set_current_txn_handle(DbEnv *env, + DbTxn *newtxn) + +``` + +Set environment env's current transaction handle to be newtxn. + +The original transaction handle returned without aborting or commiting. This function is used for users to use one transaction among multiple threads. + +#### Parameters + +##### newtxn + +The new transaction to be as the current transaction of env. + +##### env + +The environment whose current transaction to replace. + +#### Return Value + +The old current transaction of env. It is not resolved. + +### Group: Transaction control global functions. + +dbstl transaction API. + +You should call these API rather than DB C/C++ API to use Berkeley DB transaction features. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsset_global_dbfile_suffix_number.md b/docs-src/api/stl/stldbstl_global_functionsset_global_dbfile_suffix_number.md new file mode 100644 index 000000000..fc86c69f0 --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsset_global_dbfile_suffix_number.md @@ -0,0 +1,25 @@ +--- +title: "set_global_dbfile_suffix_number" +api-name: "set_global_dbfile_suffix_number" +source: docs/api_reference/STL/stldbstl_global_functionsset_global_dbfile_suffix_number.html +--- +## set_global_dbfile_suffix_number + +### Function Details + +``` c + void set_global_dbfile_suffix_number(u_int32_t num) + +``` + +If exisiting random temporary database name generation mechanism is still causing name clashes, users can set this global suffix number which will be append to each temporary database file name and incremented after each append, and by default it is 0. + +#### Parameters + +##### num + +Starting number to append to each temporary db file name. + +### Class + +dbstl_global_functions diff --git a/docs-src/api/stl/stldbstl_global_functionsunlock_mutex.md b/docs-src/api/stl/stldbstl_global_functionsunlock_mutex.md new file mode 100644 index 000000000..a36bfa59e --- /dev/null +++ b/docs-src/api/stl/stldbstl_global_functionsunlock_mutex.md @@ -0,0 +1,35 @@ +--- +title: "unlock_mutex" +api-name: "unlock_mutex" +source: docs/api_reference/STL/stldbstl_global_functionsunlock_mutex.html +--- +## unlock_mutex + +### Function Details + +``` c + int unlock_mutex(db_mutex_t mtx) + +``` + +Unlock a mutex, and return immediately. + +#### Parameters + +##### mtx + +The mutex handle to unlock. + +#### Return Value + +0 if succeed, non-zero otherwise, call db_strerror to get message. + +### Group: Mutex API based on Berkeley DB mutex. + +These functions are in-process mutex support which uses Berkeley DB mutex mechanisms. + +You can call these functions to do portable synchronization for your code. + +### Class + +dbstl_global_functions diff --git a/docs-src/build.py b/docs-src/build.py index 0d5e43200..c84882175 100644 --- a/docs-src/build.py +++ b/docs-src/build.py @@ -25,8 +25,12 @@ SRC = HERE OUT = REPO / "docs-build/html" TEMPLATE = HERE / "_templates/page.html.tmpl" +MAN_TEMPLATE = HERE / "_templates/man.tmpl" SITE_TOML = HERE / "_data/site.toml" RELEASE = REPO / "dist/RELEASE" +MAN_OUT = REPO / "docs-build/man/man3" +# API .md trees whose refentry pages become section-3 man pages. +API_DIRS = [HERE / "api/c", HERE / "api/stl"] # Directories under docs-src/ that are machinery/data, not content. SKIP_DIRS = {"_data", "_templates", "_migrate"} @@ -137,11 +141,192 @@ def build_html(version, site, tmpl): return n -# --- Phase 3/4 seams: implemented later, kept here so the shape is fixed. --- +# --- Phase 3 seam: man pages (implemented). Phase 4 (PDF) stays stubbed. --- + +# Man pages skip the sidebar/frameset stub pages and the tree index pages +# (those are nav, not an API entry). +MAN_SKIP_STEMS = {"frame_index", "frame_main", "index"} + +_HEADING = re.compile(r"^(#{2,4}) +(.*)$", re.M) +_FIRST_SENTENCE = re.compile(r"(.+?[.!?])(?:\s|$)", re.S) + + +def _man_escape_name(s): + """Plain text for a NAME/.TH field: strip md/link noise, collapse spaces.""" + s = re.sub(r"]*>", "", s) # raw HTML tags + s = re.sub(r"\\([<>*_`~\[\]()])", r"\1", s) # gfm punct escapes + s = re.sub(r"`([^`]*)`", r"\1", s) # inline code + s = re.sub(r"\s+", " ", s).strip() + return s + + +def _to_man_markdown(body, title): + """Reshape an API refentry .md into man sections. + + The schema is fixed: a single `##` title heading, then the SYNOPSIS code + block + DESCRIPTION prose, then `###` sections (Parameters/Errors/Class/ + See Also/...) with `####` sub-items. Man wants top-level `.SH`, so: + - synthesize NAME + SYNOPSIS + DESCRIPTION from the title block, + - promote `###` -> `#` (.SH) and `####` -> `##` (.SS). + Returns (man_markdown, name_line_desc). + """ + # Everything before the first `##` is discarded (there is none but the + # title); split the title heading off. + m = re.search(r"^## +(.*)$", body, re.M) + intro = body[m.end():] if m else body + # Split intro into [synopsis code + description] vs the first `###`. + nxt = re.search(r"^### ", intro, re.M) + head = intro[: nxt.start()] if nxt else intro + rest = intro[nxt.start():] if nxt else "" + + # SYNOPSIS = the first fenced code block in head; DESCRIPTION = the prose. + syn = "" + cm = re.search(r"^``` ?[a-zA-Z]*\n.*?^```\s*$", head, re.S | re.M) + if cm: + syn = cm.group(0) + desc = (head[: cm.start()] + head[cm.end():]).strip() + else: + desc = head.strip() + + # NAME one-liner: first sentence of the description, else the title. + fs = _FIRST_SENTENCE.match(desc.lstrip()) + purpose = _man_escape_name(fs.group(1)) if fs else _man_escape_name(title) + # Trim the boilerplate "The `X()` method " lead-in so NAME reads as a + # purpose phrase (man convention), not a repeat of the signature. + purpose = re.sub(r"^The\s+.+?\s+(?:method|function|class)\s+", "", purpose) + purpose = purpose[:1].upper() + purpose[1:] if purpose else _man_escape_name(title) + purpose = purpose[:200] + + # Promote the trailing `###`/`####` sections one level (-> .SH/.SS). But a + # heading INDENTED inside a list item (` ### Note`) would emit an .SS in an + # open .RS block and unbalance mandoc's blocks — turn those into a bold + # label paragraph so they stay inline in the list. + rest = re.sub(r"^([ \t]+)#{2,4} +(.*)$", r"\1**\2**", rest, flags=re.M) + rest = _HEADING.sub(lambda h: ("#" * (len(h.group(1)) - 2)) + " " + h.group(2) + if len(h.group(1)) >= 3 else h.group(0), rest) + + parts = [f"# NAME", f"{title} \\- {purpose}", ""] + if syn: + parts += ["# SYNOPSIS", syn, ""] + if desc: + parts += ["# DESCRIPTION", desc, ""] + parts.append(rest) + return "\n".join(parts).strip() + "\n", purpose + + +def _pandoc_md_to_man(man_md, meta): + # Let pandoc wrap prose (default ~72 cols) so mandoc -Tlint stays quiet on + # long-line STYLE warnings; man readers reflow anyway. + cmd = ["pandoc", "-f", "gfm", "-t", "man", + "--template", str(MAN_TEMPLATE)] + for k, v in meta.items(): + cmd += ["-M", f"{k}={v}"] + p = subprocess.run(cmd, input=man_md, capture_output=True, text=True) + if p.returncode != 0: + raise RuntimeError(f"pandoc md->man failed: {p.stderr[:500]}") + return _tidy_roff(p.stdout) + + +def _tidy_roff(man): + """Drop a `.PP` that immediately follows a section/subsection heading — + pandoc emits it before tables/content and mandoc flags it ("skipping + paragraph macro: PP after SS"). Safe: an empty paragraph after a heading is + always droppable.""" + return re.sub(r"^(\.S[HS] [^\n]*\n)\.PP\n", r"\1", man, flags=re.M) + + +def _iter_api_pages(): + for d in API_DIRS: + if not d.exists(): + continue + for p in sorted(d.glob("*.md")): + if p.stem in MAN_SKIP_STEMS: + continue + yield p + + def build_man(version, site): - """TODO(phase-3): per-API .md -> section-3 man page via pandoc -t man, - plus one libdb.3 overview from the API index. Not built this phase.""" - return 0 + """Every public-API refentry .md -> a section-3 man page, plus one + libdb.3 overview. Output: docs-build/man/man3/. Returns the page count.""" + if not MAN_TEMPLATE.exists(): + sys.exit(f"missing man template {MAN_TEMPLATE}") + MAN_OUT.mkdir(parents=True, exist_ok=True) + # .TH fields: date (field 3), OS/source (field 4), manual title (field 5). + date = site.get("man_date", "") + source = f"{site['project']} {version}" + n = 0 + names = [] + for p in _iter_api_pages(): + meta_fm, body = strip_front_matter(p.read_text(encoding="utf-8")) + title = meta_fm.get("title", p.stem) + man_md, _purpose = _to_man_markdown(body, title) + manual = "Berkeley DB STL API" if p.parent.name == "stl" else "Berkeley DB C API" + man = _pandoc_md_to_man(man_md, { + "title": p.stem, "section": "3", "date": date, + "footer": source, "header": manual, + }) + (MAN_OUT / f"{p.stem}.3").write_text(man, encoding="utf-8") + names.append((p.stem, title)) + n += 1 + _build_overview(version, site, names, date, source) + n += 1 + return n + + +def _build_overview(version, site, names, date, source): + """Synthesize libdb.3 from the API index + programmer_reference intro. + + NAME + a short DESCRIPTION of the library, and a SEE ALSO listing the major + API groups (from api/c/_meta.toml's index db.md grouping) and every + generated page. The one-line summary is pulled from existing content.""" + groups = _api_groups() + desc = ("Berkeley DB is an embedded, transactional database library that " + "stores key/data pairs in one of four access methods (Btree, Hash, " + "Heap, Queue/Recno). It provides ACID transactions with " + "write-ahead logging, fine-grained locking, and safe concurrent " + "access from multiple threads and processes, all as a library " + "linked directly into the application — there is no separate " + "server process.") + lines = [ + "# NAME", + "libdb \\- Berkeley DB embedded database library", + "", + "# DESCRIPTION", + desc, + "", + "# API GROUPS", + ] + for label, stem in groups: + lines.append(f"**{label}** ({stem}(3))") + lines.append("") + lines += ["# SEE ALSO", ""] + lines.append(", ".join(f"{s}(3)" for s, _ in sorted(names)) + ".") + man = _pandoc_md_to_man("\n".join(lines) + "\n", { + "title": "libdb", "section": "3", "date": date, + "footer": source, "header": "Berkeley DB", + }) + (MAN_OUT / "libdb.3").write_text(man, encoding="utf-8") + + +def _api_groups(): + """Major API handle groups, from the C API index sidebar (frame_index.md). + Falls back to a fixed list if the sidebar is absent.""" + fb = [("Databases", "db"), ("Cursors", "dbc"), ("Key/Data Pairs", "dbt"), + ("Environments", "env"), ("Locking", "lock"), ("Logging", "lsn"), + ("Memory Pool", "memp"), ("Mutexes", "mutex"), + ("Replication", "rep"), ("Sequences", "seq"), + ("Transactions", "txn")] + idx = HERE / "api/c/frame_index.md" + if not idx.exists(): + return fb + # frame_index sidebar lists each group as Label. + groups = [] + for stem, label in re.findall( + r']*>([^<]+)', + idx.read_text(encoding="utf-8")): + if stem in {s for _, s in fb}: + groups.append((label.strip(), stem)) + return groups or fb def build_pdf(version, site): @@ -150,6 +335,25 @@ def build_pdf(version, site): return 0 +def _selfcheck(): + """Guard the md->man reshape: NAME/SYNOPSIS/DESCRIPTION split, heading + promotion, and in-list heading demotion.""" + body = ("## DB->foo()\n\n``` c\nint DB->foo(void);\n```\n\n" + "The DB->foo() method does a thing. More prose.\n\n" + "### Parameters\n\n#### bar\n\nThe bar param.\n\n" + " ### Note\n\n An in-list note.\n\n### See Also\n\nx\n") + mm, purpose = _to_man_markdown(body, "DB->foo()") + assert purpose.lower().startswith("does a thing"), purpose + assert "# NAME" in mm and "# SYNOPSIS" in mm and "# DESCRIPTION" in mm + assert "# Parameters" in mm and "## bar" in mm # ### -> #, #### -> ## + assert "# See Also" in mm + assert "**Note**" in mm and "### Note" not in mm # in-list heading demoted + assert "int DB->foo(void);" in mm # synopsis kept + # roff tidy drops PP right after a heading + assert _tidy_roff(".SS X\n.PP\n.TS\n") == ".SS X\n.TS\n" + print("selfcheck ok") + + def main(): if not TEMPLATE.exists(): sys.exit(f"missing template {TEMPLATE}") @@ -158,7 +362,12 @@ def main(): tmpl = TEMPLATE.read_text() n = build_html(version, site, tmpl) print(f"built {n} HTML pages -> {OUT} (version {version})") + m = build_man(version, site) + print(f"built {m} man pages -> {MAN_OUT} (version {version})") if __name__ == "__main__": - main() + if "--selfcheck" in sys.argv: + _selfcheck() + else: + main() diff --git a/docs-src/guides/articles/inmemory/_meta.toml b/docs-src/guides/articles/inmemory/_meta.toml new file mode 100644 index 000000000..c92bd33e9 --- /dev/null +++ b/docs-src/guides/articles/inmemory/_meta.toml @@ -0,0 +1,8 @@ +# Nav/index metadata for the inmemory guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Writing In-Memory Berkeley DB Applications" +landing = "index.md" +order = [ +] diff --git a/docs-src/guides/articles/inmemory/index.md b/docs-src/guides/articles/inmemory/index.md new file mode 100644 index 000000000..5a76eb5ef --- /dev/null +++ b/docs-src/guides/articles/inmemory/index.md @@ -0,0 +1,717 @@ +--- +title: "Writing In-Memory Berkeley DB Applications" +api-name: "Writing In-Memory Berkeley DB Applications" +source: docs/articles/inmemory/C/index.html +--- +# Writing In-Memory Berkeley DB Applications + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Introduction](index.md#intro) + + [Resources to be Managed](index.md#resources) + + [Strategies](index.md#strategies) + + [Keeping the Database in Memory](index.md#dbfiles) + + [Keeping Environments in Memory](index.md#env) + + [Sizing the Cache](index.md#cachesize) + + [Specifying a Cache Size using the Database Handle](index.md#dbcachesize-db) + + [Specifying a Cache Size using the Environment Handle](index.md#dbcachesize-env) + + [Keeping Temporary Overflow Pages in Memory](index.md#mpool-nofile) + + [Keeping Logs in Memory](index.md#logs) + + [In-Memory Replicated Applications](index.md#in-mem-rep) + + [Example In-Memory Application](index.md#example_in-mem) + +## Introduction + +This document describes how to write a DB application that keeps its data entirely in memory. That is, the application writes no data to disk. For this reason, in-memory only applications typically discard all data durability guarantees. + +### Note + +This document assume familiarity with the *Getting Started with Berkeley DB* guide. If you are using environments or transactions, then you should also have an understanding of the concepts in *Berkeley DB Getting Started with Transaction Processing* guide. + +There are several reasons why you might want to write an in-memory only DB application. For platforms on which a disk drive is available to back your data, an in-memory application might be desirable from a performance perspective. In this case, the data that your application manages might be generated during run-time and so is of no interest across application startups. + +Other platforms are disk-less. In this case, an in-memory only configuration is the only possible choice. Note that this document's primary focus is disk-less systems for which an on-disk filesystem is not available. + +## Resources to be Managed + +Before continuing, it is worthwhile to briefly describe the DB resources that must be managed if you are going to configure an in-memory only DB application. These are resources that are by default persisted on disk, or backed by a filesystem on disk. Some configuration is therefore required to keep these resources in-memory only. + +Note that you can configure only some of these resources to be held in-memory, and allow others to be backed by disk. This might be desireable for some applications that wish to improve application performance by, for example, eliminating disk I/O for some, but not all, of these resources. However, for the purpose of this document, we assume you want to configure all of these resources to be held in memory. + +Managing these resources for an in-memory application is described in detail later in this article. + +- Database files + + Normally, DB stores your database data within on-disk files. For an entirely in-memory application, you are required to turn off this behavior. + +- Environment region files + + DB environments manage region files for a variety of purposes. Normally these are backed by the filesystem, but by using the appropriate configuration option you can cause region files to reside in memory only. + +- Database cache + + The DB cache must be configured large enough to hold all your data in memory. If you do not size your cache large enough, then DB will attempt to write pages to disk. In a disk-less system, this will result in an abnormal termination of your program. + +- Logs + + DB logs describe the write activity that has occurred in your application. They are used for a number of purposes, such as recovery operations for applications that are seeking data durability guarantees. + + For in-memory applications that do not care about durability guarantees, logs are still required if you want transactional benefits other than durability (such as isolation and atomicity). This is because DB's transactional subsystem requires logs, even if you want to discard all data durability guarantees. + + If this describes your application, you must enable logs but configure them to reside only within memory. + +- Temporary overflow pages + + You must disallow backing temporary database files with the filesystem. This is mostly a configuration option, but it is also dependent upon sizing your cache correctly. + +In addition to these, if you are writing a replicated application (see *Berkeley DB Getting Started with Replicated Applications* for an introduction to writing replicated applications), there is internal replication information that is normally kept on-disk. You can cause this information to be kept in-memory if you are willing to accept some limitations in how your replicated application operates. See In-Memory Replicated Applications for more information. + +## Strategies + +DB is an extremely flexible product that can be adapted to suit almost any data management requirements. This means that you can configure DB to operate entirely within memory, but still retain some data durability guarantees or even throw away all durability guarantees. + +Data durability guarantees describe how persistent your data is. That is, once you have made a change to the data stored in your database, how much of a guarantee do you require that that modification will persist (not be lost)? There are a great many options here. For the absolute best durability guarantee, you should fully transaction-protect your data and allow your data to be written to disk upon each transaction commit. Of course, this guarantee is not available for disk-less systems. + +At the opposite end of the spectrum, you can throw away all your data once your application is done with it (for example, at application shutdown). This is a good option if you are using DB only as a kind of caching mechanism. In this case, you obviously must either generate your data entirely during runtime, or obtain it from some remote location during application startup. + +There are also durability options that exist somewhere in between these two extremes. For example, disk-less systems are sometimes backed by some kind of flash memory (e.g. compact flash cards). These platforms may want to limit the number of writes applied to the backing media because it is capable of accepting only a limited number of writes before it must be replaced. For this reason, you might want to limit data writes to the flash media only during specific moments during your application's runtime; for example, only at application shutdown. + +Another way to improve data durability for in-memory configurations is to use DB replication to commit data to the network. This strategy takes advantage of the fact that running clients have in-memory copies of the data and can take over in the event of an outage at the master. The use of replication in this way increases durability for your data while providing the benefit of avoiding disk I/O on transaction commit. + +In-memory replicated applications are described in more detail in In-Memory Replicated Applications. + +The point here is to be aware that a great many options are available to you when writing an in-memory only application. That said, the focus of this document is strictly disk-less systems; that is, systems that provide no means by which data can be written to persistent media. + +## Keeping the Database in Memory + +Normally DB databases are backed by the filesystem. For in-memory applications, such as is required on disk-less systems, you can cause your database to only reside in-memory. That is, their contents are stored entirely within DB's cache. + +There are two requirements for keeping your database(s) in-memory. The first is to size your cache such that it is big enough to hold all your data in-memory. If your cache fills up, then DB will return `ENOMEM` on the next operation that requests additional pages in the cache. As with all errors while updating a database, the current transaction must be aborted. If the update was being done without a transaction, then the application must close its environment and database handles, reopen them, and then refresh the database from some backup data source. + +For information on setting the cache size, see Sizing the Cache. + +Beyond cache sizing, you also must tell DB not to back your database with an on-disk file. You do this by NOT providing a database file name when you open the database. Note that the database file name is different from the database name; you can name your in-memory databases even if you are not storing them in an on-disk file. + +For example: + +``` c +#include "db.h" + +... + + int ret, ret_c; + const char *db_name = "in_mem_db1"; + u_int32_t db_flags; /* For open flags */ + DB *dbp; /* Database handle */ + +... + + /* Initialize the DB handle */ + ret = db_create(&dbp, NULL, 0); + if (ret != 0) { + fprintf(stderr, "Error creating database handle: %s\n", + db_strerror(ret)); + goto err; + } + + db_flags = DB_CREATE; /* If it doesn't exist, create it */ + + /* + * Open the database. Note that the file name is NULL. + * This forces the database to be stored in the cache only. + * Also note that the database has a name, even though its + * file name is NULL. + */ + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + NULL, /* File name is not specified + * on purpose */ + db_name, /* Logical db name. */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database open failed"); + goto err; + } + +err: + /* Close the database */ + if (dbp != NULL) { + ret_c = dbp->close(dbp, 0); + if (ret_c != 0) { + fprintf(stderr, "%s database close failed.\n", + db_strerror(ret_c)); + ret = ret_c + } + } +``` + +## Keeping Environments in Memory + +Like databases, DB environments are usually backed by the filesystem. In fact, a big part of what environments do is identify the location on disk where resources (such as log and database files) are kept. + +However, environments are also used for managing resources, such as obtaining new transactions, so they are useful even when building an in-memory application. Therefore, if you are going to use an environment for your in-memory DB application, you must configure it such that it does not want to use the filesystem. There are two things you need to do here. + +First, when you open your environment, do NOT identify a home directory. To accomplish this, you must: + +- NOT provide a value for the `db_home` parameter on the `DB_ENV->open()` method. + +- NOT have a DB_HOME environment variable set. + +- NOT call any of the methods that affect file naming (`DB_ENV->set_data_dir()`, `DB_ENV->set_lg_dir()`, or `DB_ENV->set_tmp_dir()`). + +Beyond this, you must also ensure that regions are backed by heap memory instead of by the filesystem or system shared memory. You do this when you open your environment by specifying the `DB_PRIVATE` flag. Note that the use of `DB_PRIVATE` means that you can only have one open handle for your environment at a time. Consequently, your in-memory only application must be single-process, although it can be multi-threaded. + +For example: + +``` c +#include "db.h" + +... + + int ret, ret_c; + u_int32_t env_flags; /* For open flags */ + DB_ENV *envp; /* Environment handle */ + +... + + /* Initialize the ENV handle */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + + /* + * Environment flags. These are for a non-threaded + * in-memory application. + */ + env_flags = + DB_CREATE | /* Create the environment if it does not exist */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_TXN | /* Initialize the transactional subsystem. This + * also turns on logging. */ + DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ + DB_PRIVATE | /* Region files are not backed by the + * filesystem. Instead, they are backed by + * heap memory. */ + + /* + * Now open the environment. Notice that we do not provide a location + * for the environment's home directory. This is required for an + * in-memory only application. + */ + ret = envp->open(envp, NULL, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + +err: + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c + } + } +``` + +## Sizing the Cache + + [Specifying a Cache Size using the Database Handle](index.md#dbcachesize-db) + + [Specifying a Cache Size using the Environment Handle](index.md#dbcachesize-env) + +One of the most important considerations for an in-memory application is to ensure that your database cache is large enough. In a normal application that is not in-memory, the cache provides a mechanism by which frequently-used data can be accessed without resorting to disk I/O. For an in-memory application, the cache is the only location your data can exist so it is critical that you make the cache large enough for your data set. + +You specify the size of your cache at application startup. Obviously you should not specify a size that is larger than available memory. Note that the size you specify for your cache is actually a *maximum* size; DB will only use memory as required so if you specify a cache size of 1 GB but your data set is only ever 10 MB in size, then 10 MB is what DB will use. + +Note that if you specify a cache size less than 500 MB, then the cache size is automatically increased by 25% to account for internal overhead purposes. + +There are two ways to specify a cache size, depending on whether you are using a database environment. + +### Specifying a Cache Size using the Database Handle + +To select a cache size using the database handle, use the `DB->set_cachesize()` method. Note that you cannot use this method after the database has been opened. + +Also, if you are using a database environment, it is an error to use this method. See the next section for details on selecting your cache size. + +The following code fragment creates a database handle, sets the cache size to 10 MB and then opens the database: + +``` c +#include "db.h" + +... + + int ret, ret_c; + const char *db_name = "in_mem_db1"; + u_int32_t db_flags; /* For open flags */ + DB *dbp; /* Database handle */ + +... + + /* Initialize the DB handle */ + ret = db_create(&dbp, NULL, 0); + if (ret != 0) { + fprintf(stderr, "Error creating database handle: %s\n", + db_strerror(ret)); + goto err; + } + + /*************************************************************/ + /*************************************************************/ + /************* Set the cache size here **********************/ + /*************************************************************/ + /*************************************************************/ + + ret = dbp->set_cachesize(dbp, + 0, /* 0 gigabytes */ + 10 * 1024 * 1024, /* 10 megabytes */ + 1); /* Create 1 cache. All memory will + * be allocated contiguously. */ + if (ret != 0) { + dbp->err(dbp, ret, "Database open failed"); + goto err; + } + + + db_flags = DB_CREATE; /* If it doesn't exist, create it */ + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + NULL, /* File name is not specified on + * purpose */ + db_name, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database open failed"); + goto err; + } + +err: + /* Close the database */ + if (dbp != NULL) { + ret_c = dbp->close(dbp, 0); + if (ret_c != 0) { + fprintf(stderr, "%s database close failed.\n", + db_strerror(ret_c)); + ret = ret_c + } + } +``` + +### Specifying a Cache Size using the Environment Handle + +To select a cache size using the environment handle, use the `ENV->set_cachesize()` method. Note that you cannot use this method after the environment has been opened. + +The following code fragment creates an environment handle, sets the cache size to 10 MB and then opens the environment: Once opened, you can use the environment when you open your database(s). This means all your databases will use the same cache. + +``` c +#include "db.h" + +... + + int ret, ret_c; + u_int32_t env_flags; /* For open flags */ + DB_ENV *envp; /* Environment handle */ + +... + + /* Initialize the ENV handle */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + /*************************************************************/ + /*************************************************************/ + /************* Set the cache size here **********************/ + /*************************************************************/ + /*************************************************************/ + + ret = + envp->set_cachesize(envp, + 0, /* 0 gigabytes */ + 10 * 1024 * 1024, /* 10 megabytes */ + 1); /* Create 1 cache. All memory will + * be allocated contiguously. */ + if (ret != 0) { + envp->err(envp, ret, "Environment open failed"); + goto err; + } + + /* + * Environment flags. These are for a non-threaded + * in-memory application. + */ + env_flags = + DB_CREATE | /* Create the environment if it does not exist */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_TXN | /* Initialize the transactional subsystem. This + * also turns on logging. */ + DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ + DB_PRIVATE | /* Region files are not backed by the filesystem. + * Instead, they are backed by heap memory. */ + + /* + * Now open the environment. Notice that we do not provide a location + * for the environment's home directory. This is required for an + * in-memory only application. + */ + ret = envp->open(envp, NULL, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + +err: + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c + } + } +``` + +## Keeping Temporary Overflow Pages in Memory + +Normally, when a database is opened, a temporary file is opened on-disk to back the database. This file is used if the database grows so large that it fills the entire cache. At that time, database pages that do not fit into the in-memory cache file are written temporarily to this file. + +For disk-less systems, you should configure your databases so that this temporary file is not created. When you do this, any attempt to create new database pages once the cache is full will fail. + +You configure this option on a per-database handle basis. That means you must configure this for every in-memory database that your application uses. + +To set this option, obtain the `DB_MPOOLFILE` field from you `DB` and then configure `DB_MPOOL_NOFILE` using the `DB_MPOOLFILE->set_flags()` method. + +For example: + +``` c +#include "db.h" + +... + + int ret, ret_c; + u_int32_t env_flags; /* For open flags */ + DB_ENV *envp; /* Environment handle */ + +... + + /* + * Configure the cache file. This can be done + * at any point in the application's life once the + * DB handle has been created. + */ + mpf = dbp->get_mpf(dbp); + ret = mpf->set_flags(mpf, DB_MPOOL_NOFILE, 1); + + if (ret != 0) { + fprintf(stderr, + "Attempt failed to configure for no backing of temp files: %s\n", + db_strerror(ret)); + goto err; + } +``` + +## Keeping Logs in Memory + +DB logs describe the write activity that has occurred in your application. For a purely in-memory application, logs should be used only if you wish to transaction-protect your database writes as logs are required by the DB transactional subsystem. + +Note that transactions provide a number of guarantees. One of these is not interesting to a purely in-memory application (data durability). However, other transaction guarantees such as isolation and atomicity might be of interest to your application. + +If this is the case for your application, then you must configure your logs to be kept entirely in-memory. You do this by setting a configuration option that prevents DB from writing log data to disk. Do this by setting the `DB_LOG_IN_MEMORY` flag using the `DB_ENV->log_set_config()` method. + +In addition, you must configure your log buffer size so that it is capable of holding all log information that can accumulate during your longest running transaction. That is, make sure the in-memory log buffer is large enough that no transaction will ever span the entire buffer. Also, avoid a state where the in-memory buffer is full and no space can be freed because a transaction that started the first log "file" is still active. + +How much log buffer space is required is a function of the number of transactions you have running concurrently, how long they last, and how much write activity occurs within them. When in-memory logging is configured, the default log buffer space is 1 MB. + +You set your log buffer space using the `DB_ENV->set_lg_bsize()`. + +For example, the following code fragment configure in-memory log usage, and it configures the log buffer size to 10 MB: + +``` c +#include "db.h" + +... + + int ret, ret_c; + u_int32_t env_flags; /* For open flags */ + DB_ENV *envp; /* Environment handle */ + +... + + /* Initialize the ENV handle */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Environment flags. These are for a non-threaded + * in-memory application. + */ + env_flags = + DB_CREATE | /* Create the environment if it does not exist */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_TXN | /* Initialize the transactional subsystem. This + * also turns on logging. */ + DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ + DB_PRIVATE | /* Region files are not backed by the filesystem. + * Instead, they are backed by heap memory. */ + + + /* Specify in-memory logging */ + ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); + if (ret != 0) { + fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Specify the size of the in-memory log buffer. + */ + ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); + if (ret != 0) { + fprintf(stderr, "Error increasing the log buffer size: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Now open the environment. Notice that we do not provide a location + * for the environment's home directory. This is required for an + * in-memory only application. + */ + ret = envp->open(envp, NULL, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + +err: + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c + } + } +``` + +## In-Memory Replicated Applications + +If you are unfamiliar with writing DB replicated applications, or if you are simply uninterested in this topic, you can skip this section. For an introductory description of DB replication, please see the *Berkeley DB Getting Started with Replicated Applications* guide. + +In-memory replicated applications can improve transaction throughput by avoiding disk I/O. Network connections are often faster than local synchronous disk writes, so in-memory replicated applications can provide significantly improved performance without entirely sacrificing reliability. + +All of your replication participants must be configured in the same way. That is, they all must be configured for in-memory storage of data, or they must all be configured for on-disk storage of data. As a result, in-memory replicated applications can achieve improved throughput performance, but they do so at the cost of reduced durability guarantees. This is because the transaction commit is not backed by stable storage anywhere in the replication group. However, this "commit to the network" does not sacrifice all durability. Because running clients should have in-memory copies of the data, a client can take over in the event of an outage at the master and in this way avoid the loss of data. + +There are many internal resources used by the DB replication subsystem which are by default backed by disk. These internal resources help the DB subsystem ensure election accuracy. While a complete description of these resources is beyond the scope of this article, you should know that you can cause all of these resources to be held entirely in-memory. But you do so with some small chance of operational errors in your replicated application. + +If you cause a replicated application to keep its internal replication resources in-memory, you run a small risk that elections will fail or be unable to complete. However, calling additional elections should eventually yield a winner. + +In addition, there is a slight possibility that the wrong site might win an election, which could result in the loss of data. This can happen if you have a site that is repeatedly crashing and trying to come back up. A site like this might be repeatedly sending out election information, and the repeated messages might confuse other sites. For replication applications that are not in-memory, these extra messages would be ignored by other sites because they would also contain some state information that allows other sites to know which election messages are relevant. But strictly in-memory replicated applications cannot maintain this state information, and so some other sites might become confused. The result might be that the wrong site could be elected master due to the inconsistent information that is available to them. + +### Note + +This is very much a corner case that you probably will never see in your production systems, especially if your sites are all stable and well-behaved. + +If an election is won by the wrong site (site A), then some other site (site B) probably has more recent log files than the winner does. But since the wrong site A won the election, site B will sync with the new master. This will cause site B (and therefore, your entire replication group) to lose any log files it contains that are more recent than the files contained by site A. + +If you are running a master that is configured to run with internal replication resources in-memory, you should never allow that site to appoint itself master again immediately after crashing or rebooting. Doing so results in a slightly higher risk of your client sites crashing. To determine your next master, you should either hold an election or appoint a different site to be master. + +In order to cause a replication site to run entirely in-memory, do all of the things described previously in this document to place all other DB resources in-memory. Then, when configuring replication, specify `DB_REP_CONF_INMEM` to the `DB_ENV->rep_set_config()` method. + +## Example In-Memory Application + +The following brief example illustrates how to open an application that is entirely in-memory. The application opens an environment and a single database, and does this in a way that the database is transaction-protected. + +Transactions can be desirable for an in-memory application even though you discard your durability guarantees, because of the other things that transactions offer such as atomicity and isolation. + +Notice that the example does nothing other than open and close the environment and database. DB database reads and writes work identically between in-memory-only and durable applications (that is, applications that write database application to durable storage). Consequently, there is no point in illustrating those actions here. + +``` c +/* We assume an ANSI-compatible compiler */ +#include +#include +#include +#include + +int +main(void) +{ + /* Initialize our handles */ + DB *dbp = NULL; + DB_ENV *envp = NULL; + DB_MPOOLFILE *mpf = NULL; + + int ret, ret_t; + const char *db_name = "in_mem_db1"; + u_int32_t open_flags; + + /* Create the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + open_flags = + DB_CREATE | /* Create the environment if it does not exist */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ + DB_INIT_TXN | + DB_PRIVATE; /* Region files are not backed by the filesystem. + * Instead, they are backed by heap memory. */ + + /* Specify in-memory logging */ + ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); + if (ret != 0) { + fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", + db_strerror(ret)); + goto err; + } + /* + * Specify the size of the in-memory log buffer. + */ + ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); + if (ret != 0) { + fprintf(stderr, "Error increasing the log buffer size: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Specify the size of the in-memory cache. + */ + ret = envp->set_cachesize(envp, 0, 10 * 1024 * 1024, 1); + if (ret != 0) { + fprintf(stderr, "Error increasing the cache size: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Now actually open the environment. Notice that the environment home + * directory is NULL. This is required for an in-memory only + * application. + */ + ret = envp->open(envp, NULL, open_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, + "Attempt to create db handle failed."); + goto err; + } + + /* + * Set the database open flags. Autocommit is used because we are + * transactional. + */ + open_flags = DB_CREATE | DB_AUTO_COMMIT; + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + NULL, /* File name -- Must be NULL for inmemory! */ + db_name, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + open_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + + if (ret != 0) { + envp->err(envp, ret, + "Attempt to open db failed."); + goto err; + } + + /* Configure the cache file */ + mpf = dbp->get_mpf(dbp); + ret = mpf->set_flags(mpf, DB_MPOOL_NOFILE, 1); + + if (ret != 0) { + envp->err(envp, ret, + "Attempt failed to configure for no backing of temp files."); + goto err; + } + +err: + /* Close our database handle, if it was opened. */ + if (dbp != NULL) { + ret_t = dbp->close(dbp, 0); + if (ret_t != 0) { + fprintf(stderr, "%s database close failed.\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Close our environment, if it was opened. */ + if (envp != NULL) { + ret_t = envp->close(envp, 0); + if (ret_t != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Final status message and return. */ + printf("I'm all done.\n"); + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` diff --git a/docs-src/guides/articles/mssgtxt/_meta.toml b/docs-src/guides/articles/mssgtxt/_meta.toml new file mode 100644 index 000000000..c61d93498 --- /dev/null +++ b/docs-src/guides/articles/mssgtxt/_meta.toml @@ -0,0 +1,8 @@ +# Nav/index metadata for the mssgtxt guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Berkeley DB Message Reference for Stripped Libraries" +landing = "index.md" +order = [ +] diff --git a/docs-src/guides/articles/mssgtxt/index.md b/docs-src/guides/articles/mssgtxt/index.md new file mode 100644 index 000000000..7d9c0cca2 --- /dev/null +++ b/docs-src/guides/articles/mssgtxt/index.md @@ -0,0 +1,1336 @@ +--- +title: "Berkeley DB Message Reference for Stripped Libraries" +api-name: "Berkeley DB Message Reference for Stripped Libraries" +source: docs/articles/mssgtxt/index.html +--- +# Berkeley DB Message Reference for Stripped Libraries + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Introduction](index.md#intro) + + [Access Methods Messages](index.md#am) + + [Common Messages](index.md#common) + + [Database Handle Messages](index.md#db) + + [Environment Handle Messages](index.md#env) + + [Locking Subsystem Messages](index.md#lock) + + [Logging Subsystem Messages](index.md#log) + + [Memory Pool Messages](index.md#mpool) + + [Replication Messages](index.md#rep) + + [Sequences Messages](index.md#sequence) + + [Transaction Messages](index.md#txn) + + [Command Line Utilities Messages](index.md#util) + +## Introduction + +When you build Berkeley DB, it is possible to cause all the error message text to be stripped from the library. This is done so as to minimize the library's footprint on devices where memory and storage is severely constrained. + +How you strip error messages from your library depends on the platform you are building on. For \*nix platforms, you do this when you configure your build using the --enable-stripped_messages configuration option. Your messages will also be stripped if you configure for small builds. + +For Windows, you can enable stripped messages using the HAVE_STRIPPED_MESSAGES build property. Your messages will also be stripped if you build a small memory footprint library. + +Stripped libraries still issue error messages, but the only thing displayed is the error number — the text of the message is not available for the library to display. This document provides the missing error message text. + +Message text is organized into tables, where each table identifies a specific portion of the library. Each such table is then sorted by message number. + +The areas of the library which can issue messages are: + +- Access method. These are error messages related to usage of the various access methods (Btree, Heap, Queue, and so forth). + +- Common error messages. These are messages that can be commonly issued by any area of the library. + +- Database error messages. These are error messages related to the usage of database handles. + +- Environment error messages. These are error messages related to the usage of environment handles. + +- Locking subsystem error messages. + +- Logging subsystem error messages. + +- Memory pool error messages. + +- Replication error messages. These error messages are issued by methods specific to either Base API or Replication Manager applications. + +- Sequences error messages. + +- Transactions error messages. + +- Command line utility error messages. + +## Access Methods Messages + +| Message Number | Message Text | +|----|----| +| 1001 | illegal record number size | +| 1002 | illegal record number of 0 | +| 1003 | %s: write failed to backing file | +| 1004 | Existing data sorts differently from put data | +| 1005 | cursor adjustment after delete failed | +| 1006 | prefix comparison may not be specified for default comparison routine | +| 1007 | bt_minkey value of %lu too high for page size of %lu | +| 1008 | %s: btree version %lu requires a version upgrade | +| 1009 | %s: unsupported btree version: %lu | +| 1010 | %s: DB_DUP specified to open method but not set in database | +| 1011 | %s: DB_RECNUM specified to open method but not set in database | +| 1012 | %s: DB_FIXEDLEN specified to open method but not set in database | +| 1013 | %s: DB_RENUMBER specified to open method but not set in database | +| 1014 | %s: multiple databases specified but not supported by file | +| 1015 | %s: duplicate sort specified but not supported in database | +| 1016 | %s: compresssion specified to open method but not set in database | +| 1017 | %s: compression support has not been compiled in | +| 1018 | open method type is Btree, database type is Recno | +| 1019 | open method type is Recno, database type is Btree | +| 1020 | Not enough room in parent: %s: page %lu | +| 1021 | Too many btree levels: %d | +| 1022 | Unknown record format, page %lu, indx 0 | +| 1023 | Compact cannot handle zero length key | +| 1024 | DB_RECNUM cannot be used with compression | +| 1025 | DB_DUP cannot be used with compression without DB_DUPSORT | +| 1026 | compression support has not been compiled in | +| 1027 | compression cannot be used with DB_RECNUM | +| 1028 | compression cannot be used with DB_DUP without DB_DUPSORT | +| 1029 | to enable compression you need to supply both function arguments | +| 1030 | compression support has not been compiled in | +| 1031 | minimum bt_minkey value is 2 | +| 1032 | Existing data sorts differently from put data | +| 1033 | Both cursors must be initialized before calling DBC-\>cmp. | +| 1034 | Page %lu: nonsensical bt_minkey value %lu on metadata page | +| 1035 | Page %lu: nonsensical root page %lu on metadata page | +| 1036 | Page %lu: Btree metadata page has both duplicates and multiple databases | +| 1037 | Page %lu: Btree metadata page illegally has both recnums and dups | +| 1038 | Page %lu: metadata page has renumber flag set but is not recno | +| 1039 | Page %lu: Btree metadata page illegally has both recnums and compression | +| 1040 | Page %lu: Btree metadata page illegally has both ""unsorted duplicates and compression | +| 1041 | Page %lu: recno metadata page specifies duplicates | +| 1042 | Page %lu: re_len of %lu in non-fixed-length database | +| 1043 | Page %lu: Recno database has dups | +| 1044 | Page %lu: nonsensical type for item %lu | +| 1045 | Page %lu: item order check unsafe: skipping | +| 1046 | Page %lu: entries listing %lu overlaps data | +| 1047 | Page %lu: bad offset %lu at index %lu | +| 1048 | Page %lu: RINTERNAL structure at offset %lu referenced twice | +| 1049 | Page %lu: gap between items at offset %lu | +| 1050 | Page %lu: bad HOFFSET %lu, appears to be %lu | +| 1051 | Page %lu: duplicated item %lu | +| 1052 | Page %lu: duplicated item %lu | +| 1053 | Page %lu: item %lu marked deleted | +| 1054 | Page %lu: duplicate page referenced by internal btree page at item %lu | +| 1055 | Page %lu: duplicate page referenced by recno page at item %lu | +| 1056 | Page %lu: impossible tlen %lu, item %lu | +| 1057 | Page %lu: offpage item %lu has bad pgno %lu | +| 1058 | Page %lu: item %lu of invalid type %lu | +| 1059 | Page %lu: gap between items at offset %lu | +| 1060 | Page %lu: offset %lu unaligned | +| 1061 | Page %lu: overlapping items at offset %lu | +| 1062 | Page %lu: overlapping items at offset %lu | +| 1063 | Page %lu: bad HOFFSET %lu, appears to be %lu | +| 1064 | Page %lu: lowest key on internal page of nonzero length | +| 1065 | Page %lu: error %lu in fetching overflow item %lu | +| 1066 | Page %lu: out-of-order key at entry %lu | +| 1067 | Page %lu: non-dup dup key at entry %lu | +| 1068 | Page %lu: database with no duplicates has duplicated keys | +| 1069 | Page %lu: btree metadata page observed twice | +| 1070 | Page %lu: btree metadata page has no root | +| 1071 | Page %lu: recno database has bad re_len %lu | +| 1072 | Page %lu: duplicate tree referenced from metadata page | +| 1073 | Page %lu: btree root of incorrect type %lu on metadata page | +| 1074 | Page %lu: unexpected page type %lu found in leaf chain (expected %lu) | +| 1075 | Page %lu: incorrect next_pgno %lu found in leaf chain (should be %lu) | +| 1076 | Page %lu: incorrect prev_pgno %lu found in leaf chain (should be %lu) | +| 1077 | Page %lu: recno leaf page non-recno tree | +| 1078 | Page %lu: non-recno leaf page in recno tree | +| 1079 | Page %lu: duplicates in non-dup btree | +| 1080 | Page %lu: unsorted duplicate set in sorted-dup database | +| 1081 | Page %lu: btree or recno page is of inappropriate type %lu | +| 1082 | Page %lu: recno page returned bad re_len %lu | +| 1083 | Page %lu: record count incorrect: actual %lu, in record %lu | +| 1084 | Page %lu: recno level incorrect: got %lu, expected %lu | +| 1085 | Page %lu: overflow page %lu referenced more than twice from internal page | +| 1086 | Page %lu: item %lu has incorrect record count of %lu, should be %lu | +| 1087 | Page %lu: Btree level incorrect: got %lu, expected %lu | +| 1088 | Page %lu: internal page is empty and should not be | +| 1089 | Page %lu: bad record count: has %lu records, claims %lu | +| 1090 | Page %lu: linked twice | +| 1091 | Page %lu: unterminated leaf chain | +| 1092 | Page %lu: first item on page sorted greater than parent entry | +| 1093 | Page %lu: first item on page had comparison error | +| 1094 | Page %lu: last item on page sorted greater than parent entry | +| 1095 | Page %lu: last item on page had comparison error | +| 1096 | Page %lu: database has custom hash function; reverify with DB_NOORDERCHK set | +| 1097 | Page %lu: Impossible max_bucket %lu on meta page | +| 1098 | Page %lu: incorrect high_mask %lu, should be %lu | +| 1099 | Page %lu: incorrect low_mask %lu, should be %lu | +| 1100 | Page %lu: suspiciously high nelem of %lu | +| 1101 | Page %lu: spares array entry %d is invalid | +| 1102 | Page %lu: item %lu is out of order or nonsensical | +| 1103 | Page %lu: entries array collided with data | +| 1104 | Page %lu: hash key stored as duplicate item %lu | +| 1105 | Page %lu: duplicate item %lu has bad length | +| 1106 | Page %lu: duplicate item %lu has two different lengths | +| 1107 | Page %lu: offpage item %lu has bad pgno %lu | +| 1108 | Page %lu: offpage item %lu has bad page number | +| 1109 | Page %lu: item %lu has bad type | +| 1110 | Page %lu: Hash meta page referenced twice | +| 1111 | Page %lu: hash bucket %lu maps to non-hash page | +| 1112 | Page %lu: non-empty page in unused hash bucket %lu | +| 1113 | Page %lu: above max_bucket referenced | +| 1114 | Page %lu: impossible first page in bucket %lu | +| 1115 | Page %lu: first page in hash bucket %lu has a prev_pgno | +| 1116 | Page %lu: hash page referenced twice | +| 1117 | Page %lu: duplicates present in non-duplicate database | +| 1118 | Page %lu: unsorted dups in sorted-dup database | +| 1119 | Page %lu: hash page has bad next_pgno | +| 1120 | Page %lu: hash page has bad prev_pgno | +| 1121 | Page %lu: item %lu hashes incorrectly | +| 1122 | Invalid flag in \_\_ham_curadj_recover | +| 1123 | Cannot replicate prepared transactions from master running release 4.2. | +| 1124 | %s: Invalid hash meta page %lu | +| 1125 | %s: hash version %lu requires a version upgrade | +| 1126 | %s: unsupported hash version: %lu | +| 1127 | %s: DB_DUP specified to open method but not set in database | +| 1128 | %s: multiple databases specified but not supported in file | +| 1129 | %s: duplicate sort function specified but not set in database | +| 1130 | H_NOMORE returned to \_\_hamc_get | +| 1131 | Existing data sorts differently from put data | +| 1132 | Attempt to return a deleted item | +| 1133 | library build did not include support for the Hash access method | +| 1134 | Extent size may not be specified for in-memory queue database | +| 1135 | Multiversion queue databases are not supported | +| 1136 | \_\_qam_open: %s: unexpected file type or format | +| 1137 | %s: queue version %lu requires a version upgrade | +| 1138 | %s: unsupported qam version: %lu | +| 1139 | Record size of %lu too large for page size of %lu | +| 1140 | Extent size must be at least 1 | +| 1141 | Queue does not support multiple databases per file | +| 1142 | Record length error: data offset plus length larger than record size of %lu | +| 1143 | illegal record number size | +| 1144 | illegal record number of 0 | +| 1145 | library build did not include support for the Queue access method | +| 1146 | Page %lu: queue databases must be one-per-file | +| 1147 | Page %lu: queue record length %lu too high for page size and recs/page | +| 1148 | Page %lu: database contains multiple Queue metadata pages | +| 1149 | Warning: %d extra extent files found | +| 1150 | Page %lu: queue record %lu extends past end of page | +| 1151 | Page %lu: queue record %lu has bad flags (%#lx) | +| 1152 | Page %lu: queue database has no meta page | +| 1153 | Page %lu: queue database page of incorrect type %lu | +| 1154 | Page %lu: queue database page of incorrect type %lu | +| 1155 | %s: specified heap size does not match size set in database | +| 1156 | Page %lu: Heap databases must be one-per-file | +| 1157 | Page %lu: Number of heap regions incorrect | +| 1158 | Page %lu: last_pgno beyond end of fixed size heap | +| 1159 | Page %lu: incorrect number of entries in page's offset table | +| 1160 | Page %lu: record %lu (length %lu) overlaps next record | +| 1161 | Page %lu: record %lu (length %lu) beyond end of page | +| 1162 | Page %lu: heap database has no meta page | +| 1163 | Page %lu: heap database page of incorrect type %lu | +| 1164 | Page %lu: heap database missing region page (page type %lu) | +| 1165 | Page %lu: record %lu has invalid flags | +| 1166 | Page %lu heap database page beyond high page in region | +| 1167 | Incorrect record size in header: %s: rid %lu.%lu | +| 1168 | region size may not be 0 | +| 1169 | region size may not be larger than %lu | +| 1170 | The key/data pairs in the buffer are not sorted. | +| 1171 | The key/data pairs in the buffer are not sorted. | +| 1172 | The DBT items in the buffer are not sorted | + +## Common Messages + +| Message Number | Message Text | +|----|----| +| 0001 | fcntl(F_SETFD) | +| 0002 | \_\_fop_file_setup: Retry limit (%d) exceeded | +| 0003 | Transactional create on replication client disallowed | +| 0004 | fop_read_meta: %s: unexpected file type or format | +| 0005 | rename: file %s exists | +| 0040 | Encrypted environment: library build did not include cryptography support | +| 0041 | unsupported byte order, only big and little-endian supported | +| 0042 | %s: %s: Invalid numeric argument | +| 0043 | %s: Invalid numeric argument | +| 0044 | %s: %s: Less than minimum value (%ld) | +| 0045 | %s: Less than minimum value (%ld) | +| 0046 | %s: %s: Greater than maximum value (%ld) | +| 0047 | %s: Greater than maximum value (%ld) | +| 0048 | %s: %s: Invalid numeric argument | +| 0049 | %s: Invalid numeric argument | +| 0050 | %s: %s: Less than minimum value (%lu) | +| 0051 | %s: Less than minimum value (%lu) | +| 0052 | %s: %s: Greater than maximum value (%lu) | +| 0053 | %s: Greater than maximum value (%lu) | +| 0054 | illegal flag combination specified to %s | +| 0055 | illegal flag specified to %s | +| 0056 | %s: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW require locking | +| 0057 | unable to create/retrieve page %lu | +| 0058 | page %lu: illegal page type or format | +| 0059 | assert failure: %s/%d: "%s" | +| 0060 | PANIC: fatal region error detected; run recovery | +| 0061 | PANIC | +| 0062 | Successful return: 0 | +| 0063 | DB_BUFFER_SMALL: User memory too small for return value | +| 0064 | DB_DONOTINDEX: Secondary index callback returns null | +| 0065 | DB_FOREIGN_CONFLICT: A foreign database constraint has been violated | +| 0066 | DB_KEYEMPTY: Non-existent key/data pair | +| 0067 | DB_KEYEXIST: Key/data pair already exists | +| 0068 | DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock | +| 0069 | DB_LOCK_NOTGRANTED: Lock not granted | +| 0070 | DB_LOG_BUFFER_FULL: In-memory log buffer is full | +| 0071 | DB_LOG_VERIFY_BAD: Log verification failed | +| 0072 | DB_NOSERVER: No message dispatch call-back function has been configured | +| 0073 | DB_NOTFOUND: No matching key/data pair found | +| 0074 | DB_OLDVERSION: Database requires a version upgrade | +| 0075 | DB_PAGE_NOTFOUND: Requested page not found | +| 0076 | DB_REP_DUPMASTER: A second master site appeared | +| 0077 | DB_REP_HANDLE_DEAD: Handle is no longer valid | +| 0078 | DB_REP_HOLDELECTION: Need to hold an election | +| 0079 | DB_REP_IGNORE: Replication record/operation ignored | +| 0080 | DB_REP_ISPERM: Permanent record written | +| 0081 | DB_REP_JOIN_FAILURE: Unable to join replication group | +| 0082 | DB_REP_LEASE_EXPIRED: Replication leases have expired | +| 0083 | DB_REP_LOCKOUT: Waiting for replication recovery to complete | +| 0084 | DB_REP_NEWSITE: A new site has entered the system | +| 0085 | DB_REP_NOTPERM: Permanent log record not written | +| 0086 | DB_REP_UNAVAIL: Too few remote sites to complete operation | +| 0087 | DB_RUNRECOVERY: Fatal error, run database recovery | +| 0088 | DB_SECONDARY_BAD: Secondary index inconsistent with primary | +| 0089 | DB_TIMEOUT: Operation timed out | +| 0090 | DB_VERIFY_BAD: Database verification failed | +| 0091 | DB_VERSION_MISMATCH: Database environment version mismatch | +| 0092 | Unknown error: %d | +| 0093 | %s: Unknown flag: %#x | +| 0094 | %s: Unexpected database type: %s | +| 0095 | %s: Unexpected code path error | +| 0096 | Read-only transaction cannot be used for an update | +| 0097 | Transaction not specified for a transactional database | +| 0098 | Transaction specified for a non-transactional database | +| 0099 | Operation forbidden while secondary index is being created | +| 0100 | Transaction and database from different environments | +| 0101 | Transaction that opened the DB handle is still active | +| 0102 | %s%sprevious transaction deadlock return not resolved | +| 0103 | DB environment not configured for transactions | +| 0104 | %lu larger than database's maximum record length %lu | +| 0105 | Record length error: ""replacement length %lu differs from replaced length %lu | +| 0106 | dbc_logging: Client update | +| 0107 | Dbc_logging: Master non-txn update | +| 0108 | Rep: flags 0x%lx msg_th %lu | +| 0109 | Rep: handle %lu, opcnt %lu | +| 0110 | Log sequence error: page LSN %lu %lu; previous LSN %lu %lu | +| 0111 | %s: attempt to modify a read-only database | +| 0112 | %s: file limited to %lu pages | +| 0113 | Thread/process %s failed: %s | +| 0114 | architecture does not support locks inside system shared memory | +| 0115 | no base system shared memory ID specified | +| 0116 | shmget: key: %ld: shared system memory region already exists | +| 0117 | shmget: key: %ld: unable to create shared system memory region | +| 0118 | shmat: id %d: unable to attach to shared system memory region | +| 0119 | shmctl/SHM_LOCK: id %d: unable to lock down shared memory region | +| 0120 | architecture lacks mmap(2), shared environments not possible | +| 0121 | shmdt | +| 0122 | shmctl: id %d: unable to delete system shared memory region | +| 0123 | munmap | +| 0124 | fileops: munmap | +| 0125 | fileops: mmap %s | +| 0126 | mmap | +| 0127 | mlock | +| 0128 | architecture doesn't support environments in system memory | +| 0129 | fileops: mkdir %s | +| 0130 | fileops: read %s: %lu bytes at offset %lu | +| 0131 | fileops: write %s: %lu bytes at offset %lu | +| 0132 | fileops: read %s: %lu bytes | +| 0133 | read: %#lx, %lu | +| 0134 | read: %#lx, %lu | +| 0135 | fileops: write %s: %lu bytes | +| 0136 | write: %#lx, %lu | +| 0137 | write: %#lx, %lu | +| 0138 | fileops: flock %s %s offset %lu | +| 0139 | fcntl | +| 0140 | advisory file locking unavailable | +| 0141 | fileops: truncate %s to %lu | +| 0142 | ftruncate: %lu | +| 0143 | malloc: %lu | +| 0144 | user-specified malloc function returned NULL | +| 0145 | realloc: %lu | +| 0146 | User-specified realloc function returned NULL | +| 0147 | malloc: %lu | +| 0148 | realloc: %lu | +| 0149 | Guard byte incorrect during free | +| 0150 | fileops: flush %s | +| 0151 | fsync | +| 0152 | fileops: open %s | +| 0153 | %s(%u): host lookup failed: %s | +| 0154 | %s(%u): host lookup failed | +| 0155 | %s(%u): host lookup failed: %s | +| 0156 | %s(%u): host lookup failed: %d | +| 0157 | %s: buffer too small to hold environment variable %s | +| 0158 | stat: %s | +| 0159 | fileops: directory list %s | +| 0160 | fileops: unlink %s | +| 0161 | unlink: %s | +| 0162 | fcntl(F_SETFD) | +| 0163 | fileops: close %s | +| 0164 | close | +| 0165 | fileops: stat %s | +| 0166 | fstat | +| 0167 | select | +| 0168 | fileops: rename %s to %s | +| 0169 | rename %s %s | +| 0170 | fileops: seek %s to %lu | +| 0171 | seek: %lu: (%lu \* %lu) + %lu | +| 0172 | Joining non-encrypted environment with encryption key | +| 0173 | Encryption algorithm not supplied | +| 0174 | Encrypted environment: no encryption key supplied | +| 0175 | Invalid password | +| 0176 | Environment encrypted using a different algorithm | +| 0177 | No cipher structure given | +| 0178 | Encrypted database: no encryption flag specified | +| 0179 | Database encrypted using a different algorithm | +| 0180 | Invalid password | +| 0181 | Unencrypted database with a supplied encryption key | +| 0182 | IPP AES NULL pointer error | +| 0183 | IPP AES length error | +| 0184 | IPP AES context does not match operation | +| 0185 | IPP AES srclen size error | +| 0186 | AES key direction is invalid | +| 0187 | AES key material not of correct length | +| 0188 | AES key passwd not valid | +| 0189 | AES cipher in wrong state (not initialized) | +| 0190 | AES bad block length | +| 0191 | AES cipher instance is invalid | +| 0192 | AES data contents are invalid | +| 0193 | AES unknown error | +| 0194 | AES error unrecognized | +| 0195 | Unencrypted checksum with a supplied encryption key | +| 0196 | Encrypted checksum: no encryption key specified | +| 0197 | segment %s does not exist | +| 0198 | no base shared memory ID specified | +| 0199 | key: %ld: shared memory region already exists | +| 0200 | shared memory segment already exists | +| 0201 | shared memory segment not initialized | +| 0202 | shared memory segment not initialized | +| 0203 | no segment name given | +| 0204 | Invalid segment id given | +| 0205 | shared memory segment not initialized | +| 0206 | segment id %ld out of range | +| 0207 | DB_REP_WOULDROLLBACK: Client data has diverged | +| 0208 | DB_HEAP_FULL: no free space in db | + +## Database Handle Messages + +| Message Number | Message Text | +|----|----| +| 0501 | "Page %lu: %s is of inappropriate type %lu | +| 0502 | "Page %lu: totally zeroed page | +| 0503 | Rename on temporary files invalid | +| 0504 | XA applications may not specify an environment to db_create | +| 0505 | Cannot open XA database before XA is enabled | +| 0506 | call implies an access method which is inconsistent with previous calls | +| 0507 | Directory %s not in environment list. | +| 0508 | Database environment not configured for encryption | +| 0509 | page sizes may not be smaller than %lu | +| 0510 | page sizes may not be larger than %lu | +| 0511 | page sizes must be a power-of-2 | +| 0512 | Illegal application-specific record type %lu in log | +| 0513 | Illegal record type %lu in log | +| 0514 | Attempting to add application-specific record with invalid type %lu | +| 0515 | Attempting to add internal record with invalid type %lu | +| 0516 | DB_DBT_PARTIAL may not be set on key during join_get | +| 0517 | Allocation failed for join key, len = %lu | +| 0518 | DB_SALVAGE requires a an output handle | +| 0519 | DB_ORDERCHKONLY requires a database name | +| 0520 | Metadata page %lu cannot be read | +| 0521 | Page %lu: Incomplete metadata page | +| 0522 | Page %lu: metadata page corrupted | +| 0523 | Page %lu: could not check metadata page | +| 0524 | Page %lu: pgno incorrectly set to %lu | +| 0525 | Page %lu: bad magic number %lu | +| 0526 | Page %lu: unsupported DB version %lu; extraneous errors may result | +| 0527 | Page %lu: bad page size %lu | +| 0528 | Page %lu: bad page type %lu | +| 0529 | Page %lu: bad meta-data flags value %#lx | +| 0530 | Page %lu: beyond the end of the file, metadata page has last page as %lu | +| 0531 | Page %lu: old-style duplicate page | +| 0532 | Page %lu: unknown page type %lu | +| 0533 | Page %lu: overflow refcount %lu, referenced %lu times | +| 0534 | Page %lu: unreferenced page | +| 0535 | Page %lu: totally zeroed page | +| 0536 | Page %lu: bad page number %lu | +| 0537 | Page %lu: bad page type %lu | +| 0538 | Page %lu: invalid next_pgno %lu | +| 0539 | Page %lu: invalid prev_pgno %lu | +| 0540 | Page %lu: invalid next_pgno %lu | +| 0541 | Page %lu: too many entries: %lu | +| 0542 | Page %lu: bad btree level %lu | +| 0543 | Page %lu: btree leaf page has incorrect level %lu | +| 0544 | Page %lu: nonzero level %lu in non-btree database | +| 0545 | Page %lu: invalid magic number | +| 0546 | Page %lu: magic number does not match database type | +| 0547 | Page %lu: unsupported database version %lu; extraneous errors may result | +| 0548 | Page %lu: invalid pagesize %lu | +| 0549 | Page %lu: bad meta-data flags value %#lx | +| 0550 | Page %lu: nonempty free list on subdatabase metadata page | +| 0551 | Page %lu: nonsensical free list pgno %lu | +| 0552 | Page %lu: last_pgno is not correct: %lu != %lu | +| 0553 | Page %lu: invalid next_pgno %lu on free list page | +| 0554 | Page %lu: page %lu encountered a second time on free list | +| 0555 | Page %lu: non-invalid page %lu on free list | +| 0556 | Subdatabase entry not page-number size | +| 0557 | Subdatabase entry references invalid page %lu | +| 0558 | Subdatabase entry references page %lu of invalid type %lu | +| 0559 | Subdatabase entry of invalid size | +| 0560 | Page %lu: DB-\>h_internal field is NULL | +| 0561 | Page %lu: incorrect hash function for database | +| 0562 | Page %lu: database metapage of bad type %lu | +| 0563 | Page %lu: entries listing %lu overlaps data | +| 0564 | Page %lu: bad offset %lu at page index %lu | +| 0565 | Page %lu: unaligned offset %lu at page index %lu | +| 0566 | Page %lu: item %lu of unrecognizable type | +| 0567 | Page %lu: item %lu extends past page boundary | +| 0568 | Page %lu: sorted duplicate set in unsorted-dup database | +| 0569 | Page %lu: unsorted duplicate set in sorted-dup database | +| 0570 | Page %lu: duplicate page of inappropriate type %lu | +| 0571 | library build did not include support for database verification | +| 0572 | Databases may not become secondary indices while cursors are open | +| 0573 | Secondary index handles may not be re-associated | +| 0574 | Secondary indices may not be used as primary databases | +| 0575 | Primary databases may not be configured with duplicates | +| 0576 | Renumbering recno databases may not be used as primary databases | +| 0577 | The primary and secondary must be opened in the same environment | +| 0578 | The DB_THREAD setting must be the same for primary and secondary | +| 0579 | Callback function may be NULL only when database handles are read-only | +| 0580 | replication recovery unrolled committed transactions; | +| 0581 | DB-\>del with DB_MULTIPLE(\_KEY) requires multiple key records | +| 0582 | Database does not have a valid file handle | +| 0583 | %s is not supported with DB_CONSUME or DB_CONSUME_WAIT | +| 0584 | DB_DBT_READONLY should not be set on data DBT. | +| 0585 | DB_MULTIPLE requires DB_DBT_USERMEM be set | +| 0586 | DB_MULTIPLE does not support DB_DBT_PARTIAL | +| 0587 | DB_MULTIPLE buffers must be aligned, | +| 0588 | At least one secondary cursor must be specified to DB-\>join | +| 0589 | All secondary cursors must share the same transaction | +| 0590 | files containing multiple databases may only be opened read-only | +| 0591 | DB_TRUNCATE not supported on VxWorks | +| 0592 | DB_UNKNOWN type specified with DB_CREATE or DB_TRUNCATE | +| 0593 | unknown type: %lu | +| 0594 | database environment not yet opened | +| 0595 | environment did not include a memory pool | +| 0596 | environment not created using DB_THREAD | +| 0597 | DB_MULTIVERSION illegal without a transaction specified | +| 0598 | DB_MULTIVERSION illegal with queue databases | +| 0599 | DB_TRUNCATE illegal with %s specified | +| 0600 | Queue databases must be one-per-file | +| 0601 | DB-\>pget may only be used on secondary indices | +| 0602 | DB_MULTIPLE and DB_MULTIPLE_KEY may not be used on secondary indices | +| 0603 | DB_GET_BOTH on a secondary index requires a primary key | +| 0604 | DB-\>put forbidden on secondary indices | +| 0605 | DB-\>put: DB_MULTIPLE(\_KEY) can only be combined with DB_OVERWRITE_DUP | +| 0606 | DB-\>put with DB_MULTIPLE(\_KEY) requires a bulk key buffer | +| 0607 | DB-\>put with DB_MULTIPLE requires a bulk data buffer | +| 0608 | a partial put in the presence of duplicates requires a cursor operation | +| 0609 | DB-\>compact may not be called with active cursors in the transaction. | +| 0610 | Secondary indices may not be used as foreign databases | +| 0611 | Foreign databases may not be configured with duplicates | +| 0612 | Renumbering recno databases may not be used as foreign databases | +| 0613 | The associating database must be a secondary index. | +| 0614 | When specifying a delete action of nullify, a callback | +| 0615 | When not specifying a delete action of nullify, a | +| 0616 | Closing already-closed cursor | +| 0617 | DBcursor-\>cmp dbc pointer must not be null | +| 0618 | DBcursor-\>cmp both cursors must refer to the same database. | +| 0619 | DB_READ_UNCOMMITTED is not supported with DB_CONSUME or DB_CONSUME_WAIT | +| 0620 | DB_DBT_READONLY should not be set on data DBT. | +| 0621 | DB_MULTIPLE/DB_MULTIPLE_KEY require DB_DBT_USERMEM be set | +| 0622 | DB_MULTIPLE/DB_MULTIPLE_KEY do not support DB_DBT_PARTIAL | +| 0623 | DB_MULTIPLE/DB_MULTIPLE_KEY buffers must be | +| 0624 | DBcursor-\>pget may only be used on secondary indices | +| 0625 | DB_MULTIPLE and DB_MULTIPLE_KEY may not be used on secondary indices | +| 0626 | %s requires both a secondary and a primary key | +| 0627 | DB_GET_BOTH on a secondary index requires a primary key | +| 0628 | DBcursor-\>put forbidden on secondary indices | +| 0629 | Bulk and partial operations cannot be combined on %s DBT | +| 0630 | DB_THREAD mandates memory allocation flag on %s DBT | +| 0631 | Cursor position must be set before performing this operation | +| 0632 | DB_AUTO_COMMIT may not be specified along with a transaction handle | +| 0633 | DB_AUTO_COMMIT may not be specified in non-transactional environment | +| 0634 | Partitioned databases may not be in memory. | +| 0635 | DB_CREATE must be specified to create databases. | +| 0636 | DBTYPE of unknown without existing file | +| 0637 | Partitioned databases may not be included with multiple databases. | +| 0638 | %s: Invalid type %d specified | +| 0639 | Invalid subdatabase type %d specified | +| 0640 | %s: metadata page checksum error | +| 0641 | \_\_db_meta_setup: %s: unexpected file type or format | +| 0642 | Checksum failure requires catastrophic recovery | +| 0643 | Cannot replicate prepared transactions from master running release 4.2 | +| 0644 | Partition open failed to allocate %d bytes | +| 0645 | Cannot specify callback and range keys. | +| 0646 | Must specify at least 2 partitions. | +| 0647 | Must specify either keys or a callback. | +| 0648 | May not specify both keys and a callback. | +| 0649 | Directory not in environment list %s | +| 0650 | Partitioning may only specified on BTREE and HASH databases. | +| 0651 | Partitioning specified on a non-partitioned database. | +| 0652 | Incompatible partitioning specified. | +| 0653 | Partition callback not specified. | +| 0654 | Record numbers are not supported in partitioned databases. | +| 0655 | Zero paritions specified. | +| 0656 | Number of partitions does not match. | +| 0657 | Hash database must specify a partition callback. | +| 0658 | Partitioning only supported on BTREE nad HASH. | +| 0659 | No range keys found. | +| 0660 | Keys found and callback set. | +| 0661 | Partition key 0 is not empty. | +| 0662 | Partition key %d does not match | +| 0663 | A partitioned database can not be in a multiple databases file | +| 0664 | library build did not include support for the database partitioning | +| 0665 | upgrade not supported | +| 0666 | %s: unsupported btree version: %lu | +| 0667 | Attempt to upgrade an encrypted database without providing a password. | +| 0668 | %s: unsupported hash version: %lu | +| 0669 | %s: unsupported queue version: %lu | +| 0670 | %s: DB-\>upgrade only supported on native byte-order systems | +| 0671 | %s: unrecognized file type | +| 0672 | %s: file size not a multiple of the pagesize | +| 0673 | rename: database %s exists | +| 0674 | Closing a primary DB while a secondary DB has active cursors is unsafe | +| 0675 | \_\_env_fileid_reset: %s: unexpected file type or format | +| 0676 | Page %lu: overflow page has zero reference count | +| 0677 | Page %lu: overflow page of invalid type %lu | +| 0678 | Page %lu: first page in overflow chain has a prev_pgno %lu | +| 0679 | Page %lu: encountered too many times in overflow traversal | +| 0680 | Page %lu: overflow page linked twice from leaf or data page | +| 0681 | Page %lu: bad next_pgno %lu on overflow page | +| 0682 | Page %lu: bad prev_pgno %lu on overflow page (should be %lu) | +| 0683 | Page %lu: overflow item incomplete | +| 0684 | checksum error: page %lu: catastrophic recovery required | +| 0685 | DB-\>truncate forbidden on secondary indices | +| 0686 | DB-\>truncate not permitted with active cursors | +| 0687 | CDS groups do not support %s | +| 0688 | CDS group has active cursors | +| 0689 | %s page %lu is on free list with type %lu | +| 0690 | DB_LOG_NO_DATA may not be specified within a transaction. | +| 0691 | Remove on temporary files invalid | +| 0692 | Both cursors must be initialized before calling DBC-\>cmp. | +| 0693 | Both cursors must be initialized before calling DBC-\>cmp. | +| 0694 | DBCursor-\>cmp mismatched off page duplicate cursor pointers. | +| 0695 | Put results in a non-unique secondary key in an | +| 0696 | Duplicate data items are not supported with sorted data | +| 0697 | Write attempted on read-only cursor | +| 0698 | Attempt to execute cascading delete in a foreign index failed | +| 0699 | Foreign database application callback | +| 0700 | Attempt to overwrite item in foreign database with nullified value failed | +| 0701 | DB_PRIVATE is not supported by | +| 0702 | Deadlock while opening %s, retrying | +| 0708 | Invalid positioning flag combined with DB_DBT_PARTIAL | +| 0709 | The primary key returned by pget can't be partial | +| 0710 | Invalid positioning flag combined with DB_DBT_PARTIAL | +| 0711 | The primary key returned by pget can't be partial. | +| 0713 | Page %lu: page %lu on free list beyond last_pgno %lu | +| 0714 | Metadata page %lu cannot be read from mpool | +| 0715 | removing %s | +| 0716 | Target directory may not be null. | +| 0717 | %s: path too long | +| 0718 | %s: directory read | +| 0719 | highest numbered log file removed: %d | +| 0720 | %s: path too long | +| 0721 | %s: cannot create | +| 0722 | %s: path too long | +| 0723 | %s: directory read | +| 0724 | copying database %s%c%s to %s%c%s | +| 0725 | data directory '%s' is absolute path, not permitted unless backup is to a single directory | +| 0726 | copying %s to %s | +| 0727 | %lu buffer allocation | +| 0728 | %s: path too long | +| 0729 | %s%c%s not present | +| 0731 | Sync failed | +| 0732 | %s: path too long | +| 0733 | %s: path too long | +| 0734 | %s: cannot create | +| 0735 | Can't flush log | +| 0736 | Can't get log file names | +| 0737 | %s: path too long | +| 0738 | %s: path too long | +| 0739 | moving %s to %s | +| 0740 | removing %s | +| 0741 | unlink of %s failed | +| 0742 | lowest numbered log file copied: %d | +| 0743 | the largest log file removed (%d) must be greater than or equal the smallest log file copied (%d) | +| 0745 | Write failed. | +| 0746 | Exclusive database handles cannot be threaded. | +| 0747 | Exclusive database handles require transactional environments. | +| 0748 | Exclusive database handles cannot be opened on replication clients. | + +## Environment Handle Messages + +| Message Number | Message Text | +|----|----| +| 1501 | Logging region out of memory; you may need to increase its size | +| 1502 | Freeing log information for process: %s, (ref %lu) | +| 1503 | DB_ENV-\>failchk requires DB_ENV-\>is_alive be configured | +| 1504 | is_alive method specified but no thread region allocated | +| 1505 | thread table must be allocated when the database environment is created | +| 1506 | unable to allocate a thread status block | +| 1507 | Thread died in Berkeley DB library | +| 1508 | Unable to allocate thread control block | +| 1509 | Invalid recovery timestamp %s; earliest time is %s | +| 1510 | First log record not found | +| 1511 | Invalid checkpoint record at \[%ld\]\[%ld\] | +| 1512 | Last log record not found | +| 1513 | Checkpoint LSN record \[%ld\]\[%ld\] not found | +| 1514 | Recovery starting from \[%lu\]\[%lu\] | +| 1515 | Recovery continuing after non-fatal checkpoint error: %s | +| 1516 | First log record not found | +| 1517 | Invalid checkpoint record at \[%ld\]\[%ld\] | +| 1518 | Recovery complete at %.24s | +| 1519 | Maximum transaction ID %lx recovery checkpoint \[%lu\]\[%lu\] | +| 1520 | Recovery function for LSN %lu %lu failed on %s pass | +| 1521 | Recovery function for LSN %lu %lu failed | +| 1522 | Log file corrupt at LSN: \[%lu\]\[%lu\] | +| 1523 | Unknown version %lu | +| 1524 | %lu: register environment | +| 1525 | %lu: creating %s | +| 1526 | %lu: adding self to registry | +| 1527 | %02u: EMPTY | +| 1528 | DB_REGISTER limits processes to one open DB_ENV handle per environment | +| 1529 | %02u: %s: KILLED | +| 1530 | %02u: %s: FAILED | +| 1531 | %02u: %s: LOCKED | +| 1532 | %lu: locking slot %02u at offset %lu | +| 1533 | %lu: recovery completed, unlocking | +| 1534 | %s: exclusive file unlock | +| 1535 | %s: existing environment not created in system memory | +| 1536 | %s: unable to read region info | +| 1537 | %s: unable to read system-memory information | +| 1538 | Program version %d.%d doesn't match environment version %d.%d | +| 1539 | Build signature doesn't match environment | +| 1540 | configured environment flags incompatible with existing environment | +| 1542 | Minimum environment memory size %ld is bigger than spcified max %ld. | +| 1543 | unable to create new master region array | +| 1544 | %s: unable to find environment | +| 1545 | %s: unable to write out public environment ID | +| 1546 | unable to join the environment | +| 1547 | environment reference count went negative | +| 1548 | region size %lu is too large; maximum is %lu | +| 1549 | region max %lu is too large; maximum is %lu | +| 1550 | architecture does not support locks inside process-local (malloc) memory | +| 1551 | application may not specify both DB_PRIVATE and DB_THREAD | +| 1552 | region memory was not correctly aligned | +| 1553 | no room remaining for additional REGIONs | +| 1554 | Library build did not include statistics support | +| 1555 | library build did not include support for cryptography | +| 1556 | Empty password specified to set_encrypt | +| 1557 | library build did not include support for cryptography | +| 1558 | Environment panic set | +| 1559 | DB_TXN_NOSYNC and DB_TXN_WRITE_NOSYNC | +| 1560 | Attempt to decrement hotbackup counter past zero | +| 1561 | Directory %s not in environment list. | +| 1562 | is_alive method specified but no thread region allocated | +| 1563 | is_alive method specified but no thread region allocated | +| 1564 | %s: method not permitted when environment specified | +| 1565 | %s: method not permitted %s handle's open method | +| 1566 | %s interface requires an environment configured for the %s subsystem | +| 1567 | The DB_RECOVER flag was not specified, and recovery is needed | +| 1568 | Berkeley DB library does not support DB_REGISTER on this system | +| 1569 | registration requires transaction support | +| 1570 | Berkeley DB library does not support replication on this system | +| 1571 | replication requires locking support | +| 1572 | replication requires transaction support | +| 1573 | recovery requires the create flag | +| 1574 | recovery requires transaction support | +| 1575 | DB_FAILCHK requires DB_ENV-\>is_alive be configured | +| 1576 | DB_FAILCHK requires DB_ENV-\>set_thread_count be configured | +| 1577 | Berkeley DB library configured to support only private environments | +| 1578 | architecture lacks fast mutexes: applications cannot be threaded | +| 1579 | Database handles still open at environment close | +| 1580 | Open database handle: %s%s%s | +| 1581 | File handles still open at environment close | +| 1582 | Open file handle: %s | +| 1583 | block not at end of region | +| 1584 | line %d: %s: incorrect name-value pair | +| 1585 | unrecognized name-value pair: %s | +| 1586 | temporary open: %s | +| 1587 | %s interface requires an environment configured with %s | +| 1588 | Maximum memory size too large: maximum is 4GB | +| 1589 | DB_PRIVATE is not | +| 1590 | Could not add %s to environment list. | + +## Locking Subsystem Messages + +| Message Number | Message Text | +|----|----| +| 2001 | library build did not include support for mutexes | +| 2002 | Win32 create event failed | +| 2003 | Win32 lock failed: mutex already locked by %s | +| 2004 | \[%I64d\]: Lost signal on mutex %p, ""id %d, ms %d | +| 2005 | \[%I64d\]: Waiting on mutex %p, id %d | +| 2006 | Win32 lock failed | +| 2007 | \[%I64d\]: Lost signal on mutex %p, ""id %d, ms %d | +| 2008 | \[%I64d\]: Waiting on mutex %p, id %d | +| 2009 | Win32 read lock failed | +| 2010 | Win32 unlock failed: lock already unlocked: mutex %d busy %d | +| 2011 | \[%I64d\]: Signalling mutex %p, id %d | +| 2012 | Win32 unlock failed | +| 2013 | Unable to allocate memory for the mutex region | +| 2014 | Unable to allocate memory for mutexes from the region | +| 2015 | Unable to acquire/release a mutex; check configuration | +| 2016 | Unable to acquire/release a shared latch; check configuration | +| 2017 | Freeing mutex for process: %s | +| 2018 | DB_ENV-\>mutex_set_align: alignment value must be a non-zero power-of-two | +| 2019 | fcntl lock failed | +| 2020 | fcntl unlock failed: lock already unlocked | +| 2021 | unable to initialize mutex | +| 2022 | pthread lock failed: lock currently in use: pid/tid: %s | +| 2023 | pthread lock failed | +| 2024 | pthread readlock failed | +| 2025 | pthread unlock failed: lock already unlocked | +| 2026 | unable to destroy cond | +| 2027 | unable to destroy mutex | +| 2028 | TAS: mutex not appropriately aligned | +| 2029 | TAS: mutex initialize | +| 2030 | TAS lock failed: lock %ld currently in use: ID: %s | +| 2031 | shared unlock %ld already unlocked | +| 2032 | unlock %ld already unlocked | +| 2033 | Mutex allocated before mutex region. | +| 2034 | unable to allocate memory for mutex; resize mutex region | +| 2035 | Invalid lock operation: %d | +| 2036 | Locker does not exist | +| 2037 | DB_ENV-\>lock_get: invalid lock mode %lu | +| 2038 | Unexpected lock status: %d | +| 2039 | Not a child transaction | +| 2040 | Locker does not exist | +| 2041 | lock_open: incompatible deadlock detector mode | +| 2042 | unable to allocate memory for the lock table | +| 2043 | DB_ENV-\>set_lk_detect: unknown deadlock detection mode specified | +| 2044 | DB_ENV-\>set_lk_detect: incompatible deadlock detector mode | +| 2045 | Unknown locker id: %lx | +| 2046 | Locker still has locks | +| 2047 | Freeing locker with locks | +| 2048 | DB_ENV-\>lock_detect: unknown deadlock detection mode specified | +| 2049 | warning: unable to abort locker %lx | +| 2050 | Aborting locker %lx | +| 2051 | %lu lockers | +| 2052 | locker has write locks | +| 2053 | Freeing read locks for locker %#lx: %s | +| 2054 | library build did not include support for locking | +| 2055 | Lock table is out of available %s | + +## Logging Subsystem Messages + +| Message Number | Message Text | +|----|----| +| 2501 | Set either an lsn range or a time range to verify logs | +| 2502 | \[%lu\]\[%lu\] Unsupported version of log file, ""log file number: %u, log file version: %u, ""supported log version: %u. | +| 2503 | callback: initialization | +| 2504 | Log verification ended and %s. | +| 2505 | Not supported version %lu | +| 2506 | file %s has LSN %lu/%lu, past end of log at %lu/%lu | +| 2507 | Commonly caused by moving a database from one database environment | +| 2508 | to another without clearing the database LSNs, or by removing all of | +| 2509 | the log files from a database environment | +| 2510 | Logging not currently permitted | +| 2511 | DB_ENV-\>log_put is illegal on replication clients | +| 2512 | Non-replication DB_ENV handle attempting | +| 2513 | DB_ENV-\>log_put: record larger than maximum file size (%lu \> %lu) | +| 2514 | Write failed on MASTER commit. | +| 2515 | Short read while restoring log | +| 2516 | DB_ENV-\>log_flush: LSN of %lu/%lu past current end-of-log of %lu/%lu | +| 2517 | Database environment corrupt; the wrong log files may | +| 2518 | DB_ENV-\>log_file is illegal with in-memory logs | +| 2519 | DB_ENV-\>log_file: name buffer is too short | +| 2520 | %s: log file unreadable | +| 2521 | %s: log file open failed | +| 2522 | DB_ENV-\>log_put is illegal on replication clients | +| 2523 | library build did not include support for log verification | +| 2524 | unable to allocate log region memory | +| 2525 | No log files found | +| 2526 | Finding last valid log LSN: file: %lu offset %lu | +| 2527 | Invalid log file: %s | +| 2528 | ignoring log file: %s | +| 2529 | Ignoring log file: %s historic byte order | +| 2530 | Ignoring log file: %s: magic number %lx, not %lx | +| 2531 | Unacceptable log file %s: unsupported log version %lu | +| 2532 | Skipping log file %s: historic log version %lu | +| 2533 | log record checksum mismatch | +| 2534 | Warning: truncating to point beyond end of log | +| 2535 | In-memory log buffer is full (an active transaction spans the buffer) | +| 2536 | "\[%lu\]\[%lu\] Not supported type of log record %u. | +| 2537 | \[%lu\]\[%lu\] \[WARNING\] Parent txn %lx is updating its ""active child txn %lx's pages, or %lx aborted. | +| 2538 | \[%lu\]\[%lu\] \[WARNING\] Txn %lx is updating txn %lx's pages. | +| 2539 | \[%lu\]\[%lu\] Verifying log record of type %s | +| 2540 | \[%lu\]\[%lu\] Log record type does not match related database type, ""current database type: %s, expected database type according to ""the log record type: %s. | +| 2541 | \[%lu\]\[%lu\] Suspicious dbreg operation: %s, the ""database file %s's register in log region does ""not begin with an open operation. | +| 2542 | \[%lu\]\[%lu\] Wrong dbreg operation ""sequence, opening %s for id %d which is already ""open. | +| 2543 | \[%lu\]\[%lu\] Wrong dbreg operation sequence,""file %s with id %d is first seen of ""status: %s | +| 2544 | \[%lu\]\[%lu\] The dbtype of database file %s with uid %s "" and id %d has changed from %s to %s. | +| 2545 | \[%lu\]\[%lu\] Wrong dbreg operation sequence for file %s ""with id %d, current status: %s, new status: %s | +| 2546 | \[%lu\]\[%lu\] \_\_ham_groupalloc should apply only to the ""master database with meta page number 0, current meta ""page number is %d. | +| 2547 | \[%lu\]\[%lu\] Can not find an active transaction's ""information, txnid: %lx. | +| 2548 | \[%lu\]\[%lu\] The number of active, committed and aborted ""child txns of txn %lx: %u, %u, %u. | +| 2549 | \[%lu\]\[%lu\] Checkpoint record, ckp_lsn: \[%lu\]\[%lu\], ""timestamp: %s. Total checkpoint: %u | +| 2550 | \[%lu\]\[%lu\] Last known checkpoint \[%lu\]\[%lu\] not equal ""to last_ckp :\[%lu\]\[%lu\]. Some checkpoint log records ""may be missing. | +| 2551 | \[%lu\]\[%lu\] Last known checkpoint \[%lu, %lu\] has a ""timestamp %s smaller than this checkpoint timestamp %s. | +| 2552 | \[%lu\]\[%lu\] ckp log's ckp_lsn \[%lu\]\[%lu\] greater than ""active txn %lx 's first lsn \[%lu\]\[%lu\] | +| 2553 | \[%lu\]\[%lu\] Can not find an active transaction's ""information, txnid: %lx. | +| 2554 | \[%lu\]\[%lu\] Parent txn %lx ended ""before child txn %lx ends. | +| 2555 | \[%lu\]\[%lu\] Can not find an active ""transaction's information, txnid: %lx. | +| 2556 | \[%lu\]\[%lu\] Txn %lx ended before it commits. | +| 2557 | \[%lu\]\[%lu\] Can not find an active transaction's ""information, txnid: %lx. | +| 2558 | \[%lu\]\[%lu\] Multiple txn_prepare log record for ""transaction %lx, previous prepare lsn: \[%lu, %lu\]. | +| 2559 | \[%lu\]\[%lu\] \[WARNING\] This log record of type %s ""does not have a greater time stamp than ""\[%lu, %lu\] of type %s | +| 2560 | \[%lu\]\[%lu\] Transaction %lx is updating a ""db file %d not registered. | +| 2561 | \[%lu\]\[%lu\] Can not find an active transaction's ""information, txnid: %lx. | +| 2562 | \[%lu\]\[%lu\] Previous record for transaction %lx is ""\[%lu\]\[%lu\] and prev_lsn is \[%lu\]\[%lu\]. | +| 2563 | \[%lu\]\[%lu\] Update action is performed in a ""prepared transaction %lx. | +| 2564 | \[%lu\]\[%lu\] Transaction id %lx reused without ""being recycled with a \_\_txn_recycle. | +| 2565 | \[%lu\]\[%lu\] Non-transactional update, ""log type: %u, fileid: %d. | +| 2566 | \[%lu\]\[%lu\] Can not find an active transaction's ""information, txnid: %lx. | +| 2567 | \[%lu\]\[%lu\] Txn %lx aborted after this log record. | +| 2568 | The number of active, committed and aborted child txns ""of txn %lx: %u, %u, %u. | +| 2569 | log region size must be \>= %d | +| 2570 | no absolute path for the current directory | +| 2571 | log file auto-remove | +| 2572 | DB_ENV-\>log_archive: bad log record | +| 2573 | DB_ENV-\>log_archive: unable to read log record | +| 2574 | DB_LOGC-\>get: unset cursor | +| 2575 | DB_LOGC-\>get: invalid LSN: %lu/%lu | +| 2576 | Encountered zero length records while traversing backwards | +| 2577 | DB_LOGC-\>get: log record LSN %lu/%lu: ""checksum mismatch, hdr.chksum: %s, hdr.prev: %u, ""hdr.len: %u, log type: %u. Skipping it and ""continuing with the %s one | +| 2578 | DB_LOGC-\>get: log record LSN %lu/%lu: checksum mismatch | +| 2579 | DB_LOGC-\>get: catastrophic recovery may be required | +| 2580 | DB_LOGC-\>get: LSN %lu/%lu: invalid log record header | +| 2581 | DB_LOGC-\>get: LSN: %lu/%lu: read | +| 2582 | DB_LOGC-\>get: LSN: %lu/%lu: short read | +| 2583 | Log file %d not found, check log directory configuration | + +## Memory Pool Messages + +| Message Number | Message Text | +|----|----| +| 0703 | Cannot allocate space for path: %s | +| 0704 | Cannot open traget file: %s | +| 0712 | %s is already in a backup | +| 0714 | Releasing backup of %s for %s. | +| 3001 | %smethod not permitted when replication is configured | +| 3002 | %s: non-transactional update to a multiversion file | +| 3003 | individual cache size too large: maximum is 4GB | +| 3004 | individual cache size too large: maximum is 10TB | +| 3005 | Truncate beyond the end of file | +| 3006 | Can't get the required free size while | +| 3007 | DB_ENV-\>memp_trickle: %d: percent must be between 1 and 100 | +| 3008 | %s: dirty flag set for readonly file page | +| 3009 | %s: error releasing a read-only page | +| 3010 | %s: error getting a page for writing | +| 3011 | %s: more pages returned than retrieved | +| 3012 | %s: page %lu: unpinned page returned | +| 3013 | \_\_memp_fput: pinned buffer not found for thread %s | +| 3014 | unable to create temporary backing file | +| 3015 | %s: write failed for page %lu | +| 3016 | %s: %s failed for page %lu | +| 3017 | unable to allocate space from the buffer cache | +| 3018 | %s: unwritable page %d remaining in the cache after error %d | +| 3019 | cannot remove the last cache | +| 3020 | cannot resize to %lu cache regions: maximum is %lu | +| 3021 | %s: dirty flag set for readonly file page | +| 3022 | %s: page %lu: reference count overflow | +| 3023 | %s: file limited to %lu pages | +| 3024 | %s: file limited to %lu pages | +| 3025 | DB_MPOOLFILE-\>get: buffer data is NOT size_t aligned | +| 3026 | Unable to allocate memory for mpool region | +| 3027 | %s: unable to flush page: %lu | +| 3028 | %s: unable to flush | +| 3029 | DB_ENV-\>memp_fcreate: method not permitted when replication is configured | +| 3030 | get_fileid: file ID not set | +| 3031 | DB_MPOOLFILE-\>get_priority: unknown priority value: %d | +| 3032 | DB_MPOOLFILE-\>set_priority: unknown priority value: %d | +| 3033 | DB_MPOOLFILE-\>open: page sizes must be a power-of-2 | +| 3034 | DB_MPOOLFILE-\>open: clear length larger than page size | +| 3035 | DB_MPOOLFILE-\>open: temporary files can't be readonly | +| 3036 | DB_MPOOLFILE-\>open: DB_MULTIVERSION requires transactions | +| 3037 | %s: file size not a multiple of the pagesize | +| 3038 | %s: clear length, page size or LSN location changed | +| 3039 | Cannot open DURABLE and NOT DURABLE handles in the same file | +| 3040 | %s: close: %lu blocks left pinned | + +## Replication Messages + +| Message Number | Message Text | +|----|----| +| 3501 | Request for LSN \[%lu\]\[%lu\] not found | +| 3502 | Client initialization failed. Need to manually restore client | +| 3503 | rep_send_message: Unknown rep version %lu, my version %lu | +| 3504 | Operation locked out. Waiting for replication lockout to complete | +| 3509 | Operation locked out. Waiting for replication lockout to complete | +| 3510 | Waiting for %s (%lu) to complete replication lockout | +| 3511 | Waiting for %s (%lu) to complete replication lockout for %d minutes | +| 3512 | %s cannot call from Replication Manager application | +| 3513 | DB_ENV-\>rep_process_message: control argument must be specified | +| 3514 | Environment not configured as replication master or client | +| 3515 | DB_ENV-\>rep_process_message: error retrieving DBT contents | +| 3516 | unsupported old replication message version %lu, minimum version %d | +| 3517 | unexpected replication message version %lu, expected %d | +| 3518 | unsupported old replication log version %lu, minimum version %d | +| 3519 | unexpected log record version %lu, expected %d | +| 3520 | Inconsistent lease configuration | +| 3521 | DB_ENV-\>rep_process_message: unknown replication message: type %lu | +| 3522 | failed to read the log at \[%lu\]\[%lu\] | +| 3523 | transaction failed at \[%lu\]\[%lu\] | +| 3524 | collect failed at: \[%lu\]\[%lu\] | +| 3525 | Error syncing ckp \[%lu\]\[%lu\] | +| 3526 | Error processing txn \[%lu\]\[%lu\] | +| 3527 | DB_ENV-\>rep_elect: cannot call from Replication Manager application | +| 3528 | DB_ENV-\>rep_elect: must be called after DB_ENV-\>rep_set_transport | +| 3529 | DB_ENV-\>rep_elect: must be called after DB_ENV-\>rep_start | +| 3530 | DB_ENV-\>rep_elect: nsites must be zero if leases configured | +| 3531 | DB_ENV-\>rep_elect:WARNING: nvotes (%d) is sub-majority with nsites (%d) | +| 3532 | DB_ENV-\>rep_elect: nvotes (%d) is larger than nsites (%d) | +| 3533 | No electable site found: recvd %d of %d votes from %d sites | +| 3534 | Not enough votes to elect: recvd %d of %d from %d sites | +| 3535 | Application type mismatch for a replication | +| 3548 | %s cannot configure repmgr settings from base replication application | +| 3549 | %s in-memory replication must be configured before DB_ENV-\>open | +| 3550 | DB_ENV-\>rep_set_config: leases must be | +| 3551 | DB_ENV-\>rep_set_config: leases cannot be turned off | +| 3552 | DB_ENV-\>rep_start: cannot call from Replication Manager application | +| 3553 | DB_ENV-\>rep_start: must specify DB_REP_CLIENT or DB_REP_MASTER | +| 3554 | DB_ENV-\>rep_start: must be called after DB_ENV-\>rep_set_transport | +| 3555 | DB_ENV-\>rep_start: must call DB_ENV-\>rep_set_timeout for leases first | +| 3556 | DB_ENV-\>rep_start: Cannot become master during internal init | +| 3557 | rep_start: Cannot become master without being elected when using leases. | +| 3558 | rep_start: Cannot become master with outstanding lease granted. | +| 3559 | First record not found | +| 3560 | Checkpoint record at LSN \[%lu\]\[%lu\] not found | +| 3561 | Invalid checkpoint record at \[%lu\]\[%lu\] | +| 3562 | Checkpoint LSN record \[%lu\]\[%lu\] not found | +| 3563 | Attempt to get first log record failed | +| 3564 | Final log record not found | +| 3565 | DB_ENV-\>rep_set_nsites: cannot call from Replication Manager application | +| 3566 | timeout value must be \> 0 | +| 3567 | %scannot set Replication Manager timeout from base replication application | +| 3568 | %s: lease timeout must be set before DB_ENV-\>rep_start. | +| 3569 | Unknown timeout type argument to DB_ENV-\>rep_set_timeout | +| 3570 | unknown timeout type argument to DB_ENV-\>rep_get_timeout | +| 3571 | DB_ENV-\>rep_set_request: Invalid min or max values | +| 3572 | DB_ENV-\>rep_set_transport: cannot call from | +| 3573 | DB_ENV-\>rep_set_transport: no send function specified | +| 3574 | DB_ENV-\>rep_set_transport: eid must be greater than or equal to 0 | +| 3575 | DB_ENV-\>rep_set_clockskew: Zero only valid for | +| 3576 | DB_ENV-\>rep_set_clockskew: slow_clock value is | +| 3577 | DB_ENV-\>rep_set_clockskew: must be called before DB_ENV-\>rep_start | +| 3578 | DB_ENV-\>rep_flush: must be called after DB_ENV-\>rep_set_transport | +| 3579 | DB_ENV-\>rep_sync: must be called after DB_ENV-\>rep_set_transport | +| 3580 | non-replication commit token in replication env | +| 3581 | library build did not include support for replication | +| 3582 | closing socket | +| 3583 | releasing WSA event object | +| 3584 | can't create listen socket | +| 3585 | can't set REUSEADDR socket option | +| 3586 | can't bind socket to listening address | +| 3587 | listen() | +| 3588 | can't unblock listen socket | +| 3589 | unable to initialize Windows networking | +| 3590 | can't create event for listen socket | +| 3591 | can't enable event for listener | +| 3592 | can't set event bits 0x%lx | +| 3593 | EnumNetworkEvents | +| 3613 | "unexpected msg type %d in state %d | +| 3614 | select loop failed | +| 3615 | accept error | +| 3616 | can't set nonblock after accept | +| 3617 | connector thread failed | +| 3618 | set_nonblock in connnect thread | +| 3619 | illegal size for rep msg | +| 3620 | unexpected msg type %d in PARAMETERS state | +| 3621 | unexpected msg type rcvd in ready state: %d | +| 3622 | No available version between %lu and %lu | +| 3623 | Can't support confirmed version %lu | +| 3624 | handshake is missing rec part | +| 3625 | malformed V1 handshake | +| 3626 | can't set KEEPALIVE socket option | +| 3627 | bad ack msg size | +| 3628 | library build did not include support for the Replication Manager | +| 3629 | unexpected election failure | +| 3630 | pthread_attr_init in repmgr_thread_start | +| 3631 | pthread_attr_setstacksize in repmgr_thread_start | +| 3632 | can't access signal handler | +| 3633 | can't access signal handler | +| 3634 | select | +| 3635 | repmgr_start: unrecognized flags parameter value | +| 3636 | Replication Manager needs an environment with DB_THREAD | +| 3637 | A local site must be named before calling repmgr_start | +| 3638 | Could not clean up repmgr | +| 3639 | a non-zero flags value is required for initial repmgr_start() call | +| 3640 | repmgr is already started | +| 3641 | repmgr_start: nthreads parameter must be \>= %d | +| 3642 | can't configure repmgr elections from subordinate process | +| 3643 | subsequent repmgr_start() call may not specify DB_REP_ELECTION | +| 3644 | repmgr_start: nthreads parameter must be \>= 0 | +| 3645 | can't start selector thread | +| 3646 | unknown ack_policy in DB_ENV-\>repmgr_set_ack_policy | +| 3648 | repmgr_site: a host name is required | +| 3649 | repmgr_site: port out of range \[1,%u\] | +| 3650 | DB_ENV-\>repmgr_channel: must be called after DB_ENV-\>repmgr_start | +| 3651 | repmgr is stopped | +| 3652 | %d is not a valid remote EID | +| 3653 | set_nonblock channel | +| 3654 | DB_CHANNEL-\>send_request() not supported on DB_EID_BROADCAST channel | +| 3655 | No message dispatch call-back function has been configured | +| 3656 | Application failed to provide a response | +| 3657 | a response has already been sent | +| 3658 | originator does not accept multi-segment response | +| 3659 | originator's USERMEM buffer too small | +| 3660 | %s() invalid on DB_CHANNEL supplied to msg dispatch function | +| 3661 | %s: cannot call from base replication application | +| 3662 | Can't determine EID before env open | +| 3663 | Site config value not applicable to local site | +| 3665 | Unrecognized site config value | +| 3666 | A previously given local site may not be unset | +| 3667 | A (different) local site has already been set | +| 3668 | Local site cannot have HELPER or PEER attributes | +| 3669 | repmgr is not running | +| 3670 | No message dispatch call-back function has been configured | +| 3671 | Application failed to provide a response | +| 3672 | Nsites unknown before repmgr_start() | +| 3673 | rep_start | +| 3674 | A mismatching local site address has been set in the environment | +| 3675 | Not enough input bytes to fill a \_\_repmgr_connect_reject message | +| 3676 | unexpected msg type %lu in prepare_input | +| 3677 | unexpected msg type %lu in process_own_msg | +| 3678 | unexpected conn version %lu in send_handshake | +| 3679 | unexpected conn version %lu in accept_handshake | +| 3681 | invalid own buf size %lu in prepare_input | +| 3682 | invalid cur resp %lu in prepare_input | +| 3683 | unexpected connection info in record_permlsn | + +## Sequences Messages + +| Message Number | Message Text | +|----|----| +| 4001 | Zero length sequence key specified | +| 4002 | Sequences not supported in databases configured for duplicate data | +| 4003 | Sequence value out of range | +| 4004 | Sequence create failed | +| 4005 | Bad sequence record format | +| 4006 | Unsupported sequence version: %d | +| 4007 | Cache size must be \>= 0 | +| 4008 | Sequence value out of range | +| 4009 | Minimum sequence value must be less than maximum sequence value | +| 4010 | Bad sequence record format | +| 4011 | Sequence overflow | +| 4012 | Sequence update failed | +| 4013 | Sequence overflow | +| 4014 | Number of items to be cached is larger than the sequence range | +| 4015 | library build did not include support for sequences | +| 4016 | Heap databases may not be used with sequences. | + +## Transaction Messages + +| Message Number | Message Text | +|----|----| +| 4501 | Transaction has in memory logs | +| 4502 | Transaction has in memory logs | +| 4503 | Aborting txn %#lx: %s | +| 4504 | Transaction abort failed | +| 4505 | operation not permitted while in recovery | +| 4506 | Invalid checkpoint record at \[%lu\]\[%lu\] | +| 4507 | No log records | +| 4508 | Unable to allocate memory for the transaction region | +| 4509 | unable to discard txn %#lx | +| 4510 | unable to abort transaction %#lx | +| 4511 | Error: closing the transaction region with active transactions | +| 4512 | Current ID value %lu below minimum | +| 4513 | Maximum ID value %lu below minimum | +| 4514 | txnid %lx commit record found, already on commit list | +| 4515 | transaction not in list %lx | +| 4516 | Transaction not in list %x | +| 4517 | txnid %lx commit record found, already on commit list | +| 4518 | txn_checkpoint: failed to flush the buffer cache | +| 4519 | txn_checkpoint: failed to flush the buffer cache | +| 4520 | txn_checkpoint: log failed at LSN \[%ld %ld\] | +| 4521 | Family transactions cannot have parents | +| 4522 | Child transaction snapshot setting must match parent | +| 4523 | Unable to allocate transaction recycle buffer | +| 4524 | operation not permitted during recovery | +| 4525 | Unable to allocate memory for transaction detail | +| 4526 | commit token unavailable for nested txn | +| 4527 | may not be called on a replication client | +| 4528 | DB_TXN-\>prepare: log_write failed | +| 4529 | Unable to allocate memory for transaction name | +| 4530 | operation not permitted during recovery | +| 4531 | transaction has active cursors | +| 4532 | not a restored transaction | +| 4533 | Prepare disallowed on child transactions | +| 4534 | transaction already prepared | +| 4535 | transaction already %s | +| 4536 | DB_TXN-\>abort: in-memory log undo failed | +| 4537 | DB_TXN-\>abort: log undo failed for LSN: %lu %lu | +| 4538 | Child transaction is active | +| 4539 | replication commit token in non-replication env | +| 4540 | xa_get_txn: transaction begin failed | +| 4541 | xa_get_txn: XA transaction with parent | +| 4542 | xa_get_txn: transaction does not exist | +| 4543 | xa_get_txn: txn_continue fails | +| 4544 | xa_get_txn: os_malloc failed | +| 4545 | xa_open: Failure creating env handle | +| 4546 | xa_open: Failure setting thread count | +| 4547 | xa_open: Failure opening environment | +| 4548 | xa_open: Failure getting log configuration | +| 4549 | xa_open: In-memory logging not allowed in XA environment | +| 4550 | xa_start: failure mapping xid | +| 4551 | xa_end: failure mapping xid | +| 4552 | xa_end: cannot end with open cursors | +| 4553 | xa_end: txn_detail mismatch | +| 4554 | xa_end: ending transaction that is idle | +| 4555 | xa_prepare: failure mapping xid | +| 4556 | xa_prepare: xid not found | +| 4557 | xa_prepare: transaction neither active nor idle | +| 4558 | xa_prepare: txnp-\>prepare failed | +| 4559 | xa_commit: failure mapping xid | +| 4560 | xa_commit: xid not found | +| 4561 | xa_commit: commiting transaction active in branch | +| 4562 | xa_commit: attempting to commit unprepared transaction | +| 4563 | xa_commit: txnp-\>commit failed | +| 4564 | xa_recover: txn_get_prepared failed | +| 4565 | xa_rollback: failure mapping xid | +| 4566 | xa_rollback: xid not found | +| 4567 | xa_rollback: transaction in invalid state %d | +| 4568 | xa_rollback: failure aborting transaction | +| 4569 | xa_forget: failure mapping xid | +| 4570 | xa_forget: xid not found | +| 4571 | xa_forget: txnp-\>discard failed | + +## Command Line Utilities Messages + +| Message Number | Message Text | +|----|----| +| 5001 | %s: Unsupported database type | +| 5002 | %s: version %d.%d doesn't match library version %d.%d | +| 5003 | %s: version %d.%d doesn't match library version %d.%d | +| 5004 | \[%lu\]\[%lu\] App-specific log record: %lu data: | +| 5005 | %s: strdup: %s | +| 5006 | %s: illegal option combination | +| 5007 | Unknown statistics flag | +| 5008 | close | +| 5009 | %s: version %d.%d doesn't match library version %d.%d | +| 5010 | %s: strdup: %s | +| 5011 | callback: initialization | +| 5012 | callback: initialization | +| 5013 | tx: dispatch | +| 5014 | Unknown version %lu | +| 5015 | %s: version %d.%d doesn't match library version %d.%d | +| 5016 | \[%lu\]\[%lu\]application specific record: rec: %lu | +| 5017 | data: | +| 5018 | %s: strdup: %s | +| 5019 | %s: %s upgraded successfully | +| 5020 | %s: version %d.%d doesn't match library version %d.%d | +| 5021 | %s: strdup: %s | +| 5022 | recovery %d%% complete | +| 5023 | %s: localtime: %s | +| 5024 | %s: out of range or illegal time specification: \[\[CC\]YY\]MMDDhhmm\[.SS\] | +| 5025 | %s: version %d.%d doesn't match library version %d.%d | +| 5026 | strdup: %s | +| 5027 | cannot specify -d and -c | +| 5028 | cannot specify -D and -d or -l | +| 5029 | failed to get environment variable DB_HOME: %s | +| 5030 | no source database environment specified | +| 5031 | no target backup directory specified | +| 5032 | hot backup started at %s | +| 5033 | DB_CONFIG must not contain an absolute | +| 5035 | %s: force checkpoint | +| 5036 | %s: remove unnecessary log files | +| 5040 | %s: run catastrophic recovery | +| 5041 | %s: remove unnecessary log files | +| 5042 | hot backup completed at %s | +| 5043 | HOT BACKUP FAILED! | +| 5057 | %s: cannot specify -l with conflicting DB_CONFIG file | +| 5058 | use of -l with DB_CONFIG file is deprecated | +| 5071 | version %d.%d doesn't match library version %d.%d | +| 5072 | %s: %s: reopen: %s | +| 5073 | %s: strdup: %s | +| 5074 | No keys specified in file | +| 5075 | Keys specified in file | +| 5076 | Btree and Hash must specify keys | +| 5077 | improper database type conversion specified | +| 5078 | no database type specified | +| 5079 | odd number of key/data pairs | +| 5080 | %s: line %d: key already exists, not loaded: | +| 5081 | command-line configuration uses name=value format | +| 5082 | unknown command-line configuration keyword "%s" | +| 5083 | error reading db name | +| 5084 | line %lu: VERSION %d is unsupported | +| 5085 | line %lu: unknown type | +| 5086 | unknown input-file header configuration keyword "%s" | +| 5087 | line %lu: unexpected format | +| 5088 | unable to allocate memory | +| 5089 | boolean name=value pairs require a value of 0 or 1 | +| 5090 | unexpected end of input data or key/data pair | +| 5091 | %s: version %d.%d doesn't match library version %d.%d | +| 5092 | Cannot run %s without Replication Manager. | +| 5093 | Program name too long | +| 5094 | %s: strdup: %s | +| 5095 | db_replicate begin: %s | +| 5096 | received panic event | +| 5097 | ignoring event %d | +| 5098 | %s: version %d.%d doesn't match library version %d.%d | +| 5099 | %s: %lu %s %s: message too long | +| 5100 | %s: strdup: %s | +| 5101 | open | +| 5102 | running at %.24s | +| 5103 | rejected %d locks | +| 5104 | %s: version %d.%d doesn't match library version %d.%d | +| 5105 | Verification of %s %s. | +| 5106 | close | +| 5107 | %s: version %d.%d doesn't match library version %d.%d | +| 5108 | %s: %s: reopen: %s | +| 5109 | %s: strdup: %s | +| 5110 | %s: the -d and -p options may not both be specified | +| 5111 | %s: the -l and -s options may not both be specified | +| 5112 | %s: the -m option may not be specified with -l or -s | +| 5113 | %s: the -k and -r or -R options may not both be specified | +| 5114 | %s: the -r or R options may not be specified with -s | +| 5115 | open: %s | +| 5116 | %s: does not contain multiple databases | +| 5117 | close | +| 5118 | %s: version %d.%d doesn't match library version %d.%d | +| 5119 | %s: strdup: %s | +| 5120 | %s: version %d.%d doesn't match library version %d.%d | +| 5121 | %s: strdup: %s | +| 5122 | %s: at least one of -1, -k and -p must be specified | +| 5123 | checkpoint begin: %s | +| 5124 | checkpoint complete: %s | +| 5125 | %s: version %d.%d doesn't match library version %d.%d | +| 5126 | dbm: no open database. | +| 5127 | Heap must not specify keys | +| 5128 | improper database type conversion specified | +| 5129 | Cannot copy data from a PRIVATE environment | +| 5130 | Password may not be specified twice | +| 5131 | Password may not be specified twice | +| 5132 | Password may not be specified twice | +| 5133 | Password may not be specified twice | +| 5134 | Password may not be specified twice | +| 5135 | Password may not be specified twice | +| 5136 | Password may not be specified twice | +| 5137 | Password may not be specified twice | +| 5138 | Password may not be specified twice | +| 5139 | Password may not be specified twice | diff --git a/docs-src/guides/bdb-sql/_meta.toml b/docs-src/guides/bdb-sql/_meta.toml new file mode 100644 index 000000000..9292150aa --- /dev/null +++ b/docs-src/guides/bdb-sql/_meta.toml @@ -0,0 +1,37 @@ +# Nav/index metadata for the bdb-sql guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Getting Started with the Oracle Berkeley DB SQL APIs" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "dbsqlbasics", + "buildinstall", + "accessing_bdb_sql_databases.title", + "journaldirectory", + "unsupportedpragmas", + "changedpragmas", + "addedpragmas", + "miscdiff", + "bdb-concepts", + "sql_encryption", + "sequencesupport", + "normal-sql", + "lockingnotes", + "lockhandling", + "dbfeatures", + "mvcc", + "selectpage_size", + "sqlrep", + "reppragma", + "repstatistics", + "rep_usageexamples", + "admin", + "sync", + "datamigration", + "bfile-extension", + "bfile-sql", + "bfile-c", +] diff --git a/docs-src/guides/bdb-sql/accessing_bdb_sql_databases.title.md b/docs-src/guides/bdb-sql/accessing_bdb_sql_databases.title.md new file mode 100644 index 000000000..c2335160b --- /dev/null +++ b/docs-src/guides/bdb-sql/accessing_bdb_sql_databases.title.md @@ -0,0 +1,16 @@ +--- +title: "Accessing BDB SQL Databases" +api-name: "Accessing BDB SQL Databases" +source: docs/bdb-sql/accessing_bdb_sql_databases.title.html +--- +## Accessing BDB SQL Databases + +BDB SQL databases can be accessed using a number of different drivers, applications and APIs. Only some of these are supported by all major platforms, as identified in the following table. + +|   | UNIX/POSIX | Windows | Windows Mobile/CE | Android | iOS | +|---------------|:----------:|:-------:|:-----------------:|:-------:|:---:| +| DBSQL Library | x | x | x | x | x | +| DBSQL Shell | x | x | x | x | x | +| ODBC | x | x |   |   |   | +| JDBC | x | x |   |   |   | +| ADO.NET |   | x | x |   |   | diff --git a/docs-src/guides/bdb-sql/addedpragmas.md b/docs-src/guides/bdb-sql/addedpragmas.md new file mode 100644 index 000000000..de759e407 --- /dev/null +++ b/docs-src/guides/bdb-sql/addedpragmas.md @@ -0,0 +1,174 @@ +--- +title: "Added PRAGMAs" +api-name: "Added PRAGMAs" +source: docs/bdb-sql/addedpragmas.html +--- +## Added PRAGMAs + + [PRAGMA bdbsql_error_file](addedpragmas.md#bdbsql_error_file) + + [PRAGMA bdbsql_lock_tablesize](addedpragmas.md#bdbsql_lock_tablesize) + + [PRAGMA bdbsql_shared_resources](addedpragmas.md#bdbsql_shared_resources) + + [PRAGMA bdbsql_single_process](addedpragmas.md#bdbsql_single_process) + + [PRAGMA bdbsql_system_memory](addedpragmas.md#bdbsql_system_memory) + + [PRAGMA bdbsql_vacuum_fillpercent](addedpragmas.md#bdbsql_vacuum_fillpercent) + + [PRAGMA bdbsql_vacuum_pages](addedpragmas.md#bdbsql_vacuum_pages) + + [PRAGMA multiversion](addedpragmas.md#multiversion) + + [PRAGMA snapshot_isolation](addedpragmas.md#snapshot_isolation) + + [PRAGMA trickle](addedpragmas.md#trickle) + + [PRAGMA txn_bulk](addedpragmas.md#txn_bulk) + + [Replication PRAGMAs](addedpragmas.md#replication_pragmas) + +The following PRAGMAs are added in the Berkeley DB SQL interface. + +### PRAGMA bdbsql_error_file + +``` c +PRAGMA bdbsql_error_file [filename] +``` + +Redirects internal Berkeley DB error messages to the named file. If a relative path is specified to `[filename]`, then the path is interpreted as being relative to the current working directory. + +If this PRAGMA is issued with no filename, then the current target for Berkeley DB error output is returned. By default, error messages are sent to `STDERR`. + +This PRAGMA can be issued at any time; initial database access does not have to occur before this PRAGMA can be used. + +### PRAGMA bdbsql_lock_tablesize + +``` c +PRAGMA bdbsql_lock_tablesize [= N] +``` + +Sets or reports the number of buckets in the Berkeley DB environment's lock object hash table. + +This pragma must be called prior to opening/creating the database environment. + +For more details, see get_lk_tablesize and set_lk_tablesize. + +### PRAGMA bdbsql_shared_resources + +``` c +PRAGMA bdbsql_shared_resources [= N] +``` + +Sets or reports the maximum amount of memory (bytes) to be used by shared structures in the main environment region. + +This pragma must be called prior to opening/creating the database environment. + +For more details, see get_memory_max and set_memory_max. + +### PRAGMA bdbsql_single_process + +``` c +PRAGMA bdbsql_single_process = boolean +``` + +To create a private environment rather than a shared environment, enable this pragma. The cache and other region files will be created in memory rather than using file backed shared memory. + +In Berkeley DB SQL the default behavior is to allow a database to be opened and operated on by multiple processes simultaneously. When this pragma is enabled, accessing the same database from multiple processes simultaneously can lead to data corruption. Either option supports accessing a database using a single process multi-threaded application. + +By default omit sharing is disabled. This pragma must be called prior to opening/creating the database environment. Because the setting is not persistent, you may need to invoke it before every database open, or define compile option `BDBSQL_OMIT_SHARING` instead. + +For more information, see Shared memory region. Note that this pragma causes the DB_PRIVATE flag to be specified in the DB_ENV->open() method. + +### PRAGMA bdbsql_system_memory + +``` c +PRAGMA bdbsql_system_memory [base segment ID] +``` + +Queries or sets a flag that causes the database's shared resources to be created in system shared memory. By default the database's shared resources are created in file-backed shared memory. + +If a \[base segment ID\] is specified, the shared resources will be created using X/Open style shared memory interfaces. The \[base segment ID\] will be used as the starting ID for shared resources used by the database. Use different \[base segment ID\] values for different databases. It is possible for multi-process applications to use a single database by specifying the same \[base segment ID\] to this PRAGMA. Each connection needs to set this PRAGMA. + +This PRAGMA may be used to set a \[base segment ID\] only before the first table is created in the database. + +### PRAGMA bdbsql_vacuum_fillpercent + +``` c +PRAGMA bdbsql_vacuum_fillpercent [= N] +``` + +Sets or reports the page full threshold. Any page in the database that is at or below this percentage full is considered for vacuuming when `PRAGMA incremental_vacuum` is enabled. The value is specified as a percentage between 1 and 100. By default, pages 85% full and below are considered for vacuuming. + +### PRAGMA bdbsql_vacuum_pages + +``` c +PRAGMA bdbsql_vacuum_pages [= N] +``` + +Sets or reports the maximum number of pages to be returned to the file system from the free page list when incremental vacuuming is enabled. By default, up to 128 pages are removed from the free list. + +Page vacuuming is controlled using PRAGMA auto_vacuum. + +### PRAGMA multiversion + +``` c +PRAGMA multiversion +``` + +Controls whether Multiversion Concurrency Control (MVCC) is on or off. You can not use this PRAGMA at any time during your application's runtime after your database tables have been accessed. + +For more information on MVCC and snapshot isolation, see Using Multiversion Concurrency Control + +### PRAGMA snapshot_isolation + +``` c +PRAGMA snapshot_isolation +``` + +Controls whether snapshot isolation is turned on. This PRAGMA can be used at any time during your application's runtime *after* Multiversion Concurrency Control (MVCC) has been turned on. + +For more information on MVCC and snapshot isolation, see Using Multiversion Concurrency Control + +### PRAGMA trickle + +``` c +PRAGMA trickle [percent] +``` + +Ensures that at least the specified percentage of pages in the shared cache are clean. This can cause pages that have been modified to be flushed to disk. + +The trickle functionality enables an application to ensure that a page is available for reading new information into the shared cache without waiting for a write operation to complete. + +Specifying this PRAGMA without a percentage value causes the current trickle value to be displayed. Specify `0` to turn the trickle functionality off. + +### PRAGMA txn_bulk + +``` c +PRAGMA TXN_BULK +``` + +Enables transactional bulk loading optimization. For more information, see Using Bulk Loading. + +### Replication PRAGMAs + +Seven PRAGMAs were added to control replication. They are described in Using Replication with the SQL API: + +### Note + +If you are using these replication PRAGMAs and you want to perform a backup, there is an additional backup step for the pragma file. See Backing Up Berkeley DB SQL Databases for more information. + +- **PRAGMA replication** + +- **PRAGMA replication_initial_master** + +- **PRAGMA replication_local_site** + +- **PRAGMA replication_remote_site** + +- **PRAGMA replication_remove_site** + +- **PRAGMA replication_verbose_output** + +- **PRAGMA replication_verbose_file** diff --git a/docs-src/guides/bdb-sql/admin.md b/docs-src/guides/bdb-sql/admin.md new file mode 100644 index 000000000..dc2862771 --- /dev/null +++ b/docs-src/guides/bdb-sql/admin.md @@ -0,0 +1,49 @@ +--- +title: "Chapter 5. Administrating Berkeley DB SQL Databases" +api-name: "Chapter 5. Administrating Berkeley DB SQL Databases" +source: docs/bdb-sql/admin.html +--- +## Chapter 5. Administrating Berkeley DB SQL Databases + +**Table of Contents** + + [Backing Up Berkeley DB SQL Databases](admin.md#backup) + + [Backing Up Replicated Berkeley DB SQL Databases](admin.md#idp50739296) + + [Syncing with Oracle Databases](sync.md) + + [Syncing on Unix Platforms](sync.md#syncunix) + + [Syncing on Windows Platforms](sync.md#syncwin) + + [Syncing on Windows Mobile Platforms](sync.md#syncwinmobile) + + [Data Migration](datamigration.md) + + [Migration Using the Shells](datamigration.md#shellmigrate) + +This chapter provides administrative procedures that are unique to the Berkeley DB SQL interface. + +## Backing Up Berkeley DB SQL Databases + + [Backing Up Replicated Berkeley DB SQL Databases](admin.md#idp50739296) + +You can use the standard SQLite `.dump` command to backup the data managed by the BDB SQL interface. + +The BDB SQL interface supports the standard SQLite Online Backup API. However, there is a small difference between the two interfaces. In the BDB SQL interface, the value returned by the `sqlite3_backup_remaining` method and the number of pages passed to the `sqlite3_backup_step` method, are estimates of the number of pages to be copied and not exact values. To be certain that the backup process is complete, check if the `sqlite3_backup_step` method has returned `SQLITE_DONE`. To learn how to use SQLite Online Backup API, see the official SQLite Documentation Page. + +If you are using replication, you will also need to copy the file that contains the replication pragma information in order to have a full backup. To do that copy the file named ` pragma` from the database journal directory. + +### Backing Up Replicated Berkeley DB SQL Databases + +When BDB SQL interface databases are replicated the process for backing up a regular database should be followed. The user must then copy some additional files for the backup to be complete. + +The additional files can be found in the journal directory of the source database, and should be copied into the journal directory of the backup copy. The journal directory is automatically created when a Berkeley DB SQL interface database is created. The journal directory is created in the same directory as the database file, it has the name of the database file with a `-journal` appendix. + +The files that need to be copied into the backup journal directory are: + +- `__db.rep.egen` +- `__db.rep.gen` +- `__db.rep.init` +- `__db.rep.system` diff --git a/docs-src/guides/bdb-sql/bdb-concepts.md b/docs-src/guides/bdb-sql/bdb-concepts.md new file mode 100644 index 000000000..3e3493f6f --- /dev/null +++ b/docs-src/guides/bdb-sql/bdb-concepts.md @@ -0,0 +1,20 @@ +--- +title: "Berkeley DB Concepts" +api-name: "Berkeley DB Concepts" +source: docs/bdb-sql/bdb-concepts.html +--- +## Berkeley DB Concepts + +If you are a SQLite user who is migrating to the BDB SQL interface, then there are a few Berkeley DB-specific concepts you might want to know about. + +- Environments. The directory that is created alongside your database file, and which ends with the "-journal" suffix, is actually a Berkeley DB environment directory. + +- The Locking Subsystem + + The Berkeley DB library implements locking in a different way to SQLite. SQLite implements locking at a database level - any operation will take a lock on the entire database. Berkeley DB implements a scheme called page level locking. The database divides data into relatively small blocks. Each block corresponds to a page in database terms. Each block can contain multiple pieces of user information. Berkeley DB takes locks on individual pages. This allows for greater concurrency in applications, but means that applications are more likely to encounter deadlocks. + + See: Locking Notes for more information. + +- The Journal Subsystem + + The BDB SQL interface implements write ahead logging (WAL), it stores journal files differently to the SQLite WAL implementation. BDB SQL interface rolls over journal files when they get to a certain size (default 10MB). It is possible for the to be multiple journal files active at one time with BDB SQL interface. diff --git a/docs-src/guides/bdb-sql/bfile-c.md b/docs-src/guides/bdb-sql/bfile-c.md new file mode 100644 index 000000000..653792757 --- /dev/null +++ b/docs-src/guides/bdb-sql/bfile-c.md @@ -0,0 +1,186 @@ +--- +title: "BFILE C/C++ Objects and Functions" +api-name: "BFILE C/C++ Objects and Functions" +source: docs/bdb-sql/bfile-c.html +--- +## BFILE C/C++ Objects and Functions + + [sqlite3_column_bfile](bfile-c.md#sqlite3_column_bfile) + + [sqlite3_bfile_open](bfile-c.md#sqlite3_bfile_open) + + [sqlite3_bfile_close](bfile-c.md#sqlite3_bfile_close) + + [sqlite3_bfile_is_open](bfile-c.md#sqlite3_bfile_is_open) + + [sqlite3_bfile_read](bfile-c.md#sqlite3_bfile_read) + + [sqlite3_bfile_file_exists](bfile-c.md#sqlite3_bfile_file_exists) + + [sqlite3_bfile_size](bfile-c.md#sqlite3_bfile_size) + + [sqlite3_bfile_final](bfile-c.md#sqlite3_bfile_final) + +The BFILE extension can optionally make available to you some additional C language data types and functions for use with the SQLite C/C++ interface. These are available to you only if you take the proper steps when you compile Berkeley DB. See the *Berkeley DB Installation and Build Guide* for more information. + +Once enabled, the BFILE C extension makes the following new structure available to you: + +``` c +typedef struct sqlite3_bfile sqlite3_bfile; +``` + +This structure serves as the BFILE handle when you are using the BFILE extension along with the SQLite C/C++ interface. + +In addition to the new structure, you can also use the following new C functions: + +### sqlite3_column_bfile + +``` c +int +sqlite3_column_bfile(sqlite3_stmt *pStmt, int iCol, + sqlite3_bfile **ppBfile); +``` + +Returns a result set from a query against a column of type BFILE. + +On success, `SQLITE_OK` is returned and the new BFILE handle is written to **ppBfile**. Otherwise, `SQLITE_ERROR` is returned. + +Parameters are: + +- **pStmt** + + Pointer to the prepared statement that the function is evaluating. The statement is created using `sqlite3_prepare_v2()` or one of its variants. + + If this statement does not point to a valid row, the result is undefined. + +- **iCol** + + Index of the column for which information should be returned. The left-most column of the result set is index `0`. Use `sqlite3_column_count()` to discover the number of columns in the result set. + + If the column index is out of range, the result is undefined. + +- **ppBfile** + + The BFILE handle that you are using for the query. This pointer is valid only until `sqlite3_step()`, `sqlite3_reset()` or `sqlite3_finalize()` have been called. + + The memory space used to hold this handle is freed by sqlite3_bfile_final Do not pass these pointers to `sqlite3_free()`. + +This function can be called successfully only if all of the following conditions are true. If any of the following are not true, the result is undefined: + +- The most recent call to `sqlite3_step()` has returned `SQLITE_ROW`. + +- Neither `sqlite3_reset()` nor `sqlite3_finalize()` have been called since the last time `sqlite3_step()` was called. + +- `sqlite3_step()`, `sqlite3_reset()` or `sqlite3_finalize()` have not been called from a different thread while this routine is pending. + +### sqlite3_bfile_open + +``` c +int +sqlite3_bfile_open(sqlite3_bfile *pBfile); +``` + +Opens a file for incremental read. + +On success, `SQLITE_OK` is returned. Otherwise, `SQLITE_ERROR` is returned. + +To avoid a resource leak, every opened BFILE handle should eventually be closed with the sqlite3_bfile_close function. Note that **pBfile** is always initialized such that it is always safe to invoke `sqlite_bfile_close()` against it, regardless of the success or failure of this function. + +### sqlite3_bfile_close + +``` c +int +sqlite3_bfile_close(sqlite3_bfile *pBfile); +``` + +Closes an open BFILE handle. The BFILE is closed unconditionally. Even if this function returns an error, the BFILE is still closed. + +Calling this routine with a null pointer (such as would be returned by failed call to sqlite3_column_bfile()) is a harmless non-operation. + +On success, `SQLITE_OK` is returned. Otherwise, `SQLITE_ERROR` is returned. + +### sqlite3_bfile_is_open + +``` c +int +sqlite3_bfile_is_open(sqlite3_bfile *pBfile, int *open); +``` + +Checks whether a BFILE handle is open. The `open` parameter is set to 1 if the file is open, otherwise it is 0. + +On success, `SQLITE_OK` is returned. Otherwise, `SQLITE_ERROR` is returned. + +### sqlite3_bfile_read + +``` c +int +sqlite3_bfile_read(sqlite3_bfile *pBfile, void *oBuff, int nSize, + int iOffset, int *nRead); +``` + +This function is used to read data from an opened BFILE handle into a caller-supplied buffer. + +On success, `SQLITE_OK` is returned, the data that has been read is written to the output buffer, **oBuff**, and the amount of data written to the buffer is recorded in **nRead**. Otherwise, `SQLITE_ERROR` is returned. + +Parameters are: + +- **pBfile** + + The BFILE handle from which the data is read. + + This function only works on a BFILE handle which has been created by a prior successful call to sqlite3_bfile_open and which has not been closed by sqlite3_bfile_close. Passing any other pointer in to this function results in undefined and probably undesirable behavior. + +- **oBuff** + + The buffer used to contain the data that is read from **pBfile**. It must be at least **nSize** bytes in size. + +- **nSize** + + The amount of data, in bytes, to read from the BFILE. + +- **iOffset** + + The offset from the beginning of the file where the read operation is to begin. + +- **nRead** + + Contains the amount of data, in bytes, actually written to buffer **oBuff** once the read operation is completed. + +### Note + +The size of the BFILE can be determined using the sqlite3_bfile_size function. + +### sqlite3_bfile_file_exists + +``` c +int +sqlite3_bfile_file_exists(sqlite3_bfile *pBfile, int *exist); +``` + +Checks whether a BFILE exists. The `exists` parameter is set to 1 if the file is exists, otherwise it is 0. + +On success, `SQLITE_OK` is returned. Otherwise, `SQLITE_ERROR` is returned. + +### sqlite3_bfile_size + +``` c +int +sqlite3_bfile_size(sqlite3_bfile *pBfile, off_t *size); +``` + +Returns the size of the BFILE, in bytes. + +On success, `SQLITE_OK` is returned, and **size** is set to the size of the BFILE, in bytes. Otherwise, `SQLITE_ERROR` is returned. + +This function only works on a BFILE handle which has been created by a prior successful call to sqlite3_column_bfile and which has not been finalized by sqlite3_bfile_final. Passing any other pointer in to this function results in undefined and probably undesirable behavior. + +### sqlite3_bfile_final + +``` c +int +sqlite3_bfile_final(sqlite3_bfile *pBfile); +``` + +Frees a BFILE handle. + +On success, `SQLITE_OK` is returned. Otherwise, `SQLITE_ERROR` is returned. diff --git a/docs-src/guides/bdb-sql/bfile-extension.md b/docs-src/guides/bdb-sql/bfile-extension.md new file mode 100644 index 000000000..e815f6e7b --- /dev/null +++ b/docs-src/guides/bdb-sql/bfile-extension.md @@ -0,0 +1,24 @@ +--- +title: "Appendix A. Using the BFILE Extension" +api-name: "Appendix A. Using the BFILE Extension" +source: docs/bdb-sql/bfile-extension.html +--- +## Appendix A. Using the BFILE Extension + +The BFILE data type allows the BDB SQL interface to access binary files that are stored in the file system outside of the database. The binary file can be queried in exactly the same way as any other data type stored in the database, but Berkeley DB is able to save space in the database file by not embedding a large amount of binary data in it. This also helps overall database performance. + +Internally, a BFILE column or attribute stores a BFILE locater, which serves as a pointer to the binary file. The locater maintains the directory alias and the filename. You can change the path of BFILE without affecting the base table by using the BFILENAME function. BFILE is somewhat like the BLOB data type, but it does not participate in transactions and it is not recoverable. Instead, the underlying operating system is expected to provide file integrity and durability. + +The remainder of this section describes the various objects and functions that the BFILE extension makes available to you. In addition, complete examples of using these extensions are available with your Berkeley DB distribution. They are placed in the following location: + +``` c +/lang/sql/sqlite/ext/bfile/examples +``` + +## Supported Platforms and Languages + +The BFILE extension is currently only supported for \*nix platforms. + +The BFILE extension it is not available in your library by default. Instead, you must enable the extension when you compile Berkeley DB. See the *Berkeley DB Installation and Build Guide* for information on how to enable this extension when you build Berkeley DB. Once you have enabled the extension, applications will also need to load the BFILE library file: `libbfile_ext.so`. + +By default, the BFILE extension provides support for additional SQL statements. With some extra configuration at Berkeley DB compile time, you can also obtain support for extensions to the SQLite C/C++ interface. Both the SQL extensions and the extensions to the SQLite C/C++ interface are described in the following sections. diff --git a/docs-src/guides/bdb-sql/bfile-sql.md b/docs-src/guides/bdb-sql/bfile-sql.md new file mode 100644 index 000000000..344acd4f4 --- /dev/null +++ b/docs-src/guides/bdb-sql/bfile-sql.md @@ -0,0 +1,110 @@ +--- +title: "BFILE SQL Objects and Functions" +api-name: "BFILE SQL Objects and Functions" +source: docs/bdb-sql/bfile-sql.html +--- +## BFILE SQL Objects and Functions + + [BFILE_CREATE_DIRECTORY](bfile-sql.md#bfile_create_directory) + + [BFILE_REPLACE_DIRECTORY](bfile-sql.md#bfile_replace_directory) + + [BFILE_DROP_DIRECTORY](bfile-sql.md#bfile_drop_directory) + + [BFILE_NAME](bfile-sql.md#bfile_name) + + [BFILE_FULLPATH](bfile-sql.md#bfile_fullpath) + + [BFILE_OPEN](bfile-sql.md#bfile_open) + + [BFILE_READ](bfile-sql.md#bfile_read) + + [BFILE_CLOSE](bfile-sql.md#bfile_close) + + [BFILE_SIZE](bfile-sql.md#bfile_size) + +When the BFILE extension is enabled, you can create a `DIRECTORY` object. These objects are required before you can store a pointer to a file in a `BFILE` column. + +`DIRECTORY` objects are stored in a special table called `BFILE_DIRECTORY`. This table is automatically created for you when it is needed. You should *not* manually create this table. + +You manage `DIRECTORY` objects using the following SQL functions: + +| | +|----| +| BFILE_CREATE_DIRECTORY | +| BFILE_REPLACE_DIRECTORY | +| BFILE_DROP_DIRECTORY | + +The following sections describe the SQL functions that you can use when the BFILE extension is enabled. + +### BFILE_CREATE_DIRECTORY + +``` c +BFILE_CREATE_DIRECTORY(directory, path) +``` + +Creates a `DIRECTORY` object as a path. The specified path must not already exist, or `Directory already exists` is returned. + +### BFILE_REPLACE_DIRECTORY + +``` c +BFILE_REPLACE_DIRECTORY(directory, path) +``` + +Replaces the named `DIRECTORY` object using the specified path. If the object does not exist, `Directory does not exist` is returned. + +### BFILE_DROP_DIRECTORY + +``` c +BFILE_DROP_DIRECTORY(directory) +``` + +Drops the named `DIRECTORY` object. If the object does not exist, `Directory does not exist` is returned. + +### BFILE_NAME + +``` c +BFILE_NAME(directory, filename) +``` + +Returns the BFILE locator. + +### BFILE_FULLPATH + +``` c +BFILE_FULLPATH(column) +``` + +Returns the full path. + +### BFILE_OPEN + +``` c +BFILE_OPEN(column) +``` + +Extracts the directory and file names from the BFILE locator, and then opens that file. On success, a BFILE handle is returned. Otherwise, `0` is returned. + +### BFILE_READ + +``` c +BFILE_READ(BFILE handle, amt, offset) +``` + +Reads at most `amt` data from the BFILE handle, starting at `offset`. On success, `Data` is returned. Otherwise, `0` is returned to indicate that no more valid data is available. + +### BFILE_CLOSE + +``` c +BFILE_CLOSE(BFILE handle) +``` + +Closes the BFILE handle. + +### BFILE_SIZE + +``` c +BFILE_SIZE(column) +``` + +Returns the size of the BFILE. On success, the size is returned. Otherwise, -1 is returned. diff --git a/docs-src/guides/bdb-sql/buildinstall.md b/docs-src/guides/bdb-sql/buildinstall.md new file mode 100644 index 000000000..53c9a4910 --- /dev/null +++ b/docs-src/guides/bdb-sql/buildinstall.md @@ -0,0 +1,95 @@ +--- +title: "Getting and Installing BDB SQL" +api-name: "Getting and Installing BDB SQL" +source: docs/bdb-sql/buildinstall.html +--- +## Getting and Installing BDB SQL + + [On Windows Systems](buildinstall.md#onwin) + + [On Unix](buildinstall.md#onunix) + + [The BDB SQL ADO.NET Interface](buildinstall.md#ado_net) + +The BDB SQL interface comes as a part of the Oracle Berkeley DB download. This can be downloaded from the Oracle Berkeley DB download page. + +### On Windows Systems + +The BDB SQL interface is automatically built and installed whenever you build or install Berkeley DB for a Windows system. The BDB SQL interface `dll`s and the command line interpreter have names that differ from a standard SQLite distribution as follows: + +- `dbsql.exe` + + This is the command line shell. It operates identically to the SQLite **sqlite3.exe** shell. + +- `libdb_sql50.dll` + + This is the library that provides the BDB SQL interface. It is the equivalent of the SQLite `sqlite3.dll` library. + +### On Unix + +In order to build the BDB SQL interface, you download and build Berkeley DB, configuring it so that the BDB SQL interface is also built. Be aware that it is not built by default. Instead, you need to tell the Berkeley DB `configure` script to also build the BDB SQL interface. For instructions on building the BDB SQL interface, see Building the DB SQL Interface in the *Berkeley DB Installation and Build Guide*. + +The library and application names used when building the BDB SQL interface are different than those used by SQLite. If you want library and command shell names that are consistent with the names used by SQLite, configure the BDB SQL interface build using the compatibility (`--enable-sql_compat`) option. + +### Warning + +The compatibility option can break other applications on your platform that rely on standard SQLite. This is especially true of Mac OS X, which uses standard SQLite for a number of default applications. + + *Use the compatibility option only if you know exactly what you are doing.* + +Unless you built the BDB SQL interface with the compatibility option, libraries and a command line shell are built with the following names: + +- dbsql + + This is the command line shell. It operates identically to the SQLite **sqlite3** shell. + +- `libdb_sql` + + This is the library that provides the BDB SQL interface. It is the equivalent of the SQLite `libsqlite3` library. + +### The BDB SQL ADO.NET Interface + +Download the ADO.NET package from the Oracle Berkeley DB download page. + +#### Prerequisites For Building The ADO.NET Package + +- To build the Linq package, you will need to install `Microsoft .NET Framework 3.5 SP1`. +- To build SQLite.Designer, you will need to install the `Microsoft Visual Studio SDK`. +- To build on Windows Mobile you will need to install the `Microsoft Windows Mobile 6.5.3 Developer Tool Kit (DTK)`. +- To build on Windows Mobile you will need to use Visual Studio 2008. + +#### Building BDB SQL ADO.NET Interface For Windows + +- The package contains Visual Studio solution files: + + - `SQLite.NET.2008.sln` and `SQLite.NET.2010.sln` + + For use by with Visual Studio 2008 or 2010. Note that these solution files do not build support for Linq or SQLite Designer. + + - `SQLite.NET.2008.MSBuild.sln` and `SQLite.NET.2010.MSBuild.sln` + + For use with MSBuild (Microsoft Build Engine). These can also be used with Visual Studio. These solutions exclude SQLite Designer and CompactFramework. By default, these do not build support for Linq. + +- Change the current platform target to `ReleaseNativeOnly` choose either `Win32` or `x64` depending on your target platform. + +- Build the solution. + +#### Building BDB SQL ADO.NET Interface For Windows Mobile + +Building BDB SQL ADO.NET for Windows Mobile requires Windows Mobile 6.5.3 Professional DTK. Typical requirements for installing this toolkit are: + +- Visual Studio 2005 SP1 or Later + +- ActiveSync 4.5 + +- .NET CompactFramework 2.0 SP1 + +- Windows Mobile 6 SDK + +To build BDB SQL ADO.NET for Windows Mobile, do the following: + +- Open the `SQLite.NET.2008.WinCE.sln` solution file in Visual Studio 2008. +- Select `Load Project Normally` +- Change the current platform to `ReleaseNativeOnly`. +- Select `Configuration Manager`-\>`new`, then type or select the platform `Windows Mobile 6.5.3 Professional DTK (ARMV4I)`. Choose to copy settings from `Pocket PC 2003 (ARMV4I)` +- Build the solution. diff --git a/docs-src/guides/bdb-sql/changedpragmas.md b/docs-src/guides/bdb-sql/changedpragmas.md new file mode 100644 index 000000000..4974eaa59 --- /dev/null +++ b/docs-src/guides/bdb-sql/changedpragmas.md @@ -0,0 +1,56 @@ +--- +title: "Changed PRAGMAs" +api-name: "Changed PRAGMAs" +source: docs/bdb-sql/changedpragmas.html +--- +## Changed PRAGMAs + + [PRAGMA auto_vacuum](changedpragmas.md#auto_vacuum) + + [PRAGMA incremental_vacuum](changedpragmas.md#incremental_vacuum) + + [PRAGMA journal_size_limit](changedpragmas.md#journal_size_limit) + +The following PRAGMAs are available in the BDB SQL interface, but they behave differently in some way from standard SQLite. + +### PRAGMA auto_vacuum + +The syntax for this PRAGMA is: + +``` c +PRAGMA auto_vacuum +PRAGMA auto_vacuum = 0 | NONE | 1 | FULL | 2 | INCREMENTAL +``` + +Standard SQLite does not allow you to enable or disable auto-vacuum after a table has been created. Berkeley DB, however, allows you to change this at any time. + +In the previous syntax, `0` and `NONE` both turn off auto vacuuming. + +`1` or `FULL` causes full vacuuming to occur. That is, the BDB SQL interface will vacuum the entire database at each commit using a very low fill percentage (1%) in order to return emptied pages to the file system. Because Berkeley DB allows you to call this PRAGMA at any time, it is recommended that you do not turn on FULL vacuuming because doing so can result in a great deal of overhead to your transaction commits. + +If `2` or `INCREMENTAL` is used, then incremental vacuuming is enabled. The amount of vacuuming that is performed for incremental vacuum is controlled using the following PRAGMAs: + +| | +|----| +| PRAGMA bdbsql_vacuum_fillpercent | +| PRAGMA bdbsql_vacuum_pages | + +Note that you can call PRAGMA incremental_vacuum to perform an incremental vacuum operation on demand. + +When performing vacuum operations, Berkeley DB defragments and repacks individual database pages, while SQLite only truncates the freelist pages from the database file. + +For more information on auto vacuum, see PRAGMA auto_vacuum in the SQLite documentation. + +### PRAGMA incremental_vacuum + +Performs incremental vacuum operations on demand. You can cause incremental vacuum operations to be performed automatically using PRAGMA auto_vacuum. + +Note that for SQLite, this PRAGMA is used to specify the maximum number of pages to be freed during vacuuming. For Berkeley DB, you use `PRAGMA bdbsql_vacuum_pages` instead. + +### PRAGMA journal_size_limit + +For standard SQLite, this pragma identifies the maximum size that the journal file is allowed to be. + +Berkeley DB uses multiple journal files, Berkeley DB journal files are different to a SQLite journal file in that they contain information about multiple transactions, rather than a single transaction (similar to the SQLite WAL journal file). Over the course of the database's lifetime, Berkeley DB will probably create multiple journal files. A new journal file is created when the current journal file has reached the maximum size configured using the journal_size_limit pragma. + +Note that a BDB SQL interface journal file is referred to as a log file in the Berkeley DB documentation. diff --git a/docs-src/guides/bdb-sql/datamigration.md b/docs-src/guides/bdb-sql/datamigration.md new file mode 100644 index 000000000..501bcaf3a --- /dev/null +++ b/docs-src/guides/bdb-sql/datamigration.md @@ -0,0 +1,30 @@ +--- +title: "Data Migration" +api-name: "Data Migration" +source: docs/bdb-sql/datamigration.html +--- +## Data Migration + + [Migration Using the Shells](datamigration.md#shellmigrate) + +If you have a database created by SQLite, you can migrate it to a Berkeley DB database for use with the BDB SQL interface. For production applications, you should do this only when your application is shutdown. + +All data and schema supported by SQLite can be migrated to a Berkeley DB database. + +### Migration Using the Shells + +To migrate your data from SQLite to a Berkeley DB database: + +1. Make sure your application is shutdown. + +2. Open the SQLite database within the **sqlite3** shell. + +3. Execute the `.output` command to specify the location where you want to dump data. + +4. Dump the database using the SQLite `.dump` command. + +5. Close the **sqlite3** shell and open the Berkeley DB dbsql shell. + +6. Load the dumped data using the `.read` command. + +Note that you can migrate in the reverse direction as well. Dump the Berkeley DB database by calling `.dump` from within the Berkeley DB dbsql shell, and load it into SQLite by `.read` from within SQLite's **sqlite3** shell. diff --git a/docs-src/guides/bdb-sql/dbfeatures.md b/docs-src/guides/bdb-sql/dbfeatures.md new file mode 100644 index 000000000..dba6efcf1 --- /dev/null +++ b/docs-src/guides/bdb-sql/dbfeatures.md @@ -0,0 +1,30 @@ +--- +title: "Chapter 3. Berkeley DB Features" +api-name: "Chapter 3. Berkeley DB Features" +source: docs/bdb-sql/dbfeatures.html +--- +## Chapter 3. Berkeley DB Features + +**Table of Contents** + + [Using Bulk Loading](dbfeatures.md#bulkloading) + + [Using Multiversion Concurrency Control](mvcc.md) + + [Selecting the Page Size](selectpage_size.md) + +When using the Berkeley DB SQL API, there are features that you can manage using PRAGMAs that are unique to Berkeley DB. This chapter discusses these features. + +## Using Bulk Loading + +Bulk loading is an I/O optimization feature that is useful when loading large amounts of data to the database inside of a single transaction. The bulk load optimization avoids writing some log records to the journal file. This can provide significant performance benefits when loading a large number of new rows into a database. + +When you use bulk load, nested transactions are disabled. This means that you cannot undo any single operation. Instead, if you want to undo a given operation, you must undo everything performed within the transaction. + +It is not possible to use the bulk load functionality when replication is enabled. + +To enable bulk loading, use: + +**PRAGMA TXN_BULK** = 0 \| 1; + +Default value is `0`, which means the PRAGMA is turned off and so bulk loading is not in use. `1` means bulk loading is in use. diff --git a/docs-src/guides/bdb-sql/dbsqlbasics.md b/docs-src/guides/bdb-sql/dbsqlbasics.md new file mode 100644 index 000000000..5b876bc06 --- /dev/null +++ b/docs-src/guides/bdb-sql/dbsqlbasics.md @@ -0,0 +1,90 @@ +--- +title: "Chapter 1. Berkeley DB SQL: The Absolute Basics" +api-name: "Chapter 1. Berkeley DB SQL: The Absolute Basics" +source: docs/bdb-sql/dbsqlbasics.html +--- +## Chapter 1. Berkeley DB SQL: The Absolute Basics + +**Table of Contents** + + [BDB SQL Is Nearly Identical to SQLite](dbsqlbasics.md#identicalusage) + + [Getting and Installing BDB SQL](buildinstall.md) + + [On Windows Systems](buildinstall.md#onwin) + + [On Unix](buildinstall.md#onunix) + + [The BDB SQL ADO.NET Interface](buildinstall.md#ado_net) + + [Accessing BDB SQL Databases](accessing_bdb_sql_databases.title.md) + + [The Journal Directory](journaldirectory.md) + + [Unsupported PRAGMAs](unsupportedpragmas.md) + + [Changed PRAGMAs](changedpragmas.md) + + [PRAGMA auto_vacuum](changedpragmas.md#auto_vacuum) + + [PRAGMA incremental_vacuum](changedpragmas.md#incremental_vacuum) + + [PRAGMA journal_size_limit](changedpragmas.md#journal_size_limit) + + [Added PRAGMAs](addedpragmas.md) + + [PRAGMA bdbsql_error_file](addedpragmas.md#bdbsql_error_file) + + [PRAGMA bdbsql_lock_tablesize](addedpragmas.md#bdbsql_lock_tablesize) + + [PRAGMA bdbsql_shared_resources](addedpragmas.md#bdbsql_shared_resources) + + [PRAGMA bdbsql_single_process](addedpragmas.md#bdbsql_single_process) + + [PRAGMA bdbsql_system_memory](addedpragmas.md#bdbsql_system_memory) + + [PRAGMA bdbsql_vacuum_fillpercent](addedpragmas.md#bdbsql_vacuum_fillpercent) + + [PRAGMA bdbsql_vacuum_pages](addedpragmas.md#bdbsql_vacuum_pages) + + [PRAGMA multiversion](addedpragmas.md#multiversion) + + [PRAGMA snapshot_isolation](addedpragmas.md#snapshot_isolation) + + [PRAGMA trickle](addedpragmas.md#trickle) + + [PRAGMA txn_bulk](addedpragmas.md#txn_bulk) + + [Replication PRAGMAs](addedpragmas.md#replication_pragmas) + + [Miscellaneous Differences](miscdiff.md) + + [Berkeley DB Concepts](bdb-concepts.md) + + [Encryption](sql_encryption.md) + + [Using Sequences](sequencesupport.md) + + [create_sequence](sequencesupport.md#create_sequence) + + [nextval](sequencesupport.md#seq_nextval) + + [currval](sequencesupport.md#seq_currval) + + [drop_sequence](sequencesupport.md#seq_drop_sequence) + + [Differences for Users of other SQL Engines](normal-sql.md) + +Welcome to the Berkeley DB SQL interface. If you are a SQLite user who is using the BDB SQL interface for reasons other than performance enhancements, this chapter tells you the minimum things you need to know about the interface. You should simply read this chapter and then skip the rest of this book. + +If, however, you are using the BDB SQL interface for performance reasons, then you need to read this chapter, plus most of the rest of the chapters in this book (although you can probably skip most of Administrating Berkeley DB SQL Databases, unless you want to administer your database "the Berkeley DB way"). + +Also, if you are an existing Berkeley DB user who is interested in the BDB SQL interface, read this chapter plus the rest of this book. + +## BDB SQL Is Nearly Identical to SQLite + +Your interaction with the BDB SQL interface is almost identical to SQLite. You use the same APIs, the same command shell environment, the same SQL statements, and the same PRAGMAs to work with the database created by the BDB SQL interface as you would if you were using SQLite. + +To learn how to use SQLite, see the official SQLite Documentation Page. + +That said, there are a few small differences between the two interfaces. These are described in the remainder of this chapter. diff --git a/docs-src/guides/bdb-sql/index.md b/docs-src/guides/bdb-sql/index.md new file mode 100644 index 000000000..e37cf401d --- /dev/null +++ b/docs-src/guides/bdb-sql/index.md @@ -0,0 +1,214 @@ +--- +title: "Getting Started with the Oracle Berkeley DB SQL APIs" +api-name: "Getting Started with the Oracle Berkeley DB SQL APIs" +source: docs/bdb-sql/index.html +--- +# Getting Started with the Oracle Berkeley DB SQL APIs + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Berkeley DB SQL: The Absolute Basics](dbsqlbasics.md) + + [BDB SQL Is Nearly Identical to SQLite](dbsqlbasics.md#identicalusage) + + [Getting and Installing BDB SQL](buildinstall.md) + + [On Windows Systems](buildinstall.md#onwin) + + [On Unix](buildinstall.md#onunix) + + [The BDB SQL ADO.NET Interface](buildinstall.md#ado_net) + + [Accessing BDB SQL Databases](accessing_bdb_sql_databases.title.md) + + [The Journal Directory](journaldirectory.md) + + [Unsupported PRAGMAs](unsupportedpragmas.md) + + [Changed PRAGMAs](changedpragmas.md) + + [PRAGMA auto_vacuum](changedpragmas.md#auto_vacuum) + + [PRAGMA incremental_vacuum](changedpragmas.md#incremental_vacuum) + + [PRAGMA journal_size_limit](changedpragmas.md#journal_size_limit) + + [Added PRAGMAs](addedpragmas.md) + + [PRAGMA bdbsql_error_file](addedpragmas.md#bdbsql_error_file) + + [PRAGMA bdbsql_lock_tablesize](addedpragmas.md#bdbsql_lock_tablesize) + + [PRAGMA bdbsql_shared_resources](addedpragmas.md#bdbsql_shared_resources) + + [PRAGMA bdbsql_single_process](addedpragmas.md#bdbsql_single_process) + + [PRAGMA bdbsql_system_memory](addedpragmas.md#bdbsql_system_memory) + + [PRAGMA bdbsql_vacuum_fillpercent](addedpragmas.md#bdbsql_vacuum_fillpercent) + + [PRAGMA bdbsql_vacuum_pages](addedpragmas.md#bdbsql_vacuum_pages) + + [PRAGMA multiversion](addedpragmas.md#multiversion) + + [PRAGMA snapshot_isolation](addedpragmas.md#snapshot_isolation) + + [PRAGMA trickle](addedpragmas.md#trickle) + + [PRAGMA txn_bulk](addedpragmas.md#txn_bulk) + + [Replication PRAGMAs](addedpragmas.md#replication_pragmas) + + [Miscellaneous Differences](miscdiff.md) + + [Berkeley DB Concepts](bdb-concepts.md) + + [Encryption](sql_encryption.md) + + [Using Sequences](sequencesupport.md) + + [create_sequence](sequencesupport.md#create_sequence) + + [nextval](sequencesupport.md#seq_nextval) + + [currval](sequencesupport.md#seq_currval) + + [drop_sequence](sequencesupport.md#seq_drop_sequence) + + [Differences for Users of other SQL Engines](normal-sql.md) + + [2. Locking Notes](lockingnotes.md) + + [Internal Database Usage](lockingnotes.md#dbusage) + + [Lock Handling](lockhandling.md) + + [SQLite Lock Usage](lockhandling.md#sqllockmodel) + + [Lock Usage with the BDB SQL Interface](lockhandling.md#bdblockusage) + + [3. Berkeley DB Features](dbfeatures.md) + + [Using Bulk Loading](dbfeatures.md#bulkloading) + + [Using Multiversion Concurrency Control](mvcc.md) + + [Selecting the Page Size](selectpage_size.md) + + [4. Using Replication with the SQL API](sqlrep.md) + + [Replication Overview](sqlrep.md#repoverview) + + [Replication Masters](sqlrep.md#repmasters) + + [Elections](sqlrep.md#repelect) + + [Durability Guarantees](sqlrep.md#repdurability) + + [Two-Site Replication Groups](sqlrep.md#twositerep) + + [Replication PRAGMAs](reppragma.md) + + [PRAGMA replication](reppragma.md#pragma_replication) + + [PRAGMA replication_initial_master](reppragma.md#pragma_replication_initial_master) + + [PRAGMA replication_local_site](reppragma.md#pragma_replication_local_site) + + [PRAGMA replication_remote_site](reppragma.md#pragma_replication_remote_site) + + [PRAGMA replication_remove_site](reppragma.md#pragma_replication_remove_site) + + [PRAGMA replication_verbose_output](reppragma.md#pragma_replication_verbose_output) + + [PRAGMA replication_verbose_file](reppragma.md#pragma_replication_verbose_file) + + [Displaying Replication Statistics](repstatistics.md) + + [Replication Usage Examples](rep_usageexamples.md) + + [Example 1: Distributed Read at 3 Sites](rep_usageexamples.md#rep_ex1) + + [Example 2: 2-Site Failover](rep_usageexamples.md#rep_ex2) + + [5. Administrating Berkeley DB SQL Databases](admin.md) + + [Backing Up Berkeley DB SQL Databases](admin.md#backup) + + [Backing Up Replicated Berkeley DB SQL Databases](admin.md#idp50739296) + + [Syncing with Oracle Databases](sync.md) + + [Syncing on Unix Platforms](sync.md#syncunix) + + [Syncing on Windows Platforms](sync.md#syncwin) + + [Syncing on Windows Mobile Platforms](sync.md#syncwinmobile) + + [Data Migration](datamigration.md) + + [Migration Using the Shells](datamigration.md#shellmigrate) + + [A. Using the BFILE Extension](bfile-extension.md) + + [Supported Platforms and Languages](bfile-extension.md#bfile-support) + + [BFILE SQL Objects and Functions](bfile-sql.md) + + [BFILE_CREATE_DIRECTORY](bfile-sql.md#bfile_create_directory) + + [BFILE_REPLACE_DIRECTORY](bfile-sql.md#bfile_replace_directory) + + [BFILE_DROP_DIRECTORY](bfile-sql.md#bfile_drop_directory) + + [BFILE_NAME](bfile-sql.md#bfile_name) + + [BFILE_FULLPATH](bfile-sql.md#bfile_fullpath) + + [BFILE_OPEN](bfile-sql.md#bfile_open) + + [BFILE_READ](bfile-sql.md#bfile_read) + + [BFILE_CLOSE](bfile-sql.md#bfile_close) + + [BFILE_SIZE](bfile-sql.md#bfile_size) + + [BFILE C/C++ Objects and Functions](bfile-c.md) + + [sqlite3_column_bfile](bfile-c.md#sqlite3_column_bfile) + + [sqlite3_bfile_open](bfile-c.md#sqlite3_bfile_open) + + [sqlite3_bfile_close](bfile-c.md#sqlite3_bfile_close) + + [sqlite3_bfile_is_open](bfile-c.md#sqlite3_bfile_is_open) + + [sqlite3_bfile_read](bfile-c.md#sqlite3_bfile_read) + + [sqlite3_bfile_file_exists](bfile-c.md#sqlite3_bfile_file_exists) + + [sqlite3_bfile_size](bfile-c.md#sqlite3_bfile_size) + + [sqlite3_bfile_final](bfile-c.md#sqlite3_bfile_final) diff --git a/docs-src/guides/bdb-sql/journaldirectory.md b/docs-src/guides/bdb-sql/journaldirectory.md new file mode 100644 index 000000000..392a33f09 --- /dev/null +++ b/docs-src/guides/bdb-sql/journaldirectory.md @@ -0,0 +1,14 @@ +--- +title: "The Journal Directory" +api-name: "The Journal Directory" +source: docs/bdb-sql/journaldirectory.html +--- +## The Journal Directory + +When you create a database using the BDB SQL interface, a directory is created alongside of it. This directory has the same name as your database file, but with a `-journal` suffix. + +That is, if you create a database called "mydb" then the BDB SQL interface also creates a directory alongside of the "mydb" file called "mydb-journal". + +This directory contains files that are very important for the proper functioning of the BDB SQL interface. Do not delete this directory or any of its files unless you know what you are doing. + +In Berkeley DB terms, the journal directory contains the environment files that are required to provide access to databases across multiple processes. diff --git a/docs-src/guides/bdb-sql/lockhandling.md b/docs-src/guides/bdb-sql/lockhandling.md new file mode 100644 index 000000000..075f1bfec --- /dev/null +++ b/docs-src/guides/bdb-sql/lockhandling.md @@ -0,0 +1,111 @@ +--- +title: "Lock Handling" +api-name: "Lock Handling" +source: docs/bdb-sql/lockhandling.html +--- +## Lock Handling + + [SQLite Lock Usage](lockhandling.md#sqllockmodel) + + [Lock Usage with the BDB SQL Interface](lockhandling.md#bdblockusage) + +There is a difference in how applications written for the BDB SQL interface handle deadlocks as opposed to how deadlocks are handled for SQLite applications. For the SQLite developer, the following information is a necessary review in order to understand how the BDB SQL interface behaves differently. + +From a usage point of view, the BDB SQL interface behaves in the same way as SQLite in shared cache mode. The implications of this are explained below. + +### SQLite Lock Usage + +As mentioned previously in this chapter, SQLite locks the entire database while performing a transaction. It also has a locking model that is different from the BDB SQL interface, one that supports multiple readers, but only a single writer. In SQLite, transactions can start as follows: + +- `BEGIN` + + Begins the transaction, locking the entire database for reading. Use this if you only want to read from the database. + +- `BEGIN IMMEDIATE` + + Begins the transaction, acquiring a "modify" lock. This is also known as a RESERVED lock. Use this if you are modifying the database (that is, performing `INSERT`, `UPDATE`, or `DELETE`). RESERVED locks and read locks can co-exist. + +- `BEGIN EXCLUSIVE` + + Begins the transaction, acquiring a write lock. Transactions begun this way will be written to the disk upon commit. No other lock can co-exist with an exclusive lock. + +The last two statements are a kind of a contract. If you can get them to complete (that is, not return `SQLITE_LOCKED`), then you can start modifying the database (that is, change data in the in-memory cache), and you will eventually be able to commit (write) your modifications to the database. + +In order to avoid deadlocks in SQLite, programmers who want to modify a SQLite database start the transaction with `BEGIN IMMEDIATE`. If the transaction cannot acquire the necessary locks, it will fail, returning `SQLITE_BUSY`. At that point, the transaction falls back to an unlocked state whereby it holds no locks against the database. This means that any existing transactions in a RESERVED state can safely wait for the necessary EXCLUSIVE lock in order to finally write their modifications from the in-memory cache to the on-disk database. + +The important point here is that so long as the programmer uses these locks correctly, he can assume that he can proceed with his work without encountering a deadlock. (Assuming that all database readers and writers are also using these locks correctly.) + +### Lock Usage with the BDB SQL Interface + +When you use the BDB SQL interface, you can begin your transaction with `BEGIN` or `BEGIN EXCLUSIVE`. + +Note that the `IMMEDIATE` keyword is ignored in the BDB SQL interface (`BEGIN IMMEDIATE` behaves like `BEGIN`). + +When you begin your transaction with `BEGIN`, Berkeley DB decides what kind of a lock you need based on what you are doing to the database. If you perform an action that is read-only, it acquires a read lock. If you perform a write action, it acquires a write lock. + +Also, the BDB SQL interface supports multiple readers *and* multiple writers. This means that multiple transactions can acquire locks as long as they are not trying to modify the same page. For example: + +**Session 1:** + +``` c +dbsql> create table a(x int); +dbsql> begin; +dbsql> insert into a values (1); +dbsql> commit; +``` + +**Session 2:** + +``` c +dbsql> create table b(x int); +dbsql> begin; +dbsql> insert into b values (1); +dbsql> commit; +``` + +Because these two sessions are operating on different pages in the Berkeley DB cache, this example will work. If you tried this with SQLite, you could not start the second transaction until the first had completed. + +However, if you do this using the BDB SQL interface: + +**Session 1:** + +``` c +dbsql> begin; +dbsql> insert into a values (2); +``` + +**Session 2:** + +``` c +dbsql> begin; +dbsql> insert into a values (2); +``` + +The second session blocks until the first session commits the transaction. Again, this is because both sessions are operating on the same database page(s). However, if you simultaneously attempt to write pages in reverse order, you can deadlock. For example: + +**Session 1:** + +``` c +dbsql> begin; +dbsql> insert into a values (3); +dbsql> insert into b values (3); +``` + +**Session 2:** + +``` c +dbsql> begin; +dbsql> insert into b values (3); +dbsql> insert into a values (3); +Error: database table is locked +``` + +What happens here is that Session 1 is blocked waiting for a lock on table b, while Session 2 is blocked waiting for a lock on table a. The application can make no forward progress, and so it is deadlocked. + +When such a deadlock is detected one session loses the lock it got when executing its last statement, and that statement is automatically rolled back. The rest of the statements in the session will still be valid, and you can continue to execute statements in that session. The session that does not lose its lock to deadlock detection will continue to execute as if nothing happened. + +Assume Session 2 was sacrificed to deadlock detection, no value would be inserted into a and an error will be returned. But the insertion of value 3 into b would still be valid. Session 1 would continue to wait while inserting into table b until Session 2 either commits or aborts, thus freeing the lock it has on table b. + +When you begin your transaction with `BEGIN EXCLUSIVE`, the session is never aborted due to deadlock or lock contention with another transaction. Non-exclusive transactions are allowed to execute concurrently with the exclusive transaction, but the non-exclusive transactions will have their locks released if deadlock with the exclusive transaction occurs. If two or more exclusive transactions are running at the same time, they will be forced to execute in serial. + +If Session 1 was using an exclusive transaction, then Session 2 would lose its locks when deadlock is detected between the two. If both Session 1 and Session 2 start an exclusive transaction, then the last one to start the exclusive transaction would be blocked after executing `BEGIN EXCLUSIVE` until the first one is committed or aborted. diff --git a/docs-src/guides/bdb-sql/lockingnotes.md b/docs-src/guides/bdb-sql/lockingnotes.md new file mode 100644 index 000000000..3c6cd4091 --- /dev/null +++ b/docs-src/guides/bdb-sql/lockingnotes.md @@ -0,0 +1,36 @@ +--- +title: "Chapter 2. Locking Notes" +api-name: "Chapter 2. Locking Notes" +source: docs/bdb-sql/lockingnotes.html +--- +## Chapter 2. Locking Notes + +**Table of Contents** + + [Internal Database Usage](lockingnotes.md#dbusage) + + [Lock Handling](lockhandling.md) + + [SQLite Lock Usage](lockhandling.md#sqllockmodel) + + [Lock Usage with the BDB SQL Interface](lockhandling.md#bdblockusage) + +There are some important performance differences between the BDB SQL interface and SQLite, especially in a concurrent environment. This chapter gives you enough information about how the BDB SQL interface uses its database, as opposed to how SQLite uses its database, in order for you to understand the difference between the two interfaces. It then gives you some advice on how to best approach working with the BDB SQL interface in a multi-threaded environment. + +If you are an existing user of SQLite, and you care about improving your application performance when using the BDB SQL interface in a concurrent situation, you should read this chapter. Existing users of Berkeley DB may also find some interesting information in this chapter, although it is mostly geared towards SQLite users. + +## Internal Database Usage + +The BDB SQL interface and SQLite do different things when it comes to locking data in their databases. In order to provide ACID transactions, both products must prevent concurrent access during write operations. Further, both products prevent concurrent access by obtaining software level locks that allow only the current holder of the lock to perform write access to the locked data. + +The difference between the two is that when SQLite requires a lock (such as when a transaction is underway), it locks the entire database and all tables. (This is known as *database level locking*.) The BDB SQL interface, on the other hand, only locks the portion of the table being operated on within the current transactional context (this is known as *page level locking*). In most situations, this allows applications using the BDB SQL interface to operate concurrently and so have better read/write throughput than applications using SQLite. This is because there is less lock contention. + +By default, one Berkeley DB logical database is created within the single database file for every SQL table that you create. Within each such logical database, each table row is represented as a Berkeley DB key/data pair. + +This is important because the BDB SQL interface uses Berkeley DB's Transaction Data Store product. This means that Berkeley DB does not have to lock an entire database (all the tables within a database file) when it acquires a lock. Instead, it locks a single Berkeley DB database page (which usually contains a small sub-set of rows within a single table). + +The size of database pages will differ from platform to platform (you can also manually configure this), but usually a database page can hold multiple key/data pairs; that is, multiple rows from a SQL table. Exactly how many table rows fit on a database page depends on the size of your page and the size of your table rows. + +If you have an exceptionally small table, it is possible for the entire table to fit on a single database page. In this case, Berkeley DB is in essence forced to serialize access to the entire table when it requires a lock for it. + +Note, however, that the case of a single table fitting on a single database page is very rare, and it in fact represents the abnormal case. Normally tables span multiple pages and so Berkeley DB will lock only portions of your tables. This locking behavior is automatic and transparent to your application. diff --git a/docs-src/guides/bdb-sql/miscdiff.md b/docs-src/guides/bdb-sql/miscdiff.md new file mode 100644 index 000000000..70e3097df --- /dev/null +++ b/docs-src/guides/bdb-sql/miscdiff.md @@ -0,0 +1,58 @@ +--- +title: "Miscellaneous Differences" +api-name: "Miscellaneous Differences" +source: docs/bdb-sql/miscdiff.html +--- +## Miscellaneous Differences + +The following miscellaneous differences also exist between the BDB SQL interface and SQLite: + +- The BDB SQL interface does not support the `IMMEDIATE` keyword (`BEGIN IMMEDIATE` behaves just like `BEGIN`). + +- When an exclusive transaction is active, it will block any new transactions from beginning (they will be blocked during their first operation until the exclusive transactions commits or aborts). Non-exclusive transactions that are active when the exclusive transaction begins will not be able to execute any more operations without being blocked until the exclusive transactions finishes. + +- Enabling MVCC mostly disables exclusive transactions. Exclusive transactions can still be used, but they will run concurrently with regular transactions, even ones that write to the database. The only advantage of exclusive transactions in this case is that two exclusive transactions will be forced to run in serial, and that if an exclusive transaction and non-exclusive transaction experience deadlock, then the non-exclusive transaction will always be the transaction forced to release its locks. + + For more information on MVCC and snapshot isolation, see Using Multiversion Concurrency Control + +- There are differences in how the two products work in a concurrent application that will cause the BDB SQL interface to deadlock where SQLite would result in a different error. This is because the products use different locking paradigms. See Locking Notes for more information. + +- The BDB SQL does not call the busy callback when a session attempts to operate the same database page that another session has locked. It blocks instead. This means that the functions `sqlite3_busy_handler` and `sqlite3_busy_timeout` are not effective in BDB SQL. + +- The BDB SQL does not support two phase commit across databases. Attaching to multiple databases can lead to inconsistency after recovery and undetected deadlocks when accessing multiple databases from concurrent transactions in different order. Hence, applications must ensure that they access databases in the same order in any transaction that spans multiple databases. Else, a deadlock can occur that causes threads to block, and the deadlock will not be detected by Berkeley DB. + +- In BDB SQL, when two sessions accessing the same database perform conflicting operations on the same page, one session will be blocked until the conflicting operations are resolved. For example, + + **Session 1:** + + ``` c + dbsql> insert into a values (4); + dbsql> begin; + dbsql> insert into a values (5); + ``` + + **Session 2:** + + ``` c + dbsql> select * from a; + ``` + + What happens here is that Session 2 is blocked until Session 1 commits the transaction. + + **Session 1:** + + ``` c + dbsql> commit; + ``` + + **Session 2:** + + ``` c + dbsql> select * from a; + 4 + 5 + ``` + + Under such situations in SQLite, operations poll instead of blocking, and a callback is used to determine whether to continue polling. + +- By default, you always only have a single database file when you use BDB SQL interface SQL, just as you do when you use SQLite. However, you can configure BDB SQL interface at compile time to create one BDB SQL interface database file for each SQL table that you create. How to perform this configuration is described in the *Berkeley DB Installation and Build Guide*. diff --git a/docs-src/guides/bdb-sql/moreinfo.md b/docs-src/guides/bdb-sql/moreinfo.md new file mode 100644 index 000000000..f5887b189 --- /dev/null +++ b/docs-src/guides/bdb-sql/moreinfo.md @@ -0,0 +1,26 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/bdb-sql/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when using the Berkeley DB SQL interface: + +- Berkeley DB Installation and Build Guide + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Getting Started with Replicated Applications + +To download the latest documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/bdb-sql/mvcc.md b/docs-src/guides/bdb-sql/mvcc.md new file mode 100644 index 000000000..b1d436dfd --- /dev/null +++ b/docs-src/guides/bdb-sql/mvcc.md @@ -0,0 +1,26 @@ +--- +title: "Using Multiversion Concurrency Control" +api-name: "Using Multiversion Concurrency Control" +source: docs/bdb-sql/mvcc.html +--- +## Using Multiversion Concurrency Control + +Multiversion Concurrency Control (MVCC) enables snapshot isolation. Snapshot isolation means that whenever a transaction would take a read lock on a page, it makes a copy of the page instead, and then performs its operations on that copied page. This frees other writers from blocking due to a read locks held by other transactions. + +You should use snapshot isolation whenever you have a lot of read-only transactions operating at the same time that read-write transactions. In this case, snapshot isolation will improve transaction throughput, albeit at the cost of greater resource usage. + +MVCC is described in more detail in Snapshot Isolation. + +To use MVCC, you must enable it before you access any database tables. Once MVCC is enabled, you can turn snapshot isolation on and off at anytime during the life of your application. + +To turn MVCC on, use: + +**PRAGMA multiversion** = on \| off; + +This PRAGMA must be enabled before you access any database tables during your application runtime, or an error is returned. Turning MVCC on automatically enables snapshot isolation. By default MVCC is turned off. + +Once MVCC is enabled, you can turn snapshot isolation on and off using: + +**PRAGMA snapshot_isolation** = on \| off; + +This PRAGMA can be used at any time during the life of your application *after* MVCC has been turned on. If you attempt to enable or disable snapshot isolation before MVCC is enabled, an error is returned. diff --git a/docs-src/guides/bdb-sql/normal-sql.md b/docs-src/guides/bdb-sql/normal-sql.md new file mode 100644 index 000000000..22ed8e62d --- /dev/null +++ b/docs-src/guides/bdb-sql/normal-sql.md @@ -0,0 +1,20 @@ +--- +title: "Differences for Users of other SQL Engines" +api-name: "Differences for Users of other SQL Engines" +source: docs/bdb-sql/normal-sql.html +--- +## Differences for Users of other SQL Engines + +If you are used to a SQL implementation from other SQL engine (such as Oracle's RDBMS), the SQL used by the BDB SQL interface (which is the same as used by SQLite) may hold some surprises for you. + +Some things in particular to take note of: + +- Datatyping is weaker in SQLite than it is with standard SQL. For example, SQLite does not enforce the length of a `VARCHAR`. While standard SQL will truncate a `VARCHAR` that is too long, you could (for example) declare a `VARCHAR(10)` then put 500 characters in it without any truncation, ever. + + SQLite datatyping is described in detail on the Datatypes in SQLite Version 3 page. + +- Do not use autocommit with SQLite. Instead, use `begin exclusive` and then `commit`. + +- How NULLs are handled in SQLite may be different from what you are used to. See NULL Handling in SQLite Versus Other Database Engines for details. + +- There are some features of SQL that SQLite does not support. For more information, see SQL Features That SQLite Does Not Implement. diff --git a/docs-src/guides/bdb-sql/preface.md b/docs-src/guides/bdb-sql/preface.md new file mode 100644 index 000000000..ba3ee285f --- /dev/null +++ b/docs-src/guides/bdb-sql/preface.md @@ -0,0 +1,46 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/bdb-sql/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +Welcome to the Berkeley DB SQL interface. This manual describes how to configure and use the SQL interface to Berkeley DB 11*g* Release 2. This manual also describes common administrative tasks, such as backup and restore, database dump and load, and data migration when using the BDB SQL interface. + +This manual is intended for anyone who wants to use the BDB SQL interface. Because usage of the BDB SQL interface is very nearly identical to SQLite, prior knowledge of SQLite is assumed by this manual. No prior knowledge of Berkeley DB is necessary, but it is helpful. + +To learn about SQLite, see the official SQLite website at: http://www.sqlite.org + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Keywords or literal text that you are expected to type is presented in a `monospaced font`. For example: "Use the `DB_HOME` environment variable to identify the location of your environment directory." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples and literal text that you might type are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/guides/bdb-sql/rep_usageexamples.md b/docs-src/guides/bdb-sql/rep_usageexamples.md new file mode 100644 index 000000000..26d3d9b9b --- /dev/null +++ b/docs-src/guides/bdb-sql/rep_usageexamples.md @@ -0,0 +1,176 @@ +--- +title: "Replication Usage Examples" +api-name: "Replication Usage Examples" +source: docs/bdb-sql/rep_usageexamples.html +--- +## Replication Usage Examples + + [Example 1: Distributed Read at 3 Sites](rep_usageexamples.md#rep_ex1) + + [Example 2: 2-Site Failover](rep_usageexamples.md#rep_ex2) + +In this section we provide two examples of using replication with BDB SQL. The first example shows a typical startup process. The second demonstrates master site failover. + +### Example 1: Distributed Read at 3 Sites + +This example shows how a typical replication group startup sequence is performed. It initially populates the master, then starts two replicas so that read operations can be distributed. Then it shows what happens when an attempt is made to write to a replica. + +Site 1: + +``` c +# Start initial master. +# univ.db does not yet exist. +dbsql univ.db + pragma replication_local_site="site1:7000"; + pragma replication_initial_master=ON; + pragma replication=ON; + + # Create and populate university and country tables. + .read university.sql +``` + +Site 2: + +``` c +# Start first replica. +# univ.db does not yet exist. +dbsql univ.db + pragma replication_local_site="site2:7001"; + pragma replication_remote_site="site1:7000"; + pragma replication=ON; +``` + +Site 3: + +``` c +# Start second replica. +# univ.db does not yet exist. +dbsql univ.db + pragma replication_local_site="site3:7002"; + pragma replication_remote_site="site1:7000"; + pragma replication=ON; +``` + +Site 1: + +``` c + # Perform some writes and reads on master. + insert into country values ("Greenland","gl", 0, 0, 0, 2); + insert into university values (26, "University College London", + "ucl.edu", "uk", "Europe", 18, 39, 47, 30); + select * from country where abbr = "gl"; + update country set top_1000 = 1 where abbr = "gl"; +``` + +Site 2: + +``` c + # Perform some reads on first replica. + select * from university where region = "Europe"; + select count(*) from country where top_100 > 0; + + # Attempt to write on first replica. + insert into country values ("Antarctica","an", 0, 0, 0, 0); + .../univ.db: DBcursor->put: attempt to modify a read-only database + Error: attempt to write to a readonly database +``` + +### Example 2: 2-Site Failover + +This example demonstrates failover of the master from one site to another. It shows how a failed site can rejoin the replication group, and it shows that there is a window of time during which write operations cannot be performed for the replication group. Finally, it shows how to check the master's location. + +Site 1: + +``` c +# Start initial master. +# quote.db does not yet exist. +dbsql quote.db + pragma replication_local_site="site1:7000"; + pragma replication_initial_master=ON; + pragma replication=ON; + + # Create stock quote application table. + create table stock_quote (company_name text(40), price real); +``` + +Site 2: + +``` c +# Start replica. +# quote.db does not yet exist. +dbsql quote.db + pragma replication_local_site="site2:7001"; + pragma replication_remote_site="site1:7000"; + pragma replication=ON; +``` + +Site 1: + +``` c + # Perform some writes on master. + insert into stock_quote values ("General Electric", 20.25); + insert into stock_quote values ("Nabisco", 24.75); + insert into stock_quote values ("United Healthcare", 31.00); + update stock_quote set price=25.25 where company_name = "Nabisco"; +``` + +Site 2: + +``` c + # Perform some reads on replica. + select * from stock_quote where price < 30.00; + select price from stock_quote where + company_name = "General Electric"; +``` + +Site 1: + +``` c + # Stop the initial master. + .exit +``` + +Site 2: + +``` c + ########## + ### Now the remaining site does not accept write operations until + ### the other site rejoins the replication group. + ########## + insert into stock_quote values ("Prudential", 17.25); + .../quote.db: DBcursor->put: attempt to modify a read-only database + Error: attempt to write to a readonly database +``` + +Site 1: + +``` c +# Restart site, will rejoin replication group. +dbsql quote.db + # The earlier replication=ON causes replication to be + # automatically started. This site may or may not become master + # after rejoining replication group. Check status of site's + # startup and determine whether it is a master or a replica. + .stat :rep: + Replication summary statistics + Environment configured as a replication master + 1/49056 Maximum permanent LSN + 2 Number of environments in the replication group + 0 Number of failed message sends + 0 Number of messages ignored due to pending recovery + 0 Number of log records currently queued + + # Assuming this site became master, perform some writes. + # If this site is not the master, these writes will not + # succeed and must be performed at the other site. + insert into stock_quote values ("Raytheon", 9.25); + insert into stock_quote values ("Cadbury", 7.75); +``` + +Site 2: + +``` c + # Read operations can be performed on master or replica + # site. + select * from stock_quote where price < 21.00; +``` diff --git a/docs-src/guides/bdb-sql/reppragma.md b/docs-src/guides/bdb-sql/reppragma.md new file mode 100644 index 000000000..38da5556f --- /dev/null +++ b/docs-src/guides/bdb-sql/reppragma.md @@ -0,0 +1,96 @@ +--- +title: "Replication PRAGMAs" +api-name: "Replication PRAGMAs" +source: docs/bdb-sql/reppragma.html +--- +## Replication PRAGMAs + + [PRAGMA replication](reppragma.md#pragma_replication) + + [PRAGMA replication_initial_master](reppragma.md#pragma_replication_initial_master) + + [PRAGMA replication_local_site](reppragma.md#pragma_replication_local_site) + + [PRAGMA replication_remote_site](reppragma.md#pragma_replication_remote_site) + + [PRAGMA replication_remove_site](reppragma.md#pragma_replication_remove_site) + + [PRAGMA replication_verbose_output](reppragma.md#pragma_replication_verbose_output) + + [PRAGMA replication_verbose_file](reppragma.md#pragma_replication_verbose_file) + +To control replication when using the Berkeley DB SQL interface, you use the following PRAGMAs. For an example of how to use these, see Replication Usage Examples. + +### PRAGMA replication + +``` c +PRAGMA replication=ON|OFF +``` + +Enables the local environment to participate in replication. + +Before invoking this PRAGMA for a brand new database (one that has never been opened), you must invoke the `replication_local_site` PRAGMA and then either the `replication_initial_master` or the `replication_remote_site` PRAGMA. These actions define the way this site fits into the replication group. + +If you are enabling replication for an existing database, it must become the initial master for a new replication group. You must invoke the `replication_local_site` PRAGMA followed by the `replication_initial_master` PRAGMA before enabling replication. + +If you use this PRAGMA to turn off replication, then replication is completely disabled for the environment. In order to enable replication again, you follow the procedure used to enable replication on an existing database; that is, invoke the `replication_local_site` PRAGMA followed by the `replication_initial_master` PRAGMA, followed by `PRAGMA replication=ON`. + +### PRAGMA replication_initial_master + +``` c +PRAGMA replication_initial_master=ON|OFF +``` + +Causes the local environment to start up as a master site. This PRAGMA must be used once and only once in the replicated lifetime of a BDB SQL environment. + +This PRAGMA is usually invoked for the first site in a new replication group before the `replication` PRAGMA is invoked and before BDB SQL initially creates the underlying BDB environment for a SQL database. Starting replication on the initial master site establishes the new replication group so that other sites can join it. + +However, you must call this PRAGMA when enabling replication for a database that already exists. Doing so causes the existing database to become the replication master for a new replication group. + +Note that subsequent election activity can cause other sites in the replication group to become master. Do not assume that the initial master site will remain master indefinitely, or that it will rejoin the replication group as master after a shutdown. + +### PRAGMA replication_local_site + +``` c +PRAGMA replication_local_site="hostname:port" +``` + +Sets the local site information for replication. + +### PRAGMA replication_remote_site + +``` c +PRAGMA replication_remote_site="hostname:port" +``` + +Sets information about a remote helper site in the replication group. + +This PRAGMA is needed when a site first joins an existing replication group to specify a site that is already in the replication group. It must be invoked before the `replication` PRAGMA is invoked. This PRAGMA is not needed on the initial master site or when restarting a site that is already a member of the replication group. However, supplying this PRAGMA in those situations does no harm. + +Note that the information provided to this PRAGMA can be superseded by normal replication activity over the course of the environment's lifetime. + +### PRAGMA replication_remove_site + +``` c +PRAGMA replication_remove_site="hostname:port" +``` + +Removes the specified site from the replication group. Use this PRAGMA if you truly want to remove the site permanently from the group. It is not desirable to call this PRAGMA if a site has been temporarily shut down or disconnected from the rest of the replication group. + +Removing a site from the replication group means that the site is no longer counted towards the total number of sites belonging to the group. This is important when the replication group requires knowledge about whether a quorum has been reached (such as when, for example, elections are held). + +### PRAGMA replication_verbose_output + +``` c +PRAGMA replication_verbose_output=ON|OFF +``` + +If set to TRUE, additional logging information specifically related to replication is created. + +### PRAGMA replication_verbose_file + +``` c +PRAGMA replication_verbose_file="filename" +``` + +Indicates that verbose replication output should be sent to the specified file, as opposed to STDOUT. diff --git a/docs-src/guides/bdb-sql/repstatistics.md b/docs-src/guides/bdb-sql/repstatistics.md new file mode 100644 index 000000000..6f419697f --- /dev/null +++ b/docs-src/guides/bdb-sql/repstatistics.md @@ -0,0 +1,46 @@ +--- +title: "Displaying Replication Statistics" +api-name: "Displaying Replication Statistics" +source: docs/bdb-sql/repstatistics.html +--- +## Displaying Replication Statistics + +You can display a brief summary of replication statistics using `.stat :rep:`. This command displays the most basic information about replication status, as well as some information that is useful for troubleshooting. + +``` c +dbsql> .stat :rep: +Replication summary statistics +Environment configured as a replication client +Startup complete +1/50232 Maximum permanent LSN +2 Number of environments in the replication group +0 Number of failed message sends +0 Number of messages ignored due to pending recovery +0 Number of log records currently queued +``` + +In the above output: + +- `Environment configured as a replication client` + + Identifies the current role of the site within the replication group. In this example, the current site is not the master site. *Replication client* is another term for *replica*. + +- `Startup complete` + + Indicates that this replica site has completed its synchronization with the master site. Replica synchronization can take some time if there are many master transactions with which it needs to catch up. + +- `Maximum permanent LSN` + + Identifies the most recent log record that is durably replicated on a master or acknowledged by a replica. You can compare a replica's maximum permanent LSN to the master's maximum permanent LSN to determine if the replica is caught up with the master. + +- `Number of failed message sends` + + If this number is increasing, it could be an indication of network or communications problems between sites in the replication group. + +- `Number of messages ignored due to pending recovery` + + If this number is increasing, this site is ignoring messages because it is starting up or recovering and may need some time to catch up with the rest of the replication group. + +- `Number of log records currently queued` + + If this number is increasing, it means that connections to other sites may be unavailable or congested and that there may be delays in durably replicating master transactions. diff --git a/docs-src/guides/bdb-sql/selectpage_size.md b/docs-src/guides/bdb-sql/selectpage_size.md new file mode 100644 index 000000000..120c54caa --- /dev/null +++ b/docs-src/guides/bdb-sql/selectpage_size.md @@ -0,0 +1,20 @@ +--- +title: "Selecting the Page Size" +api-name: "Selecting the Page Size" +source: docs/bdb-sql/selectpage_size.html +--- +## Selecting the Page Size + +When using the BDB SQL interface, you configure your database page size in exactly the same way as you do when using SQLite. That is, use `PRAGMA page_size` to report and set the page size. This PRAGMA must be called before you create your first SQLite table. See the PRAGMA page_size documentation for more information. + +When you use `PRAGMA cache_size` to size your in-memory cache, you provide the cache size in terms of a number of pages. Therefore, your database page size influences how large your cache is, and so determines how much of your database will fit into memory. + +The size of your pages can also affect how efficient your application is at performing disk I/O. It will also determine just how fine-grained the fine-grained locking actually is. This is because Berkeley DB locks database pages when it acquires a lock. + +Note that the default value for your page size is probably correct for the physical hardware that you are using. In almost all situations, the default page size value will give your application the best possible I/O performance. For this reason, tuning the page size should rarely, if ever, be attempted. + +That said, when using the BDB SQL interface, the page size affects how much of your tables are locked when read and/or write locks are acquired. (See Internal Database Usage for more information.) Increasing your page size will typically improve the bandwidth you get accessing the disk, but it also may increase contention if too many key data pairs are on the same page. Decreasing your page size frequently improves concurrency, but may increase the number of locks you need to acquire and may decrease your disk bandwidth. + +When changing your page size, make sure the value you select is a power of 2 that is greater than 512 and less than or equal to 64KB. (Note that the standard SQLite `MAX_PAGE_SIZE` limit is not examined for this upper bound.) + +Beyond that, there are some additional things that you need to consider when selecting your page size. For a thorough treatment of selecting your page size, see the section on Selecting a page size in the *Berkeley DB Programmer's Reference Guide*. diff --git a/docs-src/guides/bdb-sql/sequencesupport.md b/docs-src/guides/bdb-sql/sequencesupport.md new file mode 100644 index 000000000..17937e5cc --- /dev/null +++ b/docs-src/guides/bdb-sql/sequencesupport.md @@ -0,0 +1,98 @@ +--- +title: "Using Sequences" +api-name: "Using Sequences" +source: docs/bdb-sql/sequencesupport.html +--- +## Using Sequences + + [create_sequence](sequencesupport.md#create_sequence) + + [nextval](sequencesupport.md#seq_nextval) + + [currval](sequencesupport.md#seq_currval) + + [drop_sequence](sequencesupport.md#seq_drop_sequence) + +You can use sequences with the SQL API. Sequences provide for an arbitrary number of increasing or decreasing integers that persist across database accesses. Use sequences if you need to create unique values in a highly efficient and persistent way. + +To create and access a sequence, you must use SQL functionality that is unique to the BDB SQL interface; no corresponding functionality exists in SQLite. The sequence functionality is implemented using SQLite function plugins, as such it is necessary to use the 'select' keyword as a prefix to all sequence APIs. + +The SQL API sequence support is a partial implementation of the sequence API defined in the SQL 2003 specification. + +The following sections describe the BDB SQL interface sequence API. + +### create_sequence + +Creates a new sequence. A name is required, all other parameters are optional. For example: + +``` c +SELECT create_sequence("my_sequence", "start", 100, "incr", 10, + "maxvalue", 300); +``` + +This creates a sequence called `my_sequence` starting at 100 and incrementing by 10 until it reaches 300. + +``` c +SELECT create_sequence("my_decr_sequence", "incr", -100, + "minvalue", -10000); +``` + +This creates a sequence call `my_decr_sequence` starting at 0 and decreasing by 100 until it reaches -10000. + +Parameters are: + +- `name` + + Required parameter that provides the name of the sequence. It is an error to create a sequence with another name that is currently in use within the database. + +- `start` + + The starting value for the sequence. If this parameter is not provided then `0` is used. + +- `minvalue` + + The lowest value generated by the sequence. If this parameter is not provided and a decrementing sequence is created, then INT64_MIN is used. + +- `maxvalue` + + The largest value generated by the sequence. If this parameter is not provided and an incrementing sequence is created, then INT64_MAX is used. + +- `incr` + + The amount the sequence is incremented for each get operation. This value can be positive or negative. If this parameter is not provided, then `1` is used. + +- `cache` + + Causes each handle to keep a cache of sequence values. So long as there are values available in the cache, retrieving the next value is cheap and does not lead to contention between handles. + + Sequences with caches cannot be created or dropped within an explicit transaction. + + Operations on caching sequences are not transactionally protected. That is, a rollback will not result in a value being returned to the sequence. + + Sequences with caches do not support the `currval` function. + + The parameter following the cache parameter must be an integer value specifying the size of the cache. + +### nextval + +Retrieves the next value from the named sequence. For example: + +``` c +SELECT nextval("my_sequence"); +``` + +### currval + +Retrieves the last value that was returned from the named sequence. For example: + +``` c +SELECT currval("my_sequence"); +``` + +### drop_sequence + +Removes the sequence. For example: + +``` c +SELECT drop_sequence("my_sequence"); +``` diff --git a/docs-src/guides/bdb-sql/sql_encryption.md b/docs-src/guides/bdb-sql/sql_encryption.md new file mode 100644 index 000000000..c07c8ec59 --- /dev/null +++ b/docs-src/guides/bdb-sql/sql_encryption.md @@ -0,0 +1,14 @@ +--- +title: "Encryption" +api-name: "Encryption" +source: docs/bdb-sql/sql_encryption.html +--- +## Encryption + +The Berkeley DB SQL interface supports the SQLite Encryption Extension (SEE) to ensure security of your data. The supported encryption algorithm is AES-128 in CBC mode. For more information on the concepts relating to BDB encryption, see the Berkeley DB Programmer's Reference Guide. + +To learn how to use the SQLite Encryption Extension (SEE), see the official SQLite Documentation Page. + +### Note + +The Berkeley DB SQL interface does not support the sqlite3_rekey method. diff --git a/docs-src/guides/bdb-sql/sqlrep.md b/docs-src/guides/bdb-sql/sqlrep.md new file mode 100644 index 000000000..a23e46a4a --- /dev/null +++ b/docs-src/guides/bdb-sql/sqlrep.md @@ -0,0 +1,104 @@ +--- +title: "Chapter 4. Using Replication with the SQL API" +api-name: "Chapter 4. Using Replication with the SQL API" +source: docs/bdb-sql/sqlrep.html +--- +## Chapter 4. Using Replication with the SQL API + +**Table of Contents** + + [Replication Overview](sqlrep.md#repoverview) + + [Replication Masters](sqlrep.md#repmasters) + + [Elections](sqlrep.md#repelect) + + [Durability Guarantees](sqlrep.md#repdurability) + + [Two-Site Replication Groups](sqlrep.md#twositerep) + + [Replication PRAGMAs](reppragma.md) + + [PRAGMA replication](reppragma.md#pragma_replication) + + [PRAGMA replication_initial_master](reppragma.md#pragma_replication_initial_master) + + [PRAGMA replication_local_site](reppragma.md#pragma_replication_local_site) + + [PRAGMA replication_remote_site](reppragma.md#pragma_replication_remote_site) + + [PRAGMA replication_remove_site](reppragma.md#pragma_replication_remove_site) + + [PRAGMA replication_verbose_output](reppragma.md#pragma_replication_verbose_output) + + [PRAGMA replication_verbose_file](reppragma.md#pragma_replication_verbose_file) + + [Displaying Replication Statistics](repstatistics.md) + + [Replication Usage Examples](rep_usageexamples.md) + + [Example 1: Distributed Read at 3 Sites](rep_usageexamples.md#rep_ex1) + + [Example 2: 2-Site Failover](rep_usageexamples.md#rep_ex2) + +The Berkeley DB SQL interface allows you to use Berkeley DB's replication feature. You configure and start replication using PRAGMAs that are specific to the task. + +This chapter provides a high-level introduction of Berkeley DB replication. It then shows how to configure and use replication with the SQL API. + +For a more detailed description of Berkeley DB replication, see: + +- *Berkeley DB Getting Started with Replicated Applications* + +- *Berkeley DB Programmer's Reference Guide* + +### Note + +You cannot access a BDB SQL database using multiple processes if you enable replication for that database. + +## Replication Overview + + [Replication Masters](sqlrep.md#repmasters) + + [Elections](sqlrep.md#repelect) + + [Durability Guarantees](sqlrep.md#repdurability) + + [Two-Site Replication Groups](sqlrep.md#twositerep) + +Berkeley DB's replication feature allows you to automatically distribute your database write operations to one or more read-only *replicas*. For this reason, BDB's replication implementation is said to be a *single master, multiple replica* replication strategy. + +A single replication master and all of its replicas are referred to as a *replication group*. Each replication group can have one and only one master site. + +When discussing Berkeley DB replication, we sometimes refer to *replication sites*. This is because most production applications place each of their replication participants on separate physical machines. In fact, each replication participant must be assigned a hostname/port pair that is unique within the replication group. + +Note that under the hood, the unit of replication is the environment. That is, data is replicated from one Berkeley DB environment to one or more other Berkeley DB environments. However, when used with the BDB SQL interface, you can think of this as replicating between Berkeley DB databases, because the BDB SQL interface results in a single database file for each environment. + +### Replication Masters + +Every replication group has one and only one master. The master site is where you perform write operations. These operations are then automatically replicated to the other sites in the replication group. Because the other replica sites in the replication group are read-only, it is an error for you to attempt to perform write operatons on them. + +The replication master is usually automatically selected by the replication group using elections. Replication elections simply determine which replication site has the most up-to-date copy of the data, and so is in the best position to serve as the master site. + +Note that when you initially start up your BDB SQL replicated application, you must explicitly designate a specific site as the master. Over time, the master site can move from one environment to the next. For example, if the master site is shut down, becomes unavailable, or a network partition causes it to lose contact with the rest of the replication group, then the replication group will elect a new master if it can successfully hold an election. When the old master comes back online, it rejoins the replication group as a read-only replica site. + +Also, if you are enabling replication for an existing database, then that database must be designated as the master. Doing this is required; otherwise the entire contents of the existing database might be deleted during the replication startup process. + +### Elections + +A replication group selects the master site by holding an election. In simplistic terms, each participant in the replication group votes on who it believes has the most up-to-date version of the data that the replication group is managing. The site that receives the most number of votes becomes the master site, and all data write activity must occur there. + +In order to hold an election, the replication group must have a quorum. In order to achieve a quorum, a simple majority of the sites must be available to select the master. That is, *n/2 + 1* sites must be available, where *n* is the total number of replication group participants. By requiring a simple majority, the replication group avoids the possibility of simultaneously running with two master sites due to a network partition. + +If a replication group cannot select a master, then it can only be used in read-only mode. + +### Durability Guarantees + +Durability is a term that means data modifications have met some pre-defined set of guarantees that the modifications will remain persistent across application run times. Usually, this means that there is some assurance that the data modification has been written to stable storage (that is, written to a hard drive). + +For replicated BDB SQL applications, the durability guarantee is extended because data modifications are also replicated to those environments that are participating in the replication group. This ensures higher data durability than non-replicated applications by placing data in multiple environments that usually reside on separate physical machines. + +### Two-Site Replication Groups + +In a replication group that consists of exactly two sites, both sites must be available in order to achieve a quorum. Without a quorum, a new master site cannot be elected. This means that if the master site is unable to participate in the replication group, then the remaining read-only replica cannot become the master site. + +In other words, if you have a group that consists of exactly two sites, if you lose your master site then the replication group must exist in read-only mode until the master site becomes available again. diff --git a/docs-src/guides/bdb-sql/sync.md b/docs-src/guides/bdb-sql/sync.md new file mode 100644 index 000000000..d840d10ee --- /dev/null +++ b/docs-src/guides/bdb-sql/sync.md @@ -0,0 +1,46 @@ +--- +title: "Syncing with Oracle Databases" +api-name: "Syncing with Oracle Databases" +source: docs/bdb-sql/sync.html +--- +## Syncing with Oracle Databases + + [Syncing on Unix Platforms](sync.md#syncunix) + + [Syncing on Windows Platforms](sync.md#syncwin) + + [Syncing on Windows Mobile Platforms](sync.md#syncwinmobile) + +Oracle's SQLite Mobile Client product allows you to synchronize a SQLite database with a back-end Oracle database. Because the BDB SQL interface is a drop-in replacement for SQLite, this means you can synchronize a Berkeley DB database with an Oracle back-end as well. + +### Note + +Berkeley DB SQL databases are not compatible with SQLite databases. In order for sync to work, you must remove any currently existing SQLite databases. + +### Syncing on Unix Platforms + +For Unix platforms, the easiest way to use Oracle's SQLite Mobile Client is to build the BDB SQL interface with the compatibility option. That is, specify both `--enable-sql` and `--enable-sql-compat` when you configure your Berkeley DB installation. This causes libraries with the exact same name as the SQLite libraries to be created when you build Berkeley DB. + +Having done that, you must then change your platform's library search path so that it finds the Berkeley DB libraries *before* any installed SQLite libraries. On many (but not all) Unix platforms, you do this by modifying the `LD_LIBRARY_PATH` environment variable. See your operating system documentation for information on how to change your search path for dynamically linked libraries. + +Once you have properly configured and built your Berkeley DB installation, and you have properly configured your operating system, you can use the Oracle SQLite Mobile Client in exactly the same way as you would if you were using standard SQLite libraries and databases with it. See the Oracle Database Lite documentation for information on using SQLite Mobile Client. + +For information on building the BDB SQL interface, see the Configuring the SQL Interface section in the *Berkeley DB Installation and Build Guide*. + +### Syncing on Windows Platforms + +For Windows platforms, you use Oracle's SQLite Mobile Client by building the BDB SQL interface in the same way as you normally do. See the Building Berkeley DB for Windows chapter in the *Berkeley DB Installation and Build Guide* for more information. + +Once you have built the product, rename the Berkeley DB SQL dlls so that they are named identically to the standard SQLite dlls (sqlite3.dll). Install the renamed Berkeley DB SQL dll along with the main Berkeley DB dll (libdb5x.dll) in the same directory as the SQLite dlls. See the Building the SQL API section for details. + +Finally, configure your Windows PATH environment variable so that it finds your Berkeley DB dlls before it finds any standard SQLite dlls that might be installed on your system. + +Once you have built your Berkeley DB installation and renamed your dlls, and you have properly configured your operating system, you can use the Oracle SQLite Mobile Client in exactly the same way as you would if you were using standard SQLite libraries and databases with it. See the Oracle Database Lite documentation for information on using SQLite Mobile Client. + +### Syncing on Windows Mobile Platforms + +For Windows Mobile platforms, you use Oracle's SQLite Mobile Client by building the BDB SQL interface in the same way as you normally do. See the Building Berkeley DB for Windows Mobile chapter in the *Berkeley DB Installation and Build Guide* for more information. + +Once you have built the product, rename the Berkeley DB SQL dll to `sqlite3.dll`. Then, copy the dll to the `\Windows` path on the phone. Note that you only need the new `sqlite3.dll`; you do not need any of the other Berkeley DB dlls. + +Once you have built your Berkeley DB installation and renamed your dlls, and you have properly configured your operating system, you can use the Oracle SQLite Mobile Client in exactly the same way as you would if you were using standard SQLite libraries and databases with it. See the Oracle Database Lite documentation for information on using SQLite Mobile Client. diff --git a/docs-src/guides/bdb-sql/unsupportedpragmas.md b/docs-src/guides/bdb-sql/unsupportedpragmas.md new file mode 100644 index 000000000..0f18bf6f0 --- /dev/null +++ b/docs-src/guides/bdb-sql/unsupportedpragmas.md @@ -0,0 +1,15 @@ +--- +title: "Unsupported PRAGMAs" +api-name: "Unsupported PRAGMAs" +source: docs/bdb-sql/unsupportedpragmas.html +--- +## Unsupported PRAGMAs + +The following PRAGMAs are not supported by the BDB SQL interface. + +| | +|----| +| PRAGMA journal_mode | +| PRAGMA legacy_file_format | + +Also, PRAGMA fullfsync is always on for the BDB SQL interface. (This is an issue only for Mac OS X platforms.) diff --git a/docs-src/guides/collections/BasicProgram.md b/docs-src/guides/collections/BasicProgram.md new file mode 100644 index 000000000..d81cd7de2 --- /dev/null +++ b/docs-src/guides/collections/BasicProgram.md @@ -0,0 +1,282 @@ +--- +title: "Chapter 2.  The Basic Program" +api-name: "Chapter 2.  The Basic Program" +source: docs/collections/tutorial/BasicProgram.html +--- +## Chapter 2.  The Basic Program + +**Table of Contents** + + [Defining Serialized Key and Value Classes](BasicProgram.md#keyandvalueclasses) + + [Opening and Closing the Database Environment](opendbenvironment.md) + + [Opening and Closing the Class Catalog](openclasscatalog.md) + + [Opening and Closing Databases](opendatabases.md) + + [Creating Bindings and Collections](createbindingscollections.md) + + [Implementing the Main Program](implementingmain.md) + + [Using Transactions](usingtransactions.md) + + [Adding Database Items](addingdatabaseitems.md) + + [Retrieving Database Items](retrievingdatabaseitems.md) + + [Handling Exceptions](handlingexceptions.md) + +The Basic example is a minimal implementation of the shipment program. It writes and reads the part, supplier and shipment databases. + +The complete source of the final version of the example program is included in the Berkeley DB distribution. + +## Defining Serialized Key and Value Classes + +The key and value classes for each type of shipment record — Parts, Suppliers and Shipments — are defined as ordinary Java classes. In this example the serialized form of the key and value objects is stored directly in the database. Therefore these classes must implement the standard Java java.io.Serializable interface. A compact form of Java serialization is used that does not duplicate the class description in each record. Instead the class descriptions are stored in the class catalog store, which is described in the next section. But in all other respects, standard Java serialization is used. + +An important point is that instances of these classes are passed and returned by value, not by reference, when they are stored and retrieved from the database. This means that changing a key or value object does not automatically change the database. The object must be explicitly stored in the database after changing it. To emphasize this point the key and value classes defined here have no field setter methods. Setter methods can be defined, but it is important to remember that calling a setter method will not cause the change to be stored in the database. How to store and retrieve objects in the database will be described later. + +Each key and value class contains a toString method that is used to output the contents of the object in the example program. This is meant for illustration only and is not required for database objects in general. + +Notice that the key and value classes defined below do not contain any references to `com.sleepycat` packages. An important characteristic of these classes is that they are independent of the database. Therefore, they may be easily used in other contexts and may be defined in a way that is compatible with other tools and libraries. + +The `PartKey` class contains only the Part's Number field. + +Note that `PartKey` (as well as `SupplierKey` below) contain only a single String field. Instead of defining a specific class for each type of key, the String class by itself could have been used. Specific key classes were used to illustrate strong typing and for consistency in the example. The use of a plain String as an index key is illustrated in the next example program. It is up to the developer to use either primitive Java classes such as String and Integer, or strongly typed classes. When there is the possibility that fields will be added later to a key or value, a specific class should be used. + +``` c +import java.io.Serializable; + +public class PartKey implements Serializable +{ + private String number; + + public PartKey(String number) { + this.number = number; + } + + public final String getNumber() { + return number; + } + + public String toString() { + return "[PartKey: number=" + number + ']'; + } +} +``` + +The `PartData` class contains the Part's Name, Color, Weight and City fields. + +``` c +import java.io.Serializable; + +public class PartData implements Serializable +{ + private String name; + private String color; + private Weight weight; + private String city; + + public PartData(String name, String color, Weight weight, String city) + { + this.name = name; + this.color = color; + this.weight = weight; + this.city = city; + } + + public final String getName() + { + return name; + } + + public final String getColor() + { + return color; + } + + public final Weight getWeight() + { + return weight; + } + + public final String getCity() + { + return city; + } + + public String toString() + { + return "[PartData: name=" + name + + " color=" + color + + " weight=" + weight + + " city=" + city + ']'; + } +} +``` + +The `Weight` class is also defined here, and is used as the type of the Part's Weight field. Just as in standard Java serialization, nothing special is needed to store nested objects as long as they are all Serializable. + +``` c +import java.io.Serializable; + +public class Weight implements Serializable +{ + public final static String GRAMS = "grams"; + public final static String OUNCES = "ounces"; + + private double amount; + private String units; + + public Weight(double amount, String units) + { + this.amount = amount; + this.units = units; + } + + public final double getAmount() + { + return amount; + } + + public final String getUnits() + { + return units; + } + + public String toString() + { + return "[" + amount + ' ' + units + ']'; + } +} +``` + +The `SupplierKey` class contains the Supplier's Number field. + +``` c +import java.io.Serializable; + +public class SupplierKey implements Serializable +{ + private String number; + + public SupplierKey(String number) + { + this.number = number; + } + + public final String getNumber() + { + return number; + } + + public String toString() + { + return "[SupplierKey: number=" + number + ']'; + } +} +``` + +The `SupplierData` class contains the Supplier's Name, Status and City fields. + +``` c +import java.io.Serializable; + +public class SupplierData implements Serializable +{ + private String name; + private int status; + private String city; + + public SupplierData(String name, int status, String city) + { + this.name = name; + this.status = status; + this.city = city; + } + + public final String getName() + { + return name; + } + + public final int getStatus() + { + return status; + } + + public final String getCity() + { + return city; + } + + public String toString() + { + return "[SupplierData: name=" + name + + " status=" + status + + " city=" + city + ']'; + } +} + +``` + +The `ShipmentKey` class contains the keys of both the Part and Supplier. + +``` c +import java.io.Serializable; + +public class ShipmentKey implements Serializable +{ + private String partNumber; + private String supplierNumber; + + public ShipmentKey(String partNumber, String supplierNumber) + { + this.partNumber = partNumber; + this.supplierNumber = supplierNumber; + } + + public final String getPartNumber() + { + return partNumber; + } + + public final String getSupplierNumber() + { + return supplierNumber; + } + + public String toString() + { + return "[ShipmentKey: supplier=" + supplierNumber + + " part=" + partNumber + ']'; + } +} +``` + +The `ShipmentData` class contains only the Shipment's Quantity field. Like `PartKey` and `SupplierKey`, `ShipmentData` contains only a single primitive field. Therefore the Integer class could have been used instead of defining a specific value class. + +``` c +import java.io.Serializable; + +public class ShipmentData implements Serializable +{ + private int quantity; + + public ShipmentData(int quantity) + { + this.quantity = quantity; + } + + public final int getQuantity() + { + return quantity; + } + + public String toString() + { + return "[ShipmentData: quantity=" + quantity + ']'; + } +} +``` diff --git a/docs-src/guides/collections/Entity.md b/docs-src/guides/collections/Entity.md new file mode 100644 index 000000000..04429070f --- /dev/null +++ b/docs-src/guides/collections/Entity.md @@ -0,0 +1,185 @@ +--- +title: "Chapter 4.  Using Entity Classes" +api-name: "Chapter 4.  Using Entity Classes" +source: docs/collections/tutorial/Entity.html +--- +## Chapter 4.  Using Entity Classes + +**Table of Contents** + + [Defining Entity Classes](Entity.md#definingentityclasses) + + [Creating Entity Bindings](creatingentitybindings.md) + + [Creating Collections with Entity Bindings](collectionswithentities.md) + + [Using Entities with Collections](entitieswithcollections.md) + +In the prior examples, the keys and values of each store were represented using separate classes. For example, a `PartKey` and a `PartData` class were used. Many times it is desirable to have a single class representing both the key and the value, for example, a `Part` class. + +Such a combined key and value class is called an *entity class* and is used along with an *entity binding*. Entity bindings combine a key and a value into an entity when reading a record from a collection, and split an entity into a key and a value when writing a record to a collection. Entity bindings are used in place of value bindings, and entity objects are used with collections in place of value objects. + +Some reasons for using entities are: + +- When the key is a property of an entity object representing the record as a whole, the object's identity and concept are often clearer than with key and value objects that are disjoint. + +- A single entity object per record is often more convenient to use than two objects. + +Of course, instead of using an entity binding, you could simply create the entity yourself after reading the key and value from a collection, and split the entity into a key and value yourself before writing it to a collection. But this would detract from the convenience of the using the Java collections API. It is convenient to obtain a `Part` object directly from Map.get and to add a `Part` object using Set.add. Collections having entity bindings can be used naturally without combining and splitting objects each time a collection method is called; however, an entity binding class must be defined by the application. + +In addition to showing how to use entity bindings, this example illustrates a key feature of all bindings: Bindings are independent of database storage parameters and formats. Compare this example to the prior Index example and you'll see that the `Sample` and `SampleViews` classes have been changed to use entity bindings, but the `SampleDatabase` class was not changed at all. In fact, the Entity program and the Index program can be used interchangeably to access the same physical database files. This demonstrates that bindings are only a "view" onto the physical stored data. + +`Warning:` When using multiple bindings for the same database, it is the application's responsibility to ensure that the same format is used for all bindings. For example, a serial binding and a tuple binding cannot be used to access the same records. + +The complete source of the final version of the example program is included in the Berkeley DB distribution. + +## Defining Entity Classes + +As described in the prior section, *entity classes* are combined key/value classes that are managed by entity bindings. In this example the `Part`, `Supplier` and `Shipment` classes are entity classes. These classes contain fields that are a union of the fields of the key and value classes that were defined earlier for each store. + +In general, entity classes may be defined in any way desired by the application. The entity binding, which is also defined by the application, is responsible for mapping between key/value objects and entity objects. + +The `Part`, `Supplier` and `Shipment` entity classes are defined below. + +An important difference between the entity classes defined here and the key and value classes defined earlier is that the entity classes are not serializable (do not implement the Serializable interface). This is because the entity classes are not directly stored. The entity binding decomposes an entity object into key and value objects, and only the key and value objects are serialized for storage. + +One advantage of using entities can already be seen in the `toString()` method of the classes below. These return debugging output for the combined key and value, and will be used later to create a listing of the database that is more readable than in the prior examples. + +``` c +public class Part +{ + private String number; + private String name; + private String color; + private Weight weight; + private String city; + + public Part(String number, String name, String color, Weight weight, + String city) + { + this.number = number; + this.name = name; + this.color = color; + this.weight = weight; + this.city = city; + } + + public final String getNumber() + { + return number; + } + + public final String getName() + { + return name; + } + + public final String getColor() + { + return color; + } + + public final Weight getWeight() + { + return weight; + } + + public final String getCity() + { + return city; + } + + public String toString() + { + return "Part: number=" + number + + " name=" + name + + " color=" + color + + " weight=" + weight + + " city=" + city + '.'; + } +} +``` + +``` c +public class Supplier +{ + private String number; + private String name; + private int status; + private String city; + + public Supplier(String number, String name, int status, String city) + { + this.number = number; + this.name = name; + this.status = status; + this.city = city; + } + + public final String getNumber() + { + return number; + } + + public final String getName() + { + return name; + } + + public final int getStatus() + { + return status; + } + + public final String getCity() + { + return city; + } + + public String toString() + { + return "Supplier: number=" + number + + " name=" + name + + " status=" + status + + " city=" + city + '.'; + } +} +``` + +``` c +public class Shipment +{ + private String partNumber; + private String supplierNumber; + private int quantity; + + public Shipment(String partNumber, String supplierNumber, int quantity) + { + this.partNumber = partNumber; + this.supplierNumber = supplierNumber; + this.quantity = quantity; + } + + public final String getPartNumber() + { + return partNumber; + } + + public final String getSupplierNumber() + { + return supplierNumber; + } + + public final int getQuantity() + { + return quantity; + } + + public String toString() + { + return "Shipment: part=" + partNumber + + " supplier=" + supplierNumber + + " quantity=" + quantity + '.'; + } +} +``` diff --git a/docs-src/guides/collections/SerializableEntity.md b/docs-src/guides/collections/SerializableEntity.md new file mode 100644 index 000000000..f3287fc0a --- /dev/null +++ b/docs-src/guides/collections/SerializableEntity.md @@ -0,0 +1,194 @@ +--- +title: "Chapter 6.  Using Serializable Entities" +api-name: "Chapter 6.  Using Serializable Entities" +source: docs/collections/tutorial/SerializableEntity.html +--- +## Chapter 6.  Using Serializable Entities + +**Table of Contents** + + [Using Transient Fields in an Entity Class](SerializableEntity.md#transientfieldsinclass) + + [Using Transient Fields in an Entity Binding](transientfieldsinbinding.md) + + [Removing the Redundant Value Classes](removingredundantvalueclasses.md) + +In the prior examples that used entities (the Entity and Tuple examples) you may have noticed the redundancy between the serializable value classes and the entity classes. An entity class by definition contains all properties of the value class as well as all properties of the key class. + +When using serializable values it is possible to remove this redundancy by changing the entity class in two ways: + +- Make the entity class serializable, so it can be used in place of the value class. + +- Make the key fields transient, so they are not redundantly stored in the record. + +The modified entity class can then serve double-duty: It can be serialized and stored as the record value, and it can be used as the entity class as usual along with the Java collections API. The `PartData`, `SupplierData` and `ShipmentData` classes can then be removed. + +Transient fields are defined in Java as fields that are not stored in the serialized form of an object. Therefore, when an object is deserialized the transient fields must be explicitly initialized. Since the entity binding is responsible for creating entity objects, it is the natural place to initialize the transient key fields. + +Note that it is not strictly necessary to make the key fields of a serializable entity class transient. If this is not done, the key will simply be stored redundantly in the record's value. This extra storage may or may not be acceptable to an application. But since we are using tuple keys and an entity binding class must be implemented anyway to extract the key from the entity, it is sensible to use transient key fields to reduce the record size. Of course there may be a reason that transient fields are not desired; for example, if an application wants to serialize the entity objects for other purposes, then using transient fields should be avoided. + +The complete source of the final version of the example program is included in the Berkeley DB distribution. + +## Using Transient Fields in an Entity Class + +The entity classes in this example are redefined such that they can be used both as serializable value classes and as entity classes. Compared to the prior example there are three changes to the `Part`, `Supplier` and `Shipment` entity classes: + +- Each class now implements the `Serializable` interface. + +- The key fields in each class are declared as `transient`. + +- A package-private `setKey()` method is added to each class for initializing the transient key fields. This method will be called from the entity bindings. + +``` c +import java.io.Serializable; +... +public class Part implements Serializable +{ + private transient String number; + private String name; + private String color; + private Weight weight; + private String city; + + public Part(String number, String name, String color, Weight weight, + String city) + { + this.number = number; + this.name = name; + this.color = color; + this.weight = weight; + this.city = city; + } + + final void setKey(String number) + { + this.number = number; + } + + public final String getNumber() + { + return number; + } + + public final String getName() + { + return name; + } + + public final String getColor() + { + return color; + } + + public final Weight getWeight() + { + return weight; + } + + public final String getCity() + { + return city; + } + + public String toString() + { + return "Part: number=" + number + + " name=" + name + + " color=" + color + + " weight=" + weight + + " city=" + city + '.'; + } +} +... +public class Supplier implements Serializable +{ + private transient String number; + private String name; + private int status; + private String city; + + public Supplier(String number, String name, int status, String city) + { + this.number = number; + this.name = name; + this.status = status; + this.city = city; + } + + void setKey(String number) + { + this.number = number; + } + + public final String getNumber() + { + return number; + } + + public final String getName() + { + return name; + } + + public final int getStatus() + { + return status; + } + + public final String getCity() + { + return city; + } + + public String toString() + { + return "Supplier: number=" + number + + " name=" + name + + " status=" + status + + " city=" + city + '.'; + } +} +... +public class Shipment implements Serializable +{ + private transient String partNumber; + private transient String supplierNumber; + private int quantity; + + public Shipment(String partNumber, String supplierNumber, int quantity) + { + this.partNumber = partNumber; + this.supplierNumber = supplierNumber; + this.quantity = quantity; + } + + void setKey(String partNumber, String supplierNumber) + { + this.partNumber = partNumber; + this.supplierNumber = supplierNumber; + } + + public final String getPartNumber() + { + return partNumber; + } + + public final String getSupplierNumber() + { + return supplierNumber; + } + + public final int getQuantity() + { + return quantity; + } + + public String toString() + { + return "Shipment: part=" + partNumber + + " supplier=" + supplierNumber + + " quantity=" + quantity + '.'; + } +} + +``` diff --git a/docs-src/guides/collections/SerializedObjectStorage.md b/docs-src/guides/collections/SerializedObjectStorage.md new file mode 100644 index 000000000..7c9b1e809 --- /dev/null +++ b/docs-src/guides/collections/SerializedObjectStorage.md @@ -0,0 +1,8 @@ +--- +title: "Serialized Object Storage" +api-name: "Serialized Object Storage" +source: docs/collections/tutorial/SerializedObjectStorage.html +--- +## Serialized Object Storage + +Serialization of an object graph includes class information as well as instance information. If more than one instance of the same class is serialized as separate serialization operations then the class information exists more than once. To eliminate this inefficiency the StoredClassCatalog class will store the class format for all database records stored using a SerialBinding. Refer to the `ship` sample code for examples (the class `SampleDatabase` in `examples_java/src/com/sleepycat/examples/collections/ship/basic/SampleDatabase.java` is a good place to start). diff --git a/docs-src/guides/collections/Summary.md b/docs-src/guides/collections/Summary.md new file mode 100644 index 000000000..e8dd973df --- /dev/null +++ b/docs-src/guides/collections/Summary.md @@ -0,0 +1,22 @@ +--- +title: "Chapter 7.  Summary" +api-name: "Chapter 7.  Summary" +source: docs/collections/tutorial/Summary.html +--- +## Chapter 7.  Summary + +In summary, the DB Java Collections API tutorial has demonstrated how to create different types of bindings, as well as how to use the basic facilities of the DB Java Collections API: the environment, databases, secondary indices, collections, and transactions. The final approach illustrated by the last example program, Serializable Entity, uses tuple keys and serial entity values. Hopefully it is clear that any type of object-to-data binding may be implemented by an application and used along with standard Java collections. + +The following table summarizes the differences between the examples in the tutorial. + +| Example | Key | Value | Entity | Comments | +|----|----|----|----|----| +| The Basic Program | Serial | Serial | No | The shipment program | +| Using Secondary Indices | Serial | Serial | No | Secondary indices | +| Using Entity Classes | Serial | Serial | Yes | Combining the key and value in a single object | +| Using Tuples | Tuple | Serial | Yes | Compact ordered keys | +| Using Serializable Entities | Tuple | Serial | Yes | One serializable class for entities and values | + +Having completed this tutorial, you may want to explore how other types of bindings can be implemented. The bindings shown in this tutorial are all *external bindings*, meaning that the data classes themselves contain none of the binding implementation. It is also possible to implement *internal bindings*, where the data classes implement the binding. + +Internal bindings are called *marshalled bindings* in the DB Java Collections API, and in this model each data class implements a marshalling interface. A single external binding class that understands the marshalling interface is used to call the internal bindings of each data object, and therefore the overall model and API is unchanged. To learn about marshalled bindings, see the `marshal` and `factory` examples that came with your DB distribution (you can find them in `/examples_java/src/com/sleepycat/examples/collections/ship` where `` is the location where you unpacked your DB distribution). These examples continue building on the example programs used in the tutorial. The Marshal program is the next program following the Serializable Entity program, and the Factory program follows the Marshal program. The source code comments in these examples explain their differences. diff --git a/docs-src/guides/collections/Tuple.md b/docs-src/guides/collections/Tuple.md new file mode 100644 index 000000000..2d5688ba3 --- /dev/null +++ b/docs-src/guides/collections/Tuple.md @@ -0,0 +1,54 @@ +--- +title: "Chapter 5.  Using Tuples" +api-name: "Chapter 5.  Using Tuples" +source: docs/collections/tutorial/Tuple.html +--- +## Chapter 5.  Using Tuples + +**Table of Contents** + + [Using the Tuple Format](Tuple.md#tupleformat) + + [Using Tuples with Key Creators](tupleswithkeycreators.md) + + [Creating Tuple Key Bindings](tuplekeybindings.md) + + [Creating Tuple-Serial Entity Bindings](tuple-serialentitybindings.md) + + [Using Sorted Collections](sortedcollections.md) + +DB Java Collections API *tuples* are sequences of primitive Java data types, for example, integers and strings. The *tuple format* is a binary format for tuples that can be used to store keys and/or values. + +Tuples are useful as keys because they have a meaningful sort order, while serialized objects do not. This is because the binary data for a tuple is written in such a way that its raw byte ordering provides a useful sort order. For example, strings in tuples are written with a null terminator rather than with a leading length. + +Tuples are useful as keys *or* values when reducing the record size to a minimum is important. A tuple is significantly smaller than an equivalent serialized object. However, unlike serialized objects, tuples cannot contain complex data types and are not easily extended except by adding fields at the end of the tuple. + +Whenever a tuple format is used, except when the key or value class is a Java primitive wrapper class, a *tuple binding* class must be implemented to map between the Java object and the tuple fields. Because of this extra requirement, and because tuples are not easily extended, a useful technique shown in this example is to use tuples for keys and serialized objects for values. This provides compact ordered keys but still allows arbitrary Java objects as values, and avoids implementing a tuple binding for each value class. + +Compare this example to the prior Entity example and you'll see that the `Sample` class has not changed. When changing a database format, while new bindings are needed to map key and value objects to the new format, the application using the objects often does not need to be modified. + +The complete source of the final version of the example program is included in the Berkeley DB distribution. + +## Using the Tuple Format + +Tuples are sequences of primitive Java values that can be written to, and read from, the raw data bytes of a stored record. The primitive values are written or read one at a time in sequence, using the DB Java Collections API TupleInput and TupleOutput classes. These classes are very similar to the standard Java DataInput and DataOutput interfaces. The primary difference is the binary format of the data, which is designed for sorting in the case of tuples. + +For example, to read and write a tuple containing two string values, the following code snippets could be used. + +``` c +import com.sleepycat.bind.tuple.TupleInput; +import com.sleepycat.bind.tuple.TupleOutput; +... +TupleInput input; +TupleOutput output; +... +String partNumber = input.readString(); +String supplierNumber = input.readString(); +... +output.writeString(partNumber); +output.writeString(supplierNumber); +``` + +Since a tuple is defined as an ordered sequence, reading and writing order must match. If the wrong data type is read (an integer instead of string, for example), an exception may be thrown or at minimum invalid data will be read. + +When the tuple format is used, bindings and key creators must read and write tuples using the tuple API as shown above. This will be illustrated in the next two sections. diff --git a/docs-src/guides/collections/UsingCollectionsAPI.md b/docs-src/guides/collections/UsingCollectionsAPI.md new file mode 100644 index 000000000..87b13b16b --- /dev/null +++ b/docs-src/guides/collections/UsingCollectionsAPI.md @@ -0,0 +1,95 @@ +--- +title: "Using the DB Java Collections API" +api-name: "Using the DB Java Collections API" +source: docs/collections/tutorial/UsingCollectionsAPI.html +--- +## Using the DB Java Collections API + + [Using Transactions](UsingCollectionsAPI.md#UsingTransactions) + + [Transaction Rollback](UsingCollectionsAPI.md#TransactionRollback) + + [Selecting Access Methods](UsingCollectionsAPI.md#SelectingAccessMethods) + + [Access Method Restrictions](UsingCollectionsAPI.md#AccessMethodRestrictions) + +An Environment manages the resources for one or more data stores. A Database object represents a single database and is created via a method on the environment object. SecondaryDatabase objects represent an index associated with a primary database. An access method must be chosen for each database and secondary database. Primary and secondary databases are then used to create stored collection objects, as described in Using Stored Collections . + +### Using Transactions + +Once you have an environment, one or more databases, and one or more stored collections, you are ready to access (read and write) stored data. For a transactional environment, a transaction must be started before accessing data, and must be committed or aborted after access is complete. The DB Java Collections API provides several ways of managing transactions. + +The recommended technique is to use the TransactionRunner class along with your own implementation of the TransactionWorker interface. TransactionRunner will call your TransactionWorker implementation class to perform the data access or work of the transaction. This technique has the following benefits: + +- Transaction exceptions will be handled transparently and retries will be performed when deadlocks are detected. + +- The transaction will automatically be committed if your TransactionWorker.doWork() method returns normally, or will be aborted if `doWork()` throws an exception. + +- `TransactionRunner` can be used for non-transactional environments as well, allowing you to write your application independently of the environment. + +If you don't want to use TransactionRunner, the alternative is to use the CurrentTransaction class. + +1. Obtain a CurrentTransaction instance by calling the CurrentTransaction.getInstance method. The instance returned can be used by all threads in a program. + +2. Use CurrentTransaction.beginTransaction(), CurrentTransaction.commitTransaction() and CurrentTransaction.abortTransaction() to directly begin, commit and abort transactions. + +If you choose to use CurrentTransaction directly you must handle the DeadlockException exception and perform retries yourself. Also note that CurrentTransaction may only be used in a transactional environment. + +The DB Java Collections API supports nested transactions. If TransactionRunner.run(com.sleepycat.collections.TransactionWorker) or CurrentTransaction.beginTransaction() , is called while another transaction is active, a child transaction is created. When TransactionRunner.run(com.sleepycat.collections.TransactionWorker) returns, or when CurrentTransaction.commitTransaction() or CurrentTransaction.abortTransaction() is called, the parent transaction becomes active again. Note that because only one transaction is active per-thread, it is impossible to accidentally use a parent transaction while a child transaction is active. + +The DB Java Collections API supports transaction auto-commit. If no transaction is active and a write operation is requested for a transactional database, auto-commit is used automatically. + +The DB Java Collections API also supports transaction dirty-read via the StoredCollections class. When dirty-read is enabled for a collection, data will be read that has been modified by another transaction but not committed. Using dirty-read can improve concurrency since reading will not wait for other transactions to complete. For a non-transactional container, dirty-read has no effect. See StoredCollections for how to create a dirty-read collection. + +### Transaction Rollback + +When a transaction is aborted (or rolled back) the application is responsible for discarding references to any data objects that were modified during the transaction. Since the DB Java Collections API treats data by value, not by reference, neither the data objects nor the DB Java Collections API objects contain status information indicating whether the data objects are 1- in sync with the database, 2- dirty (contain changes that have not been written to the database), 3- stale (were read previously but have become out of sync with changes made to the database), or 4- contain changes that cannot be committed because of an aborted transaction. + +For example, a given data object will reflect the current state of the database after reading it within a transaction. If the object is then modified it will be out of sync with the database. When the modified object is written to the database it will then be in sync again. But if the transaction is aborted the object will then be out of sync with the database. References to objects for aborted transactions should no longer be used. When these objects are needed later they should be read fresh from the database. + +When an existing stored object is to be updated, special care should be taken to read the data, then modify it, and then write it to the database, all within a single transaction. If a stale data object (an object that was read previously but has since been changed in the database) is modified and then written to the database, database changes may be overwritten unintentionally. + +When an application enforces rules about concurrent access to specific data objects or all data objects, the rules described here can be relaxed. For example, if the application knows that a certain object is only modified in one place, it may be able to reliably keep a current copy of that object. In that case, it is not necessary to reread the object before updating it. That said, if arbitrary concurrent access is to be supported, the safest approach is to always read data before modifying it within a single transaction. + +Similar concerns apply to using data that may have become stale. If the application depends on current data, it should be read fresh from the database just before it is used. + +### Selecting Access Methods + +For each data store and secondary index, you must choose from one of the access methods in the table below. The access method determines not only whether sorted keys or duplicate keys are supported, but also what types of collection views may be used and what restrictions are imposed on the collection views. + +| Access Method | Ordered | Duplicates | Record Numbers | Database Type | `DatabaseConfig` Method | +|----|----|----|----|----|----| +| BTREE-UNIQUE | Yes | No | No | BTREE | None | +| BTREE-DUP | Yes | Yes, Unsorted | No | BTREE | setUnsortedDuplicates | +| BTREE-DUPSORT | Yes | Yes, Sorted | No | BTREE | setSortedDuplicates | +| BTREE-RECNUM | Yes | No | Yes, Renumbered | BTREE | setBtreeRecordNumbers | +| HASH-UNIQUE | No | No | No | HASH | None | +| HASH-DUP | No | Yes, Unsorted | No | HASH | setUnsortedDuplicates | +| HASH-DUPSORT | No | Yes, Sorted | No | HASH | setSortedDuplicates | +| QUEUE | Yes | No | Yes, Fixed | QUEUE | None | +| RECNO | Yes | No | Yes, Fixed | RECNO | None | +| RECNO-RENUMBER | Yes | No | Yes, Renumbered | RECNO | setRenumbering | + +Please see Available Access Methods in the *Berkeley DB Programmer's Reference Guide* for more information on access method configuration. + +### Access Method Restrictions + +The restrictions imposed by the access method on the database model are: + +- If keys are ordered then data may be enumerated in key order and key ranges may be used to form subsets of a data store. The `SortedMap` and `SortedSet` interfaces are supported for collections with ordered keys. + +- If duplicates are allowed then more than one value may be associated with the same key. This means that the data store cannot be strictly considered a map — it is really a multi-map. See Using Stored Collections for implications on the use of the collection interfaces. + +- If duplicate keys are allowed for a data store then the data store may not have secondary indices. + +- For secondary indices with duplicates, the duplicates must be sorted. This restriction is imposed by the DB Java Collections API. + +- With sorted duplicates, all values for the same key must be distinct. + +- If duplicates are unsorted, then values for the same key must be distinct. + +- If record number keys are used, the the number of records is limited to the maximum value of an unsigned 32-bit integer. + +- If record number keys are renumbered, then standard List add/remove behavior is supported but concurrency/performance is reduced. + +See Using Stored Collections for more information on how access methods impact the use of stored collections. diff --git a/docs-src/guides/collections/UsingSecondaries.md b/docs-src/guides/collections/UsingSecondaries.md new file mode 100644 index 000000000..eb603bed8 --- /dev/null +++ b/docs-src/guides/collections/UsingSecondaries.md @@ -0,0 +1,169 @@ +--- +title: "Chapter 3.  Using Secondary Indices" +api-name: "Chapter 3.  Using Secondary Indices" +source: docs/collections/tutorial/UsingSecondaries.html +--- +## Chapter 3.  Using Secondary Indices + +**Table of Contents** + + [Opening Secondary Key Indices](UsingSecondaries.md#opensecondaryindices) + + [More Secondary Key Indices](openingforeignkeys.md) + + [Creating Indexed Collections](indexedcollections.md) + + [Retrieving Items by Index Key](retrievingbyindexkey.md) + +In the Basic example, each store has a single *primary key*. The Index example extends the Basic example to add the use of *secondary keys*. + +The complete source of the final version of the example program is included in the Berkeley DB distribution. + +## Opening Secondary Key Indices + +*Secondary indices* or *secondary databases* are used to access a primary database by a key other than the primary key. Recall that the Supplier Number field is the primary key of the Supplier database. In this section, the Supplier City field will be used as a secondary lookup key. Given a city value, we would like to be able to find the Suppliers in that city. Note that more than one Supplier may be in the same city. + +Both primary and secondary databases contain key-value records. The key of an index record is the secondary key, and its value is the key of the associated record in the primary database. When lookups by secondary key are performed, the associated record in the primary database is transparently retrieved by its primary key and returned to the caller. + +Secondary indices are maintained automatically when index key fields (the City field in this case) are added, modified or removed in the records of the primary database. However, the application must implement a SecondaryKeyCreator that extracts the index key from the database record. + +It is useful to contrast opening an secondary index with opening a primary database (as described earlier in Opening and Closing Databases . + +- A primary database may be associated with one or more secondary indices. A secondary index is always associated with exactly one primary database. + +- For a secondary index, a SecondaryKeyCreator must be implemented by the application to extract the index key from the record of its associated primary database. + +- A primary database is represented by a Database object and a secondary index is represented by a SecondaryDatabase object. The SecondaryDatabase class extends the Database class. + +- When a SecondaryDatabase is created it is associated with a primary Database object and a SecondaryKeyCreator. + +The `SampleDatabase` class is extended to open the Supplier-by-City secondary key index. + +``` c +import com.sleepycat.bind.serial.SerialSerialKeyCreator; +import com.sleepycat.db.SecondaryConfig; +import com.sleepycat.db.SecondaryDatabase; +... +public class SampleDatabase +{ + ... + private static final String SUPPLIER_CITY_INDEX = + "supplier_city_index"; + ... + private SecondaryDatabase supplierByCityDb; + ... + public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + ... + SecondaryConfig secConfig = new SecondaryConfig(); + secConfig.setTransactional(true); + secConfig.setAllowCreate(true); + secConfig.setType(DatabaseType.BTREE); + secConfig.setSortedDuplicates(true); + + secConfig.setKeyCreator( + new SupplierByCityKeyCreator(javaCatalog, + SupplierKey.class, + SupplierData.class, + String.class)); + + supplierByCityDb = env.openSecondaryDatabase(null, + SUPPLIER_CITY_INDEX, + null, + supplierDb, + secConfig); + ... + } +} +``` + +A SecondaryConfig object is used to configure the secondary database. The SecondaryConfig class extends the DatabaseConfig class, and most steps for configuring a secondary database are the same as for configuring a primary database. The main difference in the example above is that the `SecondaryConfig.setSortedDuplicates()` method is called to allow duplicate index keys. This is how more than one Supplier may be in the same City. If this property is not specified, the default is that the index keys of all records must be unique. + +For a primary database, duplicate keys are not normally used since a primary database with duplicate keys may not have any associated secondary indices. If primary database keys are not unique, there is no way for a secondary key to reference a specific record in the primary database. + +Note that `setSortedDuplicates()` and not `setUnsortedDuplicates()` was called. Sorted duplicates are always used for indices rather than unsorted duplicates, since sorting enables optimized equality joins. + +Opening a secondary key index requires creating a SecondaryKeyCreator. The `SupplierByCityKeyCreator` class implements the SecondaryKeyCreator interface and will be defined below. + +The SecondaryDatabase object is opened last. If you compare the `openSecondaryDatabase()` and `openDatabase()` methods you'll notice only two differences: + +- `openSecondaryDatabase()` has an extra parameter for specifying the associated primary database. The primary database is `supplierDb` in this case. + +- The last parameter of `openSecondaryDatabase()` is a `SecondaryConfig` instead of a `DatabaseConfig`. + +How to use the secondary index to access records will be shown in a later section. + +The application-defined `SupplierByCityKeyCreator` class is shown below. It was used above to configure the secondary database. + +``` c +public class SampleDatabase +{ +... + private static class SupplierByCityKeyCreator + extends SerialSerialKeyCreator + { + private SupplierByCityKeyCreator(ClassCatalog catalog, + Class primaryKeyClass, + Class valueClass, + Class indexKeyClass) + { + super(catalog, primaryKeyClass, valueClass, indexKeyClass); + } + + public Object createSecondaryKey(Object primaryKeyInput, + Object valueInput) + { + SupplierData supplierData = (SupplierData) valueInput; + return supplierData.getCity(); + } + } +... +} +``` + +In general, a key creator class must implement the SecondaryKeyCreator interface. This interface has methods that operate on the record data as raw bytes. In practice, it is easiest to use an abstract base class that performs the conversion of record data to and from the format defined for the database's key and value. The base class implements the SecondaryKeyCreator interface and has abstract methods that must be implemented in turn by the application. + +In this example the SerialSerialKeyCreator base class is used because the database record uses the serial format for both its key and its value. The abstract methods of this class have key and value parameters of type Object which are automatically converted to and from the raw record data by the base class. + +To perform the conversions properly, the key creator must be aware of all three formats involved: the key format of the primary database record, the value format of the primary database record, and the key format of the index record. The SerialSerialKeyCreator constructor is given the base classes for these three formats as parameters. + +The `SerialSerialKeyCreator.createSecondaryKey` method is given the key and value of the primary database record as parameters, and it returns the key of the index record. In this example, the index key is a field in the primary database record value. Since the record value is known to be a `SupplierData` object, it is cast to that class and the city field is returned. + +Note that the `primaryKeyInput` parameter is not used in the example. This parameter is needed only when an index key is derived from the key of the primary database record. Normally an index key is derived only from the primary database record value, but it may be derived from the key, value or both. + +The following getter methods return the secondary database object for use by other classes in the example program. The secondary database object is used to create Java collections for accessing records via their secondary keys. + +``` c +public class SampleDatabase +{ + ... + public final SecondaryDatabase getSupplierByCityDatabase() + { + return supplierByCityDb; + } + ... +} +``` + +The following statement closes the secondary database. + +``` c +public class SampleDatabase +{ + ... + public void close() + throws DatabaseException { + + supplierByCityDb.close(); + partDb.close(); + supplierDb.close(); + shipmentDb.close(); + javaCatalog.close(); + env.close(); + } + ... +} +``` + +Secondary databases must be closed before closing their associated primary database. diff --git a/docs-src/guides/collections/UsingStoredCollections.md b/docs-src/guides/collections/UsingStoredCollections.md new file mode 100644 index 000000000..a258047b2 --- /dev/null +++ b/docs-src/guides/collections/UsingStoredCollections.md @@ -0,0 +1,138 @@ +--- +title: "Using Stored Collections" +api-name: "Using Stored Collections" +source: docs/collections/tutorial/UsingStoredCollections.html +--- +## Using Stored Collections + + [Stored Collection and Access Methods](UsingStoredCollections.md#StoredCollectionAccessMethods) + + [Stored Collections Versus Standard Java Collections](UsingStoredCollections.md#StoredVersusStandardCollections) + + [Other Stored Collection Characteristics](UsingStoredCollections.md#StoredCollectionCharacteristics) + + [Why Java Collections for Berkeley DB](UsingStoredCollections.md#WhyJavaCollections) + +When a stored collection is created it is based on either a Database or a SecondaryDatabase. When a database is used, the primary key of the database is used as the collection key. When a secondary database is used, the index key is used as the collection key. Indexed collections can be used for reading elements and removing elements but not for adding or updating elements. + +### Stored Collection and Access Methods + +The use of stored collections is constrained in certain respects as described below. Most of these restrictions have to do with List interfaces; for Map interfaces, most all access modes are fully supported since the Berkeley DB model is map-like. + +- SortedSet and SortedMap interfaces may only be used if keys are ordered. This means ordered keys are required for creating a StoredSortedEntrySet, StoredSortedKeySet, StoredSortedMap, or StoredSortedValueSet. + +- All iterators for stored collections implement the ListIterator interface as well as the Iterator interface. ListIterator.hasPrevious() and ListIterator.previous() work in all cases. However, the following ListIterator method behavior is dependent on the access method. + + - ListIterator.nextIndex() and ListIterator.previousIndex() only work when record number keys are used, and throw UnsupportedOperationException otherwise. + + - ListIterator.add() inserts before the current position and renumbers following keys if the RECNO-RENUMBER access method is used. + + - For all access methods other than RECNO-RENUMBER: + + - ListIterator.add() throws UnsupportedOperationException if duplicates are not allowed. + + - ListIterator.add() inserts a duplicate before the current position if duplicates are unsorted. + + - ListIterator.add() inserts a duplicate in sorted order if duplicates are sorted. + + - ListIterator.set() throws UnsupportedOperationException if sorted duplicates are configured, since updating with sorted duplicates would change the iterator position. + +- Map.Entry.setValue() throws UnsupportedOperationException if duplicates are sorted. + +- Only the access methods that use a record number key may be used with a List `List` view. + +- To create a stored List that supports the List.add() `List.add()` method, only the RECNO-RENUMBER access method may be used. + +- For List access methods that do not support List.add() `List.add()` (RECNO, QUEUE, and BTREE-RECNUM): + + - List.add() `List.add()` and ListIterator.add() `ListIterator.add()` always throw UnsupportedOperationException . + + - List.remove() `List.remove()` and ListIterator.remove() `ListIterator.remove()` do not cause list indices to be renumbered. However, iterators will skip the removed values. + + For these access methods, stored Lists are most useful as read-only collections where indices are not required to be sequential. + +- When duplicates are allowed the Collection interfaces are modified in several ways as described in the next section. + +### Stored Collections Versus Standard Java Collections + +Stored collections have the following differences with the standard Java collection interfaces. Some of these are interface contract violations. + +The Java collections interface does not support duplicate keys (multi-maps or multi-sets). When the access method allows duplicate keys, the collection interfaces are defined as follows. + +- Map.entrySet() may contain multiple Map.Entry objects with the same key. + +- Map.keySet() always contains unique keys, it does not contain duplicates. + +- Map.values() contains all values including the values associated with duplicate keys. + +- Map.put() appends a duplicate if the key already exists rather than replacing the existing value, and always returns null. + +- Map.remove() removes all duplicates for the specified key. + +- Map.get() returns the first duplicate for the specified key. + +- StoredMap.duplicates() is an additional method for returning the values for a given key as a Collection. + +Other differences are: + +- Collection.size() and Map.size() always throws UnsupportedOperationException. This is because the number of records in a database cannot be determined reliably or cheaply. + +- Because the size() method cannot be used, the bulk operation methods of standard Java collections cannot be passed stored collections as parameters, since the implementations rely on size(). However, the bulk operation methods of stored collections can be passed standard Java collections as parameters. `storedCollection.addAll(standardCollection)` is allowed while `standardCollection.addAll(storedCollection)` is *not* allowed. This restriction applies to the standard collection constructors that take a Collection parameter (copy constructors), the Map.putAll() method, and the following Collection methods: addAll(), containsAll(), removeAll() and retainAll(). + +- The `ListIterator.nextIndex()` method returns `Integer.MAX_VALUE` for stored lists when positioned at the end of the list, rather than returning the list size as specified by the ListIterator interface. Again, this is because the database size is not available. + +- Comparator objects cannot be used and the SortedMap.comparator() and SortedSet.comparator() methods always return null. The Comparable interface is not supported. However, Comparators that operate on byte arrays may be specified using DatabaseConfig.setBtreeComparator. + +- The Object.equals() method is not used to determine whether a key or value is contained in a collection, to locate a value by key, etc. Instead the byte array representation of the keys and values are used. However, the equals() method *is* called for each key and value when comparing two collections for equality. It is the responsibility of the application to make sure that the equals() method returns true if and only if the byte array representations of the two objects are equal. Normally this occurs naturally since the byte array representation is derived from the object's fields. + +### Other Stored Collection Characteristics + +The following characteristics of stored collections are extensions of the definitions in the java.util package. These differences do not violate the Java collections interface contract. + +- All stored collections are thread safe (can be used by multiple threads concurrently) whenever the Berkeley DB Concurrent Data Store or Transactional Data Store environment is used. Locking is handled by the Berkeley DB environment. To access a collection from multiple threads, creation of synchronized collections using the Collections class is not necessary except when using the Data Store environment. Iterators, however, should always be used only by a single thread. + +- All stored collections may be read-only if desired by passing false for the writeAllowed parameter of their constructor. Creation of immutable collections using the Collections class is not necessary. + +- A stored collection is partially read-only if a secondary index is used. Specifically, values may be removed but may not be added or updated. The following methods will throw UnsupportedOperationException when an index is used: Collection.add(), List.set(), ListIterator.set() and Map.Entry.setValue(). + +- SortedMap.entrySet() and SortedMap.keySet() return a SortedSet, not just a Set as specified in Java collections interface. This allows using the SortedSet methods on the returned collection. + +- SortedMap.values() returns a SortedSet, not just a Collection, whenever the keys of the map can be derived from the values using an entity binding. Note that the sorted set returned is not really a set if duplicates are allowed, since it is technically a collection; however, the SortedSet methods (for example, subSet()), can still be used. + +- For SortedSet and SortedMap views, additional subSet() and subMap() methods are provided that allow control over whether keys are treated as inclusive or exclusive values in the key range. + +- Keys and values are stored by value, not by reference. This is because objects that are added to collections are converted to byte arrays (by bindings) and stored in the database. When they are retrieved from the collection they are read from the database and converted from byte arrays to objects. Therefore, the object reference added to a collection will not be the same as the reference later retrieved from the collection. + +- A runtime exception, RuntimeExceptionWrapper, is thrown whenever database exceptions occur which are not runtime exceptions. The RuntimeExceptionWrapper.getCause() method can be called to get the underlying exception. + +- All iterators for stored collections implement the ListIterator interface as well as the Iterator interface. This is to allow use of the ListIterator.hasPrevious() and ListIterator.previous() methods, which work for all collections since Berkeley DB provides bidirectional cursors. + +- All stored collections have a StoredCollection.iterator(boolean) method that allows creating a read-only iterator for a writable collection. For the standard Collection.iterator() method, the iterator is read-only only when the collection is read-only. Read-only iterators are important for using the Berkeley DB Concurrent Data Store environment, since only one write cursors may be open at one time. + +- Iterator stability for stored collections is greater than the iterator stability defined by the Java collections interfaces. Stored iterator stability is the same as the cursor stability defined by Berkeley DB. + +- When an entity binding is used, updating (setting) a value is not allowed if the key in the entity is not equal to the original key. For example, calling Map.put() is not allowed when the key parameter is not equal to the key of the entity parameter. Map.put(), List.set(), ListIterator.set(), and Map.Entry.setValue() will throw IllegalArgumentException in this situation. + +- Adding and removing items from stored lists is not allowed for sublists. This is simply an unimplemented feature and may be changed in the future. Currently for sublists the following methods throw UnsupportedOperationException: List.add()`List.add()`, List.remove()`List.remove()`, ListIterator.add() and ListIterator.remove()`ListIterator.remove()`. + +- The StoredList.append(java.lang.Object) and StoredMap.append(java.lang.Object) extension methods allows adding a new record with an automatically assigned key. Record number assignment by the database itself is supported for QUEUE, RECNO and RECNO-RENUMBER databases. An application-defined PrimaryKeyAssigner is used to assign the key value. + +### Why Java Collections for Berkeley DB + +The Java collections interface was chosen as the best Java API for DB given these requirements: + +1. Provide the Java developer with an API that is as familiar and easy to use as possible. + +2. Provide access to all, or a large majority, of the features of the underlying Berkeley DB storage system. + +3. Compared to the DB API, provide a higher-level API that is oriented toward Java developers. + +4. For ease of use, support object-to-data bindings, per-thread transactions, and some traditional database features such as foreign keys. + +5. Provide a thin layer that can be thoroughly tested and which does not significantly impact the reliability and performance of DB. + +Admittedly there are several things about the Java Collections API that don't quite fit with DB or with any transactional database, and therefore there are some new rules for applying the Java Collections API. However, these disadvantages are considered to be smaller than the disadvantages of the alternatives: + +- A new API not based on the Java Collections API could have been designed that maps well to DB but is higher-level. However, this would require designing an entirely new model. The exceptions for using the Java Collections API are considered easier to learn than a whole new model. A new model would also require a long design stabilization period before being as complete and understandable as either the Java Collections API or the DB API. + +- The ODMG API or another object persistence API could have been implemented on top of DB. However, an object persistence implementation would add much code and require a long stabilization period. And while it may work well for applications that require object persistence, it would probably never perform well enough for many other applications. diff --git a/docs-src/guides/collections/_meta.toml b/docs-src/guides/collections/_meta.toml new file mode 100644 index 000000000..b4b294c62 --- /dev/null +++ b/docs-src/guides/collections/_meta.toml @@ -0,0 +1,44 @@ +# Nav/index metadata for the collections guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Berkeley DB Collections Tutorial" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "intro", + "developing", + "tutorialintroduction", + "BasicProgram", + "opendbenvironment", + "openclasscatalog", + "opendatabases", + "createbindingscollections", + "implementingmain", + "usingtransactions", + "addingdatabaseitems", + "retrievingdatabaseitems", + "handlingexceptions", + "UsingSecondaries", + "openingforeignkeys", + "indexedcollections", + "retrievingbyindexkey", + "Entity", + "creatingentitybindings", + "collectionswithentities", + "entitieswithcollections", + "Tuple", + "tupleswithkeycreators", + "tuplekeybindings", + "tuple-serialentitybindings", + "sortedcollections", + "SerializableEntity", + "transientfieldsinbinding", + "removingredundantvalueclasses", + "Summary", + "collectionOverview", + "UsingCollectionsAPI", + "UsingStoredCollections", + "SerializedObjectStorage", +] diff --git a/docs-src/guides/collections/addingdatabaseitems.md b/docs-src/guides/collections/addingdatabaseitems.md new file mode 100644 index 000000000..434950cdb --- /dev/null +++ b/docs-src/guides/collections/addingdatabaseitems.md @@ -0,0 +1,139 @@ +--- +title: "Adding Database Items" +api-name: "Adding Database Items" +source: docs/collections/tutorial/addingdatabaseitems.html +--- +## Adding Database Items + +Adding (as well as updating, removing, and deleting) information in the database is accomplished via the standard Java collections API. In the example, the Map.put method is used to add objects. All standard Java methods for modifying a collection may be used with the DB Java Collections API. + +The `PopulateDatabase.doWork()` method calls private methods for adding objects to each of the three database stores. It is called via the TransactionRunner class and was outlined in the previous section. + +``` c +import java.util.Map; +import com.sleepycat.collections.TransactionWorker; +... +public class Sample +{ + ... + private SampleViews views; + ... + private class PopulateDatabase implements TransactionWorker + { + public void doWork() + throws Exception + { + addSuppliers(); + addParts(); + addShipments(); + } + } + ... + + private void addSuppliers() + { + } + + private void addParts() + { + } + + private void addShipments() + { + } +} +``` + +The `addSuppliers()`, `addParts()` and `addShipments()` methods add objects to the Suppliers, Parts and Shipments stores. The Map for each store is obtained from the `SampleViews` object. + +``` c + private void addSuppliers() + { + Map suppliers = views.getSupplierMap(); + if (suppliers.isEmpty()) + { + System.out.println("Adding Suppliers"); + suppliers.put(new SupplierKey("S1"), + new SupplierData("Smith", 20, "London")); + suppliers.put(new SupplierKey("S2"), + new SupplierData("Jones", 10, "Paris")); + suppliers.put(new SupplierKey("S3"), + new SupplierData("Blake", 30, "Paris")); + suppliers.put(new SupplierKey("S4"), + new SupplierData("Clark", 20, "London")); + suppliers.put(new SupplierKey("S5"), + new SupplierData("Adams", 30, "Athens")); + } + } + + private void addParts() + { + Map parts = views.getPartMap(); + if (parts.isEmpty()) + { + System.out.println("Adding Parts"); + parts.put(new PartKey("P1"), + new PartData("Nut", "Red", + new Weight(12.0, Weight.GRAMS), + "London")); + parts.put(new PartKey("P2"), + new PartData("Bolt", "Green", + new Weight(17.0, Weight.GRAMS), + "Paris")); + parts.put(new PartKey("P3"), + new PartData("Screw", "Blue", + new Weight(17.0, Weight.GRAMS), + "Rome")); + parts.put(new PartKey("P4"), + new PartData("Screw", "Red", + new Weight(14.0, Weight.GRAMS), + "London")); + parts.put(new PartKey("P5"), + new PartData("Cam", "Blue", + new Weight(12.0, Weight.GRAMS), + "Paris")); + parts.put(new PartKey("P6"), + new PartData("Cog", "Red", + new Weight(19.0, Weight.GRAMS), + "London")); + } + } + + private void addShipments() + { + Map shipments = views.getShipmentMap(); + if (shipments.isEmpty()) + { + System.out.println("Adding Shipments"); + shipments.put(new ShipmentKey("P1", "S1"), + new ShipmentData(300)); + shipments.put(new ShipmentKey("P2", "S1"), + new ShipmentData(200)); + shipments.put(new ShipmentKey("P3", "S1"), + new ShipmentData(400)); + shipments.put(new ShipmentKey("P4", "S1"), + new ShipmentData(200)); + shipments.put(new ShipmentKey("P5", "S1"), + new ShipmentData(100)); + shipments.put(new ShipmentKey("P6", "S1"), + new ShipmentData(100)); + shipments.put(new ShipmentKey("P1", "S2"), + new ShipmentData(300)); + shipments.put(new ShipmentKey("P2", "S2"), + new ShipmentData(400)); + shipments.put(new ShipmentKey("P2", "S3"), + new ShipmentData(200)); + shipments.put(new ShipmentKey("P2", "S4"), + new ShipmentData(200)); + shipments.put(new ShipmentKey("P4", "S4"), + new ShipmentData(300)); + shipments.put(new ShipmentKey("P5", "S4"), + new ShipmentData(400)); + } + } +} +``` + +The key and value classes used above were defined in the Defining Serialized Key and Value Classes . + +In each method above, objects are added only if the map is not empty. This is a simple way of allowing the example program to be run repeatedly. In real-life applications another technique — checking the Map.containsKey method, for example — might be used. diff --git a/docs-src/guides/collections/collectionOverview.md b/docs-src/guides/collections/collectionOverview.md new file mode 100644 index 000000000..88e1334c6 --- /dev/null +++ b/docs-src/guides/collections/collectionOverview.md @@ -0,0 +1,99 @@ +--- +title: "Appendix A.  API Notes and Details" +api-name: "Appendix A.  API Notes and Details" +source: docs/collections/tutorial/collectionOverview.html +--- +## Appendix A.  API Notes and Details + +This appendix contains information useful to the collections programmer that is too detailed to easily fit into the format of a tutorial. Specifically, this appendix contains the following information: + +- Using Data Bindings + +- Using the DB Java Collections API + +- Using Stored Collections + +- Serialized Object Storage + +## Using Data Bindings + + [Selecting Binding Formats](collectionOverview.md#SelectingBindingFormats) + + [Record Number Bindings](collectionOverview.md#RecordNumberBindings) + + [Selecting Data Bindings](collectionOverview.md#SelectingDataBindings) + + [Implementing Bindings](collectionOverview.md#ImplementingBindings) + + [Using Bindings](collectionOverview.md#UsingBindings) + + [Secondary Key Creators](collectionOverview.md#SecondaryKeyCreators) + +Data bindings determine how keys and values are represented as stored data (byte arrays) in the database, and how stored data is converted to and from Java objects. + +The selection of data bindings is, in general, independent of the selection of access methods and collection views. In other words, any binding can be used with any access method or collection. One exception to this rule is described under Record Number Bindings below. + +### Note + +In this document, bindings are described in the context of their use for stored data in a database. However, bindings may also be used independently of a database to operate on an arbitrary byte array. This allows using bindings when data is to be written to a file or sent over a network, for example. + +### Selecting Binding Formats + +For the key and value of each stored collection, you may select one of the following types of bindings. + +| Binding Format | Ordered | Description | +|----|----|----| +| SerialBinding | No | The data is stored using a compact form of Java serialization, where the class descriptions are stored separately in a catalog database. Arbitrary Java objects are supported. | +| TupleBinding | Yes | The data is stored using a series of fixed length primitive values or zero terminated character arrays (strings). Class/type evolution is not supported. | +| RecordNumberBinding | Yes | The data is a 32-bit integer stored in a platform-dependent format. | +| Custom binding format | User-defined | The data storage format and ordering is determined by the custom binding implementation. | + +As shown in the table above, the tuple format supports built-in ordering (without specifying a custom comparator), while the serial format does not. This means that when a specific key order is needed, tuples should be used instead of serial data. Alternatively, a custom Btree comparator should be specified using `DatabaseConfig.setBtreeComparator()`. Note that a custom Btree comparator will usually execute more slowly than the default byte-by-byte comparison. This makes using tuples an attractive option, since they provide ordering along with optimal performance. + +The tuple binding uses less space and executes faster than the serial binding. But once a tuple is written to a database, the order of fields in the tuple may not be changed and fields may not be deleted. The only type evolution allowed is the addition of fields at the end of the tuple, and this must be explicitly supported by the custom binding implementation. + +The serial binding supports the full generality of Java serialization including type evolution. But serialized data can only be accessed by Java applications, its size is larger, and its bindings are slower to execute. + +### Record Number Bindings + +Any use of an access method with record number keys, and therefore any use of a stored list view, requires using RecordNumberBinding as the key binding. Since Berkeley DB stores record number keys using a platform-dependent byte order, RecordNumberBinding is needed to store record numbers properly. See logical record numbers in the *Berkeley DB Programmer's Reference Guide* for more information on storing DB record numbers. + +### Note + +You may not use RecordNumberBinding except with record number keys, as determined by the access method. Using RecordNumberBinding in other cases will create a database that is not portable between platforms. When constructing the stored collection, the DB Java Collections API will throw an IllegalArgumentException in such cases. + +### Selecting Data Bindings + +There are two types of binding interfaces. Simple entry bindings implement the EntryBinding interface and can be used for key or value objects. Entity bindings implement the EntityBinding interface and are used for combined key and value objects called entities. + +Simple entry bindings map between the key or value data stored by Berkeley DB and a key or value object. This is a simple one-to-one mapping. + +Simple entry bindings are easy to implement and in some cases require no coding. For example, a SerialBinding can be used for keys or values without writing any additional code. A tuple binding for a single-item tuple can also be used without writing any code; see the TupleBinding.getPrimitiveBinding method. + +Entity bindings must divide an entity object into its key and value data, and then combine the key and value data to re-create the entity object. This is a two-to-one mapping. + +Entity bindings are useful when a stored application object naturally has its primary key as a property, which is very common. For example, an Employee object would naturally have an EmployeeNumber property (its primary key) and an entity binding would then be needed. Of course, entity bindings are more complex to implement, especially if their key and data formats are different. + +Note that even when an entity binding is used a key binding is also usually needed. For example, a key binding is used to create key objects that are passed to the Map.get() method. A key object is passed to this method even though it may return an entity that also contains the key. + +### Implementing Bindings + +There are two ways to implement bindings. The first way is to create a binding class that implements one of the two binding interfaces, EntryBinding or EntityBinding. For tuple bindings and serial bindings there are a number of abstract classes that make this easier. For example, you can extend TupleBinding to implement a simple binding for a tuple key or value. Abstract classes are also provided for entity bindings and are named after the format names of the key and value. For example, you can extend TupleSerialBinding to implement an entity binding with a tuple key and serial value. + +Another way to implement bindings is with marshalling interfaces. These are interfaces which perform the binding operations and are implemented by the key, value or entity classes themselves. With marshalling you use a binding which calls the marshalling interface and you implement the marshalling interface for each key, value or entity class. For example, you can use TupleMarshalledBinding along with key or value classes that implement the MarshalledTupleEntry interface. + +### Using Bindings + +Bindings are specified whenever a stored collection is created. A key binding must be specified for map, key set and entry set views. A value binding or entity binding must be specified for map, value set and entry set views. + +Any number of bindings may be created for the same stored data. This allows multiple views over the same data. For example, a tuple might be bound to an array of values or to a class with properties for each object. + +It is important to be careful of bindings that only use a subset of the stored data. This can be useful to simplify a view or to hide information that should not be accessible. However, if you write records using these bindings you may create stored data that is invalid from the application's point of view. It is up to the application to guard against this by creating a read-only collection when such bindings are used. + +### Secondary Key Creators + +Secondary Key Creators are needed whenever database indices are used. For each secondary index (SecondaryDatabase) a key creator is used to derive index key data from key/value data. Key creators are objects whose classes implement the SecondaryKeyCreator interface. + +Like bindings, key creators may be implemented using a separate key creator class or using a marshalling interface. Abstract key creator classes and marshalling interfaces are provided in the com.sleepycat.bind.tuple and com.sleepycat.bind.serial packages. + +Unlike bindings, key creators fundamentally operate on key and value data, not necessarily on the objects derived from the data by bindings. In this sense key creators are a part of a database definition, and may be independent of the various bindings that may be used to view data in a database. However, key creators are not prohibited from using higher level objects produced by bindings, and doing so may be convenient for some applications. For example, marshalling interfaces, which are defined for objects produced by bindings, are a convenient way to define key creators. diff --git a/docs-src/guides/collections/collectionswithentities.md b/docs-src/guides/collections/collectionswithentities.md new file mode 100644 index 000000000..e8532da3a --- /dev/null +++ b/docs-src/guides/collections/collectionswithentities.md @@ -0,0 +1,60 @@ +--- +title: "Creating Collections with Entity Bindings" +api-name: "Creating Collections with Entity Bindings" +source: docs/collections/tutorial/collectionswithentities.html +--- +## Creating Collections with Entity Bindings + +Stored map objects are created in this example in the same way as in prior examples, but using entity bindings in place of value bindings. All value objects passed and returned to the Java collections API are then actually entity objects (`Part`, `Supplier` and `Shipment`). The application no longer deals directly with plain value objects (`PartData`, `SupplierData` and `ShipmentData`). + +Since the `partValueBinding`, `supplierValueBinding` and `shipmentValueBinding` were defined as entity bindings in the prior section, there are no source code changes necessary for creating the stored map objects. + +``` c +public class SampleViews +{ + ... + public SampleViews(SampleDatabase db) + { + ... + partMap = + new StoredMap(db.getPartDatabase(), + partKeyBinding, partValueBinding, true); + supplierMap = + new StoredMap(db.getSupplierDatabase(), + supplierKeyBinding, supplierValueBinding, true); + shipmentMap = + new StoredMap(db.getShipmentDatabase(), + shipmentKeyBinding, shipmentValueBinding, true); + ... + } +``` + +Specifying an EntityBinding will select a different StoredMap constructor, but the syntax is the same. In general, an entity binding may be used anywhere that a value binding is used. + +The following getter methods are defined for use by other classes in the example program. Instead of returning the map's entry set (Map.entrySet), the map's value set (Map.values) is returned. The entry set was convenient in prior examples because it allowed enumerating all key/value pairs in the collection. Since an entity contains the key and the value, enumerating the value set can now be used more conveniently for the same purpose. + +``` c +import com.sleepycat.collections.StoredValueSet; +... +public class SampleViews +{ + ... + public StoredValueSet getPartSet() + { + return (StoredValueSet) partMap.values(); + } + + public StoredValueSet getSupplierSet() + { + return (StoredValueSet) supplierMap.values(); + } + + public StoredValueSet getShipmentSet() + { + return (StoredValueSet) shipmentMap.values(); + } + ... +} +``` + +Notice that the collection returned by the StoredMap.values method is actually a StoredValueSet and not just a Collection as defined by the Map.values interface. As long as duplicate keys are not allowed, this collection will behave as a true set and will disallow the addition of duplicates, etc. diff --git a/docs-src/guides/collections/createbindingscollections.md b/docs-src/guides/collections/createbindingscollections.md new file mode 100644 index 000000000..a149e16b3 --- /dev/null +++ b/docs-src/guides/collections/createbindingscollections.md @@ -0,0 +1,138 @@ +--- +title: "Creating Bindings and Collections" +api-name: "Creating Bindings and Collections" +source: docs/collections/tutorial/createbindingscollections.html +--- +## Creating Bindings and Collections + +*Bindings* translate between stored records and Java objects. In this example, Java serialization bindings are used. Serial bindings are the simplest type of bindings because no mapping of fields or type conversion is needed. Tuple bindings — which are more difficult to create than serial bindings but have some advantages — will be introduced later in the Tuple example program. + +Standard Java *collections* are used to access records in a database. Stored collections use bindings transparently to convert the records to objects when they are retrieved from the collection, and to convert the objects to records when they are stored in the collection. + +An important characteristic of stored collections is that they do *not* perform object caching. Every time an object is accessed via a collection it will be added to or retrieved from the database, and the bindings will be invoked to convert the data. Objects are therefore always passed and returned by value, not by reference. Because Berkeley DB is an embedded database, efficient caching of stored raw record data is performed by the database library. + +The `SampleViews` class is used to create the bindings and collections. This class is separate from the `SampleDatabase` class to illustrate the idea that a single set of stored data can be accessed via multiple bindings and collections, or *views*. The skeleton for the `SampleViews` class follows. + +``` c +import com.sleepycat.bind.EntryBinding; +import com.sleepycat.bind.serial.ClassCatalog; +import com.sleepycat.bind.serial.SerialBinding; +import com.sleepycat.collections.StoredEntrySet; +import com.sleepycat.collections.StoredMap; +... + +public class SampleViews +{ + private StoredMap partMap; + private StoredMap supplierMap; + private StoredMap shipmentMap; + + ... + public SampleViews(SampleDatabase db) + { + } +} +``` + +A StoredMap field is used for each database. The StoredMap class implements the standard Java Map interface, which has methods for obtaining a Set of keys, a Collection of values, or a Set of Map.Entry key/value pairs. Because databases contain key/value pairs, any Berkeley DB database may be represented as a Java map. + +The following statements create the key and data bindings using the SerialBinding class. + +``` c + public SampleViews(SampleDatabase db) + { + ClassCatalog catalog = db.getClassCatalog(); + EntryBinding partKeyBinding = + new SerialBinding(catalog, PartKey.class); + EntryBinding partValueBinding = + new SerialBinding(catalog, PartData.class); + EntryBinding supplierKeyBinding = + new SerialBinding(catalog, SupplierKey.class); + EntryBinding supplierValueBinding = + new SerialBinding(catalog, SupplierData.class); + EntryBinding shipmentKeyBinding = + new SerialBinding(catalog, ShipmentKey.class); + EntryBinding shipmentValueBinding = + new SerialBinding(catalog, ShipmentData.class); + ... + } +``` + +The first parameter of the SerialBinding constructor is the class catalog, and is used to store the class descriptions of the serialized objects. + +The second parameter is the base class for the serialized objects and is used for type checking of the stored objects. If `null` or `Object.class` is specified, then any Java class is allowed. Otherwise, all objects stored in that format must be instances of the specified class or derived from the specified class. In the example, specific classes are used to enable strong type checking. + +The following statements create standard Java maps using the StoredMap class. + +``` c + public SampleViews(SampleDatabase db) + { + ... + partMap = + new StoredMap(db.getPartDatabase(), + partKeyBinding, partValueBinding, true); + supplierMap = + new StoredMap(db.getSupplierDatabase(), + supplierKeyBinding, supplierValueBinding, true); + shipmentMap = + new StoredMap(db.getShipmentDatabase(), + shipmentKeyBinding, shipmentValueBinding, true); + ... + } +``` + +The first parameter of the StoredMap constructor is the database. In a StoredMap, the database keys (the primary keys) are used as the map keys. The Index example shows how to use secondary index keys as map keys. + +The second and third parameters are the key and value bindings to use when storing and retrieving objects via the map. + +The fourth and last parameter specifies whether changes will be allowed via the collection. If false is passed, the collection will be read-only. + +The following getter methods return the stored maps for use by other classes in the example program. Convenience methods for returning entry sets are also included. + +``` c +public class SampleViews +{ + ... + public final StoredMap getPartMap() + { + return partMap; + } + + public final StoredMap getSupplierMap() + { + return supplierMap; + } + + public final StoredMap getShipmentMap() + { + return shipmentMap; + } + + public final StoredEntrySet getPartEntrySet() + { + return (StoredEntrySet) partMap.entrySet(); + } + + public final StoredEntrySet getSupplierEntrySet() + { + return (StoredEntrySet) supplierMap.entrySet(); + } + + public final StoredEntrySet getShipmentEntrySet() + { + return (StoredEntrySet) shipmentMap.entrySet(); + } + ... +} +``` + +Note that StoredMap and StoredEntrySet are returned rather than just returning Map and Set. Since StoredMap implements the Map interface and StoredEntrySet implements the Set interface, you may ask why Map and Set were not returned directly. + +`StoredMap`, `StoredEntrySet`, and other stored collection classes have a small number of extra methods beyond those in the Java collection interfaces. The stored collection types are therefore returned to avoid casting when using the extended methods. Normally, however, only a Map or Set is needed, and may be used as follows. + +``` c + SampleDatabase sd = new SampleDatabase(new String("/home")); + SampleViews views = new SampleViews(sd); + Map partMap = views.getPartMap(); + Set supplierEntries = views.getSupplierEntrySet(); +``` diff --git a/docs-src/guides/collections/creatingentitybindings.md b/docs-src/guides/collections/creatingentitybindings.md new file mode 100644 index 000000000..632c8fe21 --- /dev/null +++ b/docs-src/guides/collections/creatingentitybindings.md @@ -0,0 +1,165 @@ +--- +title: "Creating Entity Bindings" +api-name: "Creating Entity Bindings" +source: docs/collections/tutorial/creatingentitybindings.html +--- +## Creating Entity Bindings + +*Entity bindings* are similar to ordinary bindings in that they convert between Java objects and the stored data format of keys and values. In addition, entity bindings map between key/value pairs and entity objects. An ordinary binding is a one-to-one mapping, while an entity binding is a two-to-one mapping. + +The `partValueBinding`, `supplierValueBinding` and `shipmentValueBinding` bindings are created below as entity bindings rather than (in the prior examples) serial bindings. + +``` c +import com.sleepycat.bind.EntryBinding; +import com.sleepycat.bind.EntityBinding; +import com.sleepycat.bind.serial.SerialBinding; +import com.sleepycat.bind.serial.SerialSerialBinding; +... + +public class SampleViews +{ + ... + public SampleViews(SampleDatabase db) + { + ClassCatalog catalog = db.getClassCatalog(); + SerialBinding partKeyBinding = + new SerialBinding(catalog, PartKey.class); + EntityBinding partValueBinding = + new PartBinding(catalog, PartKey.class, PartData.class); + SerialBinding supplierKeyBinding = + new SerialBinding(catalog, SupplierKey.class); + EntityBinding supplierValueBinding = + new SupplierBinding(catalog, SupplierKey.class, + SupplierData.class); + SerialBinding shipmentKeyBinding = + new SerialBinding(catalog, ShipmentKey.class); + EntityBinding shipmentValueBinding = + new ShipmentBinding(catalog, ShipmentKey.class, + ShipmentData.class); + SerialBinding cityKeyBinding = + new SerialBinding(catalog, String.class); + ... + } +} +``` + +The entity bindings will be used in the next section to construct stored map objects. + +The `PartBinding` class is defined below. + +``` c +public class SampleViews +{ + ... + private static class PartBinding extends SerialSerialBinding { + private PartBinding(ClassCatalog classCatalog, + Class keyClass, + Class dataClass) + { + super(classCatalog, keyClass, dataClass); + } + + public Object entryToObject(Object keyInput, Object dataInput) + { + PartKey key = (PartKey) keyInput; + PartData data = (PartData) dataInput; + return new Part(key.getNumber(), data.getName(), + data.getColor(), data.getWeight(), + data.getCity()); + } + + public Object objectToKey(Object object) + { + Part part = (Part) object; + return new PartKey(part.getNumber()); + } + + public Object objectToData(Object object) + { + Part part = (Part) object; + return new PartData(part.getName(), part.getColor(), + part.getWeight(), part.getCity()); + } + } + ... +} +``` + +In general, an entity binding is any class that implements the EntityBinding interface, just as an ordinary binding is any class that implements the EntryBinding interface. In the prior examples the built-in SerialBinding class (which implements EntryBinding) was used and no application-defined binding classes were needed. + +In this example, application-defined binding classes are used that extend the SerialSerialBinding abstract base class. This base class implements EntityBinding and provides the conversions between key/value bytes and key/value objects, just as the SerialBinding class does. The application-defined entity class implements the abstract methods defined in the base class that map between key/value objects and entity objects. + +Three abstract methods are implemented for each entity binding. The `entryToObject()` method takes as input the key and data objects, which have been deserialized automatically by the base class. As output, it returns the combined `Part` entity. + +The `objectToKey()` and `objectToData()` methods take an entity object as input. As output they return the part key or data object that is extracted from the entity object. The key or data will then be serialized automatically by the base class. + +The `SupplierBinding` and `ShipmentBinding` classes are very similar to the `PartBinding` class. + +``` c +public class SampleViews +{ + ... + private static class SupplierBinding extends SerialSerialBinding { + private SupplierBinding(ClassCatalog classCatalog, + Class keyClass, + Class dataClass) + { + super(classCatalog, keyClass, dataClass); + } + + public Object entryToObject(Object keyInput, Object dataInput) + { + SupplierKey key = (SupplierKey) keyInput; + SupplierData data = (SupplierData) dataInput; + return new Supplier(key.getNumber(), data.getName(), + data.getStatus(), data.getCity()); + } + + public Object objectToKey(Object object) + { + Supplier supplier = (Supplier) object; + return new SupplierKey(supplier.getNumber()); + } + + public Object objectToData(Object object) + { + Supplier supplier = (Supplier) object; + return new SupplierData(supplier.getName(), + supplier.getStatus(), + supplier.getCity()); + } + } + + private static class ShipmentBinding extends SerialSerialBinding { + private ShipmentBinding(ClassCatalog classCatalog, + Class keyClass, + Class dataClass) + { + super(classCatalog, keyClass, dataClass); + } + + public Object entryToObject(Object keyInput, Object dataInput) + { + ShipmentKey key = (ShipmentKey) keyInput; + ShipmentData data = (ShipmentData) dataInput; + return new Shipment(key.getPartNumber(), + key.getSupplierNumber(), + data.getQuantity()); + } + + public Object objectToKey(Object object) + { + Shipment shipment = (Shipment) object; + return new ShipmentKey(shipment.getPartNumber(), + shipment.getSupplierNumber()); + } + + public Object objectToData(Object object) + { + Shipment shipment = (Shipment) object; + return new ShipmentData(shipment.getQuantity()); + } + } + ... +} +``` diff --git a/docs-src/guides/collections/developing.md b/docs-src/guides/collections/developing.md new file mode 100644 index 000000000..3edc48708 --- /dev/null +++ b/docs-src/guides/collections/developing.md @@ -0,0 +1,38 @@ +--- +title: "Developing a DB Collections Application" +api-name: "Developing a DB Collections Application" +source: docs/collections/tutorial/developing.html +--- +## Developing a DB Collections Application + +There are several important choices to make when developing an application using the DB Java Collections API. + +1. Choose the Berkeley DB Environment + + Depending on your application's concurrency and transactional requirements, you may choose one of the three Berkeley DB Environments: Data Store, Concurrent Data Store, or Transactional Data Store. For details on creating and configuring the environment, see the *Berkeley DB Programmer's Reference Guide* + +2. Choose the Berkeley DB Access Method + + For each Berkeley DB datastore, you may choose from any of the four Berkeley DB access methods — BTREE, HASH, RECNO, or QUEUE (DatabaseType.BTREE, DatabaseType.HASH, DatabaseType.RECNO, or DatabaseType.QUEUE.) — and a number of other database options. Your choice depends on several factors such as whether you need ordered keys, unique keys, record number access, and so forth. For more information on access methods, see the *Berkeley DB Programmer's Reference Guide*. + +3. Choose the Format for Keys and Values + + For each database you may choose a binding format for the keys and values. For example, the tuple format is useful for keys because it has a deterministic sort order. The serial format is useful for values if you want to store arbitrary Java objects. In some cases a custom format may be appropriate. For details on choosing a binding format see Using Data Bindings . + +4. Choose the Binding for Keys and Values + + With the serial data format you do not have to create a binding for each Java class that is stored since Java serialization is used. But for other formats a binding must be defined that translates between stored byte arrays and Java objects. For details see Using Data Bindings . + +5. Choose Secondary Indices + + Any database that has unique keys may have any number of secondary indices. A secondary index has keys that are derived from data values in the primary database. This allows lookup and iteration of objects in the database by its index keys. For each index you must define how the index keys are derived from the data values using a SecondaryKeyCreator. For details see the SecondaryDatabase, SecondaryConfig and SecondaryKeyCreator classes. + +6. Choose the Collection Interface for each Database + + The standard Java Collection interfaces are used for accessing databases and secondary indices. The Map and Set interfaces may be used for any type of database. The Iterator interface is used through the Set interfaces. For more information on the collection interfaces see Using Stored Collections . + +Any number of bindings and collections may be created for the same database. This allows multiple views of the same stored data. For example, a data store may be viewed as a Map of keys to values, a Set of keys, or a Collection of values. String values, for example, may be used with the built-in binding to the String class, or with a custom binding to another class that represents the string values differently. + +It is sometimes desirable to use a Java class that encapsulates both a data key and a data value. For example, a Part object might contain both the part number (key) and the part name (value). Using the DB Java Collections API this type of object is called an "entity". An entity binding is used to translate between the Java object and the stored data key and value. Entity bindings may be used with all Collection types. + +Please be aware that the provided DB Java Collections API collection classes do not conform completely to the interface contracts defined in the `java.util` package. For example, all iterators must be explicitly closed and the `size()` method is not available. The differences between the DB Java Collections API collections and the standard Java collections are documented in Stored Collections Versus Standard Java Collections . diff --git a/docs-src/guides/collections/entitieswithcollections.md b/docs-src/guides/collections/entitieswithcollections.md new file mode 100644 index 000000000..3e554d2b7 --- /dev/null +++ b/docs-src/guides/collections/entitieswithcollections.md @@ -0,0 +1,163 @@ +--- +title: "Using Entities with Collections" +api-name: "Using Entities with Collections" +source: docs/collections/tutorial/entitieswithcollections.html +--- +## Using Entities with Collections + +In this example entity objects, rather than key and value objects, are used for adding and enumerating the records in a collection. Because fewer classes and objects are involved, adding and enumerating is done more conveniently and more simply than in the prior examples. + +For adding and iterating entities, the collection of entities returned by Map.values is used. In general, when using an entity binding, all Java collection methods that are passed or returned a value object will be passed or returned an entity object instead. + +The `Sample` class has been changed in this example to add objects using the Set.add method rather than the Map.put method that was used in the prior examples. Entity objects are constructed and passed to Set.add. + +``` c +import java.util.Set; +... +public class Sample +{ + ... + private void addSuppliers() + { + Set suppliers = views.getSupplierSet(); + if (suppliers.isEmpty()) + { + System.out.println("Adding Suppliers"); + suppliers.add(new Supplier("S1", "Smith", 20, "London")); + suppliers.add(new Supplier("S2", "Jones", 10, "Paris")); + suppliers.add(new Supplier("S3", "Blake", 30, "Paris")); + suppliers.add(new Supplier("S4", "Clark", 20, "London")); + suppliers.add(new Supplier("S5", "Adams", 30, "Athens")); + } + } + + private void addParts() + { + Set parts = views.getPartSet(); + if (parts.isEmpty()) + { + System.out.println("Adding Parts"); + parts.add(new Part("P1", "Nut", "Red", + new Weight(12.0, Weight.GRAMS), "London")); + parts.add(new Part("P2", "Bolt", "Green", + new Weight(17.0, Weight.GRAMS), "Paris")); + parts.add(new Part("P3", "Screw", "Blue", + new Weight(17.0, Weight.GRAMS), "Rome")); + parts.add(new Part("P4", "Screw", "Red", + new Weight(14.0, Weight.GRAMS), "London")); + parts.add(new Part("P5", "Cam", "Blue", + new Weight(12.0, Weight.GRAMS), "Paris")); + parts.add(new Part("P6", "Cog", "Red", + new Weight(19.0, Weight.GRAMS), "London")); + } + } + + private void addShipments() + { + Set shipments = views.getShipmentSet(); + if (shipments.isEmpty()) + { + System.out.println("Adding Shipments"); + shipments.add(new Shipment("P1", "S1", 300)); + shipments.add(new Shipment("P2", "S1", 200)); + shipments.add(new Shipment("P3", "S1", 400)); + shipments.add(new Shipment("P4", "S1", 200)); + shipments.add(new Shipment("P5", "S1", 100)); + shipments.add(new Shipment("P6", "S1", 100)); + shipments.add(new Shipment("P1", "S2", 300)); + shipments.add(new Shipment("P2", "S2", 400)); + shipments.add(new Shipment("P2", "S3", 200)); + shipments.add(new Shipment("P2", "S4", 200)); + shipments.add(new Shipment("P4", "S4", 300)); + shipments.add(new Shipment("P5", "S4", 400)); + } + } +``` + +Instead of printing the key/value pairs by iterating over the Map.entrySet as done in the prior example, this example iterates over the entities in the Map.values collection. + +``` c +import java.util.Iterator; +import java.util.Set; +... +public class Sample +{ + ... + private class PrintDatabase implements TransactionWorker + { + public void doWork() + throws Exception + { + printValues("Parts", + views.getPartSet().iterator()); + printValues("Suppliers", + views.getSupplierSet().iterator()); + printValues("Suppliers for City Paris", + views.getSupplierByCityMap().duplicates( + "Paris").iterator()); + printValues("Shipments", + views.getShipmentSet().iterator()); + printValues("Shipments for Part P1", + views.getShipmentByPartMap().duplicates( + new PartKey("P1")).iterator()); + printValues("Shipments for Supplier S1", + views.getShipmentBySupplierMap().duplicates( + new SupplierKey("S1")).iterator()); + } + } + ... +} +``` + +The output of the example program is shown below. + +``` c +Adding Suppliers +Adding Parts +Adding Shipments + +--- Parts --- +Part: number=P1 name=Nut color=Red weight=[12.0 grams] city=London +Part: number=P2 name=Bolt color=Green weight=[17.0 grams] city=Paris +Part: number=P3 name=Screw color=Blue weight=[17.0 grams] city=Rome +Part: number=P4 name=Screw color=Red weight=[14.0 grams] city=London +Part: number=P5 name=Cam color=Blue weight=[12.0 grams] city=Paris +Part: number=P6 name=Cog color=Red weight=[19.0 grams] city=London + +--- Suppliers --- +Supplier: number=S1 name=Smith status=20 city=London +Supplier: number=S2 name=Jones status=10 city=Paris +Supplier: number=S3 name=Blake status=30 city=Paris +Supplier: number=S4 name=Clark status=20 city=London +Supplier: number=S5 name=Adams status=30 city=Athens + +--- Suppliers for City Paris --- +Supplier: number=S2 name=Jones status=10 city=Paris +Supplier: number=S3 name=Blake status=30 city=Paris + +--- Shipments --- +Shipment: part=P1 supplier=S1 quantity=300 +Shipment: part=P1 supplier=S2 quantity=300 +Shipment: part=P2 supplier=S1 quantity=200 +Shipment: part=P2 supplier=S2 quantity=400 +Shipment: part=P2 supplier=S3 quantity=200 +Shipment: part=P2 supplier=S4 quantity=200 +Shipment: part=P3 supplier=S1 quantity=400 +Shipment: part=P4 supplier=S1 quantity=200 +Shipment: part=P4 supplier=S4 quantity=300 +Shipment: part=P5 supplier=S1 quantity=100 +Shipment: part=P5 supplier=S4 quantity=400 +Shipment: part=P6 supplier=S1 quantity=100 + +--- Shipments for Part P1 --- +Shipment: part=P1 supplier=S1 quantity=300 +Shipment: part=P1 supplier=S2 quantity=300 + +--- Shipments for Supplier S1 --- +Shipment: part=P1 supplier=S1 quantity=300 +Shipment: part=P2 supplier=S1 quantity=200 +Shipment: part=P3 supplier=S1 quantity=400 +Shipment: part=P4 supplier=S1 quantity=200 +Shipment: part=P5 supplier=S1 quantity=100 +Shipment: part=P6 supplier=S1 quantity=100 +``` diff --git a/docs-src/guides/collections/handlingexceptions.md b/docs-src/guides/collections/handlingexceptions.md new file mode 100644 index 000000000..501695b44 --- /dev/null +++ b/docs-src/guides/collections/handlingexceptions.md @@ -0,0 +1,40 @@ +--- +title: "Handling Exceptions" +api-name: "Handling Exceptions" +source: docs/collections/tutorial/handlingexceptions.html +--- +## Handling Exceptions + +Exception handling was illustrated previously in Implementing the Main Program and Using Transactions exception handling in a DB Java Collections API application in more detail. + +There are two exceptions that must be treated specially: RunRecoveryException and DeadlockException. + +RunRecoveryException is thrown when the only solution is to shut down the application and run recovery. All applications must catch this exception and follow the recovery procedure. + +When DeadlockException is thrown, the application should normally retry the operation. If a deadlock continues to occur for some maximum number of retries, the application should give up and try again later or take other corrective actions. The DB Java Collections API provides two APIs for transaction execution. + +- When using the CurrentTransaction class directly, the application must catch DeadlockException and follow the procedure described previously. + +- When using the TransactionRunner class, retries are performed automatically and the application need only handle the case where the maximum number of retries has been reached. In that case, TransactionRunner.run will throw DeadlockException. + +When using the TransactionRunner class there are two other considerations. + +- First, if the application-defined TransactionWorker.doWork method throws an exception the transaction will automatically be aborted, and otherwise the transaction will automatically be committed. Applications should design their transaction processing with this in mind. + +- Second, please be aware that TransactionRunner.run unwraps exceptions in order to discover whether a nested exception is a DeadlockException. This is particularly important since all Berkeley DB exceptions that occur while calling a stored collection method are wrapped with a RuntimeExceptionWrapper. This wrapping is necessary because Berkeley DB exceptions are checked exceptions, and the Java collections API does not allow such exceptions to be thrown. + +When calling TransactionRunner.run, the unwrapped (nested) exception will be unwrapped and thrown automatically. If you are not using TransactionRunner or if you are handling exceptions directly for some other reason, use the ExceptionUnwrapper.unwrap method to get the nested exception. For example, this can be used to discover that an exception is a RunRecoveryException as shown below. + +``` c +import com.sleepycat.db.RunRecoveryException; +import com.sleepycat.util.ExceptionUnwrapper; +... + catch (Exception e) + { + e = ExceptionUnwrapper.unwrap(e); + if (e instanceof RunRecoveryException) + { + // follow recovery procedure + } + } +``` diff --git a/docs-src/guides/collections/implementingmain.md b/docs-src/guides/collections/implementingmain.md new file mode 100644 index 000000000..60768be01 --- /dev/null +++ b/docs-src/guides/collections/implementingmain.md @@ -0,0 +1,147 @@ +--- +title: "Implementing the Main Program" +api-name: "Implementing the Main Program" +source: docs/collections/tutorial/implementingmain.html +--- +## Implementing the Main Program + +The main program opens the environment and databases, stores and retrieves objects within a transaction, and finally closes the environment databases. This section describes the main program shell, and the next section describes how to run transactions for storing and retrieving objects. + +The `Sample` class contains the main program. The skeleton for the `Sample` class follows. + +``` c +import com.sleepycat.db.DatabaseException; +import java.io.FileNotFoundException; + +public class Sample +{ + private SampleDatabase db; + private SampleViews views; + + public static void main(String args) + { + } + + private Sample(String homeDir) + throws DatabaseException, FileNotFoundException + { + } + + private void close() + throws DatabaseException + { + } + + private void run() + throws Exception + { + } +} +``` + +The main program uses the `SampleDatabase` and `SampleViews` classes that were described in the preceding sections. The `main` method will create an instance of the `Sample` class, and call its `run()` and `close()` methods. + +The following statements parse the program's command line arguments. + +``` c + public static void main(String[] args) + { + System.out.println("\nRunning sample: " + Sample.class); + String homeDir = "./tmp"; + for (int i = 0; i < args.length; i += 1) + { + String arg = args[i]; + if (args[i].equals("-h") && i < args.length - 1) + { + i += 1; + homeDir = args[i]; + } + else + { + System.err.println("Usage:\n java " + + Sample.class.getName() + + "\n [-h ]"); + System.exit(2); + } + } + ... + } +``` + +The usage command is: + +``` c +java com.sleepycat.examples.bdb.shipment.basic.Sample + [-h ] +``` + +The `-h` command is used to set the `homeDir` variable, which will later be passed to the `SampleDatabase()` constructor. Normally all Berkeley DB programs should provide a way to configure their database environment home directory. + +The default for the home directory is `./tmp` — the tmp subdirectory of the current directory where the sample is run. The home directory must exist before running the sample. To re-create the sample database from scratch, delete all files in the home directory before running the sample. + +The home directory was described previously in Opening and Closing the Database Environment . + +Of course, the command line arguments shown are only examples and a real-life application may use different techniques for configuring these options. + +The following statements create an instance of the `Sample` class and call its `run()` and `close()` methods. + +``` c + public static void main(String args) + { + ... + Sample sample = null; + try + { + sample = new Sample(homeDir); + sample.run(); + } + catch (Exception e) + { + e.printStackTrace(); + } + finally + { + if (sample != null) + { + try + { + sample.close(); + } + catch (Exception e) + { + System.err.println("Exception during database close:"); + e.printStackTrace(); + } + } + } + } +``` + +The `Sample()` constructor will open the environment and databases, and the `run()` method will run transactions for storing and retrieving objects. If either of these throws an exception, then the program was unable to run and should normally terminate. (Transaction retries are handled at a lower level and will be described later.) The first `catch` statement handles such exceptions. + +The `finally` statement is used to call the `close()` method since an attempt should always be made to close the environment and databases cleanly. If an exception is thrown during close and a prior exception occurred above, then the exception during close is likely a side effect of the prior exception. + +The `Sample()` constructor creates the `SampleDatabase` and `SampleViews` objects. + +``` c + private Sample(String homeDir) + throws DatabaseException, FileNotFoundException + { + db = new SampleDatabase(homeDir); + views = new SampleViews(db); + } +``` + +Recall that creating the `SampleDatabase` object will open the environment and all databases. + +To close the database the `Sample.close()` method simply calls `SampleDatabase.close()`. + +``` c + private void close() + throws DatabaseException + { + db.close(); + } +``` + +The `run()` method is described in the next section. diff --git a/docs-src/guides/collections/index.md b/docs-src/guides/collections/index.md new file mode 100644 index 000000000..0be7058f8 --- /dev/null +++ b/docs-src/guides/collections/index.md @@ -0,0 +1,142 @@ +--- +title: "Berkeley DB Collections Tutorial" +api-name: "Berkeley DB Collections Tutorial" +source: docs/collections/tutorial/index.html +--- +# Berkeley DB Collections Tutorial + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Java™ and all Java-based marks are a trademark or registered trademark of Sun Microsystems, Inc, in the United States and other countries. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction](intro.md) + + [Features](intro.md#features) + + [Developing a DB Collections Application](developing.md) + + [Tutorial Introduction](tutorialintroduction.md) + + [2. The Basic Program](BasicProgram.md) + + [Defining Serialized Key and Value Classes](BasicProgram.md#keyandvalueclasses) + + [Opening and Closing the Database Environment](opendbenvironment.md) + + [Opening and Closing the Class Catalog](openclasscatalog.md) + + [Opening and Closing Databases](opendatabases.md) + + [Creating Bindings and Collections](createbindingscollections.md) + + [Implementing the Main Program](implementingmain.md) + + [Using Transactions](usingtransactions.md) + + [Adding Database Items](addingdatabaseitems.md) + + [Retrieving Database Items](retrievingdatabaseitems.md) + + [Handling Exceptions](handlingexceptions.md) + + [3. Using Secondary Indices](UsingSecondaries.md) + + [Opening Secondary Key Indices](UsingSecondaries.md#opensecondaryindices) + + [More Secondary Key Indices](openingforeignkeys.md) + + [Creating Indexed Collections](indexedcollections.md) + + [Retrieving Items by Index Key](retrievingbyindexkey.md) + + [4. Using Entity Classes](Entity.md) + + [Defining Entity Classes](Entity.md#definingentityclasses) + + [Creating Entity Bindings](creatingentitybindings.md) + + [Creating Collections with Entity Bindings](collectionswithentities.md) + + [Using Entities with Collections](entitieswithcollections.md) + + [5. Using Tuples](Tuple.md) + + [Using the Tuple Format](Tuple.md#tupleformat) + + [Using Tuples with Key Creators](tupleswithkeycreators.md) + + [Creating Tuple Key Bindings](tuplekeybindings.md) + + [Creating Tuple-Serial Entity Bindings](tuple-serialentitybindings.md) + + [Using Sorted Collections](sortedcollections.md) + + [6. Using Serializable Entities](SerializableEntity.md) + + [Using Transient Fields in an Entity Class](SerializableEntity.md#transientfieldsinclass) + + [Using Transient Fields in an Entity Binding](transientfieldsinbinding.md) + + [Removing the Redundant Value Classes](removingredundantvalueclasses.md) + + [7. Summary](Summary.md) + + [A. API Notes and Details](collectionOverview.md) + + [Using Data Bindings](collectionOverview.md#UsingDataBindings) + + [Selecting Binding Formats](collectionOverview.md#SelectingBindingFormats) + + [Record Number Bindings](collectionOverview.md#RecordNumberBindings) + + [Selecting Data Bindings](collectionOverview.md#SelectingDataBindings) + + [Implementing Bindings](collectionOverview.md#ImplementingBindings) + + [Using Bindings](collectionOverview.md#UsingBindings) + + [Secondary Key Creators](collectionOverview.md#SecondaryKeyCreators) + + [Using the DB Java Collections API](UsingCollectionsAPI.md) + + [Using Transactions](UsingCollectionsAPI.md#UsingTransactions) + + [Transaction Rollback](UsingCollectionsAPI.md#TransactionRollback) + + [Selecting Access Methods](UsingCollectionsAPI.md#SelectingAccessMethods) + + [Access Method Restrictions](UsingCollectionsAPI.md#AccessMethodRestrictions) + + [Using Stored Collections](UsingStoredCollections.md) + + [Stored Collection and Access Methods](UsingStoredCollections.md#StoredCollectionAccessMethods) + + [Stored Collections Versus Standard Java Collections](UsingStoredCollections.md#StoredVersusStandardCollections) + + [Other Stored Collection Characteristics](UsingStoredCollections.md#StoredCollectionCharacteristics) + + [Why Java Collections for Berkeley DB](UsingStoredCollections.md#WhyJavaCollections) + + [Serialized Object Storage](SerializedObjectStorage.md) diff --git a/docs-src/guides/collections/indexedcollections.md b/docs-src/guides/collections/indexedcollections.md new file mode 100644 index 000000000..3ccc8e5ed --- /dev/null +++ b/docs-src/guides/collections/indexedcollections.md @@ -0,0 +1,105 @@ +--- +title: "Creating Indexed Collections" +api-name: "Creating Indexed Collections" +source: docs/collections/tutorial/indexedcollections.html +--- +## Creating Indexed Collections + +In the prior Basic example, bindings and Java collections were created for accessing databases via their primary keys. In this example, bindings and collections are added for accessing the same databases via their index keys. As in the prior example, serial bindings and the Java Map class are used. + +When a map is created from a SecondaryDatabase, the keys of the map will be the index keys. However, the values of the map will be the values of the primary database associated with the index. This is how index keys can be used to access the values in a primary database. + +For example, the Supplier's City field is an index key that can be used to access the Supplier database. When a map is created using the `supplierByCityDb()` method, the key to the map will be the City field, a String object. When Map.get is called passing the City as the key parameter, a `SupplierData` object will be returned. + +The `SampleViews` class is extended to create an index key binding for the Supplier's City field and three Java maps based on the three indices created in the prior section. + +``` c +import com.sleepycat.bind.EntryBinding; +import com.sleepycat.bind.serial.SerialBinding; +import com.sleepycat.collections.StoredEntrySet; +import com.sleepycat.collections.StoredMap; +... + +public class SampleViews +{ + ... + private StoredMap supplierByCityMap; + private StoredMap shipmentByPartMap; + private StoredMap shipmentBySupplierMap; + ... + + public SampleViews(SampleDatabase db) + { + ClassCatalog catalog = db.getClassCatalog(); + ... + EntryBinding cityKeyBinding = + new SerialBinding(catalog, String.class); + ... + supplierByCityMap = + new StoredMap(db.getSupplierByCityDatabase(), + cityKeyBinding, supplierValueBinding, true); + shipmentByPartMap = + new StoredMap(db.getShipmentByPartDatabase(), + partKeyBinding, shipmentValueBinding, true); + shipmentBySupplierMap = + new StoredMap(db.getShipmentBySupplierDatabase(), + supplierKeyBinding, shipmentValueBinding, true); + ... + } +} +``` + +In general, the indexed maps are created here in the same way as the unindexed maps were created in the Basic example. The differences are: + +- The first parameter of the StoredMap constructor is a SecondaryDatabase rather than a Database. + +- The second parameter is the index key binding rather than the primary key binding. + +For the `supplierByCityMap`, the `cityKeyBinding` must first be created. This binding was not created in the Basic example because the City field is not a primary key. + +Like the bindings created earlier for keys and values, the `cityKeyBinding` is a SerialBinding. Unlike the bindings created earlier, it is an example of creating a binding for a built-in Java class, String, instead of an application-defined class. Any serializable class may be used. + +For the `shipmentByPartMap` and `shipmentBySupplierMap`, the `partKeyBinding` and `supplierKeyBinding` are used. These were created in the Basic example and used as the primary key bindings for the `partMap` and `supplierMap`. + +The value bindings — `supplierValueBinding` and `shipmentValueBinding` — were also created in the Basic example. + +This illustrates that bindings and formats may and should be reused where appropriate for creating maps and other collections. + +The following getter methods return the stored maps for use by other classes in the example program. Convenience methods for returning entry sets are also included. + +``` c +public class SampleViews +{ + ... + public final StoredMap getShipmentByPartMap() + { + return shipmentByPartMap; + } + + public final StoredMap getShipmentBySupplierMap() + { + return shipmentBySupplierMap; + } + + public final StoredMap getSupplierByCityMap() + { + return supplierByCityMap; + } + + public final StoredEntrySet getShipmentByPartEntrySet() + { + return (StoredEntrySet) shipmentByPartMap.entrySet(); + } + + public final StoredEntrySet getShipmentBySupplierEntrySet() + { + return (StoredEntrySet) shipmentBySupplierMap.entrySet(); + } + + public final StoredEntrySet getSupplierByCityEntrySet() + { + return (StoredEntrySet) supplierByCityMap.entrySet(); + } + ... +} +``` diff --git a/docs-src/guides/collections/intro.md b/docs-src/guides/collections/intro.md new file mode 100644 index 000000000..52ade70ac --- /dev/null +++ b/docs-src/guides/collections/intro.md @@ -0,0 +1,42 @@ +--- +title: "Chapter 1.  Introduction" +api-name: "Chapter 1.  Introduction" +source: docs/collections/tutorial/intro.html +--- +## Chapter 1.  Introduction + +**Table of Contents** + + [Features](intro.md#features) + + [Developing a DB Collections Application](developing.md) + + [Tutorial Introduction](tutorialintroduction.md) + +The DB Java Collections API is a Java framework that extends the well known Java Collections design pattern such that collections can now be stored, updated and queried in a transactional manner. The DB Java Collections API is a layer on top of DB. + +Together the DB Java Collections API and Berkeley DB provide an embedded data management solution with all the benefits of a full transactional storage and the simplicity of a well known Java API. Java programmers who need fast, scalable, transactional data management for their projects can quickly adopt and deploy the DB Java Collections API with confidence. + +This framework was first known as Greybird DB written by Mark Hayes. Mark collaborated with us to permanently incorporate his excellent work into our distribution and to support it as an ongoing part of Berkeley DB and Berkeley DB Java Edition. The repository of source code that remains at SourceForge at version 0.9.0 is considered the last version before incorporation and will remain intact but will not be updated to reflect changes made as part of Berkeley DB or Berkeley DB Java Edition. + +## Features + +Berkeley DB has always provided a Java API which can be roughly described as a map and cursor interface, where the keys and values are represented as byte arrays. This API is a Java (JNI) interface to the C API and it closely modeled the Berkeley DB C API's interface. The DB Java Collections API is a layer on top of that thin JNI mapping of the C API to Berkeley DB. It adds significant new functionality in several ways. + +- An implementation of the Java Collections interfaces (Map, SortedMap, Set, SortedSet, List and Iterator) is provided. + +- Transactions are supported using the conventional Java transaction-per-thread model, where the current transaction is implicitly associated with the current thread. + +- Transaction runner utilities are provided that automatically perform transaction retry and exception handling. + +- Keys and values are represented as Java objects rather than byte arrays. Bindings are used to map between Java objects and the stored byte arrays. + +- The tuple data format is provided as the simplest data representation, and is useful for keys as well as simple compact values. + +- The serial data format is provided for storing arbitrary Java objects without writing custom binding code. Java serialization is extended to store the class descriptions separately, making the data records much more compact than with standard Java serialization. + +- Custom data formats and bindings can be easily added. XML data format and XML bindings could easily be created using this feature, for example. + +- The DB Java Collections API insulates the application from minor differences in the use of the Berkeley DB Data Store, Concurrent Data Store, and Transactional Data Store products. This allows for development with one and deployment with another without significant changes to code. + +Note that the DB Java Collections API does not support caching of programming language objects nor does it keep track of their stored status. This is in contrast to "persistent object" approaches such as those defined by ODMG and JDO (JSR 12). Such approaches have benefits but also require sophisticated object caching. For simplicity the DB Java Collections API treats data objects by value, not by reference, and does not perform object caching of any kind. Since the DB Java Collections API is a thin layer, its reliability and performance characteristics are roughly equivalent to those of Berkeley DB, and database tuning is accomplished in the same way as for any Berkeley DB database. diff --git a/docs-src/guides/collections/moreinfo.md b/docs-src/guides/collections/moreinfo.md new file mode 100644 index 000000000..3bde20623 --- /dev/null +++ b/docs-src/guides/collections/moreinfo.md @@ -0,0 +1,30 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/collections/tutorial/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Getting Started with Berkeley DB for Java + +- Getting Started with Transaction Processing for Java + +- Berkeley DB Getting Started with Replicated Applications for Java + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Javadoc + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/collections/openclasscatalog.md b/docs-src/guides/collections/openclasscatalog.md new file mode 100644 index 000000000..94f1aeb86 --- /dev/null +++ b/docs-src/guides/collections/openclasscatalog.md @@ -0,0 +1,89 @@ +--- +title: "Opening and Closing the Class Catalog" +api-name: "Opening and Closing the Class Catalog" +source: docs/collections/tutorial/openclasscatalog.html +--- +## Opening and Closing the Class Catalog + +This section describes how to open and close the Java class catalog. The class catalog is a specialized database store that contains the Java class descriptions of the serialized objects that are stored in the database. The class descriptions are stored in the catalog rather than storing them redundantly in each database record. A single class catalog per environment must be opened whenever serialized objects will be stored in the database. + +The `SampleDatabase` class is extended to open and close the class catalog. The following additional imports and class members are needed. + +``` c +import com.sleepycat.bind.serial.StoredClassCatalog; +import com.sleepycat.bind.serial.ClassCatalog; +import com.sleepycat.db.Database; +import com.sleepycat.db.DatabaseConfig; +import com.sleepycat.db.DatabaseType; +import com.sleepycat.db.DatabaseException; +import com.sleepycat.db.Environment; +import com.sleepycat.db.EnvironmentConfig; +import java.io.File; +import java.io.FileNotFoundException; + +... + +public class SampleDatabase +{ + private Environment env; + private static final String CLASS_CATALOG = "java_class_catalog"; + ... + private StoredClassCatalog javaCatalog; + ... +} +``` + +While the class catalog is itself a database, it contains metadata for other databases and is therefore treated specially by the DB Java Collections API. The StoredClassCatalog class encapsulates the catalog store and implements this special behavior. + +The following statements open the class catalog by creating a `Database` and a `StoredClassCatalog` object. The catalog database is created if it does not already exist. + +``` c + public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + ... + DatabaseConfig dbConfig = new DatabaseConfig(); + dbConfig.setTransactional(true); + dbConfig.setAllowCreate(true); + dbConfig.setType(DatabaseType.BTREE); + + Database catalogDb = env.openDatabase(null, CLASS_CATALOG, null, + dbConfig); + + javaCatalog = new StoredClassCatalog(catalogDb); + ... + } + ... + public final StoredClassCatalog getClassCatalog() { + return javaCatalog; + } +``` + +The DatabaseConfig class is used to specify configuration parameters when opening a database. The first configuration option specified — `setTransactional()` — is set to true to create a transactional database. While non-transactional databases can also be created, the examples in this tutorial use transactional databases. + +`setAllowCreate()` is set to true to specify that the database will be created if it does not already exist. If this parameter is not specified, an exception will be thrown if the database does not already exist. + +`setDatabaseType()` identifies the database storage type or access method. For opening a catalog database, the `BTREE` type is required. `BTREE` is the most commonly used database type and in this tutorial is used for all databases. + +The first parameter of the `openDatabase()` method is an optional transaction that is used for creating a new database. If null is passed, auto-commit is used when creating a database. + +The second and third parameters of `openDatabase()` specify the filename and database (sub-file) name of the database. The database name is optional and is `null` in this example. + +The last parameter of `openDatabase()` specifies the database configuration object. + +Lastly, the `StoredClassCatalog` object is created to manage the information in the class catalog database. The `StoredClassCatalog` object will be used in the sections following for creating serial bindings. + +The `getClassCatalog` method returns the catalog object for use by other classes in the example program. + +When the environment is closed, the class catalog is closed also. + +``` c + public void close() + throws DatabaseException + { + javaCatalog.close(); + env.close(); + } +``` + +The `StoredClassCatalog.close()` method simply closes the underlying class catalog database and in fact the Database.close() method may be called instead, if desired. It is recommended that you close the catalog database and all other databases, before closing the environment. diff --git a/docs-src/guides/collections/opendatabases.md b/docs-src/guides/collections/opendatabases.md new file mode 100644 index 000000000..e547af95d --- /dev/null +++ b/docs-src/guides/collections/opendatabases.md @@ -0,0 +1,90 @@ +--- +title: "Opening and Closing Databases" +api-name: "Opening and Closing Databases" +source: docs/collections/tutorial/opendatabases.html +--- +## Opening and Closing Databases + +This section describes how to open and close the Part, Supplier and Shipment databases. A *database* is a collection of records, each of which has a key and a value. The keys and values are stored in a selected format, which defines the syntax of the stored data. Two examples of formats are Java serialization format and tuple format. In a given database, all keys have the same format and all values have the same format. + +The `SampleDatabase` class is extended to open and close the three databases. The following additional class members are needed. + +``` c +public class SampleDatabase +{ + ... + private static final String SUPPLIER_STORE = "supplier_store"; + private static final String PART_STORE = "part_store"; + private static final String SHIPMENT_STORE = "shipment_store"; + ... + private Database supplierDb; + private Database partDb; + private Database shipmentDb; + ... +} +``` + +For each database there is a database name constant and a `Database` object. + +The following statements open the three databases by constructing a `Database` object. + +``` c + public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + ... + DatabaseConfig dbConfig = new DatabaseConfig(); + dbConfig.setTransactional(true); + dbConfig.setAllowCreate(true); + dbConfig.setType(DatabaseType.BTREE); + ... + partDb = env.openDatabase(null, PART_STORE, null, dbConfig); + supplierDb = env.openDatabase(null, SUPPLIER_STORE, null, + dbConfig); + shipmentDb = env.openDatabase(null, SHIPMENT_STORE, null, + dbConfig); + ... + } +``` + +The database configuration object that was used previously for opening the catalog database is reused for opening the three databases above. The databases are created if they don't already exist. The parameters of the `openDatabase()` method were described earlier when the class catalog database was opened. + +The following statements close the three databases. + +``` c + public void close() + throws DatabaseException + { + partDb.close(); + supplierDb.close(); + shipmentDb.close(); + javaCatalog.close(); + env.close(); + } +``` + +It is recommended that all databases, including the catalog database, is closed before closing the environment. + +The following getter methods return the databases for use by other classes in the example program. + +``` c +public class SampleDatabase +{ + ... + public final Database getPartDatabase() + { + return partDb; + } + + public final Database getSupplierDatabase() + { + return supplierDb; + } + + public final Database getShipmentDatabase() + { + return shipmentDb; + } + ... +} +``` diff --git a/docs-src/guides/collections/opendbenvironment.md b/docs-src/guides/collections/opendbenvironment.md new file mode 100644 index 000000000..ddbf3aaa4 --- /dev/null +++ b/docs-src/guides/collections/opendbenvironment.md @@ -0,0 +1,83 @@ +--- +title: "Opening and Closing the Database Environment" +api-name: "Opening and Closing the Database Environment" +source: docs/collections/tutorial/opendbenvironment.html +--- +## Opening and Closing the Database Environment + +This section of the tutorial describes how to open and close the database environment. The database environment manages resources (for example, memory, locks and transactions) for any number of databases. A single environment instance is normally used for all databases. + +The `SampleDatabase` class is used to open and close the environment. It will also be used in following sections to open and close the class catalog and other databases. Its constructor is used to open the environment and its `close()` method is used to close the environment. The skeleton for the `SampleDatabase` class follows. + +``` c +import com.sleepycat.db.DatabaseException; +import com.sleepycat.db.Environment; +import com.sleepycat.db.EnvironmentConfig; +import java.io.File; +import java.io.FileNotFoundException; + +public class SampleDatabase +{ + private Environment env; + + public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + } + + public void close() + throws DatabaseException + { + } +} +``` + +The first thing to notice is that the Environment class is in the com.sleepycat.db package, not the com.sleepycat.collections package. The com.sleepycat.db package contains all core Berkeley DB functionality. The com.sleepycat.collections package contains extended functionality that is based on the Java Collections API. The collections package is layered on top of the com.sleepycat.db package. Both packages are needed to create a complete application based on the DB Java Collections API. + +The following statements create an Environment object. + +``` c +public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + System.out.println("Opening environment in: " + homeDirectory); + + EnvironmentConfig envConfig = new EnvironmentConfig(); + envConfig.setTransactional(true); + envConfig.setAllowCreate(true); + envConfig.setInitializeCache(true); + envConfig.setInitializeLocking(true); + + env = new Environment(new File(homeDirectory), envConfig); + } +``` + +The EnvironmentConfig class is used to specify environment configuration parameters. The first configuration option specified — `setTransactional()` — is set to true to create an environment where transactional (and non-transactional) databases may be opened. While non-transactional environments can also be created, the examples in this tutorial use a transactional environment. + +`setAllowCreate()` is set to true to specify that the environment's files will be created if they don't already exist. If this parameter is not specified, an exception will be thrown if the environment does not already exist. A similar parameter will be used later to cause databases to be created if they don't exist. + +When an `Environment` object is constructed, a home directory and the environment configuration object are specified. The home directory is the location of the environment's log files that store all database information. + +The following statement closes the environment. The environment must be closed when database work is completed to free allocated resources and to avoid having to run recovery later. It is recommended that databases are closed before closing the environment. + +``` c + public void close() + throws DatabaseException + { + env.close(); + } +``` + +The following getter method returns the environment for use by other classes in the example program. The environment is used for opening databases and running transactions. + +``` c +public class SampleDatabase +{ + ... + public final Environment getEnvironment() + { + return env; + } + ... +} +``` diff --git a/docs-src/guides/collections/openingforeignkeys.md b/docs-src/guides/collections/openingforeignkeys.md new file mode 100644 index 000000000..571e00fe0 --- /dev/null +++ b/docs-src/guides/collections/openingforeignkeys.md @@ -0,0 +1,156 @@ +--- +title: "More Secondary Key Indices" +api-name: "More Secondary Key Indices" +source: docs/collections/tutorial/openingforeignkeys.html +--- +## More Secondary Key Indices + +This section builds on the prior section describing secondary key indices. Two more secondary key indices are defined for indexing the Shipment record by PartNumber and by SupplierNumber. + +The `SampleDatabase` class is extended to open the Shipment-by-Part and Shipment-by-Supplier secondary key indices. + +``` c +import com.sleepycat.bind.serial.SerialSerialKeyCreator; +import com.sleepycat.db.SecondaryConfig; +import com.sleepycat.db.SecondaryDatabase; +... +public class SampleDatabase +{ + ... + private static final String SHIPMENT_PART_INDEX = + "shipment_part_index"; + private static final String SHIPMENT_SUPPLIER_INDEX = + "shipment_supplier_index"; + ... + private SecondaryDatabase shipmentByPartDb; + private SecondaryDatabase shipmentBySupplierDb; + ... + public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + ... + SecondaryConfig secConfig = new SecondaryConfig(); + secConfig.setTransactional(true); + secConfig.setAllowCreate(true); + secConfig.setType(DatabaseType.BTREE); + secConfig.setSortedDuplicates(true); + ... + secConfig.setKeyCreator( + new ShipmentByPartKeyCreator(javaCatalog, + ShipmentKey.class, + ShipmentData.class, + PartKey.class)); + shipmentByPartDb = env.openSecondaryDatabase(null, + SHIPMENT_PART_INDEX, + null, + shipmentDb, + secConfig); + + secConfig.setKeyCreator( + new ShipmentBySupplierKeyCreator(javaCatalog, + ShipmentKey.class, + ShipmentData.class, + SupplierKey.class)); + shipmentBySupplierDb = env.openSecondaryDatabase(null, + SHIPMENT_SUPPLIER_INDEX, + null, + shipmentDb, + secConfig); + ... + } +} +``` + +The statements in this example are very similar to the statements used in the previous section for opening a secondary index. + +The application-defined `ShipmentByPartKeyCreator` and `ShipmentBySupplierKeyCreator` classes are shown below. They were used above to configure the secondary database objects. + +``` c +public class SampleDatabase +{ +... + private static class ShipmentByPartKeyCreator + extends SerialSerialKeyCreator + { + private ShipmentByPartKeyCreator(ClassCatalog catalog, + Class primaryKeyClass, + Class valueClass, + Class indexKeyClass) + { + super(catalog, primaryKeyClass, valueClass, indexKeyClass); + } + + public Object createSecondaryKey(Object primaryKeyInput, + Object valueInput) + { + ShipmentKey shipmentKey = (ShipmentKey) primaryKeyInput; + return new PartKey(shipmentKey.getPartNumber()); + } + } + + private static class ShipmentBySupplierKeyCreator + extends SerialSerialKeyCreator + { + private ShipmentBySupplierKeyCreator(ClassCatalog catalog, + Class primaryKeyClass, + Class valueClass, + Class indexKeyClass) + { + super(catalog, primaryKeyClass, valueClass, indexKeyClass); + } + + public Object createSecondaryKey(Object primaryKeyInput, + Object valueInput) + { + ShipmentKey shipmentKey = (ShipmentKey) primaryKeyInput; + return new SupplierKey(shipmentKey.getSupplierNumber()); + } + } + ... +} +``` + +The key creator classes above are almost identical to the one defined in the previous section for use with a secondary index. The index key fields are different, of course, but the interesting difference is that the index keys are extracted from the key, not the value, of the Shipment record. This illustrates that an index key may be derived from the primary database record key, value, or both. + +The following getter methods return the secondary database objects for use by other classes in the example program. + +``` c +public class SampleDatabase +{ + ... + public final SecondaryDatabase getShipmentByPartDatabase() + { + return shipmentByPartDb; + } + + public final SecondaryDatabase getShipmentBySupplierDatabase() + { + return shipmentBySupplierDb; + } + ... +} +``` + +The following statements close the secondary databases. + +``` c +public class SampleDatabase +{ + ... + public void close() + throws DatabaseException { + + supplierByCityDb.close(); + shipmentByPartDb.close(); + shipmentBySupplierDb.close(); + partDb.close(); + supplierDb.close(); + shipmentDb.close(); + javaCatalog.close(); + env.close(); + } + ... +} +``` + +Secondary databases must be closed before closing their associated primary database. diff --git a/docs-src/guides/collections/preface.md b/docs-src/guides/collections/preface.md new file mode 100644 index 000000000..bc5e211de --- /dev/null +++ b/docs-src/guides/collections/preface.md @@ -0,0 +1,57 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/collections/tutorial/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +Welcome to the Berkeley DB (DB) Collections API. This document provides a tutorial that introduces the collections API. The goal of this document is to provide you with an efficient mechanism with which you can quickly become efficient with this API. As such, this document is intended for Java developers and senior software architects who are looking for transactionally-protected backing of their Java collections. No prior experience with DB technologies is expected or required. + +This document reflects Berkeley DB 11*g* Release 2, which provides DB library version 11.2.5.3. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Class names are represented in `monospaced font`, as are `method names`. For example: "The `Environment.openDatabase()` method returns a `Database` class object." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALLATION_HOME* directory." + +Program examples are displayed in a monospaced font on a shaded background. For example: + +``` c +import com.sleepycat.db.Environment; +import com.sleepycat.db.EnvironmentConfig; +import java.io.File; + +... + +// Open the environment. Allow it to be created if it does not already +// exist. +Environment myDbEnvironment; +``` + +In situations in this book, programming examples are updated from one chapter to the next in this book. When this occurs, the new code is presented in **`monospaced bold`** font. For example: + +``` c +import com.sleepycat.db.Environment; +import com.sleepycat.db.EnvironmentConfig; +import java.io.File; + +... + +// Open the environment. Allow it to be created if it does not already +// exist. +Environment myDbEnv; +EnvironmentConfig envConfig = new EnvironmentConfig(); +envConfig.setAllowCreate(true); +myDbEnv = new Environment(new File("/export/dbEnv"), envConfig); +``` diff --git a/docs-src/guides/collections/removingredundantvalueclasses.md b/docs-src/guides/collections/removingredundantvalueclasses.md new file mode 100644 index 000000000..eba56a3e5 --- /dev/null +++ b/docs-src/guides/collections/removingredundantvalueclasses.md @@ -0,0 +1,65 @@ +--- +title: "Removing the Redundant Value Classes" +api-name: "Removing the Redundant Value Classes" +source: docs/collections/tutorial/removingredundantvalueclasses.html +--- +## Removing the Redundant Value Classes + +The `PartData`, `SupplierData` and `ShipmentData` classes have been removed in this example, and the `Part`, `Supplier` and `Shipment` entity classes are used in their place. + +The serial formats are created with the entity classes. + +``` c +public class SampleDatabase +{ + ... + public SampleDatabase(String homeDirectory) + throws DatabaseException, FileNotFoundException + { + ... + secConfig.setKeyCreator(new SupplierByCityKeyCreator(javaCatalog, + Supplier.class)); + ... + secConfig.setKeyCreator(new ShipmentByPartKeyCreator(javaCatalog, + Shipment.class)); + ... + secConfig.setKeyCreator(new + ShipmentBySupplierKeyCreator(javaCatalog, + Shipment.class)); + ... + } +} +``` + +The index key creator uses the entity class as well. + +``` c +public class SampleDatabase +{ + ... + + private static class SupplierByCityKeyCreator + extends TupleSerialKeyCreator + { + private SupplierByCityKeyCreator(ClassCatalog catalog, + Class valueClass) + { + super(catalog, valueClass); + } + + public boolean createSecondaryKey(TupleInput primaryKeyInput, + Object valueInput, + TupleOutput indexKeyOutput) + { + Supplier supplier = (Supplier) valueInput; + String city = supplier.getCity(); + if (city != null) { + indexKeyOutput.writeString(supplier.getCity()); + return true; + } else { + return false; + } + } + } +} +``` diff --git a/docs-src/guides/collections/retrievingbyindexkey.md b/docs-src/guides/collections/retrievingbyindexkey.md new file mode 100644 index 000000000..a2254b69c --- /dev/null +++ b/docs-src/guides/collections/retrievingbyindexkey.md @@ -0,0 +1,140 @@ +--- +title: "Retrieving Items by Index Key" +api-name: "Retrieving Items by Index Key" +source: docs/collections/tutorial/retrievingbyindexkey.html +--- +## Retrieving Items by Index Key + +Retrieving information via database index keys can be accomplished using the standard Java collections API, using a collection created from a SecondaryDatabase rather than a Database. However, the standard Java API does not support *duplicate keys*: more than one element in a collection having the same key. All three indices created in the prior section have duplicate keys because of the nature of the city, part number and supplier number index keys. More than one supplier may be in the same city, and more than one shipment may have the same part number or supplier number. This section describes how to use extended methods for stored collections to return all values for a given key. + +Using the standard Java collections API, the Map.get method for a stored collection with duplicate keys will return only the first value for a given key. To obtain all values for a given key, the StoredMap.duplicates method may be called. This returns a Collection of values for the given key. If duplicate keys are not allowed, the returned collection will have at most one value. If the key is not present in the map, an empty collection is returned. + +The `Sample` class is extended to retrieve duplicates for specific index keys that are present in the database. + +``` c +import java.util.Iterator; +... +public class Sample +{ + ... + private SampleViews views; + ... + private class PrintDatabase implements TransactionWorker + { + public void doWork() + throws Exception + { + printEntries("Parts", + views.getPartEntrySet().iterator()); + printEntries("Suppliers", + views.getSupplierEntrySet().iterator()); + printValues("Suppliers for City Paris", + views.getSupplierByCityMap().duplicates( + "Paris").iterator()); + printEntries("Shipments", + views.getShipmentEntrySet().iterator()); + printValues("Shipments for Part P1", + views.getShipmentByPartMap().duplicates( + new PartKey("P1")).iterator()); + printValues("Shipments for Supplier S1", + views.getShipmentBySupplierMap().duplicates( + new + SupplierKey("S1")).iterator()); + } + } + + private void printValues(String label, Iterator iterator) + { + System.out.println("\n--- " + label + " ---"); + while (iterator.hasNext()) + { + System.out.println(iterator.next().toString()); + } + } + ... +} +``` + +The StoredMap.duplicates method is called passing the desired key. The returned value is a standard Java Collection containing the values for the specified key. A standard Java Iterator is then obtained for this collection and all values returned by that iterator are printed. + +Another technique for retrieving duplicates is to use the collection returned by Map.entrySet. When duplicate keys are present, a Map.Entry object will be present in this collection for each duplicate. This collection can then be iterated or a subset can be created from it, all using the standard Java collection API. + +Note that we did not discuss how duplicates keys can be explicitly added or removed in a collection. For index keys, the addition and deletion of duplicate keys happens automatically when records containing the index key are added, updated, or removed. + +While not shown in the example program, it is also possible to create a store with duplicate keys in the same way as an index with duplicate keys — by calling `DatabaseConfig.setSortedDuplicates()` method. In that case, calling Map.put will add duplicate keys. To remove all duplicate keys, call Map.remove. To remove a specific duplicate key, call StoredMap.duplicates and then call Collection.remove using the returned collection. Duplicate values may also be added to this collection using Collection.add. + +The output of the example program is shown below. + +``` c +Adding Suppliers +Adding Parts +Adding Shipments + +--- Parts --- +PartKey: number=P1 +PartData: name=Nut color=Red weight=[12.0 grams] city=London +PartKey: number=P2 +PartData: name=Bolt color=Green weight=[17.0 grams] city=Paris +PartKey: number=P3 +PartData: name=Screw color=Blue weight=[17.0 grams] city=Rome +PartKey: number=P4 +PartData: name=Screw color=Red weight=[14.0 grams] city=London +PartKey: number=P5 +PartData: name=Cam color=Blue weight=[12.0 grams] city=Paris +PartKey: number=P6 +PartData: name=Cog color=Red weight=[19.0 grams] city=London + +--- Suppliers --- +SupplierKey: number=S1 +SupplierData: name=Smith status=20 city=London +SupplierKey: number=S2 +SupplierData: name=Jones status=10 city=Paris +SupplierKey: number=S3 +SupplierData: name=Blake status=30 city=Paris +SupplierKey: number=S4 +SupplierData: name=Clark status=20 city=London +SupplierKey: number=S5 +SupplierData: name=Adams status=30 city=Athens + +--- Suppliers for City Paris --- +SupplierData: name=Jones status=10 city=Paris +SupplierData: name=Blake status=30 city=Paris + +--- Shipments --- +ShipmentKey: supplier=S1 part=P1 +ShipmentData: quantity=300 +ShipmentKey: supplier=S2 part=P1 +ShipmentData: quantity=300 +ShipmentKey: supplier=S1 part=P2 +ShipmentData: quantity=200 +ShipmentKey: supplier=S2 part=P2 +ShipmentData: quantity=400 +ShipmentKey: supplier=S3 part=P2 +ShipmentData: quantity=200 +ShipmentKey: supplier=S4 part=P2 +ShipmentData: quantity=200 +ShipmentKey: supplier=S1 part=P3 +ShipmentData: quantity=400 +ShipmentKey: supplier=S1 part=P4 +ShipmentData: quantity=200 +ShipmentKey: supplier=S4 part=P4 +ShipmentData: quantity=300 +ShipmentKey: supplier=S1 part=P5 +ShipmentData: quantity=100 +ShipmentKey: supplier=S4 part=P5 +ShipmentData: quantity=400 +ShipmentKey: supplier=S1 part=P6 +ShipmentData: quantity=100 + +--- Shipments for Part P1 --- +ShipmentData: quantity=300 +ShipmentData: quantity=300 + +--- Shipments for Supplier S1 --- +ShipmentData: quantity=300 +ShipmentData: quantity=200 +ShipmentData: quantity=400 +ShipmentData: quantity=200 +ShipmentData: quantity=100 +ShipmentData: quantity=100 +``` diff --git a/docs-src/guides/collections/retrievingdatabaseitems.md b/docs-src/guides/collections/retrievingdatabaseitems.md new file mode 100644 index 000000000..fa15d9b64 --- /dev/null +++ b/docs-src/guides/collections/retrievingdatabaseitems.md @@ -0,0 +1,119 @@ +--- +title: "Retrieving Database Items" +api-name: "Retrieving Database Items" +source: docs/collections/tutorial/retrievingdatabaseitems.html +--- +## Retrieving Database Items + +Retrieving information from the database is accomplished via the standard Java collections API. In the example, the Set.iterator method is used to iterate all Map.Entry objects for each store. All standard Java methods for retrieving objects from a collection may be used with the DB Java Collections API. + +The `PrintDatabase.doWork()` method calls `printEntries()` to print the map entries for each database store. It is called via the TransactionRunner class and was outlined in the previous section. + +``` c +import java.util.Iterator; +... +public class Sample +{ + ... + private SampleViews views; + ... + private class PrintDatabase implements TransactionWorker + { + public void doWork() + throws Exception + { + printEntries("Parts", + views.getPartEntrySet().iterator()); + printEntries("Suppliers", + views.getSupplierEntrySet().iterator()); + printEntries("Shipments", + views.getShipmentEntrySet().iterator()); + } + } + ... + + private void printEntries(String label, Iterator iterator) + { + } + ... +} +``` + +The Set of Map.Entry objects for each store is obtained from the `SampleViews` object. This set can also be obtained by calling the Map.entrySet method of a stored map. + +The `printEntries()` prints the map entries for any stored map. The Object.toString method of each key and value is called to obtain a printable representation of each object. + +``` c + private void printEntries(String label, Iterator iterator) + { + System.out.println("\n--- " + label + " ---"); + while (iterator.hasNext()) + { + Map.Entry entry = (Map.Entry) iterator.next(); + System.out.println(entry.getKey().toString()); + System.out.println(entry.getValue().toString()); + } + } +``` + +This is one of a small number of behavioral differences between standard Java collections and stored collections. For a complete list see Using Stored Collections . + +The output of the example program is shown below. + +``` c +Adding Suppliers +Adding Parts +Adding Shipments + +--- Parts --- +PartKey: number=P1 +PartData: name=Nut color=Red weight=[12.0 grams] city=London +PartKey: number=P2 +PartData: name=Bolt color=Green weight=[17.0 grams] city=Paris +PartKey: number=P3 +PartData: name=Screw color=Blue weight=[17.0 grams] city=Rome +PartKey: number=P4 +PartData: name=Screw color=Red weight=[14.0 grams] city=London +PartKey: number=P5 +PartData: name=Cam color=Blue weight=[12.0 grams] city=Paris +PartKey: number=P6 +PartData: name=Cog color=Red weight=[19.0 grams] city=London + +--- Suppliers --- +SupplierKey: number=S1 +SupplierData: name=Smith status=20 city=London +SupplierKey: number=S2 +SupplierData: name=Jones status=10 city=Paris +SupplierKey: number=S3 +SupplierData: name=Blake status=30 city=Paris +SupplierKey: number=S4 +SupplierData: name=Clark status=20 city=London +SupplierKey: number=S5 +SupplierData: name=Adams status=30 city=Athens + +--- Shipments --- +ShipmentKey: supplier=S1 part=P1 +ShipmentData: quantity=300 +ShipmentKey: supplier=S2 part=P1 +ShipmentData: quantity=300 +ShipmentKey: supplier=S1 part=P2 +ShipmentData: quantity=200 +ShipmentKey: supplier=S2 part=P2 +ShipmentData: quantity=400 +ShipmentKey: supplier=S3 part=P2 +ShipmentData: quantity=200 +ShipmentKey: supplier=S4 part=P2 +ShipmentData: quantity=200 +ShipmentKey: supplier=S1 part=P3 +ShipmentData: quantity=400 +ShipmentKey: supplier=S1 part=P4 +ShipmentData: quantity=200 +ShipmentKey: supplier=S4 part=P4 +ShipmentData: quantity=300 +ShipmentKey: supplier=S1 part=P5 +ShipmentData: quantity=100 +ShipmentKey: supplier=S4 part=P5 +ShipmentData: quantity=400 +ShipmentKey: supplier=S1 part=P6 +ShipmentData: quantity=100 +``` diff --git a/docs-src/guides/collections/sortedcollections.md b/docs-src/guides/collections/sortedcollections.md new file mode 100644 index 000000000..dd12bad5c --- /dev/null +++ b/docs-src/guides/collections/sortedcollections.md @@ -0,0 +1,65 @@ +--- +title: "Using Sorted Collections" +api-name: "Using Sorted Collections" +source: docs/collections/tutorial/sortedcollections.html +--- +## Using Sorted Collections + +In general, no changes to the prior example are necessary to use collections having tuple keys. Iteration of elements in a stored collection will be ordered by the sort order of the tuples. + +In addition to using the tuple format, the DatabaseType.BTREE access method must be used when creating the database. DatabaseType.BTREE is used for the databases in all examples. The DatabaseType.HASH access method does not support sorted keys. + +Although not shown in the example, all methods of the SortedMap and SortedSet interfaces may be used with sorted collections. For example, submaps and subsets may be created. + +The output of the example program shows that records are sorted by key value. + +``` c +Adding Suppliers +Adding Parts +Adding Shipments + +--- Parts --- +Part: number=P1 name=Nut color=Red weight=[12.0 grams] city=London +Part: number=P2 name=Bolt color=Green weight=[17.0 grams] city=Paris +Part: number=P3 name=Screw color=Blue weight=[17.0 grams] city=Rome +Part: number=P4 name=Screw color=Red weight=[14.0 grams] city=London +Part: number=P5 name=Cam color=Blue weight=[12.0 grams] city=Paris +Part: number=P6 name=Cog color=Red weight=[19.0 grams] city=London + +--- Suppliers --- +Supplier: number=S1 name=Smith status=20 city=London +Supplier: number=S2 name=Jones status=10 city=Paris +Supplier: number=S3 name=Blake status=30 city=Paris +Supplier: number=S4 name=Clark status=20 city=London +Supplier: number=S5 name=Adams status=30 city=Athens + +--- Suppliers for City Paris --- +Supplier: number=S2 name=Jones status=10 city=Paris +Supplier: number=S3 name=Blake status=30 city=Paris + +--- Shipments --- +Shipment: part=P1 supplier=S1 quantity=300 +Shipment: part=P1 supplier=S2 quantity=300 +Shipment: part=P2 supplier=S1 quantity=200 +Shipment: part=P2 supplier=S2 quantity=400 +Shipment: part=P2 supplier=S3 quantity=200 +Shipment: part=P2 supplier=S4 quantity=200 +Shipment: part=P3 supplier=S1 quantity=400 +Shipment: part=P4 supplier=S1 quantity=200 +Shipment: part=P4 supplier=S4 quantity=300 +Shipment: part=P5 supplier=S1 quantity=100 +Shipment: part=P5 supplier=S4 quantity=400 +Shipment: part=P6 supplier=S1 quantity=100 + +--- Shipments for Part P1 --- +Shipment: part=P1 supplier=S1 quantity=300 +Shipment: part=P1 supplier=S2 quantity=300 + +--- Shipments for Supplier S1 --- +Shipment: part=P1 supplier=S1 quantity=300 +Shipment: part=P2 supplier=S1 quantity=200 +Shipment: part=P3 supplier=S1 quantity=400 +Shipment: part=P4 supplier=S1 quantity=200 +Shipment: part=P5 supplier=S1 quantity=100 +Shipment: part=P6 supplier=S1 quantity=100 +``` diff --git a/docs-src/guides/collections/transientfieldsinbinding.md b/docs-src/guides/collections/transientfieldsinbinding.md new file mode 100644 index 000000000..8c29898f5 --- /dev/null +++ b/docs-src/guides/collections/transientfieldsinbinding.md @@ -0,0 +1,103 @@ +--- +title: "Using Transient Fields in an Entity Binding" +api-name: "Using Transient Fields in an Entity Binding" +source: docs/collections/tutorial/transientfieldsinbinding.html +--- +## Using Transient Fields in an Entity Binding + +The entity bindings from the prior example have been changed in this example to use the entity object both as a value object and an entity object. + +Before, the `entryToObject()` method combined the deserialized value object with the key fields to create a new entity object. Now, this method uses the deserialized object directly as an entity, and initializes its key using the fields read from the key tuple. + +Before, the `objectToData()` method constructed a new value object using information in the entity. Now it simply returns the entity. Nothing needs to be changed in the entity, since the transient key fields won't be serialized. + +``` c +import com.sleepycat.bind.serial.ClassCatalog; +... +public class SampleViews +{ + ... + private static class PartBinding extends TupleSerialBinding + { + private PartBinding(ClassCatalog classCatalog, Class dataClass) + { + super(classCatalog, dataClass); + } + + public Object entryToObject(TupleInput keyInput, Object dataInput) + { + String number = keyInput.readString(); + Part part = (Part) dataInput; + part.setKey(number); + return part; + } + + public void objectToKey(Object object, TupleOutput output) + { + Part part = (Part) object; + output.writeString(part.getNumber()); + } + + public Object objectToData(Object object) + { + return object; + } + } + + private static class SupplierBinding extends TupleSerialBinding + { + private SupplierBinding(ClassCatalog classCatalog, Class dataClass) + { + super(classCatalog, dataClass); + } + + public Object entryToObject(TupleInput keyInput, Object dataInput) + { + String number = keyInput.readString(); + Supplier supplier = (Supplier) dataInput; + supplier.setKey(number); + return supplier; + } + + public void objectToKey(Object object, TupleOutput output) + { + Supplier supplier = (Supplier) object; + output.writeString(supplier.getNumber()); + } + + public Object objectToData(Object object) + { + return object; + } + } + + private static class ShipmentBinding extends TupleSerialBinding + { + private ShipmentBinding(ClassCatalog classCatalog, Class dataClass) + { + super(classCatalog, dataClass); + } + + public Object entryToObject(TupleInput keyInput, Object dataInput) + { + String partNumber = keyInput.readString(); + String supplierNumber = keyInput.readString(); + Shipment shipment = (Shipment) dataInput; + shipment.setKey(partNumber, supplierNumber); + return shipment; + } + + public void objectToKey(Object object, TupleOutput output) + { + Shipment shipment = (Shipment) object; + output.writeString(shipment.getPartNumber()); + output.writeString(shipment.getSupplierNumber()); + } + + public Object objectToData(Object object) + { + return object; + } + } +} +``` diff --git a/docs-src/guides/collections/tuple-serialentitybindings.md b/docs-src/guides/collections/tuple-serialentitybindings.md new file mode 100644 index 000000000..ef899b46e --- /dev/null +++ b/docs-src/guides/collections/tuple-serialentitybindings.md @@ -0,0 +1,103 @@ +--- +title: "Creating Tuple-Serial Entity Bindings" +api-name: "Creating Tuple-Serial Entity Bindings" +source: docs/collections/tutorial/tuple-serialentitybindings.html +--- +## Creating Tuple-Serial Entity Bindings + +In the prior example serial keys and serial values were used, and the SerialSerialBinding base class was used for entity bindings. In this example, tuple keys and serial values are used and therefore the TupleSerialBinding base class is used for entity bindings. + +As with any entity binding, a key and value is converted to an entity in the TupleSerialBinding.entryToObject method, and from an entity to a key and value in the TupleSerialBinding.objectToKey and TupleSerialBinding.objectToData methods. But since keys are stored as tuples, not as serialized objects, key fields are read and written using the TupleInput and TupleOutput parameters. + +The `SampleViews` class contains the modified entity binding classes that were defined in the prior example: `PartBinding`, `SupplierBinding` and `ShipmentBinding`. + +``` c +import com.sleepycat.bind.serial.TupleSerialBinding; +import com.sleepycat.bind.tuple.TupleInput; +import com.sleepycat.bind.tuple.TupleOutput; +... +public class SampleViews +{ + ... + private static class PartBinding extends TupleSerialBinding + { + private PartBinding(ClassCatalog classCatalog, Class dataClass) + { + super(classCatalog, dataClass); + } + public Object entryToObject(TupleInput keyInput, Object dataInput) + { + String number = keyInput.readString(); + PartData data = (PartData) dataInput; + return new Part(number, data.getName(), data.getColor(), + data.getWeight(), data.getCity()); + } + public void objectToKey(Object object, TupleOutput output) + { + Part part = (Part) object; + output.writeString(part.getNumber()); + } + public Object objectToData(Object object) + { + Part part = (Part) object; + return new PartData(part.getName(), part.getColor(), + part.getWeight(), part.getCity()); + } + } + ... + private static class SupplierBinding extends TupleSerialBinding + { + private SupplierBinding(ClassCatalog classCatalog, Class dataClass) + { + super(classCatalog, dataClass); + } + public Object entryToObject(TupleInput keyInput, Object dataInput) + { + String number = keyInput.readString(); + SupplierData data = (SupplierData) dataInput; + return new Supplier(number, data.getName(), + data.getStatus(), data.getCity()); + } + public void objectToKey(Object object, TupleOutput output) + { + Supplier supplier = (Supplier) object; + output.writeString(supplier.getNumber()); + } + public Object objectToData(Object object) + { + Supplier supplier = (Supplier) object; + return new SupplierData(supplier.getName(), + supplier.getStatus(), + supplier.getCity()); + } + } + ... + private static class ShipmentBinding extends TupleSerialBinding + { + private ShipmentBinding(ClassCatalog classCatalog, Class dataClass) + { + super(classCatalog, dataClass); + } + public Object entryToObject(TupleInput keyInput, Object dataInput) + { + String partNumber = keyInput.readString(); + String supplierNumber = keyInput.readString(); + ShipmentData data = (ShipmentData) dataInput; + return new Shipment(partNumber, supplierNumber, + data.getQuantity()); + } + public void objectToKey(Object object, TupleOutput output) + { + Shipment shipment = (Shipment) object; + output.writeString(shipment.getPartNumber()); + output.writeString(shipment.getSupplierNumber()); + } + public Object objectToData(Object object) + { + Shipment shipment = (Shipment) object; + return new ShipmentData(shipment.getQuantity()); + } + } + ... +} +``` diff --git a/docs-src/guides/collections/tuplekeybindings.md b/docs-src/guides/collections/tuplekeybindings.md new file mode 100644 index 000000000..f0fce670e --- /dev/null +++ b/docs-src/guides/collections/tuplekeybindings.md @@ -0,0 +1,114 @@ +--- +title: "Creating Tuple Key Bindings" +api-name: "Creating Tuple Key Bindings" +source: docs/collections/tutorial/tuplekeybindings.html +--- +## Creating Tuple Key Bindings + +Serial bindings were used in prior examples as key bindings, and keys were stored as serialized objects. In this example, a tuple binding is used for each key since keys will be stored as tuples. Because keys are no longer stored as serialized objects, the `PartKey`, `SupplierKey` and `ShipmentKey` classes no longer implement the Serializable interface (this is the only change to these classes and is not shown below). + +For the `Part` key, `Supplier` key, and `Shipment` key, the `SampleViews` class was changed in this example to create a custom TupleBinding instead of a SerialBinding. The custom tuple key binding classes are defined further below. + +``` c +import com.sleepycat.bind.tuple.TupleBinding; +... +public class SampleViews +{ + ... + public SampleViews(SampleDatabase db) + { + ... + ClassCatalog catalog = db.getClassCatalog(); + EntryBinding partKeyBinding = + new PartKeyBinding(); + EntityBinding partDataBinding = + new PartBinding(catalog, PartData.class); + EntryBinding supplierKeyBinding = + new SupplierKeyBinding(); + EntityBinding supplierDataBinding = + new SupplierBinding(catalog, SupplierData.class); + EntryBinding shipmentKeyBinding = + new ShipmentKeyBinding(); + EntityBinding shipmentDataBinding = + new ShipmentBinding(catalog, ShipmentData.class); + EntryBinding cityKeyBinding = + TupleBinding.getPrimitiveBinding(String.class); + ... + } +} +``` + +For the City key, however, a custom binding class is not needed because the key class is a primitive Java type, String. For any primitive Java type, a tuple binding may be created using the TupleBinding.getPrimitiveBinding static method. + +The custom key binding classes, `PartKeyBinding`, `SupplierKeyBinding` and `ShipmentKeyBinding`, are defined by extending the TupleBinding class. The TupleBinding abstract class implements the EntryBinding interface, and is used for one-to-one bindings between tuples and objects. Each binding class implements two methods for converting between tuples and objects. Tuple fields are read using the TupleInput parameter and written using the TupleOutput parameter. + +``` c +import com.sleepycat.bind.tuple.TupleBinding; +import com.sleepycat.bind.tuple.TupleInput; +import com.sleepycat.bind.tuple.TupleOutput; +... +public class SampleViews +{ +... + + private static class PartKeyBinding extends TupleBinding + { + private PartKeyBinding() + { + } + + public Object entryToObject(TupleInput input) + { + String number = input.readString(); + return new PartKey(number); + } + + public void objectToEntry(Object object, TupleOutput output) + { + PartKey key = (PartKey) object; + output.writeString(key.getNumber()); + } + } + ... + private static class SupplierKeyBinding extends TupleBinding + { + private SupplierKeyBinding() + { + } + + public Object entryToObject(TupleInput input) + { + String number = input.readString(); + return new SupplierKey(number); + } + + public void objectToEntry(Object object, TupleOutput output) + { + SupplierKey key = (SupplierKey) object; + output.writeString(key.getNumber()); + } + } + ... + private static class ShipmentKeyBinding extends TupleBinding + { + private ShipmentKeyBinding() + { + } + + public Object entryToObject(TupleInput input) + { + String partNumber = input.readString(); + String supplierNumber = input.readString(); + return new ShipmentKey(partNumber, supplierNumber); + } + + public void objectToEntry(Object object, TupleOutput output) + { + ShipmentKey key = (ShipmentKey) object; + output.writeString(key.getPartNumber()); + output.writeString(key.getSupplierNumber()); + } + } + ... +} +``` diff --git a/docs-src/guides/collections/tupleswithkeycreators.md b/docs-src/guides/collections/tupleswithkeycreators.md new file mode 100644 index 000000000..ceb43e6a4 --- /dev/null +++ b/docs-src/guides/collections/tupleswithkeycreators.md @@ -0,0 +1,94 @@ +--- +title: "Using Tuples with Key Creators" +api-name: "Using Tuples with Key Creators" +source: docs/collections/tutorial/tupleswithkeycreators.html +--- +## Using Tuples with Key Creators + +Key creators were used in prior examples to extract index keys from value objects. The keys were returned as deserialized key objects, since the serial format was used for keys. In this example, the tuple format is used for keys and the key creators return keys by writing information to a tuple. The differences between this example and the prior example are: + +- The TupleSerialKeyCreator base class is used instead of the SerialSerialKeyCreator base class. + +- For all key input and output parameters, the TupleInput and TupleOutput classes are used instead of Object (representing a deserialized object). + +- Instead of returning a key output object, these methods call tuple write methods such as TupleOutput.writeString. + +In addition to writing key tuples, the `ShipmentByPartKeyCreator` and `ShipmentBySupplierKeyCreator` classes also read the key tuple of the primary key. This is because they extract the index key from fields in the Shipment's primary key. Instead of calling getter methods on the `ShipmentKey` object, as in prior examples, these methods call TupleInput.readString. The `ShipmentKey` consists of two string fields that are read in sequence. + +The modified key creators are shown below: `SupplierByCityKeyCreator`, `ShipmentByPartKeyCreator` and `ShipmentBySupplierKeyCreator`. + +``` c +import com.sleepycat.bind.serial.TupleSerialKeyCreator; +import com.sleepycat.bind.tuple.TupleInput; +import com.sleepycat.bind.tuple.TupleOutput; +... +public class SampleDatabase +{ + ... + private static class SupplierByCityKeyCreator + extends TupleSerialKeyCreator + { + private SupplierByCityKeyCreator(ClassCatalog catalog, + Class valueClass) + { + super(catalog, valueClass); + } + + public boolean createSecondaryKey(TupleInput primaryKeyInput, + Object valueInput, + TupleOutput indexKeyOutput) + { + SupplierData supplierData = (SupplierData) valueInput; + String city = supplierData.getCity(); + if (city != null) { + indexKeyOutput.writeString(supplierData.getCity()); + return true; + } else { + return false; + } + } + } + + private static class ShipmentByPartKeyCreator + extends TupleSerialKeyCreator + { + private ShipmentByPartKeyCreator(ClassCatalog catalog, + Class valueClass) + { + super(catalog, valueClass); + } + + public boolean createSecondaryKey(TupleInput primaryKeyInput, + Object valueInput, + TupleOutput indexKeyOutput) + { + String partNumber = primaryKeyInput.readString(); + // don't bother reading the supplierNumber + indexKeyOutput.writeString(partNumber); + return true; + } + } + + private static class ShipmentBySupplierKeyCreator + extends TupleSerialKeyCreator + { + private ShipmentBySupplierKeyCreator(ClassCatalog catalog, + Class valueClass) + { + super(catalog, valueClass); + } + + public boolean createSecondaryKey(TupleInput primaryKeyInput, + Object valueInput, + TupleOutput indexKeyOutput) + { + primaryKeyInput.readString(); // skip the partNumber + String supplierNumber = primaryKeyInput.readString(); + indexKeyOutput.writeString(supplierNumber); + return true; + } + } + ... +} + +``` diff --git a/docs-src/guides/collections/tutorialintroduction.md b/docs-src/guides/collections/tutorialintroduction.md new file mode 100644 index 000000000..3fb5cecb1 --- /dev/null +++ b/docs-src/guides/collections/tutorialintroduction.md @@ -0,0 +1,80 @@ +--- +title: "Tutorial Introduction" +api-name: "Tutorial Introduction" +source: docs/collections/tutorial/tutorialintroduction.html +--- +## Tutorial Introduction + +Most of the remainder of this document illustrates the use of the DB Java Collections API by presenting a tutorial that describes usage of the API. This tutorial builds a shipment database, a familiar example from classic database texts. + +The examples illustrate the following concepts of the DB Java Collections API: + +- Object-to-data *bindings* + +- The database *environment* + +- *Databases* that contain key/value records + +- *Secondary index* databases that contain index keys + +- Java *collections* for accessing databases and indices + +- *Transactions* used to commit or undo database changes + +The examples build on each other, but at the same time the source code for each example stands alone. + +- The Basic Program + +- Using Secondary Indices + +- Using Entity Classes + +- Using Tuples + +- Using Serializable Entities + +The shipment database consists of three database stores: the part store, the supplier store, and the shipment store. Each store contains a number of records, and each record consists of a key and a value. + +| Store | Key | Value | +|----------|------------------------------|---------------------------| +| Part | Part Number | Name, Color, Weight, City | +| Supplier | Supplier Number | Name, Status, City | +| Shipment | Part Number, Supplier Number | Quantity | + +In the example programs, Java classes containing the fields above are defined for the key and value of each store: `PartKey`, `PartData`, `SupplierKey`, `SupplierData`, `ShipmentKey` and `ShipmentData`. In addition, because the Part's Weight field is itself composed of two fields — the weight value and the unit of measure — it is represented by a separate `Weight` class. These classes will be defined in the first example program. + +In general the DB Java Collections API uses bindings to describe how Java objects are stored. A binding defines the stored data syntax and the mapping between a Java object and the stored data. The example programs show how to create different types of bindings, and explains the characteristics of each type. + +The following tables show the record values that are used in all the example programs in the tutorial. + +| Number | Name | Color | Weight | City | +|--------|-------|-------|------------|--------| +| P1 | Nut | Red | 12.0 grams | London | +| P2 | Bolt | Green | 17.0 grams | Paris | +| P3 | Screw | Blue | 17.0 grams | Rome | +| P4 | Screw | Red | 14.0 grams | London | +| P5 | Cam | Blue | 12.0 grams | Paris | +| P6 | Cog | Red | 19.0 grams | London | + +| Number | Name | Status | City | +|--------|-------|--------|--------| +| S1 | Smith | 20 | London | +| S2 | Jones | 10 | Paris | +| S3 | Blake | 30 | Paris | +| S4 | Clark | 20 | London | +| S5 | Adams | 30 | Athens | + +| Part Number | Supplier Number | Quantity | +|-------------|-----------------|----------| +| P1 | S1 | 300 | +| P1 | S2 | 300 | +| P2 | S1 | 200 | +| P2 | S2 | 400 | +| P2 | S3 | 200 | +| P2 | S4 | 200 | +| P3 | S1 | 400 | +| P4 | S1 | 200 | +| P4 | S4 | 300 | +| P5 | S1 | 100 | +| P5 | S4 | 400 | +| P6 | S1 | 100 | diff --git a/docs-src/guides/collections/usingtransactions.md b/docs-src/guides/collections/usingtransactions.md new file mode 100644 index 000000000..bfee774f5 --- /dev/null +++ b/docs-src/guides/collections/usingtransactions.md @@ -0,0 +1,65 @@ +--- +title: "Using Transactions" +api-name: "Using Transactions" +source: docs/collections/tutorial/usingtransactions.html +--- +## Using Transactions + +DB transactional applications have standard transactional characteristics: recoverability, atomicity and integrity (this is sometimes also referred to generically as *ACID properties*). The DB Java Collections API provides these transactional capabilities using a *transaction-per-thread* model. Once a transaction is begun, it is implicitly associated with the current thread until it is committed or aborted. This model is used for the following reasons. + +- The transaction-per-thread model is commonly used in other Java APIs such as J2EE. + +- Since the Java collections API is used for data access, there is no way to pass a transaction object to methods such as Map.put. + +The DB Java Collections API provides two transaction APIs. The lower-level API is the CurrentTransaction class. It provides a way to get the transaction for the current thread, and to begin, commit and abort transactions. It also provides access to the Berkeley DB core API Transaction object. With CurrentTransaction, just as in the com.sleepycat.db API, the application is responsible for beginning, committing and aborting transactions, and for handling deadlock exceptions and retrying operations. This API may be needed for some applications, but it is not used in the example. + +The example uses the higher-level TransactionRunner and TransactionWorker APIs, which are build on top of CurrentTransaction. `TransactionRunner.run()` automatically begins a transaction and then calls the `TransactionWorker.doWork()` method, which is implemented by the application. + +The `TransactionRunner.run()` method automatically detects deadlock exceptions and performs retries by repeatedly calling the `TransactionWorker.doWork()` method until the operation succeeds or the maximum retry count is reached. If the maximum retry count is reached or if another exception (other than DeadlockException) is thrown by `TransactionWorker.doWork()`, then the transaction will be automatically aborted. Otherwise, the transaction will be automatically committed. + +Using this high-level API, if `TransactionRunner.run()` throws an exception, the application can assume that the operation failed and the transaction was aborted; otherwise, when an exception is not thrown, the application can assume the operation succeeded and the transaction was committed. + +The `Sample.run()` method creates a `TransactionRunner` object and calls its `run()` method. + +``` c +import com.sleepycat.collections.TransactionRunner; +import com.sleepycat.collections.TransactionWorker; +... +public class Sample +{ + private SampleDatabase db; + ... + private void run() + throws Exception + { + TransactionRunner runner = + new TransactionRunner(db.getEnvironment()); + runner.run(new PopulateDatabase()); + runner.run(new PrintDatabase()); + } + ... + private class PopulateDatabase implements TransactionWorker + { + public void doWork() + throws Exception + { + } + } + + private class PrintDatabase implements TransactionWorker + { + public void doWork() + throws Exception + { + } + } +} +``` + +The `run()` method is called by `main()` and was outlined in the previous section. It first creates a `TransactionRunner`, passing the database environment to its constructor. + +It then calls `TransactionRunner.run()` to execute two transactions, passing instances of the application-defined `PopulateDatabase` and `PrintDatabase` nested classes. These classes implement the `TransactionWorker.doWork()` method and will be fully described in the next two sections. + +For each call to `TransactionRunner.run()`, a separate transaction will be performed. The use of two transactions in the example — one for populating the database and another for printing its contents — is arbitrary. A real-life application should be designed to create transactions for each group of operations that should have ACID properties, while also taking into account the impact of transactions on performance. + +The advantage of using `TransactionRunner` is that deadlock retries and transaction begin, commit and abort are handled automatically. However, a `TransactionWorker` class must be implemented for each type of transaction. If desired, anonymous inner classes can be used to implement the `TransactionWorker` interface. diff --git a/docs-src/guides/gsg/CoreCursorUsage.md b/docs-src/guides/gsg/CoreCursorUsage.md new file mode 100644 index 000000000..653fc98ea --- /dev/null +++ b/docs-src/guides/gsg/CoreCursorUsage.md @@ -0,0 +1,243 @@ +--- +title: "Cursor Example" +api-name: "Cursor Example" +source: docs/gsg/C/CoreCursorUsage.html +--- +## Cursor Example + +In Database Usage Example we wrote an application that loaded two databases with vendor and inventory information. In this example, we will write an application to display all of the items in the inventory database. As a part of showing any given inventory item, we will look up the vendor who can provide the item and show the vendor's contact information. + +Specifically, the `example_database_read` application does the following: + +1. Opens the the inventory and vendor databases that were created by our `example_database_load` application. See example_database_load for information on how that application creates the databases and writes data to them. + +2. Obtains a cursor from the inventory database. + +3. Steps through the inventory database, displaying each record as it goes. + +4. Gets the name of the vendor for that inventory item from the inventory record. + +5. Uses the vendor name to look up the vendor record in the vendor database. + +6. Displays the vendor record. + +Remember that you can find the complete implementation of this application in: + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +**Example 4.1 example_database_read** + +To begin, we include the necessary header files and perform our forward declarations. + +``` c +/* File: example_database_read.c */ +/* gettingstarted_common.h includes db.h for us */ +#include "gettingstarted_common.h" + +/* Forward declarations */ +char * show_inventory_item(void *); +int show_all_records(STOCK_DBS *); +int show_records(STOCK_DBS *, char *); +int show_vendor_record(char *, DB *); +``` + +Next we write our `main()` function. Note that it is somewhat unnecessarily complicated here because we will be extending it in the next chapter to perform inventory item lookups. + +``` c +/* + * Displays all inventory items and the associated vendor record. + */ +int +main(int argc, char *argv[]) +{ + STOCK_DBS my_stock; + int ret; + + /* Initialize the STOCK_DBS struct */ + initialize_stockdbs(&my_stock); + + /* + * Parse the command line arguments here and determine + * the location of the database files. This step is + * omitted for brevity. + */ + + /* + * Identify the files that will hold our databases + * This function uses information obtained from the + * command line to identify the directory in which + * the database files reside. + */ + set_db_filenames(&my_stock); + + /* Open all databases */ + ret = databases_setup(&my_stock, "example_database_read", stderr); + if (ret != 0) { + fprintf(stderr, "Error opening databases\n"); + databases_close(&my_stock); + return (ret); + } + + ret = show_all_records(&my_stock); + + /* close our databases */ + databases_close(&my_stock); + return (ret); +} +``` + +Next we need to write the `show_all_records()` function. This function takes a `STOCK_DBS` structure and displays all of the inventory records found in the inventory database. Once it shows the inventory record, it retrieves the vendor's name from that record and uses it to look up and display the appropriate vendor record: + +``` c +int show_all_records(STOCK_DBS *my_stock) +{ + DBC *cursorp; + DBT key, data; + char *the_vendor; + int exit_value, ret; + + /* Initialize our DBTs. */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + /* Get a cursor to the itemname db */ + my_stock->inventory_dbp->cursor(my_stock->inventory_dbp, NULL, + &cursorp, 0); + + /* + * Iterate over the inventory database, from the first record + * to the last, displaying each in turn. + */ + exit_value = 0; + while ((ret = + cursorp->get(cursorp, &key, &data, DB_NEXT)) + == 0) + { + the_vendor = show_inventory_item(data.data); + ret = show_vendor_record(the_vendor, my_stock->vendor_dbp); + if (ret) { + exit_value = ret; + break; + } + } + + /* Close the cursor */ + cursorp->close(cursorp); + return(exit_value); +} +``` + +The `show_inventory_item()` simply extracts the inventory information from the record data and displays it. It then returns the vendor's name. Note that in order to extract the inventory information, we have to unpack it from the data buffer. How we do this is entirely dependent on how we packed the buffer in the first place. For more information, see the `load_inventory_database()` function implementation in example_database_load. + +``` c +/* + * Shows an inventory item. + */ +char * +show_inventory_item(void *vBuf) +{ + float price; + int buf_pos, quantity; + char *category, *name, *sku, *vendor_name; + char *buf = (char *)vBuf; + + /* Get the price. */ + price = *((float *)buf); + buf_pos = sizeof(float); + + /* Get the quantity. */ + quantity = *((int *)(buf + buf_pos)); + buf_pos += sizeof(int); + + /* Get the inventory item's name */ + name = buf + buf_pos; + buf_pos += strlen(name) + 1; + + /* Get the inventory item's sku */ + sku = buf + buf_pos; + buf_pos += strlen(sku) + 1; + + /* + * Get the category (fruits, vegetables, desserts) that this + * item belongs to. + */ + category = buf + buf_pos; + buf_pos += strlen(category) + 1; + + /* Get the vendor's name */ + vendor_name = buf + buf_pos; + + /* Display all this information */ + printf("name: %s\n", name); + printf("\tSKU: %s\n", sku); + printf("\tCategory: %s\n", category); + printf("\tPrice: %.2f\n", price); + printf("\tQuantity: %i\n", quantity); + printf("\tVendor:\n"); + + /* Return the vendor's name */ + return(vendor_name); +} +``` + +Having returned the vendor's name, we can now use it to look up and display the appropriate vendor record. In this case we do not need to use a cursor to display the vendor record. Using a cursor here complicates our code slightly for no good gain. Instead, we simply perform a `get()` directly against the vendor database. + +``` c +/* + * Shows a vendor record. Each vendor record is an instance of + * a vendor structure. See load_vendor_database() in + * example_database_load for how this structure was originally + * put into the database. + */ +int +show_vendor_record(char *vendor_name, DB *vendor_dbp) +{ + DBT key, data; + VENDOR my_vendor; + int ret; + + /* Zero our DBTs */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + /* Set the search key to the vendor's name */ + key.data = vendor_name; + key.size = strlen(vendor_name) + 1; + + /* + * Make sure we use the memory we set aside for the VENDOR + * structure rather than the memory that DB allocates. + * Some systems may require structures to be aligned in memory + * in a specific way, and DB may not get it right. + */ + + data.data = &my_vendor; + data.ulen = sizeof(VENDOR); + data.flags = DB_DBT_USERMEM; + + /* Get the record */ + ret = vendor_dbp->get(vendor_dbp, 0, &key, &data, 0); + if (ret != 0) { + vendor_dbp->err(vendor_dbp, ret, + "Error searching for vendor: '%s'", vendor_name); + return(ret); + } else { + printf("\t\t%s\n", my_vendor.name); + printf("\t\t%s\n", my_vendor.street); + printf("\t\t%s, %s\n", my_vendor.city, my_vendor.state); + printf("\t\t%s\n\n", my_vendor.zipcode); + printf("\t\t%s\n\n", my_vendor.phone_number); + printf("\t\tContact: %s\n", my_vendor.sales_rep); + printf("\t\t%s\n", my_vendor.sales_rep_phone); + } + return(0); +} +``` + + + +That completes the implementation of `example_database_read()`. In the next chapter, we will extend this application to make use of a secondary database so that we can query the inventory database for a specific inventory item. diff --git a/docs-src/guides/gsg/CoreDBAdmin.md b/docs-src/guides/gsg/CoreDBAdmin.md new file mode 100644 index 000000000..e3c51cb03 --- /dev/null +++ b/docs-src/guides/gsg/CoreDBAdmin.md @@ -0,0 +1,66 @@ +--- +title: "Administrative Methods" +api-name: "Administrative Methods" +source: docs/gsg/C/CoreDBAdmin.html +--- +## Administrative Methods + +The following `DB` methods may be useful to you when managing DB databases: + +- `DB->get_open_flags()` + + Returns the current open flags. It is an error to use this method on an unopened database. + + ``` c + #include + ... + DB *dbp; + u_int32_t open_flags; + + /* Database open and subsequent operations omitted for clarity */ + + dbp->get_open_flags(dbp, &open_flags); + ``` + +- `DB->remove()` + + Removes the specified database. If no value is given for the *`database`* parameter, then the entire file referenced by this method is removed. + + Never remove a database that has handles opened for it. Never remove a file that contains databases with opened handles. + + ``` c + #include + ... + DB *dbp; + + /* Database handle creation omitted for clarity */ + + dbp->remove(dbp, /* Database pointer */ + "mydb.db", /* Database file to remove */ + NULL, /* Database to remove. This is + * NULL so the entire file is + * removed. */ + 0); /* Flags. None used. */ + ``` + +- `DB->rename()` + + Renames the specified database. If no value is given for the *`database`* parameter, then the entire file referenced by this method is renamed. + + Never rename a database that has handles opened for it. Never rename a file that contains databases with opened handles. + + ``` c + #include + ... + DB *dbp; + + /* Database handle creation omitted for clarity */ + + dbp->rename(dbp, /* Database pointer */ + "mydb.db", /* Database file to rename */ + NULL, /* Database to rename. This is + * NULL so the entire file is + * renamed. */ + "newdb.db", /* New database file name */ + 0); /* Flags. None used. */ + ``` diff --git a/docs-src/guides/gsg/CoreDbUsage.md b/docs-src/guides/gsg/CoreDbUsage.md new file mode 100644 index 000000000..6d52cdd36 --- /dev/null +++ b/docs-src/guides/gsg/CoreDbUsage.md @@ -0,0 +1,216 @@ +--- +title: "Database Example" +api-name: "Database Example" +source: docs/gsg/C/CoreDbUsage.html +--- +## Database Example + +Throughout this book we will build a couple of applications that load and retrieve inventory data from DB databases. While we are not yet ready to begin reading from or writing to our databases, we can at least create some important structures and functions that we will use to manage our databases. + +Note that subsequent examples in this book will build on this code to perform the more interesting work of writing to and reading from the databases. + +Note that you can find the complete implementation of these functions in: + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +**Example 2.1 The stock_db Structure** + +To begin, we create a structure that we will use to hold all our database pointers and database names: + +``` c +/* File: gettingstarted_common.h */ +#include + +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; + +/* Function prototypes */ +int databases_setup(STOCK_DBS *, const char *, FILE *); +int databases_close(STOCK_DBS *); +void initialize_stockdbs(STOCK_DBS *); +int open_database(DB **, const char *, const char *, + FILE *); +void set_db_filenames(STOCK_DBS *my_stock); +``` + + +**Example 2.2 The stock_db Utility Functions** + +Before continuing, we want some utility functions that we use to make sure the stock_db structure is in a sane state before using it. One is a simple function that initializes all the structure's pointers to a useful default.The second is more interesting in that it is used to place a common path on all our database names so that we can explicitly identify where all the database files should reside. + +``` c +/* File: gettingstarted_common.c */ +#include "gettingstarted_common.h" + +/* Initializes the STOCK_DBS struct.*/ +void +initialize_stockdbs(STOCK_DBS *my_stock) +{ + my_stock->db_home_dir = DEFAULT_HOMEDIR; + my_stock->inventory_dbp = NULL; + my_stock->vendor_dbp = NULL; + + my_stock->inventory_db_name = NULL; + my_stock->vendor_db_name = NULL; +} + +/* Identify all the files that will hold our databases. */ +void +set_db_filenames(STOCK_DBS *my_stock) +{ + size_t size; + + /* Create the Inventory DB file name */ + size = strlen(my_stock->db_home_dir) + strlen(INVENTORYDB) + 1; + my_stock->inventory_db_name = malloc(size); + snprintf(my_stock->inventory_db_name, size, "%s%s", + my_stock->db_home_dir, INVENTORYDB); + + /* Create the Vendor DB file name */ + size = strlen(my_stock->db_home_dir) + strlen(VENDORDB) + 1; + my_stock->vendor_db_name = malloc(size); + snprintf(my_stock->vendor_db_name, size, "%s%s", + my_stock->db_home_dir, VENDORDB); +} +``` + + +**Example 2.3 open_database() Function** + +We are opening multiple databases, and we are opening those databases using identical flags and error reporting settings. It is therefore worthwhile to create a function that performs this operation for us: + +``` c +/* File: gettingstarted_common.c */ + +/* Opens a database */ +int +open_database(DB **dbpp, /* The DB handle that we are opening */ + const char *file_name, /* The file in which the db lives */ + const char *program_name, /* Name of the program calling this + * function */ + FILE *error_file_pointer) /* File where we want error messages + sent */ +{ + DB *dbp; /* For convenience */ + u_int32_t open_flags; + int ret; + + /* Initialize the DB handle */ + ret = db_create(&dbp, NULL, 0); + if (ret != 0) { + fprintf(error_file_pointer, "%s: %s\n", program_name, + db_strerror(ret)); + return(ret); + } + + /* Point to the memory malloc'd by db_create() */ + *dbpp = dbp; + + /* Set up error handling for this database */ + dbp->set_errfile(dbp, error_file_pointer); + dbp->set_errpfx(dbp, program_name); + + /* Set the open flags */ + open_flags = DB_CREATE; + + /* Now open the database */ + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name (unneeded) */ + DB_BTREE, /* Database type (using btree) */ + open_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed.", file_name); + return(ret); + } + + return (0); +} +``` + + +**Example 2.4 The databases_setup() Function** + +Now that we have our `open_database()` function, we can use it to open a database. We now create a simple function that will open all our databases for us. + +``` c +/* opens all databases */ +int +databases_setup(STOCK_DBS *my_stock, const char *program_name, + FILE *error_file_pointer) +{ + int ret; + + /* Open the vendor database */ + ret = open_database(&(my_stock->vendor_dbp), + my_stock->vendor_db_name, + program_name, error_file_pointer); + if (ret != 0) + /* + * Error reporting is handled in open_database() so just return + * the return code here. + */ + return (ret); + + /* Open the inventory database */ + ret = open_database(&(my_stock->inventory_dbp), + my_stock->inventory_db_name, + program_name, error_file_pointer); + if (ret != 0) + /* + * Error reporting is handled in open_database() so just return + * the return code here. + */ + return (ret); + + printf("databases opened successfully\n"); + return (0); +} +``` + + +**Example 2.5 The databases_close() Function** + +Finally, it is useful to have a function that can close all our databases for us: + +``` c +/* Closes all the databases. */ +int +databases_close(STOCK_DBS *my_stock) +{ + int ret; + /* + * Note that closing a database automatically flushes its cached data + * to disk, so no sync is required here. + */ + + if (my_stock->inventory_dbp != NULL) { + ret = my_stock->inventory_dbp->close(my_stock->inventory_dbp, 0); + if (ret != 0) + fprintf(stderr, "Inventory database close failed: %s\n", + db_strerror(ret)); + } + + if (my_stock->vendor_dbp != NULL) { + ret = my_stock->vendor_dbp->close(my_stock->vendor_dbp, 0); + if (ret != 0) + fprintf(stderr, "Vendor database close failed: %s\n", + db_strerror(ret)); + } + + printf("databases closed.\n"); + return (0); +} +``` diff --git a/docs-src/guides/gsg/CoreEnvUsage.md b/docs-src/guides/gsg/CoreEnvUsage.md new file mode 100644 index 000000000..1ff74ec43 --- /dev/null +++ b/docs-src/guides/gsg/CoreEnvUsage.md @@ -0,0 +1,93 @@ +--- +title: "Managing Databases in Environments" +api-name: "Managing Databases in Environments" +source: docs/gsg/C/CoreEnvUsage.html +--- +## Managing Databases in Environments + +In Environments, we introduced environments. While environments are not used in the example built in this book, they are so commonly used for a wide class of DB applications that it is necessary to show their basic usage, if only from a completeness perspective. + +To use an environment, you must first create the environment handle using , and then open it. At open time, you must identify the directory in which it resides. This directory must exist prior to the open attempt. You can also identify open properties, such as whether the environment can be created if it does not already exist. + +You will also need to initialize the in-memory cache when you open your environment. + +For example, to create an environment handle and open an environment: + +``` c +#include +... +DB_ENV *myEnv; /* Env structure handle */ +DB *dbp; /* DB structure handle */ +u_int32_t db_flags; /* database open flags */ +u_int32_t env_flags; /* env open flags */ +int ret; /* function return value */ + +/* + Create an environment object and initialize it for error + reporting. +*/ +ret = db_env_create(&myEnv, 0); +if (ret != 0) { + fprintf(stderr, "Error creating env handle: %s\n", db_strerror(ret)); + return -1; +} + +/* Open the environment. */ +env_flags = DB_CREATE | /* If the environment does not exist, + * create it. */ + DB_INIT_MPOOL; /* Initialize the in-memory cache. */ + +ret = myEnv->open(myEnv, /* DB_ENV ptr */ + "/export1/testEnv", /* env home directory */ + env_flags, /* Open flags */ + 0); /* File mode (default) */ +if (ret != 0) { + fprintf(stderr, "Environment open failed: %s", db_strerror(ret)); + return -1; +} +``` + +Once an environment is opened, you can open databases in it. Note that by default databases are stored in the environment's home directory, or relative to that directory if you provide any sort of a path in the database's file name: + +``` c +/* + * Initialize the DB structure. Pass the pointer + * to the environment in which this DB is opened. + */ +ret = db_create(&dbp, myEnv, 0); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Database open flags */ +db_flags = DB_CREATE; /* If the database does not exist, + * create it.*/ + +/* open the database */ +ret = dbp->open(dbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_db.db", /* On-disk file that holds the database. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + db_flags, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} +``` + +When you are done with an environment, you must close it. It is recommended that before closing an environment, you close any open databases. + +``` c +/* +* Close the database and environment +*/ + +if (dbp != NULL) { + dbp->close(dbp, 0); +} + +if (myEnv != NULL) { + myEnv->close(myEnv, 0); +} +``` diff --git a/docs-src/guides/gsg/Cursors.md b/docs-src/guides/gsg/Cursors.md new file mode 100644 index 000000000..b03e43bc5 --- /dev/null +++ b/docs-src/guides/gsg/Cursors.md @@ -0,0 +1,67 @@ +--- +title: "Chapter 4. Using Cursors" +api-name: "Chapter 4. Using Cursors" +source: docs/gsg/C/Cursors.html +--- +## Chapter 4. Using Cursors + +**Table of Contents** + + [Opening and Closing Cursors](Cursors.md#openCursor) + + [Getting Records Using the Cursor](Positioning.md) + + [Searching for Records](Positioning.md#cursorsearch) + + [Working with Duplicate Records](Positioning.md#getdups) + + [Putting Records Using Cursors](PutEntryWCursor.md) + + [Deleting Records Using Cursors](DeleteEntryWCursor.md) + + [Replacing Records Using Cursors](ReplacingEntryWCursor.md) + + [Cursor Example](CoreCursorUsage.md) + +Cursors provide a mechanism by which you can iterate over the records in a database. Using cursors, you can get, put, and delete database records. If a database allows duplicate records, then cursors are the easiest way that you can access anything other than the first record for a given key. + +This chapter introduces cursors. It explains how to open and close them, how to use them to modify databases, and how to use them with duplicate records. + +## Opening and Closing Cursors + +Cursors are managed using the `DBC` structure. To use a cursor, you must open it using the `DB->cursor()` method. + +For example: + +``` c +#include + +... + +DB *my_database; +DBC *cursorp; + +/* Database open omitted for clarity */ + +/* Get a cursor */ +my_database->cursor(my_database, NULL, &cursorp, 0); +``` + +When you are done with the cursor, you should close it. To close a cursor, call the `DBC->close()` method. Note that closing your database while cursors are still opened within the scope of the DB handle, especially if those cursors are writing to the database, can have unpredictable results. It is recommended that you close all cursor handles after their use to ensure concurrency and to release resources such as page locks. + +``` c +#include + +... + +DB *my_database; +DBC *cursorp; + +/* Database and cursor open omitted for clarity */ + +if (cursorp != NULL) + cursorp->close(cursorp); + +if (my_database != NULL) + my_database->close(my_database, 0); +``` diff --git a/docs-src/guides/gsg/DBEntry.md b/docs-src/guides/gsg/DBEntry.md new file mode 100644 index 000000000..07334faf9 --- /dev/null +++ b/docs-src/guides/gsg/DBEntry.md @@ -0,0 +1,89 @@ +--- +title: "Chapter 3. Database Records" +api-name: "Chapter 3. Database Records" +source: docs/gsg/C/DBEntry.html +--- +## Chapter 3. Database Records + +**Table of Contents** + + [Using Database Records](DBEntry.md#usingDbEntry) + + [Reading and Writing Database Records](usingDbt.md) + + [Writing Records to the Database](usingDbt.md#databaseWrite) + + [Getting Records from the Database](usingDbt.md#CoreDatabaseRead) + + [Deleting Records](usingDbt.md#recordDelete) + + [Data Persistence](usingDbt.md#datapersist) + + [Using C Structures with DB](cstructs.md) + + [C Structures with Pointers](cstructs.md#cstructdynamic) + + [Database Usage Example](DbUsage.md) + +DB records contain two parts — a key and some data. Both the key and its corresponding data are encapsulated in `DBT` structures. Therefore, to access a DB record, you need two such structures, one for the key and one for the data. + +`DBT` structures provide a `void *` field that you use to point to your data, and another field that identifies the data length. They can therefore be used to store anything from simple primitive data to complex structures so long as the information you want to store resides in a single contiguous block of memory. + +This chapter describes `DBT` usage. It also introduces storing and retrieving key/value pairs from a database. + +## Using Database Records + +Each database record is comprised of two `DBT` structures — one for the key and another for the data. + +To store a database record where the key and/or the data are primitive data (`int`, `float`, and so forth), or where the key and/or the data contain an array, we need only to point to the memory location where that data resides and identify its length. For example: + +``` c +#include +#include + +... + +DBT key, data; +float money = 122.45; +char *description = "Grocery bill."; + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = &money; +key.size = sizeof(float); + +data.data = description; +data.size = strlen(description) + 1; +``` + +To retrieve the record, simply assign the `void *` returned in the `DBT` to the appropriate variable. + +Note that in the following example we do not allow DB to assign the memory for the retrieval of the money value. The reason why is that some systems may require float values to have a specific alignment, and the memory as returned by DB may not be properly aligned (the same problem may exist for structures on some systems). We tell DB to use our memory instead of its own by specifying the `DB_DBT_USERMEM` flag. Be aware that when we do this, we must also identify how much user memory is available through the use of the `ulen` field. + +``` c +#include +#include + +... + +float money; +DBT key, data; +char *description; + +/* Initialize the DBTs */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = &money; +key.ulen = sizeof(float); +key.flags = DB_DBT_USERMEM; + +/* Database retrieval code goes here */ + +/* + * Money is set into the memory that we supplied. + */ +description = data.data; +``` diff --git a/docs-src/guides/gsg/DBOpenFlags.md b/docs-src/guides/gsg/DBOpenFlags.md new file mode 100644 index 000000000..a3dc19b3e --- /dev/null +++ b/docs-src/guides/gsg/DBOpenFlags.md @@ -0,0 +1,32 @@ +--- +title: "Database Open Flags" +api-name: "Database Open Flags" +source: docs/gsg/C/DBOpenFlags.html +--- +## Database Open Flags + +The following are the flags that you may want to use at database open time. Note that this list is not exhaustive — it includes only those flags likely to be of interest for introductory, single-threaded database applications. For a complete list of the flags available to you, see the *Berkeley DB C API Reference Guide.* + +### Note + +To specify more than one flag on the call to `DB->open()`, you must bitwise inclusively OR them together: + +``` c +u_int32_t open_flags = DB_CREATE | DB_EXCL; +``` + +- `DB_CREATE` + + If the database does not currently exist, create it. By default, the database open fails if the database does not already exist. + +- `DB_EXCL` + + Exclusive database creation. Causes the database open to fail if the database already exists. This flag is only meaningful when used with `DB_CREATE`. + +- `DB_RDONLY` + + Open the database for read operations only. Causes any subsequent database write operations to fail. + +- `DB_TRUNCATE` + + Physically truncate (empty) the on-disk file that contains the database. Causes DB to delete all databases physically contained in that file. diff --git a/docs-src/guides/gsg/DbUsage.md b/docs-src/guides/gsg/DbUsage.md new file mode 100644 index 000000000..5be67cc85 --- /dev/null +++ b/docs-src/guides/gsg/DbUsage.md @@ -0,0 +1,369 @@ +--- +title: "Database Usage Example" +api-name: "Database Usage Example" +source: docs/gsg/C/DbUsage.html +--- +## Database Usage Example + +In Database Example we created several functions that will open and close the databases that we will use for our inventory application. We now make use of those functions to load inventory data into the two databases that we use for this application. + +Again, remember that you can find the complete implementation for these functions in: + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +**Example 3.1 VENDOR Structure** + +We want to store data related to an inventory system. There are two types of information that we want to manage: inventory data and related vendor contact information. To manage this information, we could create a structure for each type of data, but to illustrate storing mixed data without a structure we refrain from creating one for the inventory data. + +For the vendor data, we add the VENDOR structure to the same file as holds our STOCK_DBS structure. Note that the VENDOR structure uses fixed-length fields. This is not necessary and in fact could represent a waste of resources if the number of vendors stored in our database scales to very large numbers. However, for simplicity we use fixed-length fields anyway, especially given that our sample data contains so few vendor records. + +Note that for the inventory data, we will store the data by marshaling it into a buffer, described below. + +``` c +/* File: gettingstarted_common.h */ +#include + +... + +typedef struct vendor { + char name[MAXFIELD]; /* Vendor name */ + char street[MAXFIELD]; /* Street name and number */ + char city[MAXFIELD]; /* City */ + char state[3]; /* Two-digit US state code */ + char zipcode[6]; /* US zipcode */ + char phone_number[13]; /* Vendor phone number */ + char sales_rep[MAXFIELD]; /* Name of sales representative */ + char sales_rep_phone[MAXFIELD]; /* Sales rep's phone number */ +} VENDOR; +``` + + +**Example 3.2 example_database_load** + +Our initial sample application will load database information from several flat files. To save space, we won't show all the details of this example program. However, as always you can find the complete implementation for this program here: + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +We begin with the normal include directives and forward declarations: + +``` c +/* example_database_load.c */ +#include "gettingstarted_common.h" + +/* Forward declarations */ +int load_vendors_database(STOCK_DBS, char *); +int pack_string(char *, char *, int); +int load_inventory_database(STOCK_DBS, char *); +``` + +Next we begin our `main()` function with the variable declarations and command line parsing that is normal for most command line applications: + +``` c +/* + * Loads the contents of vendors.txt and inventory.txt into + * Berkeley DB databases. + */ +int +main(int argc, char *argv[]) +{ + STOCK_DBS my_stock; + int ret, size; + char *basename, *inventory_file, *vendor_file; + + /* Initialize the STOCK_DBS struct */ + initialize_stockdbs(&my_stock); + + /* + * Initialize the base path. This path is used to + * identify the location of the flat-text data + * input files. + */ + basename = "./"; + + /* + * Parse the command line arguments here and determine + * the location of the flat text files containing the + * inventory data here. This step is omitted for clarity. + */ + + /* + * Identify the files that will hold our databases + * This function uses information obtained from the + * command line to identify the directory in which + * the database files reside. + */ + set_db_filenames(&my_stock); + + /* Find our input files */ + size = strlen(basename) + strlen(INVENTORY_FILE) + 1; + inventory_file = malloc(size); + snprintf(inventory_file, size, "%s%s", basename, INVENTORY_FILE); + + size = strlen(basename) + strlen(VENDORS_FILE) + 1; + vendor_file = malloc(size); + snprintf(vendor_file, size, "%s%s", basename, VENDORS_FILE); + + /* Open all databases */ + ret = databases_setup(&my_stock, "example_database_load", stderr); + if (ret != 0) { + fprintf(stderr, "Error opening databases\n"); + databases_close(&my_stock); + return (ret); + } + + ret = load_vendors_database(my_stock, vendor_file); + if (!ret) { + fprintf(stderr, "Error loading vendors database.\n"); + databases_close(&my_stock); + return (ret); + } + ret = load_inventory_database(my_stock, inventory_file); + if (!ret) { + fprintf(stderr, "Error loading inventory database.\n"); + databases_close(&my_stock); + return (ret); + } + + /* close our environment and databases */ + databases_close(&my_stock); + + printf("Done loading databases.\n"); + return (0); +} +``` + +Notice that there is not a lot to this function because we have pushed off all the database activity to other places. In particular our databases are all opened and configured in `databases_setup()` which we implemented in The databases_setup() Function. + +Next we show the implementation of `load_vendors_database()`. We load this data by scanning (line by line) the contents of the `vendors.txt` into a VENDOR structure. Once we have a line scanned into the structure, we can store that structure into our vendors database. + +Note that we use the vendor's name as the key here. In doing so, we assume that the vendor's name is unique in our database. If it was not, we would either have to select a different key, or architect our application such that it could cope with multiple vendor records with the same name. + +``` c +/* + * Loads the contents of the vendors.txt file into + * a database. + */ +int +load_vendors_database(STOCK_DBS my_stock, char *vendor_file) +{ + DBT key, data; + FILE *ifp; + VENDOR my_vendor; + char buf[MAXLINE]; + + /* Open the vendor file for read access */ + ifp = fopen(vendor_file, "r"); + if (ifp == NULL) { + fprintf(stderr, "Error opening file '%s'\n", vendor_file); + return(-1); + } + + /* Iterate over the vendor file */ + while(fgets(buf, MAXLINE, ifp) != NULL) { + /* zero out the structure */ + memset(&my_vendor, 0, sizeof(VENDOR)); + /* Zero out the DBTs */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + /* + * Scan the line into the structure. + * Convenient, but not particularly safe. + * In a real program, there would be a lot more + * defensive code here. + */ + sscanf(buf, + "%20[^#]#%20[^#]#%20[^#]#%3[^#]#%6[^#]#%13[^#]#%20[^#]#%20[^\n]", + my_vendor.name, my_vendor.street, + my_vendor.city, my_vendor.state, + my_vendor.zipcode, my_vendor.phone_number, + my_vendor.sales_rep, my_vendor.sales_rep_phone); + + /* + * Now that we have our structure we can load it + * into the database. + */ + + /* Set up the database record's key */ + key.data = my_vendor.name; + key.size = strlen(my_vendor.name) + 1; + + /* Set up the database record's data */ + data.data = &my_vendor; + data.size = sizeof(my_vendor); + + /* + * Note that given the way we built our struct, there are extra + * bytes in it. Essentially we're using fixed-width fields with + * the unused portion of some fields padded with zeros. This + * is the easiest thing to do, but it does result in a bloated + * database. Look at load_inventory_data() for an example of how + * to avoid this. + */ + + /* Put the data into the database. + * Omitting error handling for clarity. + */ + my_stock.vendor_dbp->put(my_stock.vendor_dbp, 0, + &key, &data, 0); + } /* end vendors database while loop */ + + /* Close the vendor.txt file */ + fclose(ifp); + return(0); +} +``` + +Finally, we need to write the `load_inventory_database()` function. We made this function a bit more complicated than is necessary by avoiding the use of a structure to manage the data. Instead, we manually pack all our inventory data into a single block of memory, and store that data in the database. + +While this complicates our code somewhat, this approach allows us to use the smallest amount of space possible for the data that we want to store. The result is that our cache can be smaller than it might otherwise be and our database will take less space on disk than if we used a structure with fixed-length fields. + +For a trivial dataset such as what we use for these examples, these resource savings are negligible. But if we were storing hundreds of millions of records, then the cost savings may become significant. + +Before we actually implement our inventory loading function, it is useful to create a simple utility function that copies a character array into a buffer at a designated offset: + +``` c +/* + * Simple little convenience function that takes a buffer, a string, + * and an offset and copies that string into the buffer at the + * appropriate location. Used to ensure that all our strings + * are contained in a single contiguous chunk of memory. + */ +int +pack_string(char *buffer, char *string, int start_pos) +{ + int string_size = strlen(string) + 1; + + memcpy(buffer+start_pos, string, string_size); + + return(start_pos + string_size); +} +``` + +That done, we can now load the inventory database: + +``` c +/* + * Loads the contents of the inventory.txt file into + * a database. + */ +int +load_inventory_database(STOCK_DBS my_stock, char *inventory_file) +{ + DBT key, data; + char buf[MAXLINE]; + void *databuf; + int bufLen, dataLen; + FILE *ifp; + + /* + * Rather than lining everything up nicely in a struct, we're being + * deliberately a bit sloppy here. This function illustrates how to + * store mixed data that might be obtained from various locations + * in your application. + */ + float price; + int quantity; + char category[MAXFIELD], name[MAXFIELD]; + char vendor[MAXFIELD], sku[MAXFIELD]; + + /* Load the inventory database */ + ifp = fopen(inventory_file, "r"); + if (ifp == NULL) { + fprintf(stderr, "Error opening file '%s'\n", inventory_file); + return(-1); + } + + /* Get our buffer. MAXDATABUF is some suitably large number */ + databuf = malloc(MAXDATABUF); + + /* + * Read the inventory.txt file line by line, saving each line off to + * the database as we go. + */ + while(fgets(buf, MAXLINE, ifp) != NULL) { + /* + * Scan the line into the appropriate buffers and variables. + * Convenient, but not particularly safe. In a real + * program, there would be a lot more defensive code here. + */ + sscanf(buf, + "%20[^#]#%20[^#]#%f#%i#%20[^#]#%20[^\n]", + name, sku, &price, &quantity, category, vendor); + + /* + * Now pack it into a single contiguous memory location for + * storage. + */ + memset(databuf, 0, MAXDATABUF); + bufLen = 0; + dataLen = 0; + + /* + * We first store the fixed-length elements. This makes our code + * to retrieve this data from the database a little bit easier. + */ + + /* First discover how long the data element is. */ + dataLen = sizeof(float); + /* Then copy it to our buffer */ + memcpy(databuf, &price, dataLen); + /* + * Then figure out how much data is actually in our buffer. + * We repeat this pattern for all the data we want to store. + */ + bufLen += dataLen; + + /* Rinse, lather, repeat. */ + dataLen = sizeof(int); + memcpy(databuf + bufLen, &quantity, dataLen); + bufLen += dataLen; + + bufLen = pack_string(databuf, name, bufLen); + bufLen = pack_string(databuf, sku, bufLen); + bufLen = pack_string(databuf, category, bufLen); + bufLen = pack_string(databuf, vendor, bufLen); + + /* + * Now actually save the contents of the buffer off + * to our database. + */ + + /* Zero out the DBTs */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + /* + * The key is the item's SKU. This is a unique value, so we need + * not support duplicates for this database. + */ + key.data = sku; + key.size = strlen(sku) + 1; + + /* The data is the information that we packed into databuf. */ + data.data = databuf; + data.size = bufLen; + + /* Put the data into the database */ + my_stock.vendor_dbp->put(my_stock.inventory_dbp, 0, + &key, &data, 0); + } /* end vendors database while loop */ + + /* Cleanup */ + fclose(ifp); + if (databuf != NULL) + free(databuf); + + return(0); +} +``` + +In the next chapter we provide an example that shows how to read the inventory and vendor databases. diff --git a/docs-src/guides/gsg/DeleteEntryWCursor.md b/docs-src/guides/gsg/DeleteEntryWCursor.md new file mode 100644 index 000000000..ee6306805 --- /dev/null +++ b/docs-src/guides/gsg/DeleteEntryWCursor.md @@ -0,0 +1,49 @@ +--- +title: "Deleting Records Using Cursors" +api-name: "Deleting Records Using Cursors" +source: docs/gsg/C/DeleteEntryWCursor.html +--- +## Deleting Records Using Cursors + +To delete a record using a cursor, simply position the cursor to the record that you want to delete and then call `DBC->del()`. + +For example: + +``` c +#include +#include + +... + +DB *dbp; +DBC *cursorp; +DBT key, data; +char *key1str = "My first string"; +int ret; + +/* Database open omitted */ + +/* Initialize our DBTs. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +/* Set up our DBTs */ +key.data = key1str; +key.size = strlen(key1str) + 1; + +/* Get the cursor */ +dbp->cursor(dbp, NULL, &cursorp, 0); + +/* Iterate over the database, deleting each record in turn. */ +while ((ret = cursorp->get(cursorp, &key, + &data, DB_SET)) == 0) { + cursorp->del(cursorp, 0); +} + +/* Cursors must be closed */ +if (cursorp != NULL) + cursorp->close(cursorp); + +if (dbp != NULL) + dbp->close(dbp, 0); +``` diff --git a/docs-src/guides/gsg/Positioning.md b/docs-src/guides/gsg/Positioning.md new file mode 100644 index 000000000..4052a7999 --- /dev/null +++ b/docs-src/guides/gsg/Positioning.md @@ -0,0 +1,272 @@ +--- +title: "Getting Records Using the Cursor" +api-name: "Getting Records Using the Cursor" +source: docs/gsg/C/Positioning.html +--- +## Getting Records Using the Cursor + + [Searching for Records](Positioning.md#cursorsearch) + + [Working with Duplicate Records](Positioning.md#getdups) + +To iterate over database records, from the first record to the last, simply open the cursor and then use the `DBC->get()` method. Note that you need to supply the `DB_NEXT` flag to this method. For example: + +``` c +#include +#include + +... + +DB *my_database; +DBC *cursorp; +DBT key, data; +int ret; + +/* Database open omitted for clarity */ + +/* Get a cursor */ +my_database->cursor(my_database, NULL, &cursorp, 0); + +/* Initialize our DBTs. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +/* Iterate over the database, retrieving each record in turn. */ +while ((ret = cursorp->get(cursorp, &key, &data, DB_NEXT)) == 0) { + /* Do interesting things with the DBTs here. */ +} +if (ret != DB_NOTFOUND) { + /* Error handling goes here */ +} + +/* Cursors must be closed */ +if (cursorp != NULL) + cursorp->close(cursorp); + +if (my_database != NULL) + my_database->close(my_database, 0); +``` + +To iterate over the database from the last record to the first, use `DB_PREV` instead of `DB_NEXT`: + +``` c +#include +#include + +... + +DB *my_database; +DBC *cursorp; +DBT key, data; +int ret; + +/* Database open omitted for clarity */ + +/* Get a cursor */ +my_database->cursor(my_database, NULL, &cursorp, 0); + +/* Initialize our DBTs. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +/* Iterate over the database, retrieving each record in turn. */ +while ((ret = cursorp->get(cursorp, &key, + &data, DB_PREV)) == 0) { + /* Do interesting things with the DBTs here. */ +} +if (ret != DB_NOTFOUND) { + /* Error handling goes here */ +} + +// Cursors must be closed +if (cursorp != NULL) + cursorp->close(cursorp); + +if (my_database != NULL) + my_database->close(my_database, 0); +``` + +### Searching for Records + +You can use cursors to search for database records. You can search based on just a key, or you can search based on both the key and the data. You can also perform partial matches if your database supports sorted duplicate sets. In all cases, the key and data parameters of these methods are filled with the key and data values of the database record to which the cursor is positioned as a result of the search. + +Also, if the search fails, then cursor's state is left unchanged and `DB_NOTFOUND` is returned. + +To use a cursor to search for a record, use DBT-\>get(). When you use this method, you can provide the following flags: + +### Note + +Notice in the following list that the cursor flags use the keyword `SET` when the cursor examines just the key portion of the records (in this case, the cursor is set to the record whose key matches the value provided to the cursor). Moreover, when the cursor uses the keyword `GET`, then the cursor is positioned to both the key *and* the data values provided to the cursor. + +Regardless of the keyword you use to get a record with a cursor, the cursor's key and data `DBT`s are filled with the data retrieved from the record to which the cursor is positioned. + +- `DB_SET` + + Moves the cursor to the first record in the database with the specified key. + +- `DB_SET_RANGE` + + Identical to `DB_SET` unless you are using the BTree access. In this case, the cursor moves to the first record in the database whose key is greater than or equal to the specified key. This comparison is determined by the comparison function that you provide for the database. If no comparison function is provided, then the default lexicographical sorting is used. + + For example, suppose you have database records that use the following strings as keys: + + ``` c + Alabama + Alaska + Arizona + ``` + + Then providing a search key of `Alaska` moves the cursor to the second key noted above. Providing a key of `Al` moves the cursor to the first key (`Alabama`), providing a search key of `Alas` moves the cursor to the second key (`Alaska`), and providing a key of `Ar` moves the cursor to the last key (`Arizona`). + +- `DB_GET_BOTH` + + Moves the cursor to the first record in the database that uses the specified key and data. + +- `DB_GET_BOTH_RANGE` + + Moves the cursor to the first record in the database whose key matches the specified key and whose data is greater than or equal to the specified data. If the database supports duplicate records, then on matching the key, the cursor is moved to the duplicate record with the smallest data that is greater than or equal to the specified data. + + For example, suppose your database uses BTree and it has database records that use the following key/data pairs: + + ``` c + Alabama/Athens + Alabama/Florence + Alaska/Anchorage + Alaska/Fairbanks + Arizona/Avondale + Arizona/Florence + ``` + + then providing: + + | a search key of ... | and a search data of ... | moves the cursor to ... | + |---------------------|--------------------------|-------------------------| + | Alaska | Fa | Alaska/Fairbanks | + | Arizona | Fl | Arizona/Florence | + | Alaska | An | Alaska/Anchorage | + +For example, assuming a database containing sorted duplicate records of U.S. States/U.S Cities key/data pairs (both as strings), then the following code fragment can be used to position the cursor to any record in the database and print its key/data values: + +``` c +#include +#include + +... + +DBC *cursorp; +DBT key, data; +DB *dbp; +int ret; +char *search_data = "Fa"; +char *search_key = "Alaska"; + +/* database open omitted for clarity */ + +/* Get a cursor */ +dbp->cursor(dbp, NULL, &cursorp, 0); + +/* Set up our DBTs */ +key.data = search_key; +key.size = strlen(search_key) + 1; +data.data = search_data; +data.size = strlen(search_data) + 1; + +/* + * Position the cursor to the first record in the database whose + * key matches the search key and whose data begins with the + * search data. + */ +ret = cursorp->get(cursorp, &key, &data, DB_GET_BOTH_RANGE); +if (!ret) { + /* Do something with the data */ +} else { + /* Error handling goes here */ +} + +/* Close the cursor */ +if (cursorp != NULL) + cursorp->close(cursorp); + +/* Close the database */ +if (dbp != NULL) + dbp->close(dbp, 0); +``` + +### Working with Duplicate Records + +A record is a duplicate of another record if the two records share the same key. For duplicate records, only the data portion of the record is unique. + +Duplicate records are supported only for the BTree or Hash access methods. For information on configuring your database to use duplicate records, see Allowing Duplicate Records. + +If your database supports duplicate records, then it can potentially contain multiple records that share the same key. By default, normal database get operations will only return the first such record in a set of duplicate records. Typically, subsequent duplicate records are accessed using a cursor. The following `DBC->get()` flags are interesting when working with databases that support duplicate records: + +- `DB_NEXT`, `DB_PREV` + + Shows the next/previous record in the database, regardless of whether it is a duplicate of the current record. For an example of using these methods, see Getting Records Using the Cursor. + +- `DB_GET_BOTH_RANGE` + + Useful for seeking the cursor to a specific record, regardless of whether it is a duplicate record. See Searching for Records for more information. + +- `DB_NEXT_NODUP`, `DB_PREV_NODUP` + + Gets the next/previous non-duplicate record in the database. This allows you to skip over all the duplicates in a set of duplicate records. If you call `DBC->get()` with `DB_PREV_NODUP`, then the cursor is positioned to the last record for the previous key in the database. For example, if you have the following records in your database: + + ``` c + Alabama/Athens + Alabama/Florence + Alaska/Anchorage + Alaska/Fairbanks + Arizona/Avondale + Arizona/Florence + ``` + + and your cursor is positioned to `Alaska/Fairbanks`, and you then call `DBC->get()` with `DB_PREV_NODUP`, then the cursor is positioned to Alabama/Florence. Similarly, if you call `DBC->get()` with `DB_NEXT_NODUP`, then the cursor is positioned to the first record corresponding to the next key in the database. + + If there is no next/previous key in the database, then `DB_NOTFOUND` is returned, and the cursor is left unchanged. + +- `DB_NEXT_DUP` + + Gets the next record that shares the current key. If the cursor is positioned at the last record in the duplicate set and you call `DBC->get()` with `DB_NEXT_DUP`, then `DB_NOTFOUND` is returned and the cursor is left unchanged. + +For example, the following code fragment positions a cursor to a key and displays it and all its duplicates. + +``` c +#include +#include + +... + +DB *dbp; +DBC *cursorp; +DBT key, data; +int ret; +char *search_key = "Al"; + +/* database open omitted for clarity */ + +/* Get a cursor */ +dbp->cursor(dbp, NULL, &cursorp, 0); + +/* Set up our DBTs */ +key.data = search_key; +key.size = strlen(search_key) + 1; + +/* + * Position the cursor to the first record in the database whose + * key and data begin with the correct strings. + */ +ret = cursorp->get(cursorp, &key, &data, DB_SET); +while (ret != DB_NOTFOUND) { + printf("key: %s, data: %s\n", (char *)key.data, (char *)data.data); + ret = cursorp->get(cursorp, &key, &data, DB_NEXT_DUP); +} + +/* Close the cursor */ +if (cursorp != NULL) + cursorp->close(cursorp); + +/* Close the database */ +if (dbp != NULL) + dbp->close(dbp, 0); +``` diff --git a/docs-src/guides/gsg/PutEntryWCursor.md b/docs-src/guides/gsg/PutEntryWCursor.md new file mode 100644 index 000000000..15f07db0f --- /dev/null +++ b/docs-src/guides/gsg/PutEntryWCursor.md @@ -0,0 +1,103 @@ +--- +title: "Putting Records Using Cursors" +api-name: "Putting Records Using Cursors" +source: docs/gsg/C/PutEntryWCursor.html +--- +## Putting Records Using Cursors + +You can use cursors to put records into the database. DB's behavior when putting records into the database differs depending on the flags that you use when writing the record, on the access method that you are using, and on whether your database supports sorted duplicates. + +Note that when putting records to the database using a cursor, the cursor is positioned at the record you inserted. + +You use `DBC->put()` to put (write) records to the database. You can use the following flags with this method: + +- `DB_NODUPDATA` + + If the provided key already exists in the database, then this method returns `DB_KEYEXIST`. + + If the key does not exist, then the order that the record is put into the database is determined by the insertion order in use by the database. If a comparison function has been provided to the database, the record is inserted in its sorted location. Otherwise (assuming BTree), lexicographical sorting is used, with shorter items collating before longer items. + + This flag can only be used for the BTree and Hash access methods, and only if the database has been configured to support sorted duplicate data items (`DB_DUPSORT` was specified at database creation time). + + This flag cannot be used with the Queue or Recno access methods. + + For more information on duplicate records, see Allowing Duplicate Records. + +- `DB_KEYFIRST` + + For databases that do not support duplicates, this method behaves exactly the same as if a default insertion was performed. If the database supports duplicate records, and a duplicate sort function has been specified, the inserted data item is added in its sorted location. If the key already exists in the database and no duplicate sort function has been specified, the inserted data item is added as the first of the data items for that key. + +- `DB_KEYLAST` + + Behaves exactly as if `DB_KEYFIRST` was used, except that if the key already exists in the database and no duplicate sort function has been specified, the inserted data item is added as the last of the data items for that key. + +For example: + +``` c +#include +#include + +... + +DB *dbp; +DBC *cursorp; +DBT data1, data2, data3; +DBT key1, key2; +char *key1str = "My first string"; +char *data1str = "My first data"; +char *key2str = "A second string"; +char *data2str = "My second data"; +char *data3str = "My third data"; +int ret; + +/* Set up our DBTs */ +key1.data = key1str; +key1.size = strlen(key1str) + 1; +data1.data = data1str; +data1.size = strlen(data1str) + 1; + +key2.data = key2str; +key2.size = strlen(key2str) + 1; +data2.data = data2str; +data2.size = strlen(data2str) + 1; +data3.data = data3str; +data3.size = strlen(data3str) + 1; + +/* Database open omitted */ + +/* Get the cursor */ +dbp->cursor(dbp, NULL, &cursorp, 0); + +/* + * Assuming an empty database, this first put places + * "My first string"/"My first data" in the first + * position in the database + */ +ret = cursorp->put(cursorp, &key1, + &data1, DB_KEYFIRST); + +/* + * This put places "A second string"/"My second data" in the + * the database according to its key sorts against the key + * used for the currently existing database record. Most likely + * this record would appear first in the database. + */ +ret = cursorp->put(cursorp, &key2, + &data2, DB_KEYFIRST); /* Added according to sort order */ + +/* + * If duplicates are not allowed, the currently existing record that + * uses "key2" is overwritten with the data provided on this put. + * That is, the record "A second string"/"My second data" becomes + * "A second string"/"My third data" + * + * If duplicates are allowed, then "My third data" is placed in the + * duplicates list according to how it sorts against "My second data". + */ +ret = cursorp->put(cursorp, &key2, + &data3, DB_KEYFIRST); /* If duplicates are not allowed, record + * is overwritten with new data. Otherwise, + * the record is added to the beginning of + * the duplicates list. + */ +``` diff --git a/docs-src/guides/gsg/ReplacingEntryWCursor.md b/docs-src/guides/gsg/ReplacingEntryWCursor.md new file mode 100644 index 000000000..0fb90a54a --- /dev/null +++ b/docs-src/guides/gsg/ReplacingEntryWCursor.md @@ -0,0 +1,56 @@ +--- +title: "Replacing Records Using Cursors" +api-name: "Replacing Records Using Cursors" +source: docs/gsg/C/ReplacingEntryWCursor.html +--- +## Replacing Records Using Cursors + +You replace the data for a database record by using `DBC->put()` with the `DB_CURRENT` flag. + +``` c +#include +#include + +... + +DB *dbp; +DBC *cursorp; +DBT key, data; +char *key1str = "My first string"; +char *replacement_data = "replace me"; +int ret; + +/* Initialize our DBTs. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +/* Set up our DBTs */ +key.data = key1str; +key.size = strlen(key1str) + 1; + +/* Database open omitted */ + +/* Get the cursor */ +dbp->cursor(dbp, NULL, &cursorp, 0); + +/* Position the cursor */ +ret = cursorp->get(cursorp, &key, &data, DB_SET); +if (ret == 0) { + data.data = replacement_data; + data.size = strlen(replacement_data) + 1; + cursorp->put(cursorp, &key, &data, DB_CURRENT); +} + +/* Cursors must be closed */ +if (cursorp != NULL) + cursorp->close(cursorp); + +if (dbp != NULL) + dbp->close(dbp, 0); +``` + +Note that you cannot change a record's key using this method; the key parameter is always ignored when you replace a record. + +When replacing the data portion of a record, if you are replacing a record that is a member of a sorted duplicates set, then the replacement will be successful only if the new record sorts identically to the old record. This means that if you are replacing a record that is a member of a sorted duplicates set, and if you are using the default lexicographic sort, then the replacement will fail due to violating the sort order. However, if you provide a custom sort routine that, for example, sorts based on just a few bytes out of the data item, then potentially you can perform a direct replacement and still not violate the restrictions described here. + +Under these circumstances, if you want to replace the data contained by a duplicate record, and you are not using a custom sort routine, then delete the record and create a new record with the desired key and data. diff --git a/docs-src/guides/gsg/_meta.toml b/docs-src/guides/gsg/_meta.toml new file mode 100644 index 000000000..fc3829c9c --- /dev/null +++ b/docs-src/guides/gsg/_meta.toml @@ -0,0 +1,44 @@ +# Nav/index metadata for the gsg guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Getting Started with Berkeley DB" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "introduction", + "concepts", + "accessmethods", + "databaseLimits", + "environments", + "returns", + "gettingit", + "databases", + "coredbclose", + "DBOpenFlags", + "CoreDBAdmin", + "dbErrorReporting", + "CoreEnvUsage", + "CoreDbUsage", + "DBEntry", + "usingDbt", + "cstructs", + "DbUsage", + "Cursors", + "Positioning", + "PutEntryWCursor", + "DeleteEntryWCursor", + "ReplacingEntryWCursor", + "CoreCursorUsage", + "indexes", + "keyCreator", + "readSecondary", + "secondaryDelete", + "secondaryCursor", + "joins", + "coreindexusage", + "dbconfig", + "cachesize", + "btree", +] diff --git a/docs-src/guides/gsg/accessmethods.md b/docs-src/guides/gsg/accessmethods.md new file mode 100644 index 000000000..37da11455 --- /dev/null +++ b/docs-src/guides/gsg/accessmethods.md @@ -0,0 +1,80 @@ +--- +title: "Access Methods" +api-name: "Access Methods" +source: docs/gsg/C/accessmethods.html +--- +## Access Methods + + [Selecting Access Methods](accessmethods.md#selectAM) + + [Choosing between BTree and Hash](accessmethods.md#BTreeVSHash) + + [Choosing between Queue and Recno](accessmethods.md#QueueVSRecno) + +While this manual will focus primarily on the BTree access method, it is still useful to briefly describe all of the access methods that DB makes available. + +Note that an access method can be selected only when the database is created. Once selected, actual API usage is generally identical across all access methods. That is, while some exceptions exist, mechanically you interact with the library in the same way regardless of which access method you have selected. + +The access method that you should choose is gated first by what you want to use as a key, and then secondly by the performance that you see for a given access method. + +The following are the available access methods: + + + + + + + + + + + + + + + + + + + + + + + + + + +
Access MethodDescription
BTree

Data is stored in a sorted, balanced tree structure. Both the key and the data for BTree records can be arbitrarily complex. That is, they can contain single values such as an integer or a string, or complex types such as a structure. Also, although not the default behavior, it is possible for two records to use keys that compare as equals. When this occurs, the records are considered to be duplicates of one another.

Hash

Data is stored in an extended linear hash table. Like BTree, the key and the data used for Hash records can be of arbitrarily complex data. Also, like BTree, duplicate records are optionally supported.

Queue

Data is stored in a queue as fixed-length records. Each record uses a logical record number as its key. This access method is designed for fast inserts at the tail of the queue, and it has a special operation that deletes and returns a record from the head of the queue.

+

This access method is unusual in that it provides record level locking. This can provide beneficial performance improvements in applications requiring concurrent access to the queue.

Recno

Data is stored in either fixed or variable-length records. Like Queue, Recno records use logical record numbers as keys.

+ +### Selecting Access Methods + +To select an access method, you should first consider what you want to use as a key for you database records. If you want to use arbitrary data (even strings), then you should use either BTree or Hash. If you want to use logical record numbers (essentially integers) then you should use Queue or Recno. + +Once you have made this decision, you must choose between either BTree or Hash, or Queue or Recno. This decision is described next. + +### Choosing between BTree and Hash + +For small working datasets that fit entirely in memory, there is no difference between BTree and Hash. Both will perform just as well as the other. In this situation, you might just as well use BTree, if for no other reason than the majority of DB applications use BTree. + +Note that the main concern here is your working dataset, not your entire dataset. Many applications maintain large amounts of information but only need to access some small portion of that data with any frequency. So what you want to consider is the data that you will routinely use, not the sum total of all the data managed by your application. + +However, as your working dataset grows to the point where you cannot fit it all into memory, then you need to take more care when choosing your access method. Specifically, choose: + +- BTree if your keys have some locality of reference. That is, if they sort well and you can expect that a query for a given key will likely be followed by a query for one of its neighbors. + +- Hash if your dataset is extremely large. For any given access method, DB must maintain a certain amount of internal information. However, the amount of information that DB must maintain for BTree is much greater than for Hash. The result is that as your dataset grows, this internal information can dominate the cache to the point where there is relatively little space left for application data. As a result, BTree can be forced to perform disk I/O much more frequently than would Hash given the same amount of data. + + Moreover, if your dataset becomes so large that DB will almost certainly have to perform disk I/O to satisfy a random request, then Hash will definitely out perform BTree because it has fewer internal records to search through than does BTree. + +### Choosing between Queue and Recno + +Queue or Recno are used when the application wants to use logical record numbers for the primary database key. Logical record numbers are essentially integers that uniquely identify the database record. They can be either mutable or fixed, where a mutable record number is one that might change as database records are stored or deleted. Fixed logical record numbers never change regardless of what database operations are performed. + +When deciding between Queue and Recno, choose: + +- Queue if your application requires high degrees of concurrency. Queue provides record-level locking (as opposed to the page-level locking that the other access methods use), and this can result in significantly faster throughput for highly concurrent applications. + + Note, however, that Queue provides support only for fixed length records. So if the size of the data that you want to store varies widely from record to record, you should probably choose an access method other than Queue. + +- Recno if you want mutable record numbers. Queue is only capable of providing fixed record numbers. Also, Recno provides support for databases whose permanent storage is a flat text file. This is useful for applications looking for fast, temporary storage while the data is being read or modified. diff --git a/docs-src/guides/gsg/btree.md b/docs-src/guides/gsg/btree.md new file mode 100644 index 000000000..822174546 --- /dev/null +++ b/docs-src/guides/gsg/btree.md @@ -0,0 +1,210 @@ +--- +title: "BTree Configuration" +api-name: "BTree Configuration" +source: docs/gsg/C/btree.html +--- +## BTree Configuration + + [Allowing Duplicate Records](btree.md#duplicateRecords) + + [Setting Comparison Functions](btree.md#comparators) + +In going through the previous chapters in this book, you may notice that we touch on some topics that are specific to BTree, but we do not cover those topics in any real detail. In this section, we will discuss configuration issues that are unique to BTree. + +Specifically, in this section we describe: + +- Allowing duplicate records. + +- Setting comparator callbacks. + +### Allowing Duplicate Records + +BTree databases can contain duplicate records. One record is considered to be a duplicate of another when both records use keys that compare as equal to one another. + +By default, keys are compared using a lexicographical comparison, with shorter keys collating higher than longer keys. You can override this default using the `DB->set_bt_compare()` method. See the next section for details. + +By default, DB databases do not allow duplicate records. As a result, any attempt to write a record that uses a key equal to a previously existing record results in the previously existing record being overwritten by the new record. + +Allowing duplicate records is useful if you have a database that contains records keyed by a commonly occurring piece of information. It is frequently necessary to allow duplicate records for secondary databases. + +For example, suppose your primary database contained records related to automobiles. You might in this case want to be able to find all the automobiles in the database that are of a particular color, so you would index on the color of the automobile. However, for any given color there will probably be multiple automobiles. Since the index is the secondary key, this means that multiple secondary database records will share the same key, and so the secondary database must support duplicate records. + +#### Sorted Duplicates + +Duplicate records can be stored in sorted or unsorted order. You can cause DB to automatically sort your duplicate records by specifying the `DB_DUPSORT` flag at database creation time. + +If sorted duplicates are supported, then the sorting function specified on `DB->set_dup_compare()` is used to determine the location of the duplicate record in its duplicate set. If no such function is provided, then the default lexicographical comparison is used. + +#### Unsorted Duplicates + +For performance reasons, BTrees should always contain sorted records. (BTrees containing unsorted entries must potentially spend a great deal more time locating an entry than does a BTree that contains sorted entries). That said, DB provides support for suppressing automatic sorting of duplicate records because it may be that your application is inserting records that are already in a sorted order. + +That is, if the database is configured to support unsorted duplicates, then the assumption is that your application will manually perform the sorting. In this event, expect to pay a significant performance penalty. Any time you place records into the database in a sort order not know to DB, you will pay a performance penalty + +That said, this is how DB behaves when inserting records into a database that supports non-sorted duplicates: + +- If your application simply adds a duplicate record using `DB->put()`, then the record is inserted at the end of its sorted duplicate set. + +- If a cursor is used to put the duplicate record to the database, then the new record is placed in the duplicate set according to the flags that are provided on the `DBC->put()` method. The relevant flags are: + + - `DB_AFTER` + + The data provided on the call to `DBC->put()` is placed into the database as a duplicate record. The key used for this operation is the key used for the record to which the cursor currently refers. Any key provided on the call to `DBC->put()` is therefore ignored. + + The duplicate record is inserted into the database immediately after the cursor's current position in the database. + + This flag is ignored if sorted duplicates are supported for the database. + + - `DB_BEFORE` + + Behaves the same as `DB_AFTER` except that the new record is inserted immediately before the cursor's current location in the database. + + - `DB_KEYFIRST` + + If the key provided on the call to `DBC->put()` already exists in the database, and the database is configured to use duplicates without sorting, then the new record is inserted as the first entry in the appropriate duplicates list. + + - `DB_KEYLAST` + + Behaves identically to `DB_KEYFIRST` except that the new duplicate record is inserted as the last record in the duplicates list. + +#### Configuring a Database to Support Duplicates + +Duplicates support can only be configured at database creation time. You do this by specifying the appropriate flags to `DB->set_flags()` before the database is opened for the first time. + +The flags that you can use are: + +- `DB_DUP` + + The database supports non-sorted duplicate records. + +- `DB_DUPSORT` + + The database supports sorted duplicate records. Note that this flag also sets the `DB_DUP` flag for you. + +The following code fragment illustrates how to configure a database to support sorted duplicate records: + +``` c +#include +... + +DB *dbp; +FILE *error_file_pointer; +int ret; +char *program_name = "my_prog"; +char *file_name = "mydb.db"; + +/* Variable assignments omitted for brevity */ + +/* Initialize the DB handle */ +ret = db_create(&dbp, NULL, 0); +if (ret != 0) { + fprintf(error_file_pointer, "%s: %s\n", program_name, + db_strerror(ret)); + return(ret); +} + +/* Set up error handling for this database */ +dbp->set_errfile(dbp, error_file_pointer); +dbp->set_errpfx(dbp, program_name); + +/* + * Configure the database for sorted duplicates + */ +ret = dbp->set_flags(dbp, DB_DUPSORT); +if (ret != 0) { + dbp->err(dbp, ret, "Attempt to set DUPSORT flag failed."); + dbp->close(dbp, 0); + return(ret); +} + +/* Now open the database */ +ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name (unneeded) */ + DB_BTREE, /* Database type (using btree) */ + DB_CREATE, /* Open flags */ + 0); /* File mode. Using defaults */ +if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed.", file_name); + dbp->close(dbp, 0); + return(ret); +} +``` + +### Setting Comparison Functions + +By default, DB uses a lexicographical comparison function where shorter records collate before longer records. For the majority of cases, this comparison works well and you do not need to manage it in any way. + +However, in some situations your application's performance can benefit from setting a custom comparison routine. You can do this either for database keys, or for the data if your database supports sorted duplicate records. + +Some of the reasons why you may want to provide a custom sorting function are: + +- Your database is keyed using strings and you want to provide some sort of language-sensitive ordering to that data. Doing so can help increase the locality of reference that allows your database to perform at its best. + +- You are using a little-endian system (such as x86) and you are using integers as your database's keys. Berkeley DB stores keys as byte strings and little-endian integers do not sort well when viewed as byte strings. There are several solutions to this problem, one being to provide a custom comparison function. See http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/am_misc_faq.html for more information. + +- You you do not want the entire key to participate in the comparison, for whatever reason. In this case, you may want to provide a custom comparison function so that only the relevant bytes are examined. + +#### Creating Comparison Functions + +You set a BTree's key comparison function using `DB->set_bt_compare()`. You can also set a BTree's duplicate data comparison function using `DB->set_dup_compare()`. + +You cannot use these methods after the database has been opened. Also, if the database already exists when it is opened, the function provided to these methods must be the same as that historically used to create the database or corruption can occur. + +The value that you provide to the `set_bt_compare()` method is a pointer to a function that has the following signature: + +``` c +int (*function)(DB *db, const DBT *key1, const DBT *key2) +``` + +This function must return an integer value less than, equal to, or greater than 0. If key1 is considered to be greater than key2, then the function must return a value that is greater than 0. If the two are equal, then the function must return 0, and if the first key is less than the second then the function must return a negative value. + +The function that you provide to `set_dup_compare()` works in exactly the same way, except that the `DBT` parameters hold record data items instead of keys. + +For example, an example routine that is used to sort integer keys in the database is: + +``` c +int +compare_int(DB *dbp, const DBT *a, const DBT *b) +{ + int ai, bi; + + /* + * Returns: + * < 0 if a < b + * = 0 if a = b + * > 0 if a > b + */ + memcpy(&ai, a->data, sizeof(int)); + memcpy(&bi, b->data, sizeof(int)); + return (ai - bi); +} +``` + +Note that the data must first be copied into memory that is appropriately aligned, as Berkeley DB does not guarantee any kind of alignment of the underlying data, including for comparison routines. When writing comparison routines, remember that databases created on machines of different architectures may have different integer byte orders, for which your code may need to compensate. + +To cause DB to use this comparison function: + +``` c +#include +#include + +... + +DB *dbp; +int ret; + +/* Create a database */ +ret = db_create(&dbp, NULL, 0); +if (ret != 0) { + fprintf(stderr, "%s: %s\n", "my_program", + db_strerror(ret)); + return(-1); +} + +/* Set up the btree comparison function for this database */ +dbp->set_bt_compare(dbp, compare_int); + +/* Database open call follows sometime after this. */ +``` diff --git a/docs-src/guides/gsg/cachesize.md b/docs-src/guides/gsg/cachesize.md new file mode 100644 index 000000000..66c81af93 --- /dev/null +++ b/docs-src/guides/gsg/cachesize.md @@ -0,0 +1,14 @@ +--- +title: "Selecting the Cache Size" +api-name: "Selecting the Cache Size" +source: docs/gsg/C/cachesize.html +--- +## Selecting the Cache Size + +Cache size is important to your application because if it is set to too small of a value, your application's performance will suffer from too much disk I/O. On the other hand, if your cache is too large, then your application will use more memory than it actually needs. Moreover, if your application uses too much memory, then on most operating systems this can result in your application being swapped out of memory, resulting in extremely poor performance. + +You select your cache size using either `DB->set_cachesize()`, or `DB_ENV->set_cachesize()`, depending on whether you are using a database environment or not. You cache size must be a power of 2, but it is otherwise limited only by available memory and performance considerations. + +Selecting a cache size is something of an art, but fortunately you can change it any time, so it can be easily tuned to your application's changing data requirements. The best way to determine how large your cache needs to be is to put your application into a production environment and watch to see how much disk I/O is occurring. If your application is going to disk quite a lot to retrieve database records, then you should increase the size of your cache (provided that you have enough memory to do so). + +You can use the `db_stat` command line utility with the `-m` option to gauge the effectiveness of your cache. In particular, the number of pages found in the cache is shown, along with a percentage value. The closer to 100% that you can get, the better. If this value drops too low, and you are experiencing performance problems, then you should consider increasing the size of your cache, assuming you have memory to support it. diff --git a/docs-src/guides/gsg/concepts.md b/docs-src/guides/gsg/concepts.md new file mode 100644 index 000000000..768f3bf62 --- /dev/null +++ b/docs-src/guides/gsg/concepts.md @@ -0,0 +1,34 @@ +--- +title: "Berkeley DB Concepts" +api-name: "Berkeley DB Concepts" +source: docs/gsg/C/concepts.html +--- +## Berkeley DB Concepts + +Before continuing, it is useful to describe some of the larger concepts that you will encounter when building a DB application. + +Conceptually, DB databases contain *records*. Logically each record represents a single entry in the database. Each such record contains two pieces of information: a key and a data. This manual will on occasion describe a *a record's key* or a *record's data* when it is necessary to speak to one or the other portion of a database record. + +Because of the key/data pairing used for DB databases, they are sometimes thought of as a two-column table. However, data (and sometimes keys, depending on the access method) can hold arbitrarily complex data. Frequently, C structures and other such mechanisms are stored in the record. This effectively turns a 2-column table into a table with *n* columns, where *n-1* of those columns are provided by the structure's fields. + +Note that a DB database is very much like a table in a relational database system in that most DB applications use more than one database (just as most relational databases use more than one table). + +Unlike relational systems, however, a DB database contains a single collection of records organized according to a given access method (BTree, Queue, Hash, and so forth). In a relational database system, the underlying access method is generally hidden from you. + +In any case, frequently DB applications are designed so that a single database stores a specific type of data (just as in a relational database system, a single table holds entries containing a specific set of fields). Because most applications are required to manage multiple kinds of data, a DB application will often use multiple databases. + +For example, consider an accounting application. This kind of an application may manage data based on bank accounts, checking accounts, stocks, bonds, loans, and so forth. An accounting application will also have to manage information about people, banking institutions, customer accounts, and so on. In a traditional relational database, all of these different kinds of information would be stored and managed using a (probably very) complex series of tables. In a DB application, all of this information would instead be divided out and managed using multiple databases. + +DB applications can efficiently use multiple databases using an optional mechanism called an *environment*. For more information, see Environments. + +You interact with most DB APIs using special structures that contain pointers to functions. These callbacks are called *methods* because they look so much like a method on a C++ class. The variable that you use to access these methods is often referred to as a *handle*. For example, to use a database you will obtain a handle to that database. + +Retrieving a record from a database is sometimes called *getting the record* because the method that you use to retrieve the records is called `get()`. Similarly, storing database records is sometimes called *putting the record* because you use the `put()` method to do this. + +When you store, or put, a record to a database using its handle, the record is stored according to whatever sort order is in use by the database. Sorting is mostly performed based on the key, but sometimes the data is considered too. If you put a record using a key that already exists in the database, then the existing record is replaced with the new data. However, if the database supports duplicate records (that is, records with identical keys but different data), then that new record is stored as a duplicate record and any existing records are not overwritten. + +If a database supports duplicate records, then you can use a database handle to retrieve only the first record in a set of duplicate records. + +In addition to using a database handle, you can also read and write data using a special mechanism called a *cursor*. Cursors are essentially iterators that you can use to walk over the records in a database. You can use cursors to iterate over a database from the first record to the last, and from the last to the first. You can also use cursors to seek to a record. In the event that a database supports duplicate records, cursors are the only way you can access all the records in a set of duplicates. + +Finally, DB provides a special kind of a database called a *secondary database*. Secondary databases serve as an index into normal databases (called primary database to distinguish them from secondaries). Secondary databases are interesting because DB records can hold complex data types, but seeking to a given record is performed only based on that record's key. If you wanted to be able to seek to a record based on some piece of information that is not the key, then you enable this through the use of secondary databases. diff --git a/docs-src/guides/gsg/coredbclose.md b/docs-src/guides/gsg/coredbclose.md new file mode 100644 index 000000000..457c3e1d8 --- /dev/null +++ b/docs-src/guides/gsg/coredbclose.md @@ -0,0 +1,34 @@ +--- +title: "Closing Databases" +api-name: "Closing Databases" +source: docs/gsg/C/coredbclose.html +--- +## Closing Databases + +Once you are done using the database, you must close it. You use the `DB->close()` method to do this. + +Closing a database causes it to become unusable until it is opened again. It is recommended that you close any open cursors before closing your database. Active cursors during a database close can cause unexpected results, especially if any of those cursors are writing to the database. You should always make sure that all your database accesses have completed before closing your database. + +Cursors are described in Using Cursors later in this manual. + +Be aware that when you close the last open handle for a database, then by default its cache is flushed to disk. This means that any information that has been modified in the cache is guaranteed to be written to disk when the last handle is closed. You can manually perform this operation using the `DB->sync()` method, but for normal shutdown operations it is not necessary. For more information about syncing your cache, see Data Persistence. + +The following code fragment illustrates a database close: + +``` c +#include +... +DB *dbp; /* DB struct handle */ +... + +/* + * Database open and access operations + * happen here. + */ + +... + +/* When we're done with the database, close it. */ +if (dbp != NULL) + dbp->close(dbp, 0); +``` diff --git a/docs-src/guides/gsg/coreindexusage.md b/docs-src/guides/gsg/coreindexusage.md new file mode 100644 index 000000000..b679dd31b --- /dev/null +++ b/docs-src/guides/gsg/coreindexusage.md @@ -0,0 +1,498 @@ +--- +title: "Secondary Database Example" +api-name: "Secondary Database Example" +source: docs/gsg/C/coreindexusage.html +--- +## Secondary Database Example + + [Secondary Databases with example_database_load](coreindexusage.md#edlWIndexes) + + [Secondary Databases with example_database_read](coreindexusage.md#edrWIndexes) + +In previous chapters in this book, we built applications that load and display several DB databases. In this example, we will extend those examples to use secondary databases. Specifically: + +- In Database Usage Example we built an application that can open and load data into several databases. In Secondary Databases with example_database_load we will extend that application to also open a secondary database for the purpose of indexing inventory item names. + +- In Cursor Example we built an application to display our inventory database (and related vendor information). In Secondary Databases with example_database_read we will extend that application to show inventory records based on the index we cause to be loaded using `example_database_load`. + +### Secondary Databases with example_database_load + +`example_database_load` uses several utility functions to open and close its databases. In order to cause `example_database_load` to maintain an index of inventory item names, all we really need to do is update the utility functions to: + +1. Create a new database to be used as a secondary database. + +2. Associate our new database to the inventory primary database. + +3. Close the secondary database when we close the rest of our databases. + +We also need a function that can create our secondary keys for us. + +Because DB maintains secondary databases for us; once this work is done we need not make any other changes to `example_database_load`. Therefore, we can limit all our work to the code found in `gettingstarted_common.h` and `gettingstarted_common.c`. + +Remember that you can find the complete implementation of these functions in: + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +To begin, we need to update the `stock_dbs` structure to accommodate the additional database. We defined this structure in `gettingstarted_common.h`. We can limit our update to this file to just that structure definition: + +Remember that new code is in **`bold`**. + +``` c +/* file: gettingstarted_common.h */ +#include + +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + DB *itemname_sdbp; /* Index based on the item name index */ + + char *db_home_dir; /* Directory containing the database files */ + char *itemname_db_name; /* Itemname secondary database */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; + +/* Function prototypes */ +int databases_setup(STOCK_DBS *, const char *, FILE *); +int databases_close(STOCK_DBS *); +void initialize_stockdbs(STOCK_DBS *); +int open_database(DB **, const char *, const char *, FILE *, int); +void set_db_filenames(STOCK_DBS *my_stock); +``` + +Because we updated our stock_dbs structure, we need to update our stock_dbs utility functions (The stock_db Utility Functions) accordingly. The updates are trivial and so we won't show them here in the interest of space. You can find their complete implementation in the `gettingstarted_common.c` file accompanying this example in your DB distribution. + +More importantly, however, we need to go to `gettingstarted_common.c` and create our secondary key extractor function. When we store our inventory items, we place the item name in the buffer immediately after a `float` and an `int`, so retrieving the string from the buffer is fairly easy to do: + +``` c +/* file: gettingstarted_common.c */ +#include "gettingstarted_common.h" + +/* + * Used to extract an inventory item's name from an + * inventory database record. This function is used to create + * keys for secondary database records. + */ +int +get_item_name(DB *dbp, const DBT *pkey, const DBT *pdata, DBT *skey) +{ + int offset; + + /* + * First, obtain the buffer location where we placed the + * item's name. In this example, the item's name is located + * in the primary data. It is the first string in the + * buffer after the price (a float) and the quantity (an int). + * + * See load_inventory_database() in example_database_load.c + * for how we marshalled the inventory information into the + * data DBT. + */ + offset = sizeof(float) + sizeof(int); + + /* Check to make sure there's data */ + if (pdata->size < offset) + return (-1); /* Returning non-zero means that the + * secondary record is not created/updated. + */ + + /* Now set the secondary key's data to be the item name */ + memset(skey, 0, sizeof(DBT)); + skey->data = pdata->data + offset; + skey->size = strlen(skey->data) + 1; + + return (0); +} +``` + +Having completed that function, we need to update `set_db_filenames()` and `initialize_stockdbs()` to handle the new secondary databases that our application will now use. These functions were originally introduced in The stock_db Utility Functions. + +``` c +/* Initializes the STOCK_DBS struct.*/ +void +initialize_stockdbs(STOCK_DBS *my_stock) +{ + my_stock->db_home_dir = DEFAULT_HOMEDIR; + my_stock->inventory_dbp = NULL; + my_stock->vendor_dbp = NULL; + my_stock->itemname_sdbp = NULL; + + my_stock->inventory_db_name = NULL; + my_stock->vendor_db_name = NULL; + my_stock->itemname_db_name = NULL; +} + +/* Identify all the files that will hold our databases. */ +void +set_db_filenames(STOCK_DBS *my_stock) +{ + size_t size; + + /* Create the Inventory DB file name */ + size = strlen(my_stock->db_home_dir) + strlen(INVENTORYDB) + 1; + my_stock->inventory_db_name = malloc(size); + snprintf(my_stock->inventory_db_name, size, "%s%s", + my_stock->db_home_dir, INVENTORYDB); + + /* Create the Vendor DB file name */ + size = strlen(my_stock->db_home_dir) + strlen(VENDORDB) + 1; + my_stock->vendor_db_name = malloc(size); + snprintf(my_stock->vendor_db_name, size, "%s%s", + my_stock->db_home_dir, VENDORDB); + + /* Create the itemname DB file name */ + size = strlen(my_stock->db_home_dir) + strlen(ITEMNAMEDB) + 1; + my_stock->itemname_db_name = malloc(size); + snprintf(my_stock->itemname_db_name, size, "%s%s", + my_stock->db_home_dir, ITEMNAMEDB); +} +``` + +We also need to update the `open_database()` (as described in open_database() Function) to take special actions if we are opening a secondary database. Unlike our primary databases, we want to support sorted duplicates for our secondary database. This is because we are indexing based on an item's name, and item names are shared by multiple inventory records. As a result every key the secondary database (an item name) will be used by multiple records (pointers to records in our primary database). We allow this by configuring our secondary database to support duplicate records. Further, because BTrees perform best when their records are sorted, we go ahead and configure our secondary database for sorted duplicates. + +To do this, we add a parameter to the function that indicates whether we are opening a secondary database, and we add in the few lines of code necessary to set the sorted duplicates flags. + +``` c +/* Opens a database */ +int +open_database(DB **dbpp, /* The DB handle that we are opening */ + const char *file_name, /* The file in which the db lives */ + const char *program_name, /* Name of the program calling this + * function */ + FILE *error_file_pointer, + int is_secondary) +{ + DB *dbp; /* For convenience */ + u_int32_t open_flags; + int ret; + + /* Initialize the DB handle */ + ret = db_create(&dbp, NULL, 0); + if (ret != 0) { + fprintf(error_file_pointer, "%s: %s\n", program_name, + db_strerror(ret)); + return (ret); + } + + /* Point to the memory malloc'd by db_create() */ + *dbpp = dbp; + + /* Set up error handling for this database */ + dbp->set_errfile(dbp, error_file_pointer); + dbp->set_errpfx(dbp, program_name); + + /* + * If this is a secondary database, then we want to allow + * sorted duplicates. + */ + if (is_secondary) { + ret = dbp->set_flags(dbp, DB_DUPSORT); + if (ret != 0) { + dbp->err(dbp, ret, "Attempt to set DUPSORT flag failed.", + file_name); + return (ret); + } + } + + /* Set the open flags */ + open_flags = DB_CREATE; + + /* Now open the database */ + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name (unneeded) */ + DB_BTREE, /* Database type (using btree) */ + open_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed.", file_name); + return (ret); + } + + return (ret); +} +``` + +That done, we can now update `databases_setup()` (see The databases_setup() Function) to create and open our secondary database. To do this, we have to add a flag to each call to `open_database()` that indicates whether the database is a secondary. We also have to associate our secondary database with the inventory database (the primary). + +Note that we do not anywhere in this example show the definition of `PRIMARY_DB` and `SECONDARY_DB`. See `gettingstarted_common.h` in your DB examples directory for those definitions (they are just `0` and `1`, respectively). + +``` c +/* opens all databases */ +int +databases_setup(STOCK_DBS *my_stock, const char *program_name, + FILE *error_file_pointer) +{ + int ret; + + /* Open the vendor database */ + ret = open_database(&(my_stock->vendor_dbp), + my_stock->vendor_db_name, + program_name, error_file_pointer, + PRIMARY_DB); + if (ret != 0) + /* + * Error reporting is handled in open_database() so just return + * the return code here. + */ + return (ret); + + /* Open the inventory database */ + ret = open_database(&(my_stock->inventory_dbp), + my_stock->inventory_db_name, + program_name, error_file_pointer, + PRIMARY_DB); + if (ret != 0) + /* + * Error reporting is handled in open_database() so just return + * the return code here. + */ + return (ret); + + /* + * Open the itemname secondary database. This is used to + * index the product names found in the inventory + * database. + */ + ret = open_database(&(my_stock->itemname_sdbp), + my_stock->itemname_db_name, + program_name, error_file_pointer, + SECONDARY_DB); + if (ret != 0) + /* + * Error reporting is handled in open_database() so just return + * the return code here. + */ + return (ret); + + /* + * Associate the itemname db with its primary db + * (inventory db). + */ + my_stock->inventory_dbp->associate( + my_stock->inventory_dbp, /* Primary db */ + NULL, /* txn id */ + my_stock->itemname_sdbp, /* Secondary db */ + get_item_name, /* Secondary key extractor */ + 0); /* Flags */ + + + printf("databases opened successfully\n"); + return (0); +} +``` + +Finally, we need to update `databases_close()` (The databases_close() Function) to close our new secondary database. Note that we are careful to close the secondary before the primary, even though the database close routine is single threaded. + +``` c +/* Closes all the databases and secondary databases. */ +int +databases_close(STOCK_DBS *my_stock) +{ + int ret; + /* + * Note that closing a database automatically flushes its cached data + * to disk, so no sync is required here. + */ + + if (my_stock->itemname_sdbp != NULL) { + ret = my_stock->itemname_sdbp->close(my_stock->itemname_sdbp, 0); + if (ret != 0) + fprintf(stderr, "Itemname database close failed: %s\n", + db_strerror(ret)); + } + + if (my_stock->inventory_dbp != NULL) { + ret = my_stock->inventory_dbp->close(my_stock->inventory_dbp, 0); + if (ret != 0) + fprintf(stderr, "Inventory database close failed: %s\n", + db_strerror(ret)); + } + + if (my_stock->vendor_dbp != NULL) { + ret = my_stock->vendor_dbp->close(my_stock->vendor_dbp, 0); + if (ret != 0) + fprintf(stderr, "Vendor database close failed: %s\n", + db_strerror(ret)); + } + + printf("databases closed.\n"); + return (0); +} +``` + +And the implementation changes slightly to take advantage of the new boolean. Note that to save space, we just show the constructor where the code actually changes: + +That completes our update to `example_database_load`. Now when this program is called, it will automatically index inventory items based on their names. We can then query for those items using the new index. We show how to do that in the next section. + +### Secondary Databases with example_database_read + +In Cursor Example we wrote an application that displays every inventory item in the Inventory database. In this section, we will update that example to allow us to search for and display an inventory item given a specific name. To do this, we will make use of the secondary database that `example_database_load` now creates. + +Because we manage all our database open and close activities in `databases_setup()` and `databases_close()`, the update to `example_database_read` is relatively modest. We need only add a command line parameter on which we can specify the item name, and we will need a new function in which we will perform the query and display the results. + +To begin, we add a single forward declaration to the application, and update our usage function slightly: + +``` c +/* File: example_database_read.c */ +/* gettingstarted_common.h includes db.h for us */ +#include "gettingstarted_common.h" + +/* Forward declarations */ +char * show_inventory_item(void *); +int show_all_records(STOCK_DBS *); +int show_records(STOCK_DBS *, char *); +int show_vendor_record(char *, DB *); +``` + +Next, we update `main()` to accept the new command line switch. We also need a new variable to contain the item's name. + +``` c +/* + * Searches for a inventory item based on that item's name. The search is + * performed using the item name secondary database. Displays all + * inventory items that use the specified name, as well as the vendor + * associated with that inventory item. + * + * If no item name is provided, then all inventory items are displayed. + */ +int +main(int argc, char *argv[]) +{ + STOCK_DBS my_stock; + int ret; + char *itemname; + + /* Initialize the STOCK_DBS struct */ + initialize_stockdbs(&my_stock); + + itemname = NULL; + /* + * Parse the command line arguments here and determine + * the location of the database files as well as the + * inventory item we want displayed, if any. This step is + * omitted for brevity. + */ + + /* + * Identify the files that will hold our databases + * This function uses information obtained from the + * command line to identify the directory in which + * the database files reside. + */ + set_db_filenames(&my_stock); + + /* Open all databases */ + ret = databases_setup(&my_stock, "example_database_read", stderr); + if (ret != 0) { + fprintf(stderr, "Error opening databases\n"); + databases_close(&my_stock); + return (ret); + } +``` + +The final update to the `main()` entails a little bit of logic to determine whether we want to display all available inventory items, or just the ones that match a name provided on the `-i` command line parameter. + +``` c + /* + * Show either a single item or all items, depending + * on whether itemname is set to a value. + */ + if (itemname == NULL) + ret = show_all_records(&my_stock); + else + ret = show_records(&my_stock, itemname); + + /* Close our databases */ + databases_close(&my_stock); + return (ret); +} +``` + +The only other thing that we need to add to the application is the implementation of the `show_records()` function. + +### Note + +In the interest of space, we refrain from showing the other functions used by this application. For their implementation, please see Cursor Example. Alternatively, you can see the entire implementation of this application in: + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +``` c +/* + * Search for an inventory item given its name (using the inventory item + * secondary database) and display that record and any duplicates that may + * exist. + */ +int +show_records(STOCK_DBS *my_stock, char *itemname) +{ + DBC *itemname_cursorp; + DBT key, data; + char *the_vendor; + int ret, exit_value; + + /* Initialize our DBTs. */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + /* Get a cursor to the itemname db */ + my_stock->itemname_sdbp->cursor(my_stock->itemname_sdbp, 0, + &itemname_cursorp, 0); + + /* + * Get the search key. This is the name on the inventory + * record that we want to examine. + */ + key.data = itemname; + key.size = strlen(itemname) + 1; + + /* + * Position our cursor to the first record in the secondary + * database that has the appropriate key. + */ + exit_value = 0; + ret = itemname_cursorp->get(itemname_cursorp, &key, &data, DB_SET); + if (!ret) { + do { + /* + * Show the inventory record and the vendor responsible + * for this inventory item. + */ + the_vendor = show_inventory_item(data.data); + ret = show_vendor_record(the_vendor, my_stock->vendor_dbp); + if (ret) { + exit_value = ret; + break; + } + /* + * Our secondary allows duplicates, so we need to loop over + * the next duplicate records and show them all. This is done + * because an inventory item's name is not a unique value. + */ + } while(itemname_cursorp->get(itemname_cursorp, &key, &data, + DB_NEXT_DUP) == 0); + } else { + printf("No records found for '%s'\n", itemname); + } + + /* Close the cursor */ + itemname_cursorp->close(itemname_cursorp); + + return (exit_value); +} +``` + +This completes our update to `example_inventory_read`. Using this update, you can now search for and show all inventory items that match a particular name. For example: + +``` c + example_inventory_read -i "Zulu Nut" +``` diff --git a/docs-src/guides/gsg/cstructs.md b/docs-src/guides/gsg/cstructs.md new file mode 100644 index 000000000..7078f283e --- /dev/null +++ b/docs-src/guides/gsg/cstructs.md @@ -0,0 +1,213 @@ +--- +title: "Using C Structures with DB" +api-name: "Using C Structures with DB" +source: docs/gsg/C/cstructs.html +--- +## Using C Structures with DB + + [C Structures with Pointers](cstructs.md#cstructdynamic) + +Storing data in structures is a handy way to pack varied types of information into each database record. DB databases are sometimes thought of as a two column table where column 1 is the key and column 2 is the data. By using structures, you can effectively turn this table into *n* columns where *n-1* columns are contained in the structure. + +So long as a C structure contains fields that are not pointers, you can safely store and retrieve them in the same way as you would any primitive datatype. The following code fragment illustrates this: + +``` c +#include +#include + +typedef struct my_struct { + int id; + char familiar_name[MAXLINE]; /* Some suitably large value */ + char surname[MAXLINE]; +} MY_STRUCT; + +... + +DBT key, data; +DB *my_database; +MY_STRUCT user; +char *fname = "David"; +char *sname = "Rider"; + +/* Database open omitted for clarity */ + +user.id = 1; +strncpy(user.familiar_name, fname, strlen(fname)+1); +strncpy(user.surname, sname, strlen(sname)+1); + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = &(user.id); +key.size = sizeof(int); + +data.data = &user; +data.size = sizeof(MY_STRUCT); + +my_database->put(my_database, NULL, &key, &data, DB_NOOVERWRITE); +``` + +To retrieve the structure, make sure you supply your own memory. The reason why is that like real numbers, some systems require structures to be aligned in a specific way. Because it is possible that the memory DB provides is not aligned properly, for safest result simply use your own memory: + +``` c +#include +#include + +... + +DBT key, data; +DB *my_database; +MY_STRUCT user; + +/* Database open omitted for clarity */ + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +/* Initialize the structure */ +memset(&user, 0, sizeof(MY_STRUCT)); +user.id = 1; + +key.data = &user.id; +key.size = sizeof(int); + +/* Use our memory to retrieve the structure */ +data.data = &user; +data.ulen = sizeof(MY_STRUCT); +data.flags = DB_DBT_USERMEM; + +my_database->get(my_database, NULL, &key, &data, 0); + +printf("Familiar name: %s\n", user.familiar_name); +printf("Surname: %s\n", user.surname); +``` + +Be aware that while this is the easiest way to manage structures stored in DB databases, this approach does suffer from causing your database to be larger than is strictly necessary. Each structure stored in the database is of a fixed size, and you do not see any space savings from storing a (for example) 5 character surname versus a 20 character surname. + +For a simple example such as this, the padding stored with each record is probably not critical. However, if you are storing structures that contain a very large number of character arrays, or if you are simply storing millions of records, then you may want to avoid this approach. The wasted space in each record will only serve to make your databases larger than need be, which will in turn require a larger cache and more disk I/O than you would ordinarily need. + +An alternative approach is described next. + +### C Structures with Pointers + +It is often necessary in C structures to use fields that are pointers to dynamically allocated memory. This is particularly true if you want to store character strings (or any kind of an array for that matter), and you want to avoid any overhead caused by pre-designating the size of the array. + +When storing structures like these you need to make sure that all of the data pointed to and contained by the structure is lined up in a single contiguous block of memory. Remember that DB stores data located at a specific address and of a particular size. If your structure includes fields that are pointing to dynamically allocated memory, then the data that you want to store can be located in different, not necessarily contiguous, locations on the heap. + +The easiest way to solve this problem is to pack your data into a single memory location and then store the data in that location. (This process is sometimes called *marshalling the data*.) For example: + +``` c +#include +#include +#include + +typedef struct my_struct { + int id; + char *familiar_name; + char *surname; +} MY_STRUCT; + +... + +DBT key, data; +DB *my_database; +MY_STRUCT user; +int buffsize, bufflen; +char fname[ ] = "Pete"; +char sname[10]; +char *databuff; + +strncpy(sname, "Oar", strlen("Oar")+1); + +/* Database open omitted for clarity */ + +user.id = 1; +user.familiar_name = fname; +user.surname = sname; + +/* Some of the structure's data is on the stack, and + * some is on the heap. To store this structure's data, we + * need to marshall it -- pack it all into a single location + * in memory. + */ + +/* Get the buffer */ +buffsize = sizeof(int) + + (strlen(user.familiar_name) + strlen(user.surname) + 2); +databuff = malloc(buffsize); +memset(databuff, 0, buffsize); + +/* copy everything to the buffer */ +memcpy(databuff, &(user.id), sizeof(int)); +bufflen = sizeof(int); + +memcpy(databuff + bufflen, user.familiar_name, + strlen(user.familiar_name) + 1); +bufflen += strlen(user.familiar_name) + 1; + +memcpy(databuff + bufflen, user.surname, + strlen(user.surname) + 1); +bufflen += strlen(user.surname) + 1; + +/* Now store it */ + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = &(user.id); +key.size = sizeof(int); + +data.data = databuff; +data.size = bufflen; + +my_database->put(my_database, NULL, &key, &data, DB_NOOVERWRITE); +free(sname); +free(databuff); +``` + +To retrieve the stored structure: + +``` c +#include +#include +#include + +typedef struct my_struct { + char *familiar_name; + char *surname; + int id; +} MY_STRUCT; + +... + +int id; +DBT key, data; +DB *my_database; +MY_STRUCT user; +char *buffer; + +/* Database open omitted for clarity */ + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +id = 1; +key.data = &id; +key.size = sizeof(int); + +my_database->get(my_database, NULL, &key, &data, 0); + +/* + * Some compilers won't allow pointer arithmetic on void *'s, + * so use a char * instead. + */ +buffer = data.data; + +user.id = *((int *)data.data); +user.familiar_name = buffer + sizeof(int); +user.surname = buffer + sizeof(int) + strlen(user.familiar_name) + 1; +``` diff --git a/docs-src/guides/gsg/databaseLimits.md b/docs-src/guides/gsg/databaseLimits.md new file mode 100644 index 000000000..0cf1dfbfe --- /dev/null +++ b/docs-src/guides/gsg/databaseLimits.md @@ -0,0 +1,12 @@ +--- +title: "Database Limits and Portability" +api-name: "Database Limits and Portability" +source: docs/gsg/C/databaseLimits.html +--- +## Database Limits and Portability + +Berkeley DB provides support for managing everything from very small databases that fit entirely in memory, to extremely large databases holding millions of records and terabytes of data. DB databases can store up to 256 terabytes of data. Individual record keys or record data can store up to 4 gigabytes of data. + +DB's databases store data in a binary format that is portable across platforms, even of differing endian-ness. Be aware, however, that portability aside, some performance issues can crop up in the event that you are using little endian architecture. See Setting Comparison Functions for more information. + +Also, DB's databases and data structures are designed for concurrent access — they are thread-safe, and they share well across multiple processes. That said, in order to allow multiple processes to share databases and the cache, DB makes use of mechanisms that do not work well on network-shared drives (NFS or Windows networks shares, for example). For this reason, you cannot place your DB databases and environments on network-mounted drives. diff --git a/docs-src/guides/gsg/databases.md b/docs-src/guides/gsg/databases.md new file mode 100644 index 000000000..c1b8aafe8 --- /dev/null +++ b/docs-src/guides/gsg/databases.md @@ -0,0 +1,68 @@ +--- +title: "Chapter 2. Databases" +api-name: "Chapter 2. Databases" +source: docs/gsg/C/databases.html +--- +## Chapter 2. Databases + +**Table of Contents** + + [Opening Databases](databases.md#DBOpen) + + [Closing Databases](coredbclose.md) + + [Database Open Flags](DBOpenFlags.md) + + [Administrative Methods](CoreDBAdmin.md) + + [Error Reporting Functions](dbErrorReporting.md) + + [Managing Databases in Environments](CoreEnvUsage.md) + + [Database Example](CoreDbUsage.md) + +In Berkeley DB, a database is a collection of *records*. Records, in turn, consist of key/data pairings. + +Conceptually, you can think of a database as containing a two-column table where column 1 contains a key and column 2 contains data. Both the key and the data are managed using `DBT` structures (see Database Records for details on this structure). So, fundamentally, using a DB database involves putting, getting, and deleting database records, which in turns involves efficiently managing information contained in `DBT` structures. The next several chapters of this book are dedicated to those activities. + +## Opening Databases + +To open a database, you must first use the `db_create()` function to initialize a `DB` handle. Once you have initialized the `DB` handle, you use its `open()` method to open the database. + +Note that by default, DB does not create databases if they do not already exist. To override this behavior, specify the DB_CREATE flag on the `open()` method. + +The following code fragment illustrates a database open: + +``` c +#include + +... + +DB *dbp; /* DB structure handle */ +u_int32_t flags; /* database open flags */ +int ret; /* function return value */ + +/* Initialize the structure. This + * database is not opened in an environment, + * so the environment pointer is NULL. */ +ret = db_create(&dbp, NULL, 0); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Database open flags */ +flags = DB_CREATE; /* If the database does not exist, + * create it.*/ + +/* open the database */ +ret = dbp->open(dbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_db.db", /* On-disk file that holds the database. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + flags, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} +``` diff --git a/docs-src/guides/gsg/dbErrorReporting.md b/docs-src/guides/gsg/dbErrorReporting.md new file mode 100644 index 000000000..6bdcde010 --- /dev/null +++ b/docs-src/guides/gsg/dbErrorReporting.md @@ -0,0 +1,96 @@ +--- +title: "Error Reporting Functions" +api-name: "Error Reporting Functions" +source: docs/gsg/C/dbErrorReporting.html +--- +## Error Reporting Functions + +To simplify error reporting and handling, the `DB` structure offers several useful methods. + +- `set_errcall()` + + Defines the function that is called when an error message is issued by DB. The error prefix and message are passed to this callback. It is up to the application to display this information correctly. + +- `set_errfile()` + + Sets the C library `FILE *` to be used for displaying error messages issued by the DB library. + +- `set_errpfx()` + + Sets the prefix used for any error messages issued by the DB library. + +- `err()` + + Issues an error message. The error message is sent to the callback function as defined by `set_errcall`. If that method has not been used, then the error message is sent to the file defined by `set_errfile()`. If none of these methods have been used, then the error message is sent to standard error. + + The error message consists of the prefix string (as defined by `set_errpfx()`), an optional `printf`-style formatted message, the error message, and a trailing newline. + +- `errx()` + + Behaves identically to `err()` except that the DB message text associated with the supplied error value is not appended to the error string. + +In addition, you can use the `db_strerror()` function to directly return the error string that corresponds to a particular error number. + +For example, to send all error messages for a given database handle to a callback for handling, first create your callback. Do something like this: + +``` c +/* + * Function called to handle any database error messages + * issued by DB. + */ +void +my_error_handler(const DB_ENV *dbenv, const char *error_prefix, + const char *msg) +{ + /* + * Put your code to handle the error prefix and error + * message here. Note that one or both of these parameters + * may be NULL depending on how the error message is issued + * and how the DB handle is configured. + */ +} +``` + +And then register the callback as follows: + +``` c +#include +#include + +... + +DB *dbp; +int ret; + +/* + * Create a database and initialize it for error + * reporting. + */ +ret = db_create(&dbp, NULL, 0); +if (ret != 0) { + fprintf(stderr, "%s: %s\n", "my_program", + db_strerror(ret)); + return(ret); +} + +/* Set up error handling for this database */ +dbp->set_errcall(dbp, my_error_handler); +dbp->set_errpfx(dbp, "my_example_program"); +``` + +And to issue an error message: + +``` c +ret = dbp->open(dbp, + NULL, + "mydb.db", + NULL, + DB_BTREE, + DB_CREATE, + 0); +if (ret != 0) { + dbp->err(dbp, ret, + "Database open failed: %s", "mydb.db"); + return(ret); +} +``` diff --git a/docs-src/guides/gsg/dbconfig.md b/docs-src/guides/gsg/dbconfig.md new file mode 100644 index 000000000..34763587d --- /dev/null +++ b/docs-src/guides/gsg/dbconfig.md @@ -0,0 +1,110 @@ +--- +title: "Chapter 6. Database Configuration" +api-name: "Chapter 6. Database Configuration" +source: docs/gsg/C/dbconfig.html +--- +## Chapter 6. Database Configuration + +**Table of Contents** + + [Setting the Page Size](dbconfig.md#pagesize) + + [Overflow Pages](dbconfig.md#overflowpages) + + [Locking](dbconfig.md#Locking) + + [IO Efficiency](dbconfig.md#IOEfficiency) + + [Page Sizing Advice](dbconfig.md#pagesizeAdvice) + + [Selecting the Cache Size](cachesize.md) + + [BTree Configuration](btree.md) + + [Allowing Duplicate Records](btree.md#duplicateRecords) + + [Setting Comparison Functions](btree.md#comparators) + +This chapter describes some of the database and cache configuration issues that you need to consider when building your DB database. In most cases, there is very little that you need to do in terms of managing your databases. However, there are configuration issues that you need to be concerned with, and these are largely dependent on the access method that you are choosing for your database. + +The examples and descriptions throughout this document have mostly focused on the BTree access method. This is because the majority of DB applications use BTree. For this reason, where configuration issues are dependent on the type of access method in use, this chapter will focus on BTree only. For configuration descriptions surrounding the other access methods, see the *Berkeley DB Programmer's Reference Guide*. + +## Setting the Page Size + + [Overflow Pages](dbconfig.md#overflowpages) + + [Locking](dbconfig.md#Locking) + + [IO Efficiency](dbconfig.md#IOEfficiency) + + [Page Sizing Advice](dbconfig.md#pagesizeAdvice) + +Internally, DB stores database entries on pages. Page sizes are important because they can affect your application's performance. + +DB pages can be between 512 bytes and 64K bytes in size. The size that you select must be a power of 2. You set your database's page size using `DB->set_pagesize()`. + +Note that a database's page size can only be selected at database creation time. + +When selecting a page size, you should consider the following issues: + +- Overflow pages. + +- Locking + +- Disk I/O. + +These topics are discussed next. + +### Overflow Pages + +Overflow pages are used to hold a key or data item that cannot fit on a single page. You do not have to do anything to cause overflow pages to be created, other than to store data that is too large for your database's page size. Also, the only way you can prevent overflow pages from being created is to be sure to select a page size that is large enough to hold your database entries. + +Because overflow pages exist outside of the normal database structure, their use is expensive from a performance perspective. If you select too small of a page size, then your database will be forced to use an excessive number of overflow pages. This will significantly harm your application's performance. + +For this reason, you want to select a page size that is at least large enough to hold multiple entries given the expected average size of your database entries. In BTree's case, for best results select a page size that can hold at least 4 such entries. + +You can see how many overflow pages your database is using by using the `DB->stat()` method, or by examining your database using the `db_stat` command line utility. + +### Locking + +Locking and multi-threaded access to DB databases is built into the product. However, in order to enable the locking subsystem and in order to provide efficient sharing of the cache between databases, you must use an *environment*. Environments and multi-threaded access are not fully described in this manual (see the Berkeley DB Programmer's Reference Manual for information), however, we provide some information on sizing your pages in a multi-threaded/multi-process environment in the interest of providing a complete discussion on the topic. + +If your application is multi-threaded, or if your databases are accessed by more than one process at a time, then page size can influence your application's performance. The reason why is that for most access methods (Queue is the exception), DB implements page-level locking. This means that the finest locking granularity is at the page, not at the record. + +In most cases, database pages contain multiple database records. Further, in order to provide safe access to multiple threads or processes, DB performs locking on pages as entries on those pages are read or written. + +As the size of your page increases relative to the size of your database entries, the number of entries that are held on any given page also increase. The result is that the chances of two or more readers and/or writers wanting to access entries on any given page also increases. + +When two or more threads and/or processes want to manage data on a page, lock contention occurs. Lock contention is resolved by one thread (or process) waiting for another thread to give up its lock. It is this waiting activity that is harmful to your application's performance. + +It is possible to select a page size that is so large that your application will spend excessive, and noticeable, amounts of time resolving lock contention. Note that this scenario is particularly likely to occur as the amount of concurrency built into your application increases. + +Oh the other hand, if you select too small of a page size, then that that will only make your tree deeper, which can also cause performance penalties. The trick, therefore, is to select a reasonable page size (one that will hold a sizeable number of records) and then reduce the page size if you notice lock contention. + +You can examine the number of lock conflicts and deadlocks occurring in your application by examining your database environment lock statistics. Either use the `DB_ENV->lock_stat()` method, or use the `db_stat` command line utility. The number of unavailable locks that your application waited for is held in the lock statistic's `st_lock_wait` field. + +### IO Efficiency + +Page size can affect how efficient DB is at moving data to and from disk. For some applications, especially those for which the in-memory cache can not be large enough to hold the entire working dataset, IO efficiency can significantly impact application performance. + +Most operating systems use an internal block size to determine how much data to move to and from disk for a single I/O operation. This block size is usually equal to the filesystem's block size. For optimal disk I/O efficiency, you should select a database page size that is equal to the operating system's I/O block size. + +Essentially, DB performs data transfers based on the database page size. That is, it moves data to and from disk a page at a time. For this reason, if the page size does not match the I/O block size, then the operating system can introduce inefficiencies in how it responds to DB's I/O requests. + +For example, suppose your page size is smaller than your operating system block size. In this case, when DB writes a page to disk it is writing just a portion of a logical filesystem page. Any time any application writes just a portion of a logical filesystem page, the operating system brings in the real filesystem page, over writes the portion of the page not written by the application, then writes the filesystem page back to disk. The net result is significantly more disk I/O than if the application had simply selected a page size that was equal to the underlying filesystem block size. + +Alternatively, if you select a page size that is larger than the underlying filesystem block size, then the operating system may have to read more data than is necessary to fulfill a read request. Further, on some operating systems, requesting a single database page may result in the operating system reading enough filesystem blocks to satisfy the operating system's criteria for read-ahead. In this case, the operating system will be reading significantly more data from disk than is actually required to fulfill DB's read request. + +### Note + +While transactions are not discussed in this manual, a page size other than your filesystem's block size can affect transactional guarantees. The reason why is that page sizes larger than the filesystem's block size causes DB to write pages in block size increments. As a result, it is possible for a partial page to be written as the result of a transactional commit. For more information, see http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/transapp_reclimit.html. + +### Page Sizing Advice + +Page sizing can be confusing at first, so here are some general guidelines that you can use to select your page size. + +In general, and given no other considerations, a page size that is equal to your filesystem block size is the ideal situation. + +If your data is designed such that 4 database entries cannot fit on a single page (assuming BTree), then grow your page size to accommodate your data. Once you've abandoned matching your filesystem's block size, the general rule is that larger page sizes are better. + +The exception to this rule is if you have a great deal of concurrency occurring in your application. In this case, the closer you can match your page size to the ideal size needed for your application's data, the better. Doing so will allow you to avoid unnecessary contention for page locks. diff --git a/docs-src/guides/gsg/environments.md b/docs-src/guides/gsg/environments.md new file mode 100644 index 000000000..88d05a77a --- /dev/null +++ b/docs-src/guides/gsg/environments.md @@ -0,0 +1,36 @@ +--- +title: "Environments" +api-name: "Environments" +source: docs/gsg/C/environments.html +--- +## Environments + +This manual is meant as an introduction to the Berkeley DB library. Consequently, it describes how to build a very simple, single-threaded application and so this manual omits a great many powerful aspects of the DB database engine that are not required by simple applications. One of these is important enough that it warrants a brief overview here: environments. + +While environments are frequently not used by applications running in embedded environments where every byte counts, they will be used by virtually any other DB application requiring anything other than the bare minimum functionality. + +An *environment* is essentially an encapsulation of one or more databases. You open an environment and then you open databases in that environment. When you do so, the databases are created/located in a location relative to the environment's home directory. + +Environments offer a great many features that a stand-alone DB database cannot offer: + +- Multi-database files. + + It is possible in DB to contain multiple databases in a single physical file on disk. This is desirable for those application that open more than a few handful of databases. However, in order to have more than one database contained in a single physical file, your application *must* use an environment. + +- Multi-thread and multi-process support + + When you use an environment, resources such as the in-memory cache and locks can be shared by all of the databases opened in the environment. The environment allows you to enable subsystems that are designed to allow multiple threads and/or processes to access DB databases. For example, you use an environment to enable the concurrent data store (CDS), the locking subsystem, and/or the shared memory buffer pool. + +- Transactional processing + + DB offers a transactional subsystem that allows for full ACID-protection of your database writes. You use environments to enable the transactional subsystem, and then subsequently to obtain transaction IDs. + +- High availability (replication) support + + DB offers a replication subsystem that enables single-master database replication with multiple read-only copies of the replicated data. You use environments to enable and then manage this subsystem. + +- Logging subsystem + + DB offers write-ahead logging for applications that want to obtain a high-degree of recoverability in the face of an application or system crash. Once enabled, the logging subsystem allows the application to perform two kinds of recovery ("normal" and "catastrophic") through the use of the information contained in the log files. + +For more information on these topics, see the *Berkeley DB Getting Started with Transaction Processing* guide and the *Berkeley DB Getting Started with Replicated Applications* guide. diff --git a/docs-src/guides/gsg/gettingit.md b/docs-src/guides/gsg/gettingit.md new file mode 100644 index 000000000..6d9efbfae --- /dev/null +++ b/docs-src/guides/gsg/gettingit.md @@ -0,0 +1,12 @@ +--- +title: "Getting and Using DB" +api-name: "Getting and Using DB" +source: docs/gsg/C/gettingit.html +--- +## Getting and Using DB + +You can obtain DB by visiting the Berkeley DB download page: http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +To install DB, untar or unzip the distribution to the directory of your choice. You will then need to build the product binaries. For information on building DB, see *DB_INSTALL*`/docs/index.html`, where *DB_INSTALL* is the directory where you unpacked DB. On that page, you will find links to platform-specific build instructions. + +That page also contains links to more documentation for DB. In particular, you will find links for the *Berkeley DB Programmer's Reference Guide* as well as the API reference documentation. diff --git a/docs-src/guides/gsg/index.md b/docs-src/guides/gsg/index.md new file mode 100644 index 000000000..101f38340 --- /dev/null +++ b/docs-src/guides/gsg/index.md @@ -0,0 +1,168 @@ +--- +title: "Getting Started with Berkeley DB" +api-name: "Getting Started with Berkeley DB" +source: docs/gsg/C/index.html +--- +# Getting Started with Berkeley DB + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction to Berkeley DB](introduction.md) + + [About This Manual](introduction.md#aboutthismanual) + + [Berkeley DB Concepts](concepts.md) + + [Access Methods](accessmethods.md) + + [Selecting Access Methods](accessmethods.md#selectAM) + + [Choosing between BTree and Hash](accessmethods.md#BTreeVSHash) + + [Choosing between Queue and Recno](accessmethods.md#QueueVSRecno) + + [Database Limits and Portability](databaseLimits.md) + + [Environments](environments.md) + + [Error Returns](returns.md) + + [Getting and Using DB](gettingit.md) + + [2. Databases](databases.md) + + [Opening Databases](databases.md#DBOpen) + + [Closing Databases](coredbclose.md) + + [Database Open Flags](DBOpenFlags.md) + + [Administrative Methods](CoreDBAdmin.md) + + [Error Reporting Functions](dbErrorReporting.md) + + [Managing Databases in Environments](CoreEnvUsage.md) + + [Database Example](CoreDbUsage.md) + + [3. Database Records](DBEntry.md) + + [Using Database Records](DBEntry.md#usingDbEntry) + + [Reading and Writing Database Records](usingDbt.md) + + [Writing Records to the Database](usingDbt.md#databaseWrite) + + [Getting Records from the Database](usingDbt.md#CoreDatabaseRead) + + [Deleting Records](usingDbt.md#recordDelete) + + [Data Persistence](usingDbt.md#datapersist) + + [Using C Structures with DB](cstructs.md) + + [C Structures with Pointers](cstructs.md#cstructdynamic) + + [Database Usage Example](DbUsage.md) + + [4. Using Cursors](Cursors.md) + + [Opening and Closing Cursors](Cursors.md#openCursor) + + [Getting Records Using the Cursor](Positioning.md) + + [Searching for Records](Positioning.md#cursorsearch) + + [Working with Duplicate Records](Positioning.md#getdups) + + [Putting Records Using Cursors](PutEntryWCursor.md) + + [Deleting Records Using Cursors](DeleteEntryWCursor.md) + + [Replacing Records Using Cursors](ReplacingEntryWCursor.md) + + [Cursor Example](CoreCursorUsage.md) + + [5. Secondary Databases](indexes.md) + + [Opening and Closing Secondary Databases](indexes.md#CoreDbAssociate) + + [Implementing Key Extractors](keyCreator.md) + + [Working with Multiple Keys](keyCreator.md#multikeys) + + [Reading Secondary Databases](readSecondary.md) + + [Deleting Secondary Database Records](secondaryDelete.md) + + [Using Cursors with Secondary Databases](secondaryCursor.md) + + [Database Joins](joins.md) + + [Using Join Cursors](joins.md#joinUsage) + + [Secondary Database Example](coreindexusage.md) + + [Secondary Databases with example_database_load](coreindexusage.md#edlWIndexes) + + [Secondary Databases with example_database_read](coreindexusage.md#edrWIndexes) + + [6. Database Configuration](dbconfig.md) + + [Setting the Page Size](dbconfig.md#pagesize) + + [Overflow Pages](dbconfig.md#overflowpages) + + [Locking](dbconfig.md#Locking) + + [IO Efficiency](dbconfig.md#IOEfficiency) + + [Page Sizing Advice](dbconfig.md#pagesizeAdvice) + + [Selecting the Cache Size](cachesize.md) + + [BTree Configuration](btree.md) + + [Allowing Duplicate Records](btree.md#duplicateRecords) + + [Setting Comparison Functions](btree.md#comparators) + +**List of Examples** + +2.1. [The stock_db Structure](CoreDbUsage.md#stock-db) + +2.2. [The stock_db Utility Functions](CoreDbUsage.md#stock-db-functions) + +2.3. [open_database() Function](CoreDbUsage.md#open-db) + +2.4. [The databases_setup() Function](CoreDbUsage.md#databasesetup) + +2.5. [The databases_close() Function](CoreDbUsage.md#database_close) + +3.1. [VENDOR Structure](DbUsage.md#VENDORStruct) + +3.2. [example_database_load](DbUsage.md#exampledbload) + +4.1. [example_database_read](CoreCursorUsage.md#CoreEIR) diff --git a/docs-src/guides/gsg/indexes.md b/docs-src/guides/gsg/indexes.md new file mode 100644 index 000000000..b98559d6f --- /dev/null +++ b/docs-src/guides/gsg/indexes.md @@ -0,0 +1,136 @@ +--- +title: "Chapter 5. Secondary Databases" +api-name: "Chapter 5. Secondary Databases" +source: docs/gsg/C/indexes.html +--- +## Chapter 5. Secondary Databases + +**Table of Contents** + + [Opening and Closing Secondary Databases](indexes.md#CoreDbAssociate) + + [Implementing Key Extractors](keyCreator.md) + + [Working with Multiple Keys](keyCreator.md#multikeys) + + [Reading Secondary Databases](readSecondary.md) + + [Deleting Secondary Database Records](secondaryDelete.md) + + [Using Cursors with Secondary Databases](secondaryCursor.md) + + [Database Joins](joins.md) + + [Using Join Cursors](joins.md#joinUsage) + + [Secondary Database Example](coreindexusage.md) + + [Secondary Databases with example_database_load](coreindexusage.md#edlWIndexes) + + [Secondary Databases with example_database_read](coreindexusage.md#edrWIndexes) + +Usually you find database records by means of the record's key. However, the key that you use for your record will not always contain the information required to provide you with rapid access to the data that you want to retrieve. For example, suppose your database contains records related to users. The key might be a string that is some unique identifier for the person, such as a user ID. Each record's data, however, would likely contain a complex object containing details about people such as names, addresses, phone numbers, and so forth. While your application may frequently want to query a person by user ID (that is, by the information stored in the key), it may also on occasion want to locate people by, say, their name. + +Rather than iterate through all of the records in your database, examining each in turn for a given person's name, you create indexes based on names and then just search that index for the name that you want. You can do this using secondary databases. In DB, the database that contains your data is called a *primary database*. A database that provides an alternative set of keys to access that data is called a *secondary database*. In a secondary database, the keys are your alternative (or secondary) index, and the data corresponds to a primary record's key. + +You create a secondary database by creating the database, opening it, and then *associating* the database with the *primary* database (that is, the database for which you are creating the index). As a part of associating the secondary database to the primary, you must provide a callback that is used to create the secondary database keys. Typically this callback creates a key based on data found in the primary database record's key or data. + +Once opened, DB manages secondary databases for you. Adding or deleting records in your primary database causes DB to update the secondary as necessary. Further, changing a record's data in the primary database may cause DB to modify a record in the secondary, depending on whether the change forces a modification of a key in the secondary database. + +Note that you can not write directly to a secondary database. Any attempt to write to a secondary database results in a non-zero status return. To change the data referenced by a secondary record, modify the primary database instead. The exception to this rule is that delete operations are allowed on the secondary database. See Deleting Secondary Database Records for more information. + +### Note + +Secondary database records are updated/created by DB only if the key creator callback function returns `0`. If a value other than `0` is returned, then DB will not add the key to the secondary database, and in the event of a record update it will remove any existing key. Note that the callback can use either `DB_DONOTINDEX` or some error code outside of DB's name space to indicate that the entry should not be indexed. + +See Implementing Key Extractors for more information. + +When you read a record from a secondary database, DB automatically returns the data and optionally the key from the corresponding record in the primary database. + +## Opening and Closing Secondary Databases + +You manage secondary database opens and closes in the same way as you would any normal database. The only difference is that: + +- You must associate the secondary to a primary database using `DB->associate()`. + +- When closing your databases, it is a good idea to make sure you close your secondaries before closing your primaries. This is particularly true if your database closes are not single threaded. + +When you associate a secondary to a primary database, you must provide a callback that is used to generate the secondary's keys. These callbacks are described in the next section. + +For example, to open a secondary database and associate it to a primary database: + +``` c +#include + +... + +DB *dbp, *sdbp; /* Primary and secondary DB handles */ +u_int32_t flags; /* Primary database open flags */ +int ret; /* Function return value */ + +/* Primary */ +ret = db_create(&dbp, NULL, 0); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Secondary */ +ret = db_create(&sdbp, NULL, 0); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Usually we want to support duplicates for secondary databases */ +ret = sdbp->set_flags(sdbp, DB_DUPSORT); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Database open flags */ +flags = DB_CREATE; /* If the database does not exist, + * create it.*/ + +/* open the primary database */ +ret = dbp->open(dbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_db.db", /* On-disk file that holds the database. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + flags, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} + +/* open the secondary database */ +ret = sdbp->open(sdbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_secdb.db", /* On-disk file that holds the + database. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + flags, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} + +/* Now associate the secondary to the primary */ +dbp->associate(dbp, /* Primary database */ + NULL, /* TXN id */ + sdbp, /* Secondary database */ + get_sales_rep, /* Callback used for key creation. Not + * defined in this example. See the next + * section. */ + 0); /* Flags */ +``` + +Closing the primary and secondary databases is accomplished exactly as you would for any database: + +``` c +/* Close the secondary before the primary */ +if (sdbp != NULL) + sdbp->close(sdbp, 0); +if (dbp != NULL) + dbp->close(dbp, 0); +``` diff --git a/docs-src/guides/gsg/introduction.md b/docs-src/guides/gsg/introduction.md new file mode 100644 index 000000000..7faeed240 --- /dev/null +++ b/docs-src/guides/gsg/introduction.md @@ -0,0 +1,66 @@ +--- +title: "Chapter 1. Introduction to Berkeley DB" +api-name: "Chapter 1. Introduction to Berkeley DB" +source: docs/gsg/C/introduction.html +--- +## Chapter 1. Introduction to Berkeley DB + +**Table of Contents** + + [About This Manual](introduction.md#aboutthismanual) + + [Berkeley DB Concepts](concepts.md) + + [Access Methods](accessmethods.md) + + [Selecting Access Methods](accessmethods.md#selectAM) + + [Choosing between BTree and Hash](accessmethods.md#BTreeVSHash) + + [Choosing between Queue and Recno](accessmethods.md#QueueVSRecno) + + [Database Limits and Portability](databaseLimits.md) + + [Environments](environments.md) + + [Error Returns](returns.md) + + [Getting and Using DB](gettingit.md) + +Welcome to Berkeley DB (DB). DB is a general-purpose embedded database engine that is capable of providing a wealth of data management services. It is designed from the ground up for high-throughput applications requiring in-process, bullet-proof management of mission-critical data. DB can gracefully scale from managing a few bytes to terabytes of data. For the most part, DB is limited only by your system's available physical resources. + +You use DB through a series of programming APIs which give you the ability to read and write your data, manage your database(s), and perform other more advanced activities such as managing transactions. + +Because DB is an embedded database engine, it is extremely fast. You compile and link it into your application in the same way as you would any third-party library. This means that DB runs in the same process space as does your application, allowing you to avoid the high cost of interprocess communications incurred by stand-alone database servers. + +To further improve performance, DB offers an in-memory cache designed to provide rapid access to your most frequently used data. Once configured, cache usage is transparent. It requires very little attention on the part of the application developer. + +Beyond raw speed, DB is also extremely configurable. It provides several different ways of organizing your data in its databases. Known as *access methods*, each such data organization mechanism provides different characteristics that are appropriate for different data management profiles. (Note that this manual focuses almost entirely on the BTree access method as this is the access method used by the vast majority of DB applications). + +To further improve its configurability, DB offers many different subsystems, each of which can be used to extend DB's capabilities. For example, many applications require write-protection of their data so as to ensure that data is never left in an inconsistent state for any reason (such as software bugs or hardware failures). For those applications, a transaction subsystem can be enabled and used to transactional-protect database writes. + +The list of operating systems on which DB is available is too long to detail here. Suffice to say that it is available on all major commercial operating systems, as well as on many embedded platforms. + +Finally, DB is available in a wealth of programming languages. DB is officially supported in C, C++, and Java, but the library is also available in many other languages, especially scripting languages such as Perl and Python. + +### Note + +Before going any further, it is important to mention that DB is not a relational database (although you could use it to build a relational database). Out of the box, DB does not provide higher-level features such as triggers, or a high-level query language such as SQL. Instead, DB provides just those minimal APIs required to store and retrieve your data as efficiently as possible. + +## About This Manual + +This manual introduces DB. As such, this book does not examine intermediate or advanced features such as threaded library usage or transactional usage. Instead, this manual provides a step-by-step introduction to DB's basic concepts and library usage. + +Specifically, this manual introduces DB environments, databases, database records, and storage and retrieval of database records. This book also introduces cursors and their usage, and it describes secondary databases. + +For the most part, this manual focuses on the BTree access method. A chapter is given at the end of this manual that describes some of the concepts involving BTree usage, such as duplicate record management and comparison routines. + +Examples are given throughout this book that are designed to illustrate API usage. At the end of each chapter, a complete example is given that is designed to reinforce the concepts covered in that chapter. In addition to being presented in this book, these final programs are also available in the DB software distribution. You can find them in + +``` c +DB_INSTALL/examples_c/getting_started +``` + +where *`DB_INSTALL`* is the location where you placed your DB distribution. + +This book uses the C programming languages for its examples. Note that versions of this book exist for the C++ and Java languages as well. diff --git a/docs-src/guides/gsg/joins.md b/docs-src/guides/gsg/joins.md new file mode 100644 index 000000000..99a2e91bd --- /dev/null +++ b/docs-src/guides/gsg/joins.md @@ -0,0 +1,139 @@ +--- +title: "Database Joins" +api-name: "Database Joins" +source: docs/gsg/C/joins.html +--- +## Database Joins + + [Using Join Cursors](joins.md#joinUsage) + +If you have two or more secondary databases associated with a primary database, then you can retrieve primary records based on the intersection of multiple secondary entries. You do this using a join cursor. + +Throughout this document we have presented a structure that stores information on grocery vendors. That structure is fairly simple with a limited number of data members, few of which would be interesting from a query perspective. But suppose, instead, that we were storing information on something with many more characteristics that can be queried, such as an automobile. In that case, you may be storing information such as color, number of doors, fuel mileage, automobile type, number of passengers, make, model, and year, to name just a few. + +In this case, you would still likely be using some unique value to key your primary entries (in the United States, the automobile's VIN would be ideal for this purpose). You would then create a structure that identifies all the characteristics of the automobiles in your inventory. + +To query this data, you might then create multiple secondary databases, one for each of the characteristics that you want to query. For example, you might create a secondary for color, another for number of doors, another for number of passengers, and so forth. Of course, you will need a unique key extractor function for each such secondary database. You do all of this using the concepts and techniques described throughout this chapter. + +Once you have created this primary database and all interesting secondaries, what you have is the ability to retrieve automobile records based on a single characteristic. You can, for example, find all the automobiles that are red. Or you can find all the automobiles that have four doors. Or all the automobiles that are minivans. + +The next most natural step, then, is to form compound queries, or joins. For example, you might want to find all the automobiles that are red, and that were built by Toyota, and that are minivans. You can do this using a join cursor. + +### Using Join Cursors + +To use a join cursor: + +- Open two or more cursors for secondary databases that are associated with the same primary database. + +- Position each such cursor to the secondary key value in which you are interested. For example, to build on the previous description, the cursor for the color database is positioned to the `red` records while the cursor for the model database is positioned to the `minivan` records, and the cursor for the make database is positioned to `Toyota`. + +- Create an array of cursors, and place in it each of the cursors that are participating in your join query. Note that this array must be null terminated. + +- Obtain a join cursor. You do this using the `DB->join()` method. You must pass this method the array of secondary cursors that you opened and positioned in the previous steps. + +- Iterate over the set of matching records until the return code is not `0`. + +- Close your cursor. + +- If you are done with them, close all your cursors. + +For example: + +``` c +#include +#include + +... + +DB *automotiveDB; +DB *automotiveColorDB; +DB *automotiveMakeDB; +DB *automotiveTypeDB; +DBC *color_curs, *make_curs, *type_curs, *join_curs; +DBC *carray[4]; +DBT key, data; +int ret; + +char *the_color = "red"; +char *the_type = "minivan"; +char *the_make = "Toyota"; + +/* Database and secondary database opens omitted for brevity. + * Assume a primary database handle: + * automotiveDB + * Assume 3 secondary database handles: + * automotiveColorDB -- secondary database based on automobile color + * automotiveMakeDB -- secondary database based on the manufacturer + * automotiveTypeDB -- secondary database based on automobile type + */ + +/* initialize pointers and structures */ +color_curs = NULL; +make_curs = NULL; +type_curs = NULL; +join_curs = NULL; + +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +/* open the cursors */ +if (( ret = + automotiveColorDB->cursor(automotiveColorDB, NULL, + &color_curs, 0)) != 0) { + /* Error handling goes here */ +} + +if (( ret = + automotiveMakeDB->cursor(automotiveMakeDB, NULL, + &make_curs, 0)) != 0) { + /* Error handling goes here */ +} + +if (( ret = + automotiveTypeDB->cursor(automotiveTypeDB, NULL, + &type_curs, 0)) != 0) { + /* Error handling goes here */ +} + +/* Position the cursors */ +key.data = the_color; +key.size = strlen(the_color) + 1; +if ((ret = color_curs->get(color_curs, &key, &data, DB_SET)) != 0) + /* Error handling goes here */ + +key.data = the_make; +key.size = strlen(the_make) + 1; +if ((ret = make_curs->get(make_curs, &key, &data, DB_SET)) != 0) + /* Error handling goes here */ + +key.data = the_type; +key.size = strlen(the_type) + 1; +if ((ret = type_curs->get(type_curs, &key, &data, DB_SET)) != 0) + /* Error handling goes here */ + +/* Set up the cursor array */ +carray[0] = color_curs; +carray[1] = make_curs; +carray[2] = type_curs; +carray[3] = NULL; + +/* Create the join */ +if ((ret = automotiveDB->join(automotiveDB, carray, &join_curs, 0)) != 0) + /* Error handling goes here */ + +/* Iterate using the join cursor */ +while ((ret = join_curs->get(join_curs, &key, &data, 0)) == 0) { + /* Do interesting things with the key and data */ +} + +/* + * If we exited the loop because we ran out of records, + * then it has completed successfully. + */ +if (ret == DB_NOTFOUND) { + /* + * Close all our cursors and databases as is appropriate, and + * then exit with a normal exit status (0). + */ +} +``` diff --git a/docs-src/guides/gsg/keyCreator.md b/docs-src/guides/gsg/keyCreator.md new file mode 100644 index 000000000..7eb9eadc1 --- /dev/null +++ b/docs-src/guides/gsg/keyCreator.md @@ -0,0 +1,131 @@ +--- +title: "Implementing Key Extractors" +api-name: "Implementing Key Extractors" +source: docs/gsg/C/keyCreator.html +--- +## Implementing Key Extractors + + [Working with Multiple Keys](keyCreator.md#multikeys) + +You must provide every secondary database with a callback that creates keys from primary records. You identify this callback when you associate your secondary database to your primary. + +You can create keys using whatever data you want. Typically you will base your key on some information found in a record's data, but you can also use information found in the primary record's key. How you build your keys is entirely dependent upon the nature of the index that you want to maintain. + +You implement a key extractor by writing a function that extracts the necessary information from a primary record's key or data. This function must conform to a specific prototype, and it must be provided as a callback to the `associate()` method. + +For example, suppose your primary database records contain data that uses the following structure: + +``` c +typedef struct vendor { + char name[MAXFIELD]; /* Vendor name */ + char street[MAXFIELD]; /* Street name and number */ + char city[MAXFIELD]; /* City */ + char state[3]; /* Two-digit US state code */ + char zipcode[6]; /* US zipcode */ + char phone_number[13]; /* Vendor phone number */ + char sales_rep[MAXFIELD]; /* Name of sales representative */ + char sales_rep_phone[MAXFIELD]; /* Sales rep's phone number */ +} VENDOR; +``` + +Further suppose that you want to be able to query your primary database based on the name of a sales representative. Then you would write a function that looks like this: + +``` c +#include + +... + +int +get_sales_rep(DB *sdbp, /* secondary db handle */ + const DBT *pkey, /* primary db record's key */ + const DBT *pdata, /* primary db record's data */ + DBT *skey) /* secondary db record's key */ +{ + VENDOR *vendor; + + /* First, extract the structure contained in the primary's data */ + vendor = pdata->data; + + /* Now set the secondary key's data to be the representative's name */ + memset(skey, 0, sizeof(DBT)); + skey->data = vendor->sales_rep; + skey->size = strlen(vendor->sales_rep) + 1; + + /* Return 0 to indicate that the record can be created/updated. */ + return (0); +} +``` + +In order to use this function, you provide it on the `associate()` method after the primary and secondary databases have been created and opened: + +``` c +dbp->associate(dbp, /* Primary database */ + NULL, /* TXN id */ + sdbp, /* Secondary database */ + get_sales_rep, /* Callback used for key creation. */ + 0); /* Flags */ +``` + +### Working with Multiple Keys + +Until now we have only discussed indexes as if there is a one-to-one relationship between the secondary key and the primary database record. In fact, it is possible to generate multiple keys for any given record, provided that you take appropriate steps in your key creator to do so. + +For example, suppose you had a database that contained information about books. Suppose further that you sometimes want to look up books by author. Because sometimes books have multiple authors, you may want to return multiple secondary keys for every book that you index. + +To do this, you write a key extractor that returns a DBT whose `data` member points to an array of DBTs. Each such member of this array contains a single secondary key. In addition, the DBT returned by your key extractor must have a size field equal to the number of elements contained in the DBT array. Also, the flag field for the DBT returned by the callback must include `DB_DBT_MULTIPLE`. For example: + +### Note + +It is important that the array of secondary keys created by your callback not contain repeats. That is, every element in the array must be unique. If the array does not contain a unique set, then the secondary can get out of sync with the primary. + +``` c +int +my_callback(DB *dbp, const DBT *pkey, const DBT *pdata, DBT *skey) +{ + DBT *tmpdbt; + char *tmpdata1, tmpdata2; + + /* + * This example skips the step of extracting the data you + * want to use for building your secondary keys from the + * pkey or pdata DBT. + * + * Assume for the purpose of this example that the data + * is temporarily stored in two variables, + * tmpdata1 and tmpdata2. + */ + + /* + * Create an array of DBTs that is large enough for the + * number of keys that you want to return. In this case, + * we go with an array of size two. + */ + + tmpdbt = malloc(sizeof(DBT) * 2); + memset(tmpdbt, 0, sizeof(DBT) * 2); + + /* Now assign secondary keys to each element of the array. */ + tmpdbt[0].data = tmpdata1; + tmpdbt[0].size = (u_int32_t)strlen(tmpdbt[0].data) + 1; + tmpdbt[1].data = tmpdata2; + tmpdbt[1].size = (u_int32_t)strlen(tmpdbt[1].data) + 1; + + /* + * Now we set flags for the returned DBT. DB_DBT_MULTIPLE is + * required in order for DB to know that the DBT references an + * array. In addition, we set DB_DBT_APPMALLOC because we + * dynamically allocated memory for the DBT's data field. + * DB_DBT_APPMALLOC causes DB to release that memory once it + * is done with the returned DBT. + */ + skey->flags = DB_DBT_MULTIPLE | DB_DBT_APPMALLOC; + + /* Point the results data field to the arrays of DBTs */ + skey->data = tmpdbt; + + /* Indicate the returned array is of size 2 */ + skey->size = 2; + + return (0); +} +``` diff --git a/docs-src/guides/gsg/moreinfo.md b/docs-src/guides/gsg/moreinfo.md new file mode 100644 index 000000000..c91c44ece --- /dev/null +++ b/docs-src/guides/gsg/moreinfo.md @@ -0,0 +1,32 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/gsg/C/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Getting Started with Transaction Processing for C + +- Berkeley DB Getting Started with Replicated Applications for C + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Installation and Build Guide + +- Berkeley DB Getting Started with the SQL APIs + +- Berkeley DB C API Reference Guide + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/gsg/preface.md b/docs-src/guides/gsg/preface.md new file mode 100644 index 000000000..1dc953524 --- /dev/null +++ b/docs-src/guides/gsg/preface.md @@ -0,0 +1,58 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/gsg/C/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +Welcome to Berkeley DB (DB). This document introduces Berkeley DB 11*g* Release 2, which provides DB library version 11.2.5.3. + +This document is intended to provide a rapid introduction to the DB API set and related concepts. The goal of this document is to provide you with an efficient mechanism with which you can evaluate DB against your project's technical requirements. As such, this document is intended for C developers and senior software architects who are looking for an in-process data management solution. No prior experience with Berkeley DB is expected or required. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Structure names are represented in `monospaced font`, as are `method names`. For example: "`DB->open()` is a method on a `DB` handle." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +In some situations, programming examples are updated from one chapter to the next. When this occurs, the new code is presented in **`monospaced bold`** font. For example: + +``` c +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + DB *itemname_sdbp; /* Index based on the item name index */ + char *db_home_dir; /* Directory containing the database files */ + char *itemname_db_name; /* Itemname secondary database */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/guides/gsg/readSecondary.md b/docs-src/guides/gsg/readSecondary.md new file mode 100644 index 000000000..ae830a992 --- /dev/null +++ b/docs-src/guides/gsg/readSecondary.md @@ -0,0 +1,46 @@ +--- +title: "Reading Secondary Databases" +api-name: "Reading Secondary Databases" +source: docs/gsg/C/readSecondary.html +--- +## Reading Secondary Databases + +Like a primary database, you can read records from your secondary database either by using the `DB->get()` or `DB->pget()` methods, or by using a cursor on the secondary database. The main difference between reading secondary and primary databases is that when you read a secondary database record, the secondary record's data is not returned to you. Instead, the primary key and data corresponding to the secondary key are returned to you. + +For example, assuming your secondary database contains keys related to a person's full name: + +``` c +#include +#include + +... + +DB *my_secondary_database; +DBT key; /* Used for the search key */ +DBT pkey, pdata; /* Used to return the primary key and data */ +char *search_name = "John Doe"; + +/* Primary and secondary database opens omitted for brevity */ + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&pkey, 0, sizeof(DBT)); +memset(&pdata, 0, sizeof(DBT)); + +key.data = search_name; +key.size = strlen(search_name) + 1; + +/* Returns the key from the secondary database, and the data from the + * associated primary database entry. + */ +my_secondary_database->get(my_secondary_database, NULL, + &key, &pdata, 0); + +/* Returns the key from the secondary database, and the key and data + * from the associated primary database entry. + */ +my_secondary_database->pget(my_secondary_database, NULL, + &key, &pkey, &pdata, 0); +``` + +Note that, just like a primary database, if your secondary database supports duplicate records then `DB->get()` and `DB->pget()` only return the first record found in a matching duplicates set. If you want to see all the records related to a specific secondary key, then use a cursor opened on the secondary database. Cursors are described in Using Cursors. diff --git a/docs-src/guides/gsg/returns.md b/docs-src/guides/gsg/returns.md new file mode 100644 index 000000000..f57c2a4cf --- /dev/null +++ b/docs-src/guides/gsg/returns.md @@ -0,0 +1,18 @@ +--- +title: "Error Returns" +api-name: "Error Returns" +source: docs/gsg/C/returns.html +--- +## Error Returns + +Before continuing, it is useful to spend a few moments on error returns in DB. + +The DB interfaces always return a value of 0 on success. If the operation does not succeed for any reason, the return value will be non-zero. + +If a system error occurred (for example, DB ran out of disk space, or permission to access a file was denied, or an illegal argument was specified to one of the interfaces), DB returns an `errno` value. All of the possible values of `errno` are greater than 0. + +If the operation did not fail due to a system error, but was not successful either, DB returns a special error value. For example, if you tried to retrieve data from the database and the record for which you are searching does not exist, DB would return `DB_NOTFOUND`, a special error value that means the requested key does not appear in the database. All of the possible special error values are less than 0. + +DB also offers programmatic support for displaying error return values. First, the `db_strerror` function returns a pointer to the error message corresponding to any DB error return, similar to the ANSI C `strerror` function, but is able to handle both system error returns and DB-specific return values. + +Second, there are two error functions, `DB->err` and `DB->errx`. These functions work like the ANSI C `printf` function, taking a printf-style format string and argument list, and optionally appending the standard error string to a message constructed from the format string and other arguments. diff --git a/docs-src/guides/gsg/secondaryCursor.md b/docs-src/guides/gsg/secondaryCursor.md new file mode 100644 index 000000000..b50001f88 --- /dev/null +++ b/docs-src/guides/gsg/secondaryCursor.md @@ -0,0 +1,47 @@ +--- +title: "Using Cursors with Secondary Databases" +api-name: "Using Cursors with Secondary Databases" +source: docs/gsg/C/secondaryCursor.html +--- +## Using Cursors with Secondary Databases + +Just like cursors on a primary database, you can use cursors on secondary databases to iterate over the records in a secondary database. Like cursors used with primary databases, you can also use cursors with secondary databases to search for specific records in a database, to seek to the first or last record in the database, to get the next duplicate record, and so forth. For a complete description on cursors and their capabilities, see Using Cursors. + +However, when you use cursors with secondary databases: + +- Any data returned is the data contained on the primary database record referenced by the secondary record. + +- You cannot use `DB_GET_BOTH` and related flags with `DB->get()` and a secondary database. Instead, you must use `DB->pget()`. Also, in that case the primary and secondary key given on the call to `DB->pget()` must match the secondary key and associated primary record key in order for that primary record to be returned as a result of the call. + +For example, suppose you are using the databases, classes, and key extractors described in Implementing Key Extractors . Then the following searches for a person's name in the secondary database, and deletes all secondary and primary records that use that name. + +``` c +#include +#include + +... + +DB *sdbp; /* Secondary DB handle */ +DBC *cursorp; /* Cursor */ +DBT key, data; /* DBTs used for the delete */ +char *search_name = "John Doe"; /* Name to delete */ + +/* Primary and secondary database opens omitted for brevity. */ + +/* Get a cursor on the secondary database */ +sdbp->cursor(sdbp, NULL, &cursorp, 0); + +/* + * Zero out the DBT before using it. + */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = search_name; +key.size = strlen(search_name) + 1; + + +/* Position the cursor */ +while (cursorp->get(cursorp, &key, &data, DB_SET) == 0) + cursorp->del(cursorp, 0); +``` diff --git a/docs-src/guides/gsg/secondaryDelete.md b/docs-src/guides/gsg/secondaryDelete.md new file mode 100644 index 000000000..5f63b3155 --- /dev/null +++ b/docs-src/guides/gsg/secondaryDelete.md @@ -0,0 +1,96 @@ +--- +title: "Deleting Secondary Database Records" +api-name: "Deleting Secondary Database Records" +source: docs/gsg/C/secondaryDelete.html +--- +## Deleting Secondary Database Records + +In general, you will not modify a secondary database directly. In order to modify a secondary database, you should modify the primary database and simply allow DB to manage the secondary modifications for you. + +However, as a convenience, you can delete secondary database records directly. Doing so causes the associated primary key/data pair to be deleted. This in turn causes DB to delete all secondary database records that reference the primary record. + +You can use the `DB->del()` method to delete a secondary database record. Note that if your secondary database contains duplicate records, then deleting a record from the set of duplicates causes all of the duplicates to be deleted as well. + +### Note + +You can delete a secondary database record using the previously described mechanism only if the primary database is opened for write access. + +For example: + +``` c +#include +#include + +... + +DB *dbp, *sdbp; /* Primary and secondary DB handles */ +DBT key; /* DBTs used for the delete */ +int ret; /* Function return value */ +char *search_name = "John Doe"; /* Name to delete */ + +/* Primary */ +ret = db_create(&dbp, NULL, 0); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Secondary */ +ret = db_create(&sdbp, NULL, 0); +if (ret != 0) { + /* Error handling goes here */ +} + +/* Usually we want to support duplicates for secondary databases */ +ret = sdbp->set_flags(sdbp, DB_DUPSORT); +if (ret != 0) { + /* Error handling goes here */ +} + +/* open the primary database */ +ret = dbp->open(dbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_db.db", /* On-disk file that holds the database. + * Required. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + 0, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} + +/* open the secondary database */ +ret = sdbp->open(sdbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_secdb.db", /* On-disk file that holds the database. + * Required. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + 0, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} + +/* Now associate the secondary to the primary */ +dbp->associate(dbp, /* Primary database */ + NULL, /* TXN id */ + sdbp, /* Secondary database */ + get_sales_rep, /* Callback used for key creation. */ + 0); /* Flags */ + +/* + * Zero out the DBT before using it. + */ +memset(&key, 0, sizeof(DBT)); + +key.data = search_name; +key.size = strlen(search_name) + 1; + +/* Now delete the secondary record. This causes the associated primary + * record to be deleted. If any other secondary databases have secondary + * records referring to the deleted primary record, then those secondary + * records are also deleted. + */ + sdbp->del(sdbp, NULL, &key, 0); +``` diff --git a/docs-src/guides/gsg/usingDbt.md b/docs-src/guides/gsg/usingDbt.md new file mode 100644 index 000000000..8337a1e20 --- /dev/null +++ b/docs-src/guides/gsg/usingDbt.md @@ -0,0 +1,160 @@ +--- +title: "Reading and Writing Database Records" +api-name: "Reading and Writing Database Records" +source: docs/gsg/C/usingDbt.html +--- +## Reading and Writing Database Records + + [Writing Records to the Database](usingDbt.md#databaseWrite) + + [Getting Records from the Database](usingDbt.md#CoreDatabaseRead) + + [Deleting Records](usingDbt.md#recordDelete) + + [Data Persistence](usingDbt.md#datapersist) + +When reading and writing database records, be aware that there are some slight differences in behavior depending on whether your database supports duplicate records. Two or more database records are considered to be duplicates of one another if they share the same key. The collection of records sharing the same key are called a *duplicates set.* In DB, a given key is stored only once for a single duplicates set. + +By default, DB databases do not support duplicate records. Where duplicate records are supported, cursors (see below) are typically used to access all of the records in the duplicates set. + +DB provides two basic mechanisms for the storage and retrieval of database key/data pairs: + +- The `DBT->put()` and `DBT->get()` methods provide the easiest access for all non-duplicate records in the database. These methods are described in this section. + +- Cursors provide several methods for putting and getting database records. Cursors and their database access methods are described in Using Cursors. + +### Writing Records to the Database + +Records are stored in the database using whatever organization is required by the access method that you have selected. In some cases (such as BTree), records are stored in a sort order that you may want to define (see Setting Comparison Functions for more information). + +In any case, the mechanics of putting and getting database records do not change once you have selected your access method, configured your sorting routines (if any), and opened your database. From your code's perspective, a simple database put and get is largely the same no matter what access method you are using. + +You use `DB->put()` to put, or write, a database record. This method requires you to provide the record's key and data in the form of a pair of `DBT` structures. You can also provide one or more flags that control DB's behavior for the database write. + +Of the flags available to this method, `DB_NOOVERWRITE` may be interesting to you. This flag disallows overwriting (replacing) an existing record in the database. If the provided key already exists in the database, then this method returns `DB_KEYEXIST` even if the database supports duplicates. + +For example: + +``` c +#include +#include + +... + +char *description = "Grocery bill."; +DBT key, data; +DB *my_database; +int ret; +float money; + +/* Database open omitted for clarity */ + +money = 122.45; + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = &money; +key.size = sizeof(float); + +data.data = description; +data.size = strlen(description) +1; + +ret = my_database->put(my_database, NULL, &key, &data, DB_NOOVERWRITE); +if (ret == DB_KEYEXIST) { + my_database->err(my_database, ret, + "Put failed because key %f already exists", money); +} +``` + +### Getting Records from the Database + +You can use the `DB->get()` method to retrieve database records. Note that if your database supports duplicate records, then by default this method will only return the first record in a duplicate set. For this reason, if your database supports duplicates, the common solution is to use a cursor to retrieve records from it. Cursors are described in Using Cursors. + +(You can also retrieve a set of duplicate records using a bulk get. To do this, you use the `DB_MULTIPLE` flag on the call to `DB->get()`. For more information, see the DB Programmer's Reference Guide). + +By default, `DB->get()` returns the first record found whose key matches the key provide on the call to this method. If your database supports duplicate records, you can change this behavior slightly by supplying the `DB_GET_BOTH` flag. This flag causes `DB->get()` to return the first record that matches the provided key and data. + +If the specified key and/or data does not exist in the database, this method returns `DB_NOTFOUND`. + +``` c +#include +#include + +... +#define DESCRIPTION_SIZE 199 +DBT key, data; +DB *my_database; +float money; +char description[DESCRIPTION_SIZE + 1]; + +/* Database open omitted for clarity */ + +money = 122.45; + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); + +key.data = &money; +key.size = sizeof(float); + +data.data = description; +data.ulen = DESCRIPTION_SIZE + 1; +data.flags = DB_DBT_USERMEM; +my_database->get(my_database, NULL, &key, &data, 0); + +/* + * Description is set into the memory that we supplied. + */ +``` + +Note that in this example, the `data.size` field would be automatically set to the size of the retrieved data. + +### Deleting Records + +You can use the `DB->del()` method to delete a record from the database. If your database supports duplicate records, then all records associated with the provided key are deleted. To delete just one record from a list of duplicates, use a cursor. Cursors are described in Using Cursors. + +You can also delete every record in the database by using `DB->truncate().` + +For example: + +``` c +#include +#include + +... + +DBT key; +DB *my_database; +float money = 122.45; + +/* Database open omitted for clarity */ + +/* Zero out the DBTs before using them. */ +memset(&key, 0, sizeof(DBT)); + +key.data = &money; +key.size = sizeof(float); + +my_database->del(my_database, NULL, &key, 0); +``` + +### Data Persistence + +When you perform a database modification, your modification is made in the in-memory cache. This means that your data modifications are not necessarily flushed to disk, and so your data may not appear in the database after an application restart. + +Note that as a normal part of closing a database, its cache is written to disk. However, in the event of an application or system failure, there is no guarantee that your databases will close cleanly. In this event, it is possible for you to lose data. Under extremely rare circumstances, it is also possible for you to experience database corruption. + +Therefore, if you care if your data is durable across system failures, and to guard against the rare possibility of database corruption, you should use transactions to protect your database modifications. Every time you commit a transaction, DB ensures that the data will not be lost due to application or system failure. Transaction usage is described in the *Berkeley DB Getting Started with Transaction Processing* guide. + +If you do not want to use transactions, then the assumption is that your data is of a nature that it need not exist the next time your application starts. You may want this if, for example, you are using DB to cache data relevant only to the current application runtime. + +If, however, you are not using transactions for some reason and you still want some guarantee that your database modifications are persistent, then you should periodically call `DB->sync()`. Syncs cause any dirty entries in the in-memory cache and the operating system's file cache to be written to disk. As such, they are quite expensive and you should use them sparingly. + +Remember that by default a sync is performed any time a non-transactional database is closed cleanly. (You can override this behavior by specifying `DB_NOSYNC` on the call to `DB->close()`.) That said, you can manually run a sync by calling `DB->sync().` + +### Note + +If your application or system crashes and you are not using transactions, then you should either discard and recreate your databases, or verify them. You can verify a database using DB-\>verify(). If your databases do not verify cleanly, use the **db_dump** command to salvage as much of the database as is possible. Use either the `-R` or `-r` command line options to control how aggressive **db_dump** should be when salvaging your databases. diff --git a/docs-src/guides/gsg_db_rep/_meta.toml b/docs-src/guides/gsg_db_rep/_meta.toml new file mode 100644 index 000000000..63ef40c59 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/_meta.toml @@ -0,0 +1,33 @@ +# Nav/index metadata for the gsg_db_rep guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Getting Started with Replicated Berkeley DB Applications" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "introduction", + "repadvantage", + "apioverview", + "elections", + "permmessages", + "txnapp", + "simpleprogramlisting", + "repapp", + "rep_init_code", + "repmgr_init_example_c", + "fwrkpermmessage", + "electiontimes", + "fmwrkconnectretry", + "heartbeats", + "fwrkmasterreplica", + "processingloop", + "exampledoloop", + "addfeatures", + "manageblock", + "autoinit", + "rywc", + "c2ctransfer", + "bulk", +] diff --git a/docs-src/guides/gsg_db_rep/addfeatures.md b/docs-src/guides/gsg_db_rep/addfeatures.md new file mode 100644 index 000000000..9e4e94508 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/addfeatures.md @@ -0,0 +1,50 @@ +--- +title: "Chapter 5. Additional Features" +api-name: "Chapter 5. Additional Features" +source: docs/gsg_db_rep/C/addfeatures.html +--- +## Chapter 5. Additional Features + +**Table of Contents** + + [Delayed Synchronization](addfeatures.md#delayedsync) + + [Managing Blocking Operations](manageblock.md) + + [Stop Auto-Initialization](autoinit.md) + + [Read-Your-Writes Consistency](rywc.md) + + [Client to Client Transfer](c2ctransfer.md) + + [Identifying Peers](c2ctransfer.md#fmwrkpeerserver) + + [Bulk Transfers](bulk.md) + +Beyond the basic functionality that we have discussed so far in this book, there are several replication features that you should understand. These are all optional to use, but provide useful functionality under the right circumstances. + +These additional features are: + +1. Delayed Synchronization + +2. Managing Blocking Operations + +3. Stop Auto-Initialization + +4. Client to Client Transfer + +5. Bulk Transfers + +## Delayed Synchronization + +When a replication group has a new master, all replicas must synchronize with that master. This means they must ensure that the contents of their local database(s) are identical to that contained by the new master. + +This synchronization process can result in quite a lot of network activity. It can also put a large strain on the master server, especially if is part of a large replication group or if there is somehow a large difference between the master's database(s) and the contents of its replicas. + +It is therefore possible to delay synchronization for any replica that discovers it has a new master. You would do this so as to give the master time to synchronize other replicas before proceeding with the delayed replicas. + +To delay synchronization of a replica environment, you specify `DB_REP_CONF_DELAYCLIENT` to `DB_ENV->rep_set_config()` and then specify `1` to the `onoff` parameter. (Specify `0` to turn the feature off.) + +If you use delayed synchronization, then you must manually synchronize the replica at some future time. Until you do this, the replica is out of sync with the master, and it will ignore all database changes forwarded to it from the master. + +You synchronize a delayed replica by calling `DB_ENV->rep_sync()` on the replica that has been delayed. diff --git a/docs-src/guides/gsg_db_rep/apioverview.md b/docs-src/guides/gsg_db_rep/apioverview.md new file mode 100644 index 000000000..2423cfa9e --- /dev/null +++ b/docs-src/guides/gsg_db_rep/apioverview.md @@ -0,0 +1,46 @@ +--- +title: "The Replication APIs" +api-name: "The Replication APIs" +source: docs/gsg_db_rep/C/apioverview.html +--- +## The Replication APIs + + [Replication Manager Overview](apioverview.md#repframeworkoverview) + + [Replication Base API Overview](apioverview.md#repapioverview) + +There are two ways that you can choose to implement replication in your transactional application. The first, and preferred, mechanism is to use the pre-packaged Replication Manager that comes with the DB distribution. This framework should be sufficient for most customers. + +If for some reason the Replication Manager does not meet your application's technical requirements, you will have to use the Replication Base APIs available through the Berkeley DB library to write your own custom replication framework. + +Both of these approaches are described in slightly greater detail in this section. The bulk of the chapters later in this book are dedicated to these two replication implementation mechanisms. + +### Replication Manager Overview + +DB's pre-packaged Replication Manager exists as a layer on top of the DB library. The Replication Manager is a multi-threaded implementation that allows you to easily add replication to your existing transactional application. You access and manage the Replication Manager using methods that are available off the `DB_ENV` class. + +The Replication Manager: + +- Provides a multi-threaded communications layer using pthreads (on Unix-style systems and similar derivatives such as Mac OS X), or Windows threads on Microsoft Windows systems. + +- Uses TCP/IP sockets. Network traffic is handled via threads that handle inbound and outbound messages. However, each process uses a single socket that is shared using `select()`. + + Note that for this reason, the Replication Manager is limited to a maximum of 60 replicas (on Windows) and approximately 1000 replicas (on Unix and related systems), depending on how your system is configured. + +- Requires that only one instance of the environment handle be used. + +- Upon application startup, a master can be selected either manually or via elections. After startup time, however, during the course of normal operations it is possible for the replication group to need to locate a new master (due to network or other hardware related problems, for example) and in this scenario elections are always used to select the new master. + +If your application has technical requirements that do not conform to the implementation provided by the Replication Manager, you must write implement replication using the DB Replication Base APIs. See the next section for introductory details. + +### Replication Base API Overview + +The Replication Base API is a series of Berkeley DB library classes and methods that you can use to build your own replication infrastructure. You should use the Base API only if the Replication Manager does not meet your application's technical requirements. + +To make use of the Base API, you must write your own networking code. This frees you from the technical constraints imposed by the Replication Manager. For example, by writing your own framework, you can: + +- Use a threading package other than pthreads (Unix) or Windows threads (Microsoft Windows). This might be interesting to you if you are using a platform whose preferred threading package is something other than (for example) pthreads, such as is the case for Sun Microsystem's Solaris operating systems. + +- Implement your own sockets. The Replication Manager uses TCP/IP sockets. While this should be acceptable for the majority of applications, sometimes UDP or even raw sockets might be desired. + +For information on writing a replicated application using the Berkeley DB Replication Base APIs, see the *Berkeley DB Programmer's Reference Guide*. diff --git a/docs-src/guides/gsg_db_rep/autoinit.md b/docs-src/guides/gsg_db_rep/autoinit.md new file mode 100644 index 000000000..efab55bf5 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/autoinit.md @@ -0,0 +1,12 @@ +--- +title: "Stop Auto-Initialization" +api-name: "Stop Auto-Initialization" +source: docs/gsg_db_rep/C/autoinit.html +--- +## Stop Auto-Initialization + +As stated in the previous section, when a replication replica is synchronizing with its master, it will block DB operations at some points during this process until the synchronization is completed. You can turn off this behavior (see Managing Blocking Operations), but for replicas that have been out of touch from their master for a very long time, this may not be enough. + +If a replica has been out of touch from its master long enough, it may find that it is not possible to perform synchronization. When this happens, by default the master and replica internally decide to completely re-initialize the replica. This re-initialization involves discarding the replica's current database(s) and transferring new ones to it from the master. Depending on the size of the master's databases, this can take a long time, during which time the replica will be completely non-responsive when it comes to performing database operations. + +It is possible that there is a time of the day when it is better to perform a replica re-initialization. Or, you simply might want to decide to bring the replica up to speed by restoring its databases using a hot-backup taken from the master. Either way, you can decide to prevent automatic-initialization of your replica. To do this specify `DB_REP_CONF_AUTOINIT` to `DB_ENV->rep_set_config()` and then specify `0` to the `onoff` parameter. diff --git a/docs-src/guides/gsg_db_rep/bulk.md b/docs-src/guides/gsg_db_rep/bulk.md new file mode 100644 index 000000000..839fa307d --- /dev/null +++ b/docs-src/guides/gsg_db_rep/bulk.md @@ -0,0 +1,32 @@ +--- +title: "Bulk Transfers" +api-name: "Bulk Transfers" +source: docs/gsg_db_rep/C/bulk.html +--- +## Bulk Transfers + +By default, messages are sent from the master to replicas as they are generated. This can degrade replication performance because the various participating environments must handle a fair amount of network I/O activity. + +You can alleviate this problem by configuring your master environment for bulk transfers. Bulk transfers simply cause replication messages to accumulate in a buffer until a triggering event occurs. When this event occurs, the entire contents of the buffer is sent to the replica, thereby eliminating excessive network I/O. + +Note that if you are using replica to replica transfers, then you might want any replica that can service replication requests to also be configured for bulk transfers. + +The events that result in a bulk transfer of replication messages to a replica will differ depending on if the transmitting environment is a master or a replica. + +If the servicing environment is a master environment, then bulk transfer occurs when: + +1. Bulk transfers are configured for the master environment, and + +2. the message buffer is full or + +3. a permanent record (for example, a transaction commit or a checkpoint record) is placed in the buffer for the replica. + +If the servicing environment is a replica environment (that is, replica to replica transfers are in use), then a bulk transfer occurs when: + +1. Bulk transfers are configured for the transmitting replica, and + +2. the message buffer is full or + +3. the replica servicing the request is able to completely satisfy the request with the contents of the message buffer. + +To configure bulk transfers, specify `DB_REP_CONF_BULK` to `DB_ENV->rep_set_config()` and then specify `1` to the `onoff` parameter. (Specify `0` to turn the feature off.) diff --git a/docs-src/guides/gsg_db_rep/c2ctransfer.md b/docs-src/guides/gsg_db_rep/c2ctransfer.md new file mode 100644 index 000000000..43b850617 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/c2ctransfer.md @@ -0,0 +1,26 @@ +--- +title: "Client to Client Transfer" +api-name: "Client to Client Transfer" +source: docs/gsg_db_rep/C/c2ctransfer.html +--- +## Client to Client Transfer + + [Identifying Peers](c2ctransfer.md#fmwrkpeerserver) + +It is possible to use a replica instead of a master to synchronize another replica. This serves to take the request load off a master that might otherwise occur if multiple replicas attempted to synchronize with the master at the same time. + +For best results, use this feature combined with the delayed synchronization feature (see Delayed Synchronization). + +For example, suppose your replication group consists of four environments. Upon application startup, all three replicas will immediately attempt to synchronize with the master. But at the same time, the master itself might be busy with a heavy database write load. + +To solve this problem, delay synchronization for two of the three replicas. Allow the third replica to synchronize as normal with the master. Then, start synchronization for each of the delayed replicas (since this is a manual process, you can do them one at a time if that best suits your application). Assuming you have configured replica to replica synchronization correctly, the delayed replicas will synchronize using the up-to-date replica, rather than using the master. + +When you are using the Replication Manager, you configure replica to replica synchronization by declaring an environment to be a peer of another environment. If an environment is a peer, then it can be used for synchronization purposes. + +### Identifying Peers + +You can designate one replica to be a peer of another for replica to replica synchronization. You might want to do this if you have machines that you know are on fast, reliable network connections and so you are willing to accept the overhead of waiting for acknowledgments from those specific machines. + +Note that peers are not required to be a bi-directional. That is, just because machine A declares machine B to be a peer, that does not mean machine B must also declare machine A to be a peer. + +You declare a peer for the current environment when you add that environment to the list of known sites. You do this by specifying the `DB_REPMGR_PEER` flag to `DB_ENV->repmgr_add_remote_site()`. diff --git a/docs-src/guides/gsg_db_rep/elections.md b/docs-src/guides/gsg_db_rep/elections.md new file mode 100644 index 000000000..f0391b6b1 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/elections.md @@ -0,0 +1,56 @@ +--- +title: "Holding Elections" +api-name: "Holding Elections" +source: docs/gsg_db_rep/C/elections.html +--- +## Holding Elections + + [Influencing Elections](elections.md#influencingelections) + + [Winning Elections](elections.md#winningelections) + + [Switching Masters](elections.md#switchingmasters) + +Finding a master environment is one of the fundamental activities that every replication replica must perform. Upon startup, the underlying DB replication code will attempt to locate a master. If a master cannot be found, then the environment should initiate an election. + +### Note + +In some rare situations, it is desireable for the application to manually select its master. For these cases, elections can be turned off. + +Manually selecting a master is an activity that should be performed infrequently, if ever. You turn elections off by using the `DB_ENV->rep_set_config()` and `DB_ENV->repmgr_start()` methods. + +How elections are held depends upon the API that you use to implement replication. For example, if you are using the Replication Manager elections are held transparently without any input from your application's code. In this case, DB will determine which environment is the master and which are replicas. + +### Influencing Elections + +If you want to control the election process, you can declare a specific environment to be the master. Note that for the Replication Manager, it is only possible to do this at application startup. Should the master become unavailable during run-time for any reason, an election is held. The environment that receives the most number of votes, wins the election and becomes the master. A machine receives a vote because it has the most up-to-date log records. + +Because ties are possible when elections are held, it is possible to influence which environment will win the election. How you do this depends on which API you are using. In particular, if you are writing a custom replication layer, then there are a great many ways to manually influence elections. + +One such mechanism is priorities. When votes are cast during an election, the winner is determined first by the environment with the most up-to-date log records. But if this is a tie, the the environment's priority is considered. So given two environments with log records that are equally recent, votes are cast for the environment with the higher priority. + +Therefore, if you have a machine that you prefer to become a master in the event of an election, assign it a high priority. Assuming that the election is held at a time when the preferred machine has up-to-date log records, that machine will win the election. + +### Winning Elections + +To win an election: + +1. There cannot currently be a master environment. + +2. The environment must have the most recent log records. Part of holding the election is determining which environments have the most recent log records. This process happens automatically; your code does not need to involve itself in this process. + +3. The environment must receive the most number of votes from the replication environments that are participating in the election. + +If you are using the Replication Manager, then in the event of a tie vote the environment with the highest priority wins the election. If two or more environments receive the same number of votes and have the same priority, then the underlying replication code picks one of the environments to be the winner. Which winner will be picked by the replication code is unpredictable from the perspective of your application code. + +### Switching Masters + +To switch masters: + +1. Start up the environment that you want to be master as normal. At this time it is a replica. Make sure this environment has a higher priority than all the other environments. + +2. Allow the new environment to run for a time as a replica. This allows it to obtain the most recent copies of the log files. + +3. Shut down the current master. This should force an election. Because the new environment has the highest priority, it will win the election, provided it has had enough time to obtain all the log records. + +4. Optionally restart the old master environment. Because there is currently a master environment, an election will not be held and the old master will now run as a replica environment. diff --git a/docs-src/guides/gsg_db_rep/electiontimes.md b/docs-src/guides/gsg_db_rep/electiontimes.md new file mode 100644 index 000000000..ec87bbfae --- /dev/null +++ b/docs-src/guides/gsg_db_rep/electiontimes.md @@ -0,0 +1,34 @@ +--- +title: "Managing Election Times" +api-name: "Managing Election Times" +source: docs/gsg_db_rep/C/electiontimes.html +--- +## Managing Election Times + + [Managing Election Timeouts](electiontimes.md#electiontimeout) + + [Managing Election Retry Times](electiontimes.md#electretrytime) + +Where it comes to elections, there are two timeout values with which you should be concerned: election timeouts and election retries. + +### Managing Election Timeouts + +When an environment calls for an election, it will wait some amount of time for the other replicas in the replication group to respond. The amount of time that the environment will wait before declaring the election completed is the *election timeout*. + +If the environment hears from all other known replicas before the election timeout occurs, the election is considered a success and a master is elected. + +If only a subset of replicas respond, then the success or failure of the election is determined by how many replicas have participated in the election. It only takes a simple majority of replicas to elect a master. If there are enough votes for a given environment to meet that standard, then the master has been elected and the election is considered a success. + +However, if not enough replicas have participated in the election when the election timeout value is reached, the election is considered a failure and a master is not elected. At this point, your replication group is operating without a master, which means that, essentially, your replicated application has been placed in read-only mode. + +Note, however, that the Replication Manager will attempt a new election after a given amount of time has passed. See the next section for details. + +You set the election timeout value using `DB_ENV->rep_set_timeout()`. To do so, specify the `DB_REP_ELECTION_TIMEOUT` value to the `which` parameter and then a timeout value in microseconds to the `timeout` parameter. + +### Managing Election Retry Times + +In the event that a election fails (see the previous section), an election will not be attempted again until the election retry timeout value has expired. + +You set the retry timeout value using `DB_ENV->rep_set_timeout()`. To do so, specify the `DB_REP_ELECTION_RETRY` value to the `which` parameter and then a retry value in microseconds to the `timeout` parameter. + +Note that this flag is only valid when you are using the Replication Manager. If you are using the Base APIs, then this flag is ignored. diff --git a/docs-src/guides/gsg_db_rep/exampledoloop.md b/docs-src/guides/gsg_db_rep/exampledoloop.md new file mode 100644 index 000000000..e2c6291c5 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/exampledoloop.md @@ -0,0 +1,497 @@ +--- +title: "Example Processing Loop" +api-name: "Example Processing Loop" +source: docs/gsg_db_rep/C/exampledoloop.html +--- +## Example Processing Loop + + [Running It](exampledoloop.md#runningit) + +In this section we take the example processing loop that we presented in the previous section and we flesh it out to provide a more complete example. We do this by updating the `doloop()` function that our original transaction application used (see Function: doloop()) to fully support our replicated application. + +In the following example code, code that we add to the original example is presented in **`bold`**. + +To begin, we include a new header file into our application so that we can check for the `ENOENT` return value later in our processing loop. We also define our `APP_DATA` structure, and we define a `sleeptime` value. Finally, we add a new forward declaration for our event callback. + +``` c +/* + * File: ex_rep_gsg_repmgr.c + */ + +#include +#include +#include +#ifndef _WIN32 +#include +#endif + +#include + +#ifdef _WIN32 +extern int getopt(int, char * const *, const char *); +#endif + +#define CACHESIZE (10 * 1024 * 1024) +#define DATABASE "quote.db" +#define SLEEPTIME 3 + +const char *progname = "ex_rep_gsg_repmgr"; + +typedef struct { + int is_master; +} APP_DATA; + +int create_env(const char *, DB_ENV **); +int env_init(DB_ENV *, const char *); +int doloop (DB_ENV *); +static int print_stocks(DBC *); +void *event_callback(DB_ENV *, u_int32_t, void *); +``` + +In our `main()` function, most of what we have to add to it is some new variable declarations and initializations: + +``` c +int +main(int argc, char *argv[]) +{ + DB_ENV *dbenv; + DB_SITE *dbsite; + extern char *optarg; + const char *home; + char ch, *host, *portstr; + int ret, local_is_set, is_group_creator; + u_int32_t port; + /* Used to track whether this is a replica or a master */ + APP_DATA my_app_data; + + my_app_data.is_master = 0; /* Assume that we start as a replica */ + dbenv = NULL; + + ret = local_is_set = is_group_creator = 0; + home = NULL; +``` + +The rest of our `main()` function is unchanged, except that we make our `APP_DATA` structure available through our environment handle's `app_private` field: + +``` c + if ((ret = create_env(progname, &dbenv)) != 0) + goto err; + + /* Make APP_DATA available through the environment handle */ + dbenv->app_private = &my_app_data; + + /* Default priority is 100 */ + dbenv->rep_set_priority(dbenv, 100); + /* Permanent messages require at least one ack */ + dbenv->repmgr_set_ack_policy(dbenv, DB_REPMGR_ACKS_ONE); + /* Give 500 microseconds to receive the ack */ + dbenv->rep_set_timeout(dbenv, DB_REP_ACK_TIMEOUT, 500); + + while ((ch = getopt(argc, argv, "h:l:L:p:r:")) != EOF) + switch (ch) { + case 'h': + home = optarg; + break; + /* Set the host and port used by this environment */ + case 'l': + host = strtok(optarg, ":"); + if ((portstr = strtok(NULL, ":")) == NULL) { + fprintf(stderr, "Bad host specification.\n"); + goto err; + } + port = (unsigned short)atoi(portstr); + if ((ret = dbenv->repmgr_site(dbenv, host, port, &dbsite + 0)) != 0 ) { + fprintf(stderr, + "Could not set local address %s.\n", host); + goto err; + } + dbsite->set_config(dbsite, DB_LOCAL_SITE, 1); + if (is_group_creator) + dbsite->set_config(dbsite, DB_GROUP_CREATOR, 1); + + if ((ret = dbsite->close(dbsite)) != 0) { + dbenv->(dbenv, ret, "DB_SITE->close"); + goto err; + } + local_is_set = 1; + break; + /* Set this replica's election priority */ + case 'p': + dbenv->rep_set_priority(dbenv, atoi(optarg)); + break; + /* Identify another site in the replication group */ + case 'r': + host = strtok(optarg, ":"); + if ((portstr = strtok(NULL, ":")) == NULL) { + fprintf(stderr, "Bad host specification.\n"); + goto err; + } + port = (unsigned short)atoi(portstr); + if ((dbenv->repmgr_site(dbenv, host, port, &dbsite, + 0)) != 0) { + fprintf(stderr, + "Could not add site %s.\n", host); + goto err; + } + dbenv->set_config(dbsite, DB_BOOTSTRAP_HELPER, 1); + if ((dbenv->close(dbsite)) != 0) { + dbenv->err(dbenv, ret, "DB_SITE->close"); + goto err; + } + break; + case '?': + default: + usage(); + } + + /* Error check command line. */ + if (home == NULL || !local_is_set) + usage(); + + if ((ret = env_init(dbenv, home)) != 0) + goto err; + + if ((ret = dbenv->repmgr_start(dbenv, 3, DB_REP_ELECTION)) != 0) + goto err; + + /* Sleep to give ourselves time to find a master. */ + sleep(5); + + if ((ret = doloop(dbenv)) != 0) { + dbenv->err(dbenv, ret, "Application failed"); + goto err; + } + +err: if (dbenv != NULL) + (void)dbenv->close(dbenv, 0); + + return (ret); +} +``` + +Having updated our `main()`, we must also update our `create_env()` function to register our `event_callback` callback. Notice that our `env_init()` function, which is responsible for actually opening our environment handle, is unchanged: + +``` c +int +create_env(const char *progname, DB_ENV **dbenvp) +{ + DB_ENV *dbenv; + int ret; + + if ((ret = db_env_create(&dbenv, 0)) != 0) { + fprintf(stderr, "can't create env handle: %s\n", + db_strerror(ret)); + return (ret); + } + + dbenv->set_errfile(dbenv, stderr); + dbenv->set_errpfx(dbenv, progname); + (void)dbenv->set_event_notify(dbenv, event_callback); + + *dbenvp = dbenv; + return (0); +} + +int +env_init(DB_ENV *dbenv, const char *home) +{ + u_int32_t flags; + int ret; + + (void)dbenv->set_cachesize(dbenv, 0, CACHESIZE, 0); + (void)dbenv->set_flags(dbenv, DB_TXN_NOSYNC, 1); + + flags = DB_CREATE | + DB_INIT_LOCK | + DB_INIT_LOG | + DB_INIT_MPOOL | + DB_INIT_REP | + DB_INIT_TXN | + DB_RECOVER | + DB_THREAD; + if ((ret = dbenv->open(dbenv, home, flags, 0)) != 0) + dbenv->err(dbenv, ret, "can't open environment"); + return (ret); +} +``` + +That done, we need to implement our `event_callback()` callback. Note that what we use here is no different from the callback that we described in the previous section. However, for the sake of completeness we provide the implementation here again. + +``` c + + /* + * A callback used to determine whether the local environment is a + * replica or a master. This is called by the Replication Manager + * when the local replication environment changes state. + */ +void * +event_callback(DB_ENV *dbenv, u_int32_t which, void *info) +{ + APP_DATA *app = dbenv->app_private; + + info = NULL; /* Currently unused. */ + + switch (which) { + case DB_EVENT_REP_MASTER: + app->is_master = 1; + break; + + case DB_EVENT_REP_CLIENT: + app->is_master = 0; + break; + + case DB_EVENT_REP_STARTUPDONE: /* fallthrough */ + case DB_EVENT_REP_NEWMASTER: + /* Ignore. */ + break; + + default: + dbenv->errx(dbenv, "ignoring event %d", which); + } +} + + +``` + +That done, we need to update our `doloop()` function. This is the place where we most heavily modify our application. + +We begin by introducing `APP_DATA` to the function: + +``` c +/* + * Provides the main data processing function for our application. + * This function provides a command line prompt to which the user + * can provide a ticker string and a stock price. Once a value is + * entered to the application, the application writes the value to + * the database and then displays the entire database. + */ +#define BUFSIZE 1024 +int +doloop(DB_ENV *dbenv) +{ + DB *dbp; + APP_DATA *app_data; + DBT key, data; + char buf[BUFSIZE], *rbuf; + int ret; + u_int32_t flags; + + dbp = NULL; + ret = 0; + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + app_data = dbenv->app_private; +``` + +Next we begin to modify our main loop. To start, upon entering the loop we create the database handle and configure it as normal. But we also have to decide what flags we will use for the open. Again, it depends on whether we are a replica or a master. + +``` c + for (;;) { + if (dbp == NULL) { + if ((ret = db_create(&dbp, dbenv, 0)) != 0) + return (ret); + + flags = DB_AUTO_COMMIT; + if (app_data->is_master) + flags |= DB_CREATE; +``` + +When we open the database, we modify our error handling to account for the case where the database does not yet exist. This can happen if our code is running as a replica and the Replication Manager has not yet had a chance to create the databases for us. Recall that replicas never write to their own databases directly, and so they cannot create databases on their own. + +If we detect that the database does not yet exist, we simply close the database handle, sleep for a short period of time and then continue processing. This gives the Replication Manager a chance to create the database so that our replica can continue operations. + +``` c + if ((ret = dbp->open(dbp, + NULL, DATABASE, NULL, DB_BTREE, flags, 0)) != 0) { + if (ret == ENOENT) { + printf( + "No stock database yet available.\n"); + if ((ret = dbp->close(dbp, 0)) != 0) { + dbenv->err(dbenv, ret, + "DB->close"); + goto err; + } + dbp = NULL; + sleep(SLEEPTIME); + continue; + } + dbenv->err(dbenv, ret, "DB->open"); + goto err; + } + } +``` + +Next we modify our prompt, so that if the local process is running as a replica, we can tell from the shell that the prompt is for a read-only process. + +``` c + printf("QUOTESERVER%s> ", + app_data->is_master ? "" : " (read-only)"); + fflush(stdout); +``` + +When we collect data from the prompt, there is a case that says if no data is entered then show the entire stocks database. This display is performed by our `print_stocks()` function (which has not required a modification since we first introduced it in Function: print_stocks() ). + +When we call `print_stocks()`, we check for a dead replication handle. Dead replication handles happen whenever a replication election results in a previously committed transaction becoming invalid. This is an error scenario caused by a new master having a slightly older version of the data than the original master and so all replicas must modify their database(s) to reflect that of the new master. In this situation, some number of previously committed transactions may have to be unrolled. From the replica's perspective, the database handles should all be closed and then opened again. + +``` c + if (fgets(buf, sizeof(buf), stdin) == NULL) + break; + if (strtok(&buf[0], " \t\n") == NULL) { + switch ((ret = print_stocks(dbp))) { + case 0: + continue; + case DB_REP_HANDLE_DEAD: + (void)dbp->close(dbp, DB_NOSYNC); + dbp = NULL; + dbenv->errx(dbenv, "Got a dead replication handle"); + continue; + default: + dbp->err(dbp, ret, "Error traversing data"); + goto err; + } + } + rbuf = strtok(NULL, " \t\n"); + if (rbuf == NULL || rbuf[0] == '\0') { + if (strncmp(buf, "exit", 4) == 0 || + strncmp(buf, "quit", 4) == 0) + break; + dbenv->errx(dbenv, "Format: TICKER VALUE"); + continue; + } +``` + +That done, we need to add a little error checking to our command prompt to make sure the user is not attempting to modify the database at a replica. Remember, replicas must never modify their local databases on their own. This guards against that happening due to user input at the prompt. + +``` c + if (!app_data->is_master) { + dbenv->errx(dbenv, "Can't update at client"); + continue; + } + key.data = buf; + key.size = (u_int32_t)strlen(buf); + + data.data = rbuf; + data.size = (u_int32_t)strlen(rbuf); + + if ((ret = dbp->put(dbp, + NULL, &key, &data, 0)) != 0) { + dbp->err(dbp, ret, "DB->put"); + goto err; + } + } + +err: if (dbp != NULL) + (void)dbp->close(dbp, DB_NOSYNC); + + return (ret); +} +``` + +With that completed, we are all done updating our application for replication. The only remaining function, `print_stocks()`, is unmodified from when we originally introduced it. For details on that function, see Function: print_stocks() . + +### Running It + +To run our replicated application, we need to make sure each participating environment has its own unique home directory. We can do this by running each site on a separate networked machine, but that is not strictly necessary; multiple instances of this code can run on the same machine provided the environment home restriction is observed. + +To run a process, make sure the environment home exists and then start the process using the `-h` option to specify that directory. You must also use the `-l` or `-L` option to identify the local host and port that this process will use to listen for replication messages (-L means that this is a group creator), and the `-r` option to identify the other processes in the replication group. Finally, use the `-p` option to specify a priority. The process that you designate to have the highest priority will become the master. + +``` c +> mkdir env1 +> ./ex_rep_gsg_repmgr -h env1 -L localhost:8080 -p 10 +No stock database yet available. +No stock database yet available. +``` + +Now, start another process. This time, change the environment home to something else, use the `-l` flag to at least change the port number the process is listening on, and use the `-r` option to identify the host and port of the other replication process: + +``` c +> mkdir env2 +> ./ex_rep_gsg_repmgr -h env2 -l localhost:8081 \ +-r localhost:8080 -p 20 +``` + +After a short pause, the second process should display the master prompt: + +``` c +QUOTESERVER > +``` + +And the first process should display the read-only prompt: + +``` c +QUOTESERVER (read-only)> +``` + +Now go to the master process and give it a couple of stocks and stock prices: + +``` c +QUOTESERVER> FAKECO 9.87 +QUOTESERVER> NOINC .23 +QUOTESERVER> +``` + +Then, go to the replica and hit **`return`** at the prompt to see the new values: + +``` c +QUOTESERVER (read-only)> + Symbol Price + ====== ===== + FAKECO 9.87 + NOINC .23 +QUOTESERVER (read-only)> +``` + +Doing the same at the master results in the same thing: + +``` c +QUOTESERVER> + Symbol Price + ====== ===== + FAKECO 9.87 + NOINC .23 +QUOTESERVER> +``` + +You can change a stock by simply entering the stock value and new price at the master's prompt: + +``` c +QUOTESERVER> FAKECO 10.01 +QUOTESERVER> +``` + +Then, go to either the master or the replica to see the updated database. On the master: + +``` c +QUOTESERVER> + Symbol Price + ====== ===== + FAKECO 10.01 + NOINC .23 +QUOTESERVER> +``` + +And on the replica: + +``` c +QUOTESERVER (read-only)> + Symbol Price + ====== ===== + FAKECO 10.01 + NOINC .23 +QUOTESERVER (read-only)> +``` + +Finally, to quit the applications, simply type `quit` at both prompts. On the replica: + +``` c +QUOTESERVER (read-only)> quit +> +``` + +And on the master as well: + +``` c +QUOTESERVER> quit +> +``` diff --git a/docs-src/guides/gsg_db_rep/fmwrkconnectretry.md b/docs-src/guides/gsg_db_rep/fmwrkconnectretry.md new file mode 100644 index 000000000..ad3e7ca89 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/fmwrkconnectretry.md @@ -0,0 +1,8 @@ +--- +title: "Managing Connection Retries" +api-name: "Managing Connection Retries" +source: docs/gsg_db_rep/C/fmwrkconnectretry.html +--- +## Managing Connection Retries + +In the event that a communication failure occurs between two environments in a replication group, the Replication Manager will wait a set amount of time before attempting to re-establish the connection. You can configure this wait value using `DB_ENV->rep_set_timeout()`. To do so, specify the `DB_REP_CONNECTION_RETRY` value to the `which` parameter and then a retry value in microseconds to the `timeout` parameter. diff --git a/docs-src/guides/gsg_db_rep/fwrkmasterreplica.md b/docs-src/guides/gsg_db_rep/fwrkmasterreplica.md new file mode 100644 index 000000000..a7b53f5e9 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/fwrkmasterreplica.md @@ -0,0 +1,188 @@ +--- +title: "Chapter 4. Replica versus Master Processes" +api-name: "Chapter 4. Replica versus Master Processes" +source: docs/gsg_db_rep/C/fwrkmasterreplica.html +--- +## Chapter 4. Replica versus Master Processes + +**Table of Contents** + + [Determining State](fwrkmasterreplica.md#determinestate) + + [Processing Loop](processingloop.md) + + [Example Processing Loop](exampledoloop.md) + + [Running It](exampledoloop.md#runningit) + +Every environment participating in a replicated application must know whether it is a *master* or *replica*. The reason for this is because, simply, the master can modify the database while replicas cannot. As a result, not only will you open databases differently depended on whether the environment is running as a master, but the environment will frequently behave quite a bit differently depending on whether it thinks it is operating as the read/write interface for your database. + +Moreover, an environment must also be capable of gracefully switching between master and replica states. This means that the environment must be able to detect when it has switched states. + +Not surprisingly, a large part of your application's code will be tied up in knowing which state a given environment is in and then in the logic of how to behave depending on its state. + +This chapter shows you how to determine your environment's state, and it then shows you some sample code on how an application might behave depending on whether it is a master or a replica in a replicated application. + +## Determining State + +In order to determine whether your code is running as a master or a replica, you implement a callback whose function it is to respond to events that happen within the DB library. Note that these events are raised whenever the state is established. For example, when the current environment becomes a client — including at application startup — the `DB_EVENT_REP_CLIENT` event is raised. Also, when an election is held and a replica is elected to be a master, the `DB_EVENT_REP_MASTER` event is raised on the newly elected master and the `DB_EVENT_REP_NEWMASTER` is raised on the other replicas. + +Note that this callback is usable for events beyond those required for replication purposes. In this section, however, we only discuss the replication-specific events. + +The callback is required to determine which event has been passed to it, and then take action depending on the event. For replication, the events that we care about are: + +Some of the more commonly handled events are described below. For a complete list of events, see the `DB_ENV->set_event_notify()` method in the *Berkeley DB C API Reference Guide*. + +- `DB_EVENT_REP_CLIENT` + + The local environment is now a replica. + +- `DB_EVENT_REP_CONNECT_BROKEN` + + A previously established connection between two sites in the replication group has been broken. + +- `DB_EVENT_REP_CONNECT_ESTD` + + A connection has been established between two sites in the replication group. + +- `DB_EVENT_REP_CONNECT_RETRY_ESTABLISHED` + + An attempt was made to establish a connection to a known remote site, but the connection attempt failed. + +- `DB_EVENT_REP_DUPMASTER` + + A duplicate master has been discovered in the replication group. + +- `DB_EVENT_REP_ELECTED` + + The local site has just won an election and is now the master. Your code should now reconfigure itself to operation as a master site. + +- `DB_EVENT_REP_ELECTION_FAILED` + + The local site's attempt to initiate or participate in a replication master election failed, due to the lack of timely message response from a sufficient number of remote sites. + +- `DB_EVENT_REP_ELECTION_STARTED` + + Replication Manager has started an election to choose a master site. + +- `DB_EVENT_REP_LOCAL_SITE_REMOVED` + + The local site has been removed from the group. + +- `DB_EVENT_REP_NEWMASTER` + + An election was held and a new environment was made a master. However, the current environment *is not* the master. This event exists so that you can cause your code to take some unique action in the event that the replication groups switches masters. + +- `DB_EVENT_REP_MASTER` + + The local environment is now a master. + +- `DB_EVENT_REP_MASTER_FAILURE` + + The connection to the remote master replication site has failed. + +- `DB_EVENT_REP_PERM_FAILED` + + The Replication Manager did not receive enough acknowledgements to ensure the transaction's durability within the replicationg group. The Replication Manager has therefore flushed the transaction to the master's local disk for storage. + + How the Replication Manager knows whether the acknowledgements it has received is determined by the ack policy you have set for your applicaton. See Identifying Permanent Message Policies for more information. + +- `DB_EVENT_REP_SITE_ADDED` + + A new site has joined the replication group. + +- `DB_EVENT_REP_SITE_REMOVED` + + An existing site has been removed from the replication group. + +- `DB_EVENT_REP_STARTUPDONE` + + The replica has completed startup synchronization and is now processing log records received from the master. + +- `DB_EVENT_WRITE_FAILED` + + A Berkeley DB write to stable storage failed. + +Note that these events are raised whenever the state is established. That is, when the current environment becomes a replica, and that includes at application startup, the event is raised. Also, when an election is held and a replica is elected to be a master, then the event occurs. + +The implementation of this callback is fairly simple. First you pass a structure to the environment handle that you can use to record the environment's state, and then you implement a switch statement within the callback that you use to record the current state, depending on the arriving event. + +For example: + +``` c +#include +/* Forward declaration */ +void *event_callback(DB_ENV *, u_int32_t, void *); + +... + +/* The structure we use to track our environment's state */ +typedef struct { + int is_master; +} APP_DATA; + +... + +/* + * Inside our main() function, we declare an APP_DATA variable. + */ +APP_DATA my_app_data; +my_app_data.is_master = 0; /* Assume we start as a replica */ + +... + +/* + * Now we create our environment handle and set the APP_DATA structure + * to it's app_private member. + */ +if ((ret = db_env_create(&dbenv, 0)) != 0 ) { + fprintf(stderr, "Error creating handles: %s\n", + db_strerror(ret)); + goto err; +} +dbenv->app_private = &my_app_data; + +/* Having done that, register the callback with the + * Berkeley DB library + */ +dbenv->set_event_notify(dbenv, event_callback); +``` + +That done, we still need to implement the callback itself. This implementation can be fairly trivial. + +``` c +/* + * A callback used to determine whether the local environment is a + * replica or a master. This is called by the Replication Manager + * when the local environment changes state. + */ +void * +event_callback(DB_ENV *dbenv, u_int32_t which, void *info) +{ + APP_DATA *app = dbenv->app_private; + + info = NULL; /* Currently unused. */ + + switch (which) { + case DB_EVENT_REP_MASTER: + app->is_master = 1; + break; + + case DB_EVENT_REP_CLIENT: + app->is_master = 0; + break; + + case DB_EVENT_REP_STARTUPDONE: /* fallthrough */ + case DB_EVENT_REP_NEWMASTER: + /* Ignore. */ + break; + + default: + dbenv->errx(dbenv, "ignoring event %d", which); + } +} +``` + +Notice how we access the `APP_DATA` information using the environment handle's `app_private` data member. We also ignore the `DB_EVENT_REP_NEWMASTER` and `DB_EVENT_REP_STARTUPDONE` cases since these are not relevant for simple replicated applications. + +Of course, this only gives us the current state of the environment. We still need the code that determines what to do when the environment changes state and how to behave depending on the state (described in the next section). diff --git a/docs-src/guides/gsg_db_rep/fwrkpermmessage.md b/docs-src/guides/gsg_db_rep/fwrkpermmessage.md new file mode 100644 index 000000000..52e6dd4af --- /dev/null +++ b/docs-src/guides/gsg_db_rep/fwrkpermmessage.md @@ -0,0 +1,99 @@ +--- +title: "Permanent Message Handling" +api-name: "Permanent Message Handling" +source: docs/gsg_db_rep/C/fwrkpermmessage.html +--- +## Permanent Message Handling + + [Identifying Permanent Message Policies](fwrkpermmessage.md#fmwrkpermpolicy) + + [Setting the Permanent Message Timeout](fwrkpermmessage.md#fmwrkpermtimeout) + + [Adding a Permanent Message Policy to ex_rep_gsg_repmgr](fwrkpermmessage.md#perm2fmwrkexample) + +As described in Permanent Message Handling, messages are marked permanent if they contain database modifications that should be committed at the replica. DB's replication code decides if it must flush its transaction logs to disk depending on whether it receives sufficient permanent message acknowledgments from the participating replicas. More importantly, the thread performing the transaction commit blocks until it either receives enough acknowledgments, or the acknowledgment timeout expires. + +The Replication Manager is fully capable of managing permanent messages for you if your application requires it (most do). Almost all of the details of this are handled by the Replication Manager for you. However, you do have to set some policies that tell the Replication Manager how to handle permanent messages. + +There are two things that you have to do: + +- Determine how many acknowledgments must be received by the master. + +- Identify the amount of time that replicas have to send their acknowledgments. + +### Identifying Permanent Message Policies + +You identify permanent message policies using the `DB_ENV->repmgr_set_ack_policy()` method. Note that you can set permanent message policies at any time during the life of the application. + +The following permanent message policies are available when you use the Replication Manager: + +### Note + +The following list mentions *electable peer* several times. This is simply another environment that can be elected to be a master (that is, it has a priority greater than 0). Do not confuse this with the concept of a peer as used for client to client transfers. See Client to Client Transfer for more information on client to client transfers. + +- `DB_REPMGR_ACKS_NONE` + + No permanent message acknowledgments are required. If this policy is selected, permanent message handling is essentially "turned off." That is, the master will never wait for replica acknowledgments. In this case, transaction log data is either flushed or not strictly depending on the type of commit that is being performed (synchronous or asynchronous). + +- `DB_REPMGR_ACKS_ONE` + + At least one replica must acknowledge the permanent message within the timeout period. + +- `DB_REPMGR_ACKS_ONE_PEER` + + At least one electable peer must acknowledge the permanent message within the timeout period. + +- `DB_REPMGR_ACKS_ALL` + + All replicas must acknowledge the message within the timeout period. This policy should be selected only if your replication group has a small number of replicas, and those replicas are on extremely reliable networks and servers. + +- `DB_REPMGR_ACKS_ALL_AVAILABLE` + + All currently connected replication clients must acknowledge the message. This policy will invoke the `DB_EVENT_REP_PERM_FAILED` event if fewer than a quorum of clients acknowledged during that time. + +- `DB_REPMGR_ACKS_ALL_PEERS` + + All electable peers must acknowledge the message within the timeout period. This policy should be selected only if your replication group is small, and its various environments are on extremely reliable networks and servers. + +- `DB_REPMGR_ACKS_QUORUM` + + A quorum of electable peers must acknowledge the message within the timeout period. A quorum is reached when acknowledgments are received from the minimum number of environments needed to ensure that the record remains durable if an election is held. That is, the master wants to hear from enough electable replicas that they have committed the record so that if an election is held, the master knows the record will exist even if a new master is selected. + +By default, a quorum of electable peers must must acknowledge a permanent message in order for it considered to have been successfully transmitted. + +### Setting the Permanent Message Timeout + +The permanent message timeout represents the maximum amount of time the committing thread will block waiting for message acknowledgments. If sufficient acknowledgments arrive before this timeout has expired, the thread continues operations as normal. However, if this timeout expires, the committing thread flushes its transaction log buffer before continuing with normal operations. + +You set the timeout value using the `DB_ENV->rep_set_timeout()` method. When you do this, you provide the `DB_REP_ACK_TIMEOUT` value to the `which` parameter, and the timeout value in microseconds to the `timeout` parameter. + +For example: + +``` c + dbenv->rep_set_timeout(dbenv, DB_REP_ACK_TIMEOUT, 100); +``` + +This timeout value can be set at anytime during the life of the application. + +### Adding a Permanent Message Policy to ex_rep_gsg_repmgr + +For illustration purposes, we will now update `ex_rep_gsg_repmgr` such that it requires only one acknowledgment from a replica on transactional commits. Also, we will give this acknowledgment a 500 microsecond timeout value. This means that our application's main thread will block for up to 500 microseconds waiting for an acknowledgment. If it does not receive at least one acknowledgment in that amount of time, DB will flush the transaction logs to disk before continuing on. + +This is a very simple update. We can perform the entire thing immediately before we parse our command line options. This is where we configure our environment handle anyway, so it is a good place to put it. + +``` c + if ((ret = create_env(progname, &dbenv)) != 0) + goto err; + + /* Default priority is 100 */ + dbenv->rep_set_priority(dbenv, 100); + /* Permanent messages require at least one ack */ + dbenv->repmgr_set_ack_policy(dbenv, DB_REPMGR_ACKS_ONE); + /* Give 500 microseconds to receive the ack */ + dbenv->rep_set_timeout(dbenv, DB_REP_ACK_TIMEOUT, 500); + + /* Collect the command line options */ + while ((ch = getopt(argc, argv, "h:l:n:p:r:")) != EOF) + + ... +``` diff --git a/docs-src/guides/gsg_db_rep/heartbeats.md b/docs-src/guides/gsg_db_rep/heartbeats.md new file mode 100644 index 000000000..37854a9ea --- /dev/null +++ b/docs-src/guides/gsg_db_rep/heartbeats.md @@ -0,0 +1,16 @@ +--- +title: "Managing Heartbeats" +api-name: "Managing Heartbeats" +source: docs/gsg_db_rep/C/heartbeats.html +--- +## Managing Heartbeats + +If your replicated application experiences few updates, it is possible for the replication group to lose a master without noticing it. This is because normally a replicated application only knows that a master has gone missing when update activity causes messages to be passed between the master and replicas. + +To guard against this, you can configure a heartbeat. The heartbeat must be configured for both the master and each of the replicas. + +On the master, you configure the application to send a heartbeat on a defined interval when it is otherwise idle. Do this by using the `DB_REP_HEARTBEAT_SEND` value to the `which` parameter of the `DB_ENV->rep_set_timeout()` method. You must also provide the method a value representing the period between heartbeats in microseconds. Note that the heartbeat is sent only if the system is idle. + +On the replica, you configure the application to listen for a heartbeat. The time that you configure here is the amount of time the replica will wait for some message from the master (either the heartbeat or some other message) before concluding that the connection is lost. You do this using the `DB_REP_HEARTBEAT_MONITOR` value to the `which` parameter of the `DB_ENV->rep_set_timeout()` method and a timeout value in microseconds. + +For best results, configure the heartbeat monitor for a longer time interval than the heartbeat send interval. diff --git a/docs-src/guides/gsg_db_rep/index.md b/docs-src/guides/gsg_db_rep/index.md new file mode 100644 index 000000000..39d39329d --- /dev/null +++ b/docs-src/guides/gsg_db_rep/index.md @@ -0,0 +1,138 @@ +--- +title: "Getting Started with Replicated Berkeley DB Applications" +api-name: "Getting Started with Replicated Berkeley DB Applications" +source: docs/gsg_db_rep/C/index.html +--- +# Getting Started with Replicated Berkeley DB Applications + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction](introduction.md) + + [Overview](introduction.md#overview) + + [Replication Environments](introduction.md#repenvirons) + + [Replication Databases](introduction.md#repdbs) + + [Communications Layer](introduction.md#commlayer) + + [Selecting a Master](introduction.md#masterselect) + + [Replication Benefits](repadvantage.md) + + [The Replication APIs](apioverview.md) + + [Replication Manager Overview](apioverview.md#repframeworkoverview) + + [Replication Base API Overview](apioverview.md#repapioverview) + + [Holding Elections](elections.md) + + [Influencing Elections](elections.md#influencingelections) + + [Winning Elections](elections.md#winningelections) + + [Switching Masters](elections.md#switchingmasters) + + [Permanent Message Handling](permmessages.md) + + [When Not to Manage Permanent Messages](permmessages.md#permmessagenot) + + [Managing Permanent Messages](permmessages.md#permmanage) + + [Implementing Permanent Message Handling](permmessages.md#permimplement) + + [2. Transactional Application](txnapp.md) + + [Application Overview](txnapp.md#appoverview) + + [Program Listing](simpleprogramlisting.md) + + [Function: main()](simpleprogramlisting.md#main_c) + + [Function: create_env()](simpleprogramlisting.md#create_env_c) + + [Function: env_init()](simpleprogramlisting.md#env_init_c) + + [Function: doloop()](simpleprogramlisting.md#doloop_c) + + [Function: print_stocks()](simpleprogramlisting.md#printstocks_c) + + [3. The DB Replication Manager](repapp.md) + + [The DB_SITE Handle](repapp.md#repmgr_grpmgmt) + + [Starting and Stopping Replication](rep_init_code.md) + + [Managing Election Policies](rep_init_code.md#election_flags) + + [Selecting the Number of Threads](rep_init_code.md#thread_count) + + [Adding the Replication Manager to ex_rep_gsg_simple](repmgr_init_example_c.md) + + [Permanent Message Handling](fwrkpermmessage.md) + + [Identifying Permanent Message Policies](fwrkpermmessage.md#fmwrkpermpolicy) + + [Setting the Permanent Message Timeout](fwrkpermmessage.md#fmwrkpermtimeout) + + [Adding a Permanent Message Policy to ex_rep_gsg_repmgr](fwrkpermmessage.md#perm2fmwrkexample) + + [Managing Election Times](electiontimes.md) + + [Managing Election Timeouts](electiontimes.md#electiontimeout) + + [Managing Election Retry Times](electiontimes.md#electretrytime) + + [Managing Connection Retries](fmwrkconnectretry.md) + + [Managing Heartbeats](heartbeats.md) + + [4. Replica versus Master Processes](fwrkmasterreplica.md) + + [Determining State](fwrkmasterreplica.md#determinestate) + + [Processing Loop](processingloop.md) + + [Example Processing Loop](exampledoloop.md) + + [Running It](exampledoloop.md#runningit) + + [5. Additional Features](addfeatures.md) + + [Delayed Synchronization](addfeatures.md#delayedsync) + + [Managing Blocking Operations](manageblock.md) + + [Stop Auto-Initialization](autoinit.md) + + [Read-Your-Writes Consistency](rywc.md) + + [Client to Client Transfer](c2ctransfer.md) + + [Identifying Peers](c2ctransfer.md#fmwrkpeerserver) + + [Bulk Transfers](bulk.md) diff --git a/docs-src/guides/gsg_db_rep/introduction.md b/docs-src/guides/gsg_db_rep/introduction.md new file mode 100644 index 000000000..62cd30b58 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/introduction.md @@ -0,0 +1,102 @@ +--- +title: "Chapter 1. Introduction" +api-name: "Chapter 1. Introduction" +source: docs/gsg_db_rep/C/introduction.html +--- +## Chapter 1. Introduction + +**Table of Contents** + + [Overview](introduction.md#overview) + + [Replication Environments](introduction.md#repenvirons) + + [Replication Databases](introduction.md#repdbs) + + [Communications Layer](introduction.md#commlayer) + + [Selecting a Master](introduction.md#masterselect) + + [Replication Benefits](repadvantage.md) + + [The Replication APIs](apioverview.md) + + [Replication Manager Overview](apioverview.md#repframeworkoverview) + + [Replication Base API Overview](apioverview.md#repapioverview) + + [Holding Elections](elections.md) + + [Influencing Elections](elections.md#influencingelections) + + [Winning Elections](elections.md#winningelections) + + [Switching Masters](elections.md#switchingmasters) + + [Permanent Message Handling](permmessages.md) + + [When Not to Manage Permanent Messages](permmessages.md#permmessagenot) + + [Managing Permanent Messages](permmessages.md#permmanage) + + [Implementing Permanent Message Handling](permmessages.md#permimplement) + +This book provides a thorough introduction and discussion on replication as used with Berkeley DB (DB). It begins by offering a general overview to replication and the benefits it provides. It also describes the APIs that you use to implement replication, and it describes architecturally the things that you need to do to your application code in order to use the replication APIs. Finally, it discusses the differences in backup and restore strategies that you might pursue when using replication, especially where it comes to log file removal. + +You should understand the concepts from the *Berkeley DB Getting Started with Transaction Processing* guide before reading this book. + +## Overview + + [Replication Environments](introduction.md#repenvirons) + + [Replication Databases](introduction.md#repdbs) + + [Communications Layer](introduction.md#commlayer) + + [Selecting a Master](introduction.md#masterselect) + +The DB replication APIs allow you to distribute your database write operations (performed on a read-write master) to one or more read-only *replicas*. For this reason, DB's replication implementation is said to be a *single master, multiple replica* replication strategy. + +Note that your database write operations can occur only on the master; any attempt to write to a replica results in an error being returned to the DB API used to perform the write. + +A single replication master and all of its replicas are referred to as a *replication group*. While all members of the replication group can reside on the same machine, usually each replication participant is placed on a separate physical machine somewhere on the network. + +Note that all replication applications must first be transactional applications. The data that the master transmits to its replicas are log records that are generated as records are updated. Upon transactional commit, the master transmits a transaction record which tells the replicas to commit the records they previously received from the master. In order for all of this to work, your replicated application must also be a transactional application. For this reason, it is recommended that you write and debug your DB application as a stand-alone transactional application before introducing the replication layer to your code. + +### Replication Environments + +The most important requirement for a replication participant is that it must use a unique Berkeley DB database environment independent of all other replication participants. So while multiple replication participants can reside on the same physical machine, no two such participants can share the same environment home directory. + +For this reason, technically replication occurs between unique *database environments*. So in the strictest sense, a replication group consists of a *master environment* and one or more *replica environments*. However, the reality is that for production code, each such environment will usually be located on its own unique machine. Consequently, this manual sometimes talks about *replication sites*, meaning the unique combination of environment home directory, host and port that a specific replication application is using. + +There is no DB-specified limit to the number of environments which can participate in a replication group. The only limitation here is one of resources — network bandwidth, for example. + +(Note, however, that the Replication Manager does place a limit on the number of environments you can use. See Replication Manager Overview for details.) + +Also, DB's replication implementation requires all participating environments to be assigned IDs that are locally unique to the given environment. Depending on the replication APIs that you choose to use, you may or may not need to manage this particular detail. + +For detailed information on database environments, see the *Berkeley DB Getting Started with Transaction Processing* guide. For more information on environment IDs, see the *Berkeley DB Programmer's Reference Guide*. + +### Replication Databases + +DB's databases are managed and used in exactly the same way as if you were writing a non-replicated application, with a couple of caveats. First, the databases maintained in a replicated environment must reside either in the `ENV_HOME` directory, or in the directory identified by the `DB_ENV->set_data_dir()` method. Unlike non-replication applications, you cannot place your databases in a subdirectory below these locations. You should also not use full path names for your databases or environments as these are likely to break when they are replicated to other machines. + +### Communications Layer + +In order to transmit database writes to the replication replicas, DB requires a communications layer. DB is agnostic as to what this layer should look like. The only requirement is that it be capable of passing two opaque data objects and an environment ID from the master to its replicas without corruption. + +Because replicas are usually placed on different machines on the network, the communications layer is usually some kind of a network-aware implementation. Beyond that, its implementation details are largely up to you. It could use TCP/IP sockets, for example, or it could use raw sockets if they perform better for your particular application. + +Note that you may not have to write your own communications layer. DB provides a Replication Manager that includes a fully-functional TCP/IP-based communications layer. See The Replication APIs for more information. + +See the *Berkeley DB Programmer's Reference Guide* for a description of how to write your own custom replication communications layer. + +### Selecting a Master + +Every replication group is allowed one and only one master environment. Usually masters are selected by holding an *election*, although it is possible to turn elections off and manually select masters (this is not recommended for most replicated applications). + +When elections are being used, they are performed by the underlying Berkeley DB replication code so you have to do very little to implement them. + +When holding an election, replicas "vote" on who should be the master. Among replicas participating in the election, the one with the most up-to-date set of log records will win the election. Note that it's possible for there to be a tie. When this occurs, priorities are used to select the master. See Holding Elections for details. + +For more information on holding and managing elections, see Holding Elections. diff --git a/docs-src/guides/gsg_db_rep/manageblock.md b/docs-src/guides/gsg_db_rep/manageblock.md new file mode 100644 index 000000000..9daa7e95f --- /dev/null +++ b/docs-src/guides/gsg_db_rep/manageblock.md @@ -0,0 +1,12 @@ +--- +title: "Managing Blocking Operations" +api-name: "Managing Blocking Operations" +source: docs/gsg_db_rep/C/manageblock.html +--- +## Managing Blocking Operations + +When a replica is in the process of synchronizing with its master, DB operations are blocked at some points during this process until the synchronization is completed. For replicas with a heavy read load, these blocked operations may represent an unacceptable loss in throughput. + +You can configure DB so that it will not block when synchronization is in process. Instead, the DB operation will fail, immediately returning a `DB_REP_LOCKOUT` error. When this happens, it is up to your application to determine what action to take (that is, logging the event, making an appropriate user response, retrying the operation, and so forth). + +To turn off blocking on synchronization, specify `DB_REP_CONF_NOWAIT` to `DB_ENV->rep_set_config()` and then specify `1` to the `onoff` parameter. (Specify `0` to turn the feature off.) diff --git a/docs-src/guides/gsg_db_rep/moreinfo.md b/docs-src/guides/gsg_db_rep/moreinfo.md new file mode 100644 index 000000000..65cf7566a --- /dev/null +++ b/docs-src/guides/gsg_db_rep/moreinfo.md @@ -0,0 +1,28 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/gsg_db_rep/C/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a transactional DB application: + +- Getting Started with Transaction Processing for C + +- Getting Started with Berkeley DB for C + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB C API Reference Guide + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/gsg_db_rep/permmessages.md b/docs-src/guides/gsg_db_rep/permmessages.md new file mode 100644 index 000000000..2eb0d7f19 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/permmessages.md @@ -0,0 +1,86 @@ +--- +title: "Permanent Message Handling" +api-name: "Permanent Message Handling" +source: docs/gsg_db_rep/C/permmessages.html +--- +## Permanent Message Handling + + [When Not to Manage Permanent Messages](permmessages.md#permmessagenot) + + [Managing Permanent Messages](permmessages.md#permmanage) + + [Implementing Permanent Message Handling](permmessages.md#permimplement) + +Messages received by a replica may be marked with special flag that indicates the message is permanent. Custom replicated applications will receive notification of this flag via the `DB_REP_ISPERM` return value from the `DB_ENV->rep_process_message()` method. There is no hard requirement that a replication application look for, or respond to, this return code. However, because robust replicated applications typically do manage permanent messages, we introduce the concept here. + +A message is marked as being permanent if the message affects transactional integrity. For example, transaction commit messages are an example of a message that is marked permanent. What the application does about the permanent message is driven by the durability guarantees required by the application. + +For example, consider what the Replication Manager does when it has permanent message handling turned on and a transactional commit record is sent to the replicas. First, the replicas must transactional-commit the data modifications identified by the message. And then, upon a successful commit, the Replication Manager sends the master a message acknowledgment. + +For the master (again, using the Replication Manager), things are a little more complicated than simple message acknowledgment. Usually in a replicated application, the master commits transactions asynchronously; that is, the commit operation does not block waiting for log data to be flushed to disk before returning. So when a master is managing permanent messages, it typically blocks the committing thread immediately before `commit()` returns. The thread then waits for acknowledgments from its replicas. If it receives enough acknowledgments, it continues to operate as normal. + +If the master does not receive message acknowledgments — or, more likely, it does not receive *enough* acknowledgments — the committing thread flushes its log data to disk and then continues operations as normal. The master application can do this because replicas that fail to handle a message, for whatever reason, will eventually catch up to the master. So by flushing the transaction logs to disk, the master is ensuring that the data modifications have made it to stable storage in one location (its own hard drive). + +### When Not to Manage Permanent Messages + +There are two reasons why you might choose to not implement permanent messages. In part, these go to why you are using replication in the first place. + +One class of applications uses replication so that the application can improve transaction through-put. Essentially, the application chooses a reduced transactional durability guarantee so as to avoid the overhead forced by the disk I/O required to flush transaction logs to disk. However, the application can then regain that durability guarantee to a certain degree by replicating the commit to some number of replicas. + +Using replication to improve an application's transactional commit guarantee is called *replicating to the network.* + +In extreme cases where performance is of critical importance to the application, the master might choose to both use asynchronous commits *and* decide not to wait for message acknowledgments. In this case the master is simply broadcasting its commit activities to its replicas without waiting for any sort of a reply. An application like this might also choose to use something other than TCP/IP for its network communications since that protocol involves a fair amount of packet acknowledgment all on its own. Of course, this sort of an application should also be very sure about the reliability of both its network and the machines that are hosting its replicas. + +At the other extreme, there is a class of applications that use replication purely to improve read performance. This sort of application might choose to use synchronous commits on the master because write performance there is not of critical performance. In any case, this kind of an application might not care to know whether its replicas have received and successfully handled permanent messages because the primary storage location is assumed to be on the master, not the replicas. + +### Managing Permanent Messages + +With the exception of a rare breed of replicated applications, most masters need some view as to whether commits are occurring on replicas as expected. At a minimum, this is because masters will not flush their log buffers unless they have reason to expect that permanent messages have not been committed on the replicas. + +That said, it is important to remember that managing permanent messages involves a fair amount of network traffic. The messages must be sent to the replicas and the replicas must acknowledge them. This represents a performance overhead that can be worsened by congested networks or outright outages. + +Therefore, when managing permanent messages, you must first decide on how many of your replicas must send acknowledgments before your master decides that all is well and it can continue normal operations. When making this decision, you could decide that *all* replicas must send acknowledgments. But unless you have only one or two replicas, or you are replicating over a very fast and reliable network, this policy could prove very harmful to your application's performance. + +Therefore, a common strategy is to wait for an acknowledgment from a simple majority of replicas. This ensures that commit activity has occurred on enough machines that you can be reliably certain that data writes are preserved across your network. + +Remember that replicas that do not acknowledge a permanent message are not necessarily unable to perform the commit; it might be that network problems have simply resulted in a delay at the replica. In any case, the underlying DB replication code is written such that a replica that falls behind the master will eventually take action to catch up. + +Depending on your application, it may be possible for you to code your permanent message handling such that acknowledgment must come from only one or two replicas. This is a particularly attractive strategy if you are closely managing which machines are eligible to become masters. Assuming that you have one or two machines designated to be a master in the event that the current master goes down, you may only want to receive acknowledgments from those specific machines. + +Finally, beyond simple message acknowledgment, you also need to implement an acknowledgment timeout for your application. This timeout value is simply meant to ensure that your master does not hang indefinitely waiting for responses that will never come because a machine or router is down. + +### Implementing Permanent Message Handling + +How you implement permanent message handling depends on which API you are using to implement replication. If you are using the Replication Manager, then permanent message handling is configured using policies that you specify to the framework. In this case, you can configure your application to: + +- Ignore permanent messages (the master does not wait for acknowledgments). + +- Require acknowledgments from a quorum. A quorum is reached when acknowledgments are received from the minimum number of electable peers needed to ensure that the record remains durable if an election is held. + + An *electable peer* is any other site that potentially can be elected master. + + The goal here is to be absolutely sure the record is durable. The master wants to hear from enough electable peer that they have committed the record so that if an election is held, the master knows the record will exist even if a new master is selected. + + This is the default policy. + +- Require an acknowledgment from at least one replica. + +- Require acknowledgments from all replicas. + +- Require an acknowledgment from at least one electable peer. + +- Require acknowledgments from all electable peers. + +Note that the Replication Manager simply flushes its transaction logs and moves on if a permanent message is not sufficiently acknowledged. + +For details on permanent message handling with the Replication Manager, see Permanent Message Handling. + +If these policies are not sufficient for your needs, or if you want your application to take more corrective action than simply flushing log buffers in the event of an unsuccessful commit, then you must use implement replication using the Base APIs. + +When using the Base APIs, messages are sent from the master to its replica using a `send()` callback that you implement. Note, however, that DB's replication code automatically sets the permanent flag for you where appropriate. + +If the `send()` callback returns with a non-zero status, DB flushes the transaction log buffers for you. Therefore, you must cause your `send()` callback to block waiting for acknowledgments from your replicas. As a part of implementing the `send()` callback, you implement your permanent message handling policies. This means that you identify how many replicas must acknowledge the message before the callback can return `0`. You must also implement the acknowledgment timeout, if any. + +Further, message acknowledgments are sent from the replicas to the master using a communications channel that you implement (the replication code does not provide a channel for acknowledgments). So implementing permanent messages means that when you write your replication communications channel, you must also write it in such a way as to also handle permanent message acknowledgments. + +For more information on implementing permanent message handling using a custom replication layer, see the *Berkeley DB Programmer's Reference Guide*. diff --git a/docs-src/guides/gsg_db_rep/preface.md b/docs-src/guides/gsg_db_rep/preface.md new file mode 100644 index 000000000..af3aa7ab6 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/preface.md @@ -0,0 +1,62 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/gsg_db_rep/C/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +This document describes how to write replicated applications for Berkeley DB 11*g* Release 2 (library version 11.2.5.3). The APIs used to implement replication in your application are described here. This book describes the concepts surrounding replication, the scenarios under which you might choose to use it, and the architectural requirements that a replication application has over a transactional application. + +This book is aimed at the software engineer responsible for writing a replicated DB application. + +This book assumes that you have already read and understood the concepts contained in the *Berkeley DB Getting Started with Transaction Processing* guide. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Structure names are represented in `monospaced font`, as are `method names`. For example: "`DB->open()` is a method on a `DB` handle." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database + * files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +In some situations, programming examples are updated from one chapter to the next. When this occurs, the new code is presented in **`monospaced bold`** font. For example: + +``` c +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + DB *itemname_sdbp; /* Index based on the item name index */ + char *db_home_dir; /* Directory containing the database + * files */ + char *itemname_db_name; /* Itemname secondary database */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of special interest are represented using a note block such as this. diff --git a/docs-src/guides/gsg_db_rep/processingloop.md b/docs-src/guides/gsg_db_rep/processingloop.md new file mode 100644 index 000000000..cf304a576 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/processingloop.md @@ -0,0 +1,110 @@ +--- +title: "Processing Loop" +api-name: "Processing Loop" +source: docs/gsg_db_rep/C/processingloop.html +--- +## Processing Loop + +Typically the central part of any replication application is some sort of a continuous loop that constantly checks the state of the environment (whether it is a replica or a master), opens and/or closes the databases as is necessary, and performs other useful work. A loop such as this one must of necessity take special care to know whether it is operating on a master or a replica environment because all of its activities are dependent upon that state. + +The flow of activities through the loop will generally be as follows: + +1. Check whether the environment has changed state. If it has, you might want to reopen your database handles, especially if you opened your replica's database handles as read-only. In this case, you might need to reopen them as read-write. However, if you always open your database handles as read-write, then it is not automatically necessary to reopen the databases due to a state change. Instead, you could check for a `DB_REP_HANDLE_DEAD` return code when you use your database handle(s). If you see this, then you need to reopen your database handle(s). + +2. If the databases are closed, create new database handles, configure the handle as is appropriate, and then open the databases. Note that handle configuration will be different, depending on whether the handle is opened as a replica or a master. At a minimum, the master should be opened with database creation privileges, whereas the replica does not need to be. You must also open the master such that its databases are read-write. You *can* open replicas with read-only databases, so long as you are prepared to close and then reopen the handle in the event the client becomes a master. + + Also, note that if the local environment is a replica, then it is possible that databases do not currently exist. In this case, the database open attempts will fail. Your code will have to take this corner case into account (described below). + +3. Once the databases are opened, check to see if the local environment is a master. If it is, do whatever it is a master should do for your application. + + Remember that the code for your master should include some way for you to tell the master to exit gracefully. + +4. If the local environment is not a master, then do whatever it is your replica environments should do. Again, like the code for your master environments, you should provide a way for your replicas to exit the processing loop gracefully. + +The following code fragment illustrates these points (note that we fill out this fragment with a working example next in this chapter): + +``` c +/* loop to manage replication activities */ + +DB *dbp; +int ret; +APP_DATA *app_data; +u_int32_t flags; + +dbp = NULL; +ret = 0; + +/* + * Remember that for this to work, an APP_DATA struct would have first + * had to been set to the environment handle's app_private data + * member. (dbenv is presumably declared and opened in another part of + * the code.) + */ +app_data = dbenv->app_private; + +/* + * Infinite loop. We exit depending on how the master and replica code + * is written. + */ +for (;;) { + /* If dbp is not opened, we need to open it. */ + if (dbp == NULL) { + /* + * Create the handle and then configure it. Before you open + * it, you have to decide what open flags to use: + */ + if ((ret = db_create(&dbp, dbenv, 0)) != 0) + return (ret); + + flags = DB_AUTO_COMMIT; + if (app_data->is_master) + flags |= DB_CREATE + /* + * Now you can open your database handle, passing to it the + * flags selected above. + * + * One thing to watch out for is a case where the databases + * you are trying to open do not yet exist. This can happen + * for replicas where the databases are being opened + * read-only. If this happens, ENOENT is returned by the + * open() call. + */ + + if (( ret = dbp->open(...)) != 0) { + if (ret == ENOENT) { + /* Close the database handle, then null it out, then + * sleep for some amount of time in order to give + * replication a chance to create the databases. + */ + dbp->close(dbp, 0); // Ignoring ret code. + // Not robust! + dbp = NULL; + sleep(SOME_SLEEPTIME); + continue; + } + /* + * Otherwise, some other error has happened and general + * error handling should be used. + */ + goto err; + } + } + + /* + * Now that the databases have been opened, continue with general + * processing, depending on whether we are a master or a replica. + */ + if (app_data->is_master) { + /* + * Do master stuff here. Don't forget to include a way to + * gracefully exit the loop. */ + */ + } else { + /* + * Do replica stuff here. As is the case with the master + * code, be sure to include a way to gracefully exit the + * loop. + */ + } +} +``` diff --git a/docs-src/guides/gsg_db_rep/rep_init_code.md b/docs-src/guides/gsg_db_rep/rep_init_code.md new file mode 100644 index 000000000..417f87747 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/rep_init_code.md @@ -0,0 +1,244 @@ +--- +title: "Starting and Stopping Replication" +api-name: "Starting and Stopping Replication" +source: docs/gsg_db_rep/C/rep_init_code.html +--- +## Starting and Stopping Replication + + [Managing Election Policies](rep_init_code.md#election_flags) + + [Selecting the Number of Threads](rep_init_code.md#thread_count) + +As described above, you introduce replication to an application by starting with a transactional application, performing some basic replication configuration, and then starting replication using `DB_ENV->repmgr_start()`. + +You stop replication by closing your environment cleanly in the same way you would for any DB application. + +For example, the following code fragment initializes, then stops and starts replication. Note that other replication activities are omitted for brevity. + +``` c +#include + +/* Use a 10mb cache */ +#define CACHESIZE (10 * 1024 * 1024) + +... + + DB_ENV *dbenv; /* Environment handle. */ + DB_SITE *dbsite; /* Replication manager site handle. */ + const char *progname; /* Program name. */ + const char *envHome; /* Environment home directory. */ + const char *listen_host; /* A TCP/IP hostname. */ + const char *other_host; /* A TCP/IP hostname. */ + int ret; /* Error return code. */ + int is_group_creator; /* A flag */ + u_int16 listen_port; /* A TCP/IP port. */ + u_int16 other_port; /* A TCP/IP port. */ + + /* Initialize variables */ + dbenv = NULL; + progname = "example_replication"; + envHome = "ENVIRONMENT_HOME"; + listen_host = "mymachine.sleepycat.com"; + listen_port = 5001; + other_host = "anothermachine.sleepycat.com"; + other_port = 4555; + ret = 0; + is_group_creator = 1; /* This is usually set via a command line + argument or some other external + configuration mechanism. */ + + /* Create the environment handle */ + if ((ret = db_env_create(&dbenv, 0)) != 0 ) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Configure the environment handle. Here we configure + * asynchronous transactional commits for performance reasons. + */ + dbenv->set_errfile(dbenv, stderr); + dbenv->set_errpfx(dbenv, progname); + (void)dbenv->set_cachesize(dbenv, 0, CACHESIZE, 0); + (void)dbenv->set_flags(dbenv, DB_TXN_NOSYNC, 1); + + /* + * Configure the local address. This is the local hostname and + * port that this replication environment will use to receive + * incoming replication messages. Note that this can be + * performed only once for the replication environment. + * It is required. + + * First: Create a DB_SITE handle to identify the site's + * host/port network address. + */ + if ((ret = dbenv->repmgr_site(dbenv, listen_host, listen_port, + &dbsite;, 0)) != 0) { + fprintf(stderr, "Could not set local address (%d).\n", ret); + goto err; + } + + /* + * Second: Configure this site as the local site within the + * replication group. + */ + dbsite->set_config(dbsite, DB_LOCAL_SITE, 1); + + /* + * Third: Set DB_GROUP_CREATOR if applicable. This can be done + * only for the local site. It should also only be peformed + * for one and only one site in a replication group, so + * typically this is driven by an externally-supplied + * configuration option. + * + * DB_GROUP_CREATOR only has meaning if you are starting the + * very first site for the very first time in a replication + * group. It is otherwise ignored. + */ + if (is_group_creator) + dbsite->set_config(dbsite, DB_GROUP_CREATOR, 1); + + /* + * Having configured the local site, we can immediately + * deallocate the DB_SITE handle. + */ + if ((ret = dbsite->close(dbsite)) != 0) { + dbenv->err(dbenv, ret, "DB_SITE->close"); + goto err; + } + + /* + * Set this replication environment's priority. This is used + * for elections. + * + * Set this number to a positive integer, or 0 if you do not want + * this site to be able to become a master. + */ + dbenv->rep_set_priority(dbenv, 100); + + /* + * Configure a bootstrap helper. This information is used only + * if the site currently exists, and the local site has never + * been started before. Otherwise, this configuration + * information is ignored. + * + */ + if (!is_group_creator) { + if ((ret = dbenv->repmgr_site(dbenv, other_host, other_port, + &dbsite, 0)) != 0) { + dbenv->err(dbenv, ret, "Could not add site %s:%d\n", + other_host, other_port); + goto err; + } + + dbsite->set_config(dbsite, DB_BOOTSTRAP_HELPER, 1); + if ((ret = dbsite->close(dbsite)) != 0) { + dbenv->err(dbenv, ret, "DB_SITE->close"); + goto err; + } + + /* + * Having configured the bootstrap helper site, we can + * immediately deallocate the DB_SITE handle. + */ + if ((ret = dbsite->close(dbsite)) != 0) { + dbenv->err(dbenv, ret, "DB_SITE->close"); + goto err; + } + } + + /* Open the environment handle. Note that we add DB_THREAD and + * DB_INIT_REP to the list of flags. These are required. + */ + if ((ret = dbenv->open(dbenv, home, DB_CREATE | DB_RECOVER | + DB_INIT_LOCK | DB_INIT_LOG | + DB_INIT_MPOOL | DB_INIT_TXN | + DB_THREAD | DB_INIT_REP, + 0)) != 0) { + goto err; + } + + /* Start the replication manager such that it uses 3 threads. */ + if ((ret = dbenv->repmgr_start(dbenv, 3, DB_REP_ELECTION)) != 0) + goto err; + + /* Sleep to give ourselves time to find a master */ + sleep(5); + + /* + ********************************************************** + *** All other application code goes here, including ***** + *** database opens ***** + ********************************************************** + */ + +err: /* + * Make sure all your database and dbsite handles are closed + * (omitted from this example). + */ + + /* Close the environment */ + if (dbenv != NULL) + (void)dbenv->close(dbenv, 0); + + /* All done */ + return (ret); +``` + +### Managing Election Policies + +Before continuing, it is worth taking a look at the startup election flags accepted by `DB_ENV->repgmr_start()`. These flags control how your replication application will behave when it first starts up. + +In the previous example, we specified `DB_REP_ELECTION` when we started replication. This causes the application to try to find a master upon startup. If it cannot, it calls for an election. In the event an election is held, the environment receiving the most number of votes will become the master. + +There's some important points to make here: + +- This flag only requires that other environments in the replication group participate in the vote. There is no requirement that *all* such environments participate. In other words, if an environment starts up, it can call for an election, and select a master, even if all other environment have not yet joined the replication group. + +- It only requires a simple majority of participating environments to elect a master. This is always true of elections held using the Replication Manager. + +- As always, the environment participating in the election with the most up-to-date log files is selected as master. If an environment with more recent log files has not yet joined the replication group, it may not become the master. + +Any one of these points may be enough to cause a less-than-optimum environment to be selected as master. Therefore, to give you a better degree of control over which environment becomes a master at application startup, the Replication Manager offers the following start-up flags: + + + + + + + + + + + + + + + + + + + + + + +
FlagDescription
DB_REP_MASTER

The application starts up and declares the environment to be a master without calling for an election. It is an error for more than one environment to start up using this flag, or for an environment to use this flag when a master already exists.

+

Note that no replication group should ever operate with more than one master.

+

In the event that a environment attempts to become a master when a master already exists, the replication code will resolve the problem by holding an election. Note, however, that there is always a possibility of data loss in the face of duplicate masters, because once a master is selected, the environment that loses the election will have to roll back any transactions committed until it is in sync with the "real" master.

DB_REP_CLIENT

The application starts up and declares the environment to be a replica without calling for an election. Note that the environment can still become a master if a subsequent application starts up, calls for an election, and this environment is elected master.

DB_REP_ELECTION

As described above, the application starts up, looks for a master, and if one is not found calls for an election.

+ +### Selecting the Number of Threads + +Under the hood, the Replication Manager is threaded and you can control the number of threads used to process messages received from other replicas. The threads that the Replication Manager uses are: + +- Incoming message thread. This thread receives messages from the site's socket and passes those messages to message processing threads (see below) for handling. + +- Outgoing message thread. Outgoing messages are sent from whatever thread performed a write to the database(s). That is, the thread that called, for example, `DB->put()` is the thread that writes replication messages about that fact to the socket. + + Note that if this write activity would cause the thread to be blocked due to some condition on the socket, the Replication Manager will hand the outgoing message to the incoming message thread, and it will then write the message to the socket. This prevents your database write threads from blocking due to abnormal network I/O conditions. + +- Message processing threads are responsible for parsing and then responding to incoming replication messages. Typically, a response will include write activity to your database(s), so these threads can be busy performing disk I/O. + +Of these threads, the only ones that you have any configuration control over are the message processing threads. In this case, you can determine how many of these threads you want to run. + +It is always a bit of an art to decide on a thread count, but the short answer is you probably do not need more than three threads here, and it is likely that one will suffice. That said, the best thing to do is set your thread count to a fairly low number and then increase it if it appears that your application will benefit from the additional threads. diff --git a/docs-src/guides/gsg_db_rep/repadvantage.md b/docs-src/guides/gsg_db_rep/repadvantage.md new file mode 100644 index 000000000..9ad79c2f7 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/repadvantage.md @@ -0,0 +1,40 @@ +--- +title: "Replication Benefits" +api-name: "Replication Benefits" +source: docs/gsg_db_rep/C/repadvantage.html +--- +## Replication Benefits + +Replication offers your application a number of benefits that can be a tremendous help. Primarily replication's benefits revolve around performance, but there is also a benefit in terms of data durability guarantees. + +Briefly, the reasons why you might choose to implement replication in your DB application are: + +- Improve application reliability. + + By spreading your data across multiple machines, you can ensure that your application's data continues to be available even in the event of a hardware failure on any given machine in the replication group. + +- Improve read performance. + + By using replication you can spread data reads across multiple machines on your network. Doing so allows you to vastly improve your application's read performance. This strategy might be particularly interesting for applications that have readers on remote network nodes; you can push your data to the network's edges thereby improving application data read responsiveness. + + Additionally, depending on the portion of your data that you read on a given replica, that replica may need to cache part of your data, decreasing cache misses and reducing I/O on the replica. + +- Improve transactional commit performance + + In order to commit a transaction and achieve a transactional durability guarantee, the commit must be made *durable*. That is, the commit must be written to disk (usually, but not always, synchronously) before the application's thread of control can continue operations. + + Replication allows you to avoid this disk I/O and still maintain a degree of durability by *committing to the network*. In other words, you relax your transactional durability guarantees on the master, but by virtue of replicating the data across the network you gain some additional durability guarantees above what is provided locally. + + Usually this strategy is implemented using some form of an asynchronous transactional commit on the master. In this way your data writes will eventually be written to disk, but your application will not have to wait for the disk I/O to complete before continuing with its next operation. + + Note that it is possible to cause DB's replication implementation to wait to hear from one or more replicas as to whether they have successfully saved the write before continuing. However, in this case you might be trading performance for a even higher durability guarantee (see below). + +- Improve data durability guarantee. + + In a traditional transactional application, you commit your transactions such that data modifications are saved to disk. Beyond this, the durability of your data is dependent upon the backup strategy that you choose to implement for your site. + + Replication allows you to increase this durability guarantee by ensuring that data modifications are written to multiple machines. This means that multiple disks, disk controllers, power supplies, and CPUs are used to ensure that your data modification makes it to stable storage. In other words, replication allows you to minimize the problem of a single point of failure by using more hardware to guarantee your data writes. + + If you are using replication for this reason, then you probably will want to configure your application such that it waits to hear about a successful commit from one or more replicas before continuing with the next operation. This will obviously impact your application's write performance to some degree — with the performance penalty being largely dependent upon the speed and stability of the network connecting your replication group. + + For more information, see Permanent Message Handling. diff --git a/docs-src/guides/gsg_db_rep/repapp.md b/docs-src/guides/gsg_db_rep/repapp.md new file mode 100644 index 000000000..60f36805e --- /dev/null +++ b/docs-src/guides/gsg_db_rep/repapp.md @@ -0,0 +1,86 @@ +--- +title: "Chapter 3. The DB Replication Manager" +api-name: "Chapter 3. The DB Replication Manager" +source: docs/gsg_db_rep/C/repapp.html +--- +## Chapter 3. The DB Replication Manager + +**Table of Contents** + + [The DB_SITE Handle](repapp.md#repmgr_grpmgmt) + + [Starting and Stopping Replication](rep_init_code.md) + + [Managing Election Policies](rep_init_code.md#election_flags) + + [Selecting the Number of Threads](rep_init_code.md#thread_count) + + [Adding the Replication Manager to ex_rep_gsg_simple](repmgr_init_example_c.md) + + [Permanent Message Handling](fwrkpermmessage.md) + + [Identifying Permanent Message Policies](fwrkpermmessage.md#fmwrkpermpolicy) + + [Setting the Permanent Message Timeout](fwrkpermmessage.md#fmwrkpermtimeout) + + [Adding a Permanent Message Policy to ex_rep_gsg_repmgr](fwrkpermmessage.md#perm2fmwrkexample) + + [Managing Election Times](electiontimes.md) + + [Managing Election Timeouts](electiontimes.md#electiontimeout) + + [Managing Election Retry Times](electiontimes.md#electretrytime) + + [Managing Connection Retries](fmwrkconnectretry.md) + + [Managing Heartbeats](heartbeats.md) + +The easiest way to add replication to your transactional application is to use the Replication Manager. The Replication Manager provides a comprehensive communications layer that enables replication. For a brief listing of the Replication Manager's feature set, see Replication Manager Overview. + +To use the Replication Manager, you make use of a combination of the `DB_SITE` class and related methods, plus special methods off the `DB_ENV` class. That is: + +1. Create an environment handle as normal. + +2. Configure your environment handle as needed (e.g. set the error file and error prefix values, if desired). + +3. Use the Replication Manager replication classes and methods to configure the Replication Manager. Using these classes and methods causes DB to know that you are using the Replication Manager. + + Configuring the Replication Manager entails setting the replication environment's priority, setting the TCP/IP address that this replication environment will use for incoming replication messages, identifying TCP/IP addresses of other replication environments, setting the number of replication environments in the replication group, and so forth. These actions are discussed throughout the remainder of this chapter. + +4. Open your environment handle. When you do this, be sure to specify `DB_INIT_REP` and `DB_THREAD` to your open flags. (This is in addition to the flags that you normally use for a single-threaded transactional application). The first of these causes replication to be initialized for the application. The second causes your environment handle to be free-threaded (thread safe). Both flags are required for Replication Manager usage. + +5. Start replication by calling `DB_ENV->repmgr_start()`. + +6. Open your databases as needed. Masters must open their databases for read and write activity. Replicas can open their databases for read-only activity, but doing so means they must re-open the databases if the replica ever becomes a master. Either way, replicas should never attempt to write to the database(s) directly. + +### Note + +The Replication Manager allows you to only use one environment handle per process. + +When you are ready to shut down your application: + +1. Close any open `DB_SITE` handles that you might have open. + +2. Close your databases + +3. Close your environment. This causes replication to stop as well. + +### Note + +Before you can use the Replication Manager, you may have to enable it in your DB library. This is *not* a requirement for Microsoft Windows systems, or Unix systems that use pthread mutexes by default. Other systems, notably BSD and BSD-derived systems (such as Mac OS X), must enable the Replication Manager when you configure the DB build. + +You do this by *not* disabling replication and by configuring the library with POSIX threads support. In other words, replication must be turned on in the build (it is by default), and POSIX thread support must be enabled if it is not already by default. To do this, use the `--enable-pthread_api` switch on the configure script. + +For example: + +``` c +../dist/configure --enable-pthread-api +``` + +## The DB_SITE Handle + +Before continuing, it is useful to mention the `DB_SITE` handle. This class is used to configure important attributes about a site such as its host name and port number, and whether it is the local site. It is also used to indicate whether a site is a *group creator*, which is important when you are starting the very first site in a replication group for the very first time. + +The `DB_SITE` handle is used whenever you start up a site. It must be closed before you close your `DB_ENV` handle. + +The `DB_SITE` handle is plays an important role in replication group management. This topic is fully described in the *Berkeley DB Programmer's Reference Guide*. diff --git a/docs-src/guides/gsg_db_rep/repmgr_init_example_c.md b/docs-src/guides/gsg_db_rep/repmgr_init_example_c.md new file mode 100644 index 000000000..cc3b3f40a --- /dev/null +++ b/docs-src/guides/gsg_db_rep/repmgr_init_example_c.md @@ -0,0 +1,235 @@ +--- +title: "Adding the Replication Manager to ex_rep_gsg_simple" +api-name: "Adding the Replication Manager to ex_rep_gsg_simple" +source: docs/gsg_db_rep/C/repmgr_init_example_c.html +--- +## Adding the Replication Manager to ex_rep_gsg_simple + +We now use the methods described above to add partial support to the ex_rep_gsg_simple example that we presented in Transactional Application. That is, in this section we will: + +- Enhance our command line options to accept information of interest to a replicated application. + +- Configure our environment handle to use replication and the Replication Manager. + +- Minimally configure the Replication Manager. + +- Start replication. + +Note that when we are done with this section, we will be only partially ready to run the application. Some critical pieces will be missing; specifically, we will not yet be handling the differences between a master and a replica. (We do that in the next chapter). + +Also, note that in the following code fragments, additions and changes to the code are marked in **`bold`**. + +To begin, we copy the ex_rep_gsg_simple code to a new file called `ex_rep_gsg_repmgr.c`. We then make the corresponding change to the program name. + +``` c +/* + * File: ex_rep_gsg_repmgr.c + */ + +#include +#include +#ifndef _WIN32 +#include +#endif + +#include + +#ifdef _WIN32 +extern int getopt(int, char * const *, const char *); +#endif + +#define CACHESIZE (10 * 1024 * 1024) +#define DATABASE "quote.db" + +const char *progname = "ex_rep_gsg_repmgr"; + +int create_env(const char *, DB_ENV **); +int env_init(DB_ENV *, const char *); +int doloop (DB_ENV *); +int print_stocks(DBC *); +``` + +Next we update our usage function. The application will continue to accept the `-h` parameter so that we can identify the environment home directory used by this application. However, we also add the: + +- `-l` parameter which allows us to identify the host and port used by this application to listen for replication messages. This parameter is required unless the -L parameter is specified. + +- `-L` parameter, which allows us to identify the local site as the group creator. + +- `-r` parameter which allows us to specify other replicas. + +- `-p` option, which is used to identify this replica's priority (recall that the priority is used as a tie breaker for elections) + +``` c +/* Usage function */ +static void +usage() +{ + fprintf(stderr, "usage: %s ", progname); + fprintf(stderr, "-h home -l|-L host:port\n"); + fprintf(stderr, "\t\t[-r host:port][-p priority]\n"); + fprintf(stderr, "where:\n"); + fprintf(stderr, "\t-h identifies the environment home directory "); + fprintf(stderr, "(required).\n"); + fprintf(stderr, "\t-l identifies the host and port used by this "); + fprintf(stderr, "site (required, unless -L is specified).\n"); + fprintf(stderr, "\t-L identifies the host and port used by this "); + fprintf(stderr, "site, which is the group creator.\n"); + fprintf(stderr, "\t-r identifies another site participating in "); + fprintf(stderr, "this replication group\n"); + fprintf(stderr, "\t-p identifies the election priority used by "); + fprintf(stderr, "this replica.\n"); + exit(EXIT_FAILURE); +} +``` + +Now we can begin working on our `main()` function. We begin by adding a couple of variables that we will use to collect TCP/IP host and port information. We also declare a couple of flags that we use to make sure some required information is provided to this application. + +``` c +int +main(int argc, char *argv[]) +{ + DB_ENV *dbenv; + DB_SITE *dbsite; + extern char *optarg; + const char *home; + char ch, *host, *portstr; + int ret, local_is_set, is_group_creator; + u_int32_t port; + + dbenv = NULL; + + ret = local_is_set = is_group_creator = 0; + home = NULL; +``` + +At this time we can create our environment handle and configure it exactly as we did for `simple_txn`. The only thing that we will do differently here is that we will set a priority, arbitrarily picked to be 100, so that we can be sure the environment has a priority other than 0 (the default value). This ensures that the environment can become a master via an election. + +``` c + if ((ret = create_env(progname, &dbenv)) != 0) + goto err; + + /* Default priority is 100 */ + dbenv->rep_set_priority(dbenv, 100); +``` + +Now we collect our command line arguments. As we do so, we will configure host and port information as required, and we will configure the application's election priority if necessary. + +``` c + /* Collect the command line options */ + while ((ch = getopt(argc, argv, "h:l:L:p:r:")) != EOF) + switch (ch) { + case 'h': + home = optarg; + break; + /* Set the host and port used by this environment */ + case 'l': + host = strtok(optarg, ":"); + if ((portstr = strtok(NULL, ":")) == NULL) { + fprintf(stderr, "Bad host specification.\n"); + goto err; + } + port = (unsigned short)atoi(portstr); + if ((ret = dbenv->repmgr_site(dbenv, host, port, &dbsite + 0)) != 0 ) { + fprintf(stderr, + "Could not set local address %s.\n", host); + goto err; + } + dbsite->set_config(dbsite, DB_LOCAL_SITE, 1); + if (is_group_creator) + dbsite->set_config(dbsite, DB_GROUP_CREATOR, 1); + + if ((ret = dbsite->close(dbsite)) != 0) { + dbenv->(dbenv, ret, "DB_SITE->close"); + goto err; + } + local_is_set = 1; + break; + /* Set this replica's election priority */ + case 'p': + dbenv->rep_set_priority(dbenv, atoi(optarg)); + break; + /* Identify another site in the replication group */ + case 'r': + host = strtok(optarg, ":"); + if ((portstr = strtok(NULL, ":")) == NULL) { + fprintf(stderr, "Bad host specification.\n"); + goto err; + } + port = (unsigned short)atoi(portstr); + if ((dbenv->repmgr_site(dbenv, host, port, &dbsite, + 0)) != 0) { + fprintf(stderr, + "Could not add site %s.\n", host); + goto err; + } + dbenv->set_config(dbsite, DB_BOOTSTRAP_HELPER, 1); + if ((dbenv->close(dbsite)) != 0) { + dbenv->err(dbenv, ret, "DB_SITE->close"); + goto err; + } + break; + case '?': + default: + usage(); + } + + /* Error check command line. */ + if (home == NULL || !local_is_set) + usage(); +``` + +Having done that, we can call `env_init()`, which we use to open our environment handle. Note that this function changes slightly for this update (see below). + +``` c + if ((ret = env_init(dbenv, home)) != 0) + goto err; +``` + +Finally, we start replication before we go into the `doloop()` function (where we perform all our database access). + +``` c + if ((ret = dbenv->repmgr_start(dbenv, 3, DB_REP_ELECTION)) != 0) + goto err; + + if ((ret = doloop(dbenv)) != 0) { + dbenv->err(dbenv, ret, "Application failed"); + goto err; + } + +err: if (dbenv != NULL) + (void)dbenv->close(dbenv, 0); + + return (ret); +} +``` + +Beyond that, the rest of our application remains the same for now, with the exception of the `env_init()` function, which we use to actually open our environment handle. The flags we use to open the environment are slightly different for a replicated application than they are for a non-replicated application. Namely, replication requires the `DB_INIT_REP` flag. + +Also, because we are using the Replication Manager, we must prepare our environment for threaded usage. For this reason, we also need the `DB_THREAD` flag. + +``` c +int +env_init(DB_ENV *dbenv, const char *home) +{ + u_int32_t flags; + int ret; + + (void)dbenv->set_cachesize(dbenv, 0, CACHESIZE, 0); + (void)dbenv->set_flags(dbenv, DB_TXN_NOSYNC, 1); + + flags = DB_CREATE | + DB_INIT_LOCK | + DB_INIT_LOG | + DB_INIT_MPOOL | + DB_INIT_REP | + DB_INIT_TXN | + DB_RECOVER | + DB_THREAD; + if ((ret = dbenv->open(dbenv, home, flags, 0)) != 0) + dbenv->err(dbenv, ret, "can't open environment"); + return (ret); +} +``` + +This completes our replication updates for the moment. We are not as yet ready to actually run this program; there remains a few critical pieces left to add to it. However, the work that we performed in this section represents a solid foundation for the remainder of our replication work. diff --git a/docs-src/guides/gsg_db_rep/rywc.md b/docs-src/guides/gsg_db_rep/rywc.md new file mode 100644 index 000000000..282dd7465 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/rywc.md @@ -0,0 +1,20 @@ +--- +title: "Read-Your-Writes Consistency" +api-name: "Read-Your-Writes Consistency" +source: docs/gsg_db_rep/C/rywc.html +--- +## Read-Your-Writes Consistency + +In a distributed system, the changes made at the master are not always instantaneously available at every replica, although they eventually will be. In general, replicas not directly involved in contributing to the acknowledgement of a transaction commit will lag behind other replicas because they do not synchronize their commits with the master. + +For this reason, you might want to make use of the read-your-writes consistency feature. This feature allows you to ensure that a replica is at least current enough to have the changes made by a specific transaction. Because transactions are applied serially, by ensuring a replica has a specific commit applied to it, you know that all transaction commits occurring prior to the specified transaction have also been applied to the replica. + +You determine whether a transaction has been applied to a replica by generating a *commit token* at the master. You then transfer this commit token to the replica, where it is used to determine whether the replica is consistent enough relative to the master. + +For example, suppose the you have a web application where a replication group is implemented within a load balanced web server group. Each request to the web server consists of an update operation followed by read operations (say, from the same client), The read operations naturally expect to see the data from the updates executed by the same request. However, the read operations might have been routed to a replica that did not execute the update. + +In such a case, the update request would generate a commit token, which would be resubmitted by the browser, along with subsequent read requests. The read request could be directed at any one of the available web servers by a load balancer. The replica which services the read request would use that commit token to determine whether it can service the read operation. If the replica is current enough, it can immediately execute the transaction and satisfy the request. + +What action the replica takes if it is not consistent enough to service the read request is up to you as the application developer. You can do anything from blocking while you wait for the transaction to be applied locally, to rejecting the read request outright. + +For more information, see the `Read your writes consistency` section in the `Berkeley DB Replication` chapter of the *Berkeley DB Programmer's Reference Guide*. diff --git a/docs-src/guides/gsg_db_rep/simpleprogramlisting.md b/docs-src/guides/gsg_db_rep/simpleprogramlisting.md new file mode 100644 index 000000000..0444776f7 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/simpleprogramlisting.md @@ -0,0 +1,368 @@ +--- +title: "Program Listing" +api-name: "Program Listing" +source: docs/gsg_db_rep/C/simpleprogramlisting.html +--- +## Program Listing + + [Function: main()](simpleprogramlisting.md#main_c) + + [Function: create_env()](simpleprogramlisting.md#create_env_c) + + [Function: env_init()](simpleprogramlisting.md#env_init_c) + + [Function: doloop()](simpleprogramlisting.md#doloop_c) + + [Function: print_stocks()](simpleprogramlisting.md#printstocks_c) + +Our example program is a fairly simple transactional application. At this early stage of its development, the application contains no hint that it must be network-aware so the only command line argument that it takes is one that allows us to specify the environment home directory. (Eventually, we will specify things like host names and ports from the command line). + +Note that the application performs all writes under the protection of a transaction; however, multiple database operations are not performed per transaction. Consequently, we simplify things a bit by using autocommit for our database writes. + +Also, this application is single-threaded. It is possible to write a multi-threaded or multi-process application that performs replication. That said, the concepts described in this book are applicable to both single threaded and multi-threaded applications so nothing is gained by multi-threading this application other than distracting complexity. This manual does, however, identify where care must be taken when performing replication with a non-single threaded application. + +Finally, remember that transaction processing is not described in this manual. Rather, see the *Berkeley DB Getting Started with Transaction Processing* guide for details on that topic. + +### Function: main() + +Our program begins with the usual assortment of include statements. + +``` c +/* + * File: ex_rep_gsg_simple.c + */ + +#include +#include +#include +#ifndef _WIN32 +#include +#endif + +#include + +#ifdef _WIN32 +extern int getopt(int, char * const *, const char *); +#endif +``` + +We then define a few values. One is the size of our cache, which we keep deliberately small for this example, and the other is the name of our database. We also provide a global variable that is the name of our program; this is used for error reporting later on. + +``` c +#define CACHESIZE (10 * 1024 * 1024) +#define DATABASE "quote.db" + +const char *progname = "ex_rep_gsg_simple"; +``` + +Then we perform a couple of forward declarations. The first of these, `create_env()` and `env_init()` are used to open and initialize our environment. + +Next we declare `doloop()`, which is the function that we use to add data to the database and then display its contents. This is essentially a big `do` loop, hence the function's name. + +Finally, we have `print_stocks`, which is used to display a database record once it has been retrieved from the database. + +``` c +int create_env(const char *, DB_ENV **); +int env_init(DB_ENV *, const char *); +int doloop (DB_ENV *); +int print_stocks(DB *); +``` + +Next we need our `usage()` function, which is fairly trivial at this point: + +``` c +/* Usage function */ +static void +usage() +{ + fprintf(stderr, "usage: %s ", progname); + fprintf(stderr, "-h home\n"); + exit(EXIT_FAILURE); +} +``` + +That completed, we can jump into our application's `main()` function. If you are familiar with DB transactional applications, you will not find any surprises here. We begin by declaring and initializing the usual set of variables: + +``` c +int +main(int argc, char *argv[]) +{ + extern char *optarg; + DB_ENV *dbenv; + const char *home; + char ch; + int ret; + + dbenv = NULL; + + ret = 0; + home = NULL; +``` + +Now we create and configure our environment handle. We do this with our `create_env()` function, which we will show a little later in this example. + +``` c + if ((ret = create_env(progname, &dbenv)) != 0) + goto err; +``` + +Then we parse the command line arguments: + +``` c + while ((ch = getopt(argc, argv, "h:")) != EOF) + switch (ch) { + case 'h': + home = optarg; + break; + case '?': + default: + usage(); + } + + /* Error check command line. */ + if (home == NULL) + usage(); +``` + +Now we can open our environment. We do this with our `env_init()` function which we will describe a little later in this chapter. + +``` c + if ((ret = env_init(dbenv, home)) != 0) + goto err; +``` + +Now that we have opened the environment, we can call our `doloop()` function. This function performs the basic database interaction. Notice that we have not yet opened any databases. In a traditional transactional application we would probably open the databases before calling our our main data processing function. However, the eventual replicated application will want to handle database open and close in the main processing loop, so in a nod to what this application will eventually become we do a slightly unusual thing here. + +``` c + if ((ret = doloop(dbenv)) != 0) { + dbenv->err(dbenv, ret, "Application failed"); + goto err; + } +``` + +Finally, we provide our application shutdown code. Note, again, that in a traditional transactional application all databases would also be closed here. But, again, due to the way this application will eventually behave, we cause the database close to occur in the `doloop()` function. + +``` c +err: if (dbenv != NULL) + (void)dbenv->close(dbenv, 0); + + return (ret); +} +``` + +### Function: create_env() + +Having written our `main()` function, we now implement the first of our utility functions that we use to manage our environments. This function exists only to make our code easier to manage, and all it does is create an environment handle for us. + +``` c +int +create_env(const char *progname, DB_ENV **dbenvp) +{ + DB_ENV *dbenv; + int ret; + + if ((ret = db_env_create(&dbenv, 0)) != 0) { + fprintf(stderr, "can't create env handle: %s\n", + db_strerror(ret)); + return (ret); + } + + dbenv->set_errfile(dbenv, stderr); + dbenv->set_errpfx(dbenv, progname); + + *dbenvp = dbenv; + return (0); +} +``` + +### Function: env_init() + +Having written the function that initializes an environment handle, we now implement the function that opens the handle. Again, there should be no surprises here for anyone familiar with DB applications. The open flags that we use are those normally used for a transactional application. + +``` c +int +env_init(DB_ENV *dbenv, const char *home) +{ + u_int32_t flags; + int ret; + + (void)dbenv->set_cachesize(dbenv, 0, CACHESIZE, 0); + (void)dbenv->set_flags(dbenv, DB_TXN_NOSYNC, 1); + + flags = DB_CREATE | + DB_INIT_LOCK | + DB_INIT_LOG | + DB_INIT_MPOOL | + DB_INIT_TXN | + DB_RECOVER; + if ((ret = dbenv->open(dbenv, home, flags, 0)) != 0) + dbenv->err(dbenv, ret, "can't open environment"); + return (ret); +} +``` + +### Function: doloop() + +Having written our `main()` function and utility functions, we now implement our application's primary data processing function. This function provides a command prompt at which the user can enter a stock ticker value and a price for that value. This information is then entered to the database. + +To display the database, simply enter `return` at the prompt. + +To begin, we declare a database pointer, several `DBT` variables, and the usual assortment of variables used for buffers and return codes. We also initialize all of this. + +``` c +#define BUFSIZE 1024 +int +doloop(DB_ENV *dbenv) +{ + DB *dbp; + DBT key, data; + char buf[BUFSIZE], *rbuf; + int ret; + u_int32_t db_flags; + + dbp = NULL; + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + ret = 0; +``` + +Next, we begin the loop and we immediately open our database if it has not already been opened. Notice that we specify autocommit when we open the database. In this case, autocommit is important because we will only ever write to our database using it. There is no need for explicit transaction handles and commit/abort code in this application, because we are not combining multiple database operations together under a single transaction. + +Autocommit is described in greater detail in the *Berkeley DB Getting Started with Transaction Processing* guide. + +``` c + for (;;) { + + if (dbp == NULL) { + if ((ret = db_create(&dbp, dbenv, 0)) != 0) + return (ret); + + db_flags = DB_AUTO_COMMIT | DB_CREATE; + + if ((ret = dbp->open(dbp, NULL, DATABASE, + NULL, DB_BTREE, db_flags, 0)) != 0) { + dbenv->err(dbenv, ret, "DB->open"); + goto err; + } + } +``` + +Now we implement our command prompt. This is a simple and not very robust implementation of a command prompt. If the user enters the keywords `exit` or `quit`, the loop is exited and the application ends. If the user enters nothing and instead simply presses `return`, the entire contents of the database is displayed. We use our `print_stocks()` function to display the database. (That implementation is shown next in this chapter.) + +Notice that very little error checking is performed on the data entered at this prompt. If the user fails to enter at least one space in the value string, a simple help message is printed and the prompt is returned to the user. That is the only error checking performed here. In a real-world application, at a minimum the application would probably check to ensure that the price was in fact an integer or float value. However, in order to keep this example code as simple as possible, we refrain from implementing a thorough user interface. + +``` c + printf("QUOTESERVER > "); + fflush(stdout); + + if (fgets(buf, sizeof(buf), stdin) == NULL) + break; + if (strtok(&buf[0], " \t\n") == NULL) { + switch ((ret = print_stocks(dbp))) { + case 0: + continue; + default: + dbp->err(dbp, ret, "Error traversing data"); + goto err; + } + } + rbuf = strtok(NULL, " \t\n"); + if (rbuf == NULL || rbuf[0] == '\0') { + if (strncmp(buf, "exit", 4) == 0 || + strncmp(buf, "quit", 4) == 0) + break; + dbenv->errx(dbenv, "Format: TICKER VALUE"); + continue; + } +``` + +Now we assign data to the `DBT`s that we will use to write the new information to the database. + +``` c + key.data = buf; + key.size = (u_int32_t)strlen(buf); + + data.data = rbuf; + data.size = (u_int32_t)strlen(rbuf); +``` + +Having done that, we can write the new information to the database. Remember that this application uses autocommit, so no explicit transaction management is required. Also, the database is not configured for duplicate records, so the data portion of a record is overwritten if the provided key already exists in the database. However, in this case DB returns `DB_KEYEXIST` — which we ignore. + +``` c + if ((ret = dbp->put(dbp, NULL, &key, &data, 0)) != 0) + { + dbp->err(dbp, ret, "DB->put"); + if (ret != DB_KEYEXIST) + goto err; + } + } +``` + +Finally, we close our database before returning from the function. + +``` c +err: if (dbp != NULL) + (void)dbp->close(dbp, DB_NOSYNC); + + return (ret); +} +``` + +### Function: print_stocks() + +The `print_stocks()` function simply takes a database handle, opens a cursor, and uses it to display all the information it finds in a database. This is trivial cursor operation that should hold no surprises for you. We simply provide it here for the sake of completeness. + +If you are unfamiliar with basic cursor operations, please see the *Getting Started with Berkeley DB* guide. + +``` c +/* Displays all stock quote information in the database. */ +int +print_stocks(DB *dbp) +{ + DBC *dbc; + DBT key, data; +#define MAXKEYSIZE 10 +#define MAXDATASIZE 20 + char keybuf[MAXKEYSIZE + 1], databuf[MAXDATASIZE + 1]; + int ret, t_ret; + u_int32_t keysize, datasize; + + if ((ret = dbp->cursor(dbp, NULL, &dbc, 0)) != 0) { + dbp->err(dbp, ret, "can't open cursor"); + return (ret); + } + + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + + printf("\tSymbol\tPrice\n"); + printf("\t======\t=====\n"); + + for (ret = dbc->get(dbc, &key, &data, DB_FIRST); + ret == 0; + ret = dbc->get(dbc, &key, &data, DB_NEXT)) { + keysize = key.size > MAXKEYSIZE ? MAXKEYSIZE : key.size; + memcpy(keybuf, key.data, keysize); + keybuf[keysize] = '\0'; + + datasize = data.size >= MAXDATASIZE ? MAXDATASIZE : data.size; + memcpy(databuf, data.data, datasize); + databuf[datasize] = '\0'; + + printf("\t%s\t%s\n", keybuf, databuf); + } + printf("\n"); + fflush(stdout); + + if ((t_ret = dbc->close(dbc)) != 0 && ret == 0) + ret = t_ret; + + switch (ret) { + case 0: + case DB_NOTFOUND: + return (0); + default: + return (ret); + } +} +``` diff --git a/docs-src/guides/gsg_db_rep/txnapp.md b/docs-src/guides/gsg_db_rep/txnapp.md new file mode 100644 index 000000000..ddfec24a0 --- /dev/null +++ b/docs-src/guides/gsg_db_rep/txnapp.md @@ -0,0 +1,59 @@ +--- +title: "Chapter 2. Transactional Application" +api-name: "Chapter 2. Transactional Application" +source: docs/gsg_db_rep/C/txnapp.html +--- +## Chapter 2. Transactional Application + +**Table of Contents** + + [Application Overview](txnapp.md#appoverview) + + [Program Listing](simpleprogramlisting.md) + + [Function: main()](simpleprogramlisting.md#main_c) + + [Function: create_env()](simpleprogramlisting.md#create_env_c) + + [Function: env_init()](simpleprogramlisting.md#env_init_c) + + [Function: doloop()](simpleprogramlisting.md#doloop_c) + + [Function: print_stocks()](simpleprogramlisting.md#printstocks_c) + +In this chapter, we build a simple transaction-protected DB application. Throughout the remainder of this book, we will add replication to this example. We do this to underscore the concepts that we are presenting in this book; the first being that you should start with a working transactional program and then add replication to it. + +Note that this book assumes you already know how to write a transaction-protected DB application, so we will not be covering those concepts in this book. To learn how to write a transaction-protected application, see the *Berkeley DB Getting Started with Transaction Processing* guide. + +## Application Overview + +Our application maintains a stock market quotes database. This database contains records whose key is the stock market symbol and whose data is the stock's price. + +The application operates by presenting you with a command line prompt. You then enter the stock symbol and its value, separated by a space. The application takes this information and writes it to the database. + +To see the contents of the database, simply press `return` at the command prompt. + +To quit the application, type 'quit' or 'exit' at the command prompt. + +For example, the following illustrates the application's usage. In it, we use entirely fictitious stock market symbols and price values. + +``` c +> ./ex_rep_gsg_simple -h env_home_dir +QUOTESERVER> stock1 88 +QUOTESERVER> stock2 .08 +QUOTESERVER> + Symbol Price + ====== ===== + stock1 88 + stock2 .08 + +QUOTESERVER> stock1 88.9 +QUOTESERVER> + Symbol Price + ====== ===== + stock1 88.9 + stock2 .08 + +QUOTESERVER> quit +> +``` diff --git a/docs-src/guides/gsg_txn/_meta.toml b/docs-src/guides/gsg_txn/_meta.toml new file mode 100644 index 000000000..85875ff60 --- /dev/null +++ b/docs-src/guides/gsg_txn/_meta.toml @@ -0,0 +1,45 @@ +# Nav/index metadata for the gsg_txn guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Getting Started with Berkeley DB Transaction Processing" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "introduction", + "sysfailure", + "apireq", + "multithread-intro", + "recovery-intro", + "perftune-intro", + "enabletxn", + "envopen", + "usingtxns", + "nodurabletxn", + "abortresults", + "autocommit", + "nestedtxn", + "txncursor", + "txnindices", + "maxtxns", + "txnconcurrency", + "blocking_deadlocks", + "lockingsubsystem", + "isolation", + "txn_ccursor", + "exclusivelock", + "readmodifywrite", + "txnnowait", + "reversesplit", + "filemanagement", + "backuprestore", + "recovery", + "architectrecovery", + "hotfailover", + "logfileremoval", + "logconfig", + "wrapup", + "txnexample_c", + "inmem_txnexample_c", +] diff --git a/docs-src/guides/gsg_txn/abortresults.md b/docs-src/guides/gsg_txn/abortresults.md new file mode 100644 index 000000000..3c83552df --- /dev/null +++ b/docs-src/guides/gsg_txn/abortresults.md @@ -0,0 +1,12 @@ +--- +title: "Aborting a Transaction" +api-name: "Aborting a Transaction" +source: docs/gsg_txn/C/abortresults.html +--- +## Aborting a Transaction + +When you abort a transaction, all database modifications performed under the protection of the transaction are discarded, and all locks currently held by the transaction are released. In this event, your data is simply left in the state that it was in before the transaction began performing data modifications. + +Once you have aborted a transaction, the transaction handle that you used for the transaction is no longer valid. To perform database activities under the control of a new transaction, you must obtain a fresh transactional handle. + +To abort a transaction, call `DB_TXN->abort()`. diff --git a/docs-src/guides/gsg_txn/apireq.md b/docs-src/guides/gsg_txn/apireq.md new file mode 100644 index 000000000..8975479dd --- /dev/null +++ b/docs-src/guides/gsg_txn/apireq.md @@ -0,0 +1,38 @@ +--- +title: "Application Requirements" +api-name: "Application Requirements" +source: docs/gsg_txn/C/apireq.html +--- +## Application Requirements + +In order to use transactions, your application has certain requirements beyond what is required of non-transactional protected applications. They are: + +- Environments. + + Environments are optional for non-transactional applications, but they are required for transactional applications. + + Environment usage is described in detail in Transaction Basics. + +- Transaction subsystem. + + In order to use transactions, you must explicitly enable the transactional subsystem for your application, and this must be done at the time that your environment is first created. + +- Logging subsystem. + + The logging subsystem is required for recovery purposes, but its usage also means your application may require a little more administrative effort than it does when logging is not in use. See Managing DB Files for more information. + +- DB_TXN handles. + + In order to obtain the atomicity guarantee offered by the transactional subsystem (that is, combine multiple operations in a single unit of work), your application must use transaction handles. These handles are obtained from your DB_ENV objects. They should normally be short-lived, and their usage is reasonably simple. To complete a transaction and save the work it performed, you call its `commit()` method. To complete a transaction and discard its work, you call its `abort()` method. + + In addition, it is possible to use auto commit if you want to transactional protect a single write operation. Auto commit allows a transaction to be used without obtaining an explicit transaction handle. See Auto Commit for information on how to use auto commit. + +- Database open requirements. + + In addition to using environments and initializing the correct subsystems, your application must transaction protect the database opens, and any secondary index associations, if subsequent operations on the databases are to be transaction protected. The database open and secondary index association are commonly transaction protected using auto commit. + +- Deadlock detection. + + Typically transactional applications use multiple threads of control when accessing the database. Any time multiple threads are used on a single resource, the potential for lock contention arises. In turn, lock contention can lead to deadlocks. See Locks, Blocks, and Deadlocks for more information. + + Therefore, transactional applications must frequently include code for detecting and responding to deadlocks. Note that this requirement is not *specific* to transactions – you can certainly write concurrent non-transactional DB applications. Further, not every transactional application uses concurrency and so not every transactional application must manage deadlocks. Still, deadlock management is so frequently a characteristic of transactional applications that we discuss it in this book. See Concurrency for more information. diff --git a/docs-src/guides/gsg_txn/architectrecovery.md b/docs-src/guides/gsg_txn/architectrecovery.md new file mode 100644 index 000000000..dac1be1f2 --- /dev/null +++ b/docs-src/guides/gsg_txn/architectrecovery.md @@ -0,0 +1,108 @@ +--- +title: "Designing Your Application for Recovery" +api-name: "Designing Your Application for Recovery" +source: docs/gsg_txn/C/architectrecovery.html +--- +## Designing Your Application for Recovery + + [Recovery for Multi-Threaded Applications](architectrecovery.md#multithreadrecovery) + + [Recovery in Multi-Process Applications](architectrecovery.md#multiprocessrecovery) + +When building your DB application, you should consider how you will run recovery. If you are building a single threaded, single process application, it is fairly simple to run recovery when your application first opens its environment. In this case, you need only decide if you want to run recovery every time you open your application (recommended) or only some of the time, presumably triggered by a start up option controlled by your application's user. + +However, for multi-threaded and multi-process applications, you need to carefully consider how you will design your application's startup code so as to run recovery only when it makes sense to do so. + +### Recovery for Multi-Threaded Applications + +If your application uses only one environment handle, then handling recovery for a multi-threaded application is no more difficult than for a single threaded application. You simply open the environment in the application's main thread, and then pass that handle to each of the threads that will be performing DB operations. We illustrate this with our final example in this book (see Transaction Example for more information). + +Alternatively, you can have each worker thread open its own environment handle. However, in this case, designing for recovery is a bit more complicated. + +Generally, when a thread performing database operations fails or hangs, it is frequently best to simply restart the application and run recovery upon application startup as normal. However, not all applications can afford to restart because a single thread has misbehaved. + +If you are attempting to continue operations in the face of a misbehaving thread, then at a minimum recovery must be run if a thread performing database operations fails or hangs. + +Remember that recovery clears the environment of all outstanding locks, including any that might be outstanding from an aborted thread. If these locks are not cleared, other threads performing database operations can back up behind the locks obtained but never cleared by the failed thread. The result will be an application that hangs indefinitely. + +To run recovery under these circumstances: + +1. Suspend or shutdown all other threads performing database operations. + +2. Discarding any open environment handles. Note that attempting to gracefully close these handles may be asking for trouble; the close can fail if the environment is already in need of recovery. For this reason, it is best and easiest to simply discard the handle. + +3. Open new handles, running recovery as you open them. See Normal Recovery for more information. + +4. Restart all your database threads. + +A traditional way to handle this activity is to spawn a watcher thread that is responsible for making sure all is well with your threads, and performing the above actions if not. + +However, in the case where each worker thread opens and maintains its own environment handle, recovery is complicated for two reasons: + +1. For some applications and workloads, it might be worthwhile to give your database threads the ability to gracefully finalize any on-going transactions. If this is the case, your code must be capable of signaling each thread to halt DB activities and close its environment. If you simply run recovery against the environment, your database threads will detect this and fail in the midst of performing their database operations. + +2. Your code must be capable of ensuring only one thread runs recovery before allowing all other threads to open their respective environment handles. Recovery should be single threaded because when recovery is run against an environment, it is deleted and then recreated. This will cause all other processes and threads to "fail" when they attempt operations against the newly recovered environment. If all threads run recovery when they start up, then it is likely that some threads will fail because the environment that they are using has been recovered. This will cause the thread to have to re-execute its own recovery path. At best, this is inefficient and at worst it could cause your application to fall into an endless recovery pattern. + +### Recovery in Multi-Process Applications + +Frequently, DB applications use multiple processes to interact with the databases. For example, you may have a long-running process, such as some kind of server, and then a series of administrative tools that you use to inspect and administer the underlying databases. Or, in some web-based architectures, different services are run as independent processes that are managed by the server. + +In any case, recovery for a multi-process environment is complicated for two reasons: + +1. In the event that recovery must be run, you might want to notify processes interacting with the environment that recovery is about to occur and give them a chance to gracefully terminate. Whether it is worthwhile for you to do this is entirely dependent upon the nature of your application. Some long-running applications with multiple processes performing meaningful work might want to do this. Other applications with processes performing database operations that are likely to be harmed by error conditions in other processes will likely find it to be not worth the effort. For this latter group, the chances of performing a graceful shutdown may be low anyway. + +2. Unlike single process scenarios, it can quickly become wasteful for every process interacting with the databases to run recovery when it starts up. This is partly because recovery *does* take some amount of time to run, but mostly you want to avoid a situation where your server must reopen all its environment handles just because you fire up a command line database administrative utility that always runs recovery. + +DB offers you two methods by which you can manage recovery for multi-process DB applications. Each has different strengths and weaknesses, and they are described in the next sections. + +#### Effects of Multi-Process Recovery + +Before continuing, it is worth noting that the following sections describe recovery processes than can result in one process running recovery while other processes are currently actively performing database operations. + +When this happens, the current database operation will abnormally fail, indicating a DB_RUNRECOVERY condition. This means that your application should immediately abandon any database operations that it may have on-going, discard any environment handles it has opened, and obtain and open new handles. + +The net effect of this is that any writes performed by unresolved transactions will be lost. For persistent applications (servers, for example), the services it provides will also be unavailable for the amount of time that it takes to complete a recovery and for all participating processes to reopen their environment handles. + +#### Process Registration + +One way to handle multi-process recovery is for every process to "register" its environment. In doing so, the process gains the ability to see if any other applications are using the environment and, if so, whether they have suffered an abnormal termination. If an abnormal termination is detected, the process runs recovery; otherwise, it does not. + +Note that using process registration also ensures that recovery is serialized across applications. That is, only one process at a time has a chance to run recovery. Generally this means that the first process to start up will run recovery, and all other processes will silently not run recovery because it is not needed. + +To cause your application to register its environment, you specify the `DB_REGISTER` flag when you open your environment. You may also specify `DB_RECOVER`. However, it is an error to specify `DB_RECOVER_FATAL` when using the `DB_REGISTER` flag. If during the open, DB determines that recovery must be run, it will automatically run the correct type of recovery for you, so long as you specify normal recovery on your environment open. If you do not specify normal recovery, and you register your environment, then no recovery is run if the registration process identifies a need for it. In this case, the environment open simply fails by returning `DB_RUNRECOVERY`. + +### Note + +If you do not specify normal recovery when you open your first registered environment in the application, then that application will fail the environment open by returning `DB_RUNRECOVERY`. This is because the first process to register must create an internal registration file, and recovery is forced when that file is created. To avoid an abnormal termination of the environment open, specify recovery on the environment open for at least the first process starting in your application. + +In addition, if you specify `DB_ENV_FAILCHK` when you register your environment, then a fail check is performed on environment open (fail checks are described in the next section). If, during the fail check process, an abnormal termination is detected for any of the processes involved in the application, DB releases any read locks held by the dead process and performs transaction aborts as necessary. This is done in an attempt to clean up the environment. + +In this situation, if a general cleanup of the environment is not possible and normal recovery is not specified on environment open, then the open will abort, returning `DB_RUNRECOVERY`. However, if this situation occurs and recovery was specified, then the appropriate type of recovery (normal or fatal) is run so as to bring the environment back to a healthy state. + +Be aware that there are some limitations/requirements if you want your various processes to coordinate recovery using registration: + +1. There can be only one environment handle per environment per process. In the case of multi-threaded processes, the environment handle must be shared across threads. + +2. All processes sharing the environment must use registration. If registration is not uniformly used across all participating processes, then you can see inconsistent results in terms of your application's ability to recognize that recovery must be run. + +#### Failure Checking + +For very large and robust multi-process applications, the most common way to ensure all the processes are working as intended is to make use of a watchdog process. To assist a watchdog process, DB offers a failure checking mechanism. + +When a thread of control fails with open environment handles, the result is that there may be resources left locked or corrupted. Other threads of control may encountered these unavailable resources quickly or not at all, depending on data access patterns. + +In any case, the DB failure checking mechanism allows a watchdog to detect if an environment is unusable as a result of a thread of control failure. It should be called periodically (for example, once a minute) from the watchdog process. If the environment is deemed unusable, then the watchdog process is notified that recovery should be run. It is then up to the watchdog to actually run recovery. It is also the watchdog's responsibility to decide what to do about currently running processes before running recovery. The watchdog could, for example, attempt to gracefully shutdown or kill all relevant processes before running recovery. + +Note that failure checking need not be run from a separate process, although conceptually that is how the mechanism is meant to be used. This same mechanism could be used in a multi-threaded application that wants to have a watchdog thread. + +To use failure checking you must: + +1. Provide an `is_alive()` call back using the `DB_ENV->set_isalive()` method. DB uses this method to determine whether a specified process and thread is alive when the failure checking is performed. + +2. Possibly provide a `thread_id` callback that uniquely identifies a process and thread of control. This callback is only necessary if the standard process and thread identification functions for your platform are not sufficient to for use by failure checking. This is rarely necessary and is usually because the thread and/or process ids used by your system cannot fit into an unsigned integer. + + You provide this callback using the `DB_ENV->set_thread_id()` method. See the API reference for this method for more information on when setting a thread id callback might be necessary. + +3. Call the `DB_ENV->failchk()` method periodically. You can do this either periodically (once per minute, for example), or whenever a thread of control exits for your application. + + If this method determines that a thread of control exited holding read locks, those locks are automatically released. If the thread of control exited with an unresolved transaction, that transaction is aborted. If any other problems exist beyond these such that the environment must be recovered, the method will return `DB_RUNRECOVERY`. diff --git a/docs-src/guides/gsg_txn/autocommit.md b/docs-src/guides/gsg_txn/autocommit.md new file mode 100644 index 000000000..d9acff5ac --- /dev/null +++ b/docs-src/guides/gsg_txn/autocommit.md @@ -0,0 +1,136 @@ +--- +title: "Auto Commit" +api-name: "Auto Commit" +source: docs/gsg_txn/C/autocommit.html +--- +## Auto Commit + +While transactions are frequently used to provide atomicity to multiple database operations, it is sometimes necessary to perform a single database operation under the control of a transaction. Rather than force you to obtain a transaction, perform the single write operation, and then either commit or abort the transaction, you can automatically group this sequence of events using *auto commit*. + +To use auto commit: + +1. Open your environment and your databases so that they support transactions. See Enabling Transactions for details. + + Note that frequently auto commit is used for the environment or database open. To use auto commit for either your environment or database open, specify `DB_AUTO_COMMIT` to the `DB_ENV->set_flags()` or `DB->open()` method. If you specify auto commit for the environment open, then you do not need to also specify auto commit for the database open. + +2. Do not provide a transactional handle to the method that is performing the database write operation. + +Note that auto commit is not available for cursors. You must always open your cursor using a transaction if you want the cursor's operations to be transactional protected. See Transactional Cursors for details on using transactional cursors. + +### Note + +Never have more than one active transaction in your thread at a time. This is especially a problem if you mix an explicit transaction with another operation that uses auto commit. Doing so can result in undetectable deadlocks. + +For example, the following uses auto commit to perform the database write operation: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DBT key, data; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + const char *keystr ="thekey"; + const char *datastr = "thedata"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* Create the environment if it does + * not already exist. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_INIT_LOCK | /* Initialize locking. */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL; /* Initialize the in-memory cache. */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + db_flags = DB_CREATE | DB_AUTO_COMMIT; + /* + * Open the database. Note that we are using auto commit for the open, + * so the database is able to support transactions. + */ + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + + /* Prepare the DBTs */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + key.data = &keystr; + key.size = strlen(keystr) + 1; + data.data = &datastr; + data.size = strlen(datastr) + 1; + + /* + * Perform the database write. A txn handle is not provided, but the + * database support auto commit, so auto commit is used for the write. + */ + ret = dbp->put(dbp, NULL, &key, &data, 0); + if (ret != 0) { + envp->err(envp, ret, "Database put failed."); + goto err; + } + +err: + /* Close the database */ + if (dbp != NULL) { + ret_c = dbp->close(dbp, 0); + if (ret_c != 0) { + envp->err(envp, ret_c, "Database close failed."); + ret = ret_c + } + } + + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` diff --git a/docs-src/guides/gsg_txn/backuprestore.md b/docs-src/guides/gsg_txn/backuprestore.md new file mode 100644 index 000000000..2254781cd --- /dev/null +++ b/docs-src/guides/gsg_txn/backuprestore.md @@ -0,0 +1,98 @@ +--- +title: "Backup Procedures" +api-name: "Backup Procedures" +source: docs/gsg_txn/C/backuprestore.html +--- +## Backup Procedures + + [About Unix Copy Utilities](backuprestore.md#copyutilities) + + [Offline Backups](backuprestore.md#standardbackup) + + [Hot Backup](backuprestore.md#hotbackup) + + [Incremental Backups](backuprestore.md#incrementalbackups) + +*Durability* is an important part of your transactional guarantees. It means that once a transaction has been successfully committed, your application will always see the results of that transaction. + +Of course, no software algorithm can guarantee durability in the face of physical data loss. Hard drives can fail, and if you have not copied your data to locations other than your primary disk drives, then you will lose data when those drives fail. Therefore, in order to truly obtain a durability guarantee, you need to ensure that any data stored on disk is backed up to secondary or alternative storage, such as secondary disk drives, or offline tapes. + +There are three different types of backups that you can perform with DB databases and log files. They are: + +- Offline backups + + This type of backup is perhaps the easiest to perform as it involves simply copying database and log files to an offline storage area. It also gives you a snapshot of the database at a fixed, known point in time. However, you cannot perform this type of a backup while you are performing writes to the database. + +- Hot backups + + This type of backup gives you a snapshot of your database. Since your application can be writing to the database at the time that the snapshot is being taken, you do not necessarily know what the exact state of the database is for that given snapshot. + +- Incremental backups + + This type of backup refreshes a previously performed backup. + +Once you have performed a backup, you can perform *catastrophic recovery* to restore your databases from the backup. See Catastrophic Recovery for more information. + +Note that you can also maintain a hot failover. See Using Hot Failovers for more information. + +### About Unix Copy Utilities + +If you are copying database files you must copy databases atomically, in multiples of the database page size. In other words, the reads made by the copy program must not be interleaved with writes by other threads of control, and the copy program must read the databases in multiples of the underlying database page size. Generally, this is not a problem because operating systems already make this guarantee and system utilities normally read in power-of-2 sized chunks, which are larger than the largest possible Berkeley DB database page size. + +On some platforms (most notably, some releases of Solaris), the copy utility (`cp`) was implemented using the `mmap()` system call rather than the `read()` system call. Because `mmap()` did not make the same guarantee of read atomicity as did `read()`, the `cp` utility could create corrupted copies of the databases. + +Also, some platforms have implementations of the `tar` utility that performs 10KB block reads by default. Even when an output block size is specified, the utility will still not read the underlying databases in multiples of the specified block size. Again, the result can be a corrupted backup. + +To fix these problems, use the `dd` utility instead of `cp` or `tar`. When you use `dd`, make sure you specify a block size that is equal to, or an even multiple of, your database page size. Finally, if you plan to use a system utility to copy database files, you may want to use a system call trace utility (for example, `ktrace` or `truss`) to make sure you are not using a I/O size that is smaller than your database page size. You can also use these utilities to make sure the system utility is not using a system call other than `read()`. + +### Offline Backups + +To create an offline backup: + +1. Commit or abort all on-going transactions. + +2. Pause all database writes. + +3. Force a checkpoint. See Checkpoints for details. + +4. Copy all your database files to the backup location. Note that you can simply copy all of the database files, or you can determine which database files have been written during the lifetime of the current logs. To do this, use either the `DB_ENV->log_archive()` method with the `DB_ARCH_DATA` option, or use the **db_archive** command with the `-s` option. + + However, be aware that backing up just the modified databases only works if you have all of your log files. If you have been removing log files for any reason then using `log_archive()` can result in an unrecoverable backup because you might not be notified of a database file that was modified. + +5. Copy the *last* log file to your backup location. Your log files are named `log.`*`xxxxxxxxxx`*, where *xxxxxxxxxx* is a sequential number. The last log file is the file with the highest number. + +### Hot Backup + +To create a hot backup, you do not have to stop database operations. Transactions may be on-going and you can be writing to your database at the time of the backup. However, this means that you do not know exactly what the state of your database is at the time of the backup. + +You can use the **db_hotbackup** command line utility to create a hot backup. This program optionally runs a checkpoint, and then copies all necessary files to a target directory. + +You can also create your own hot backup facility using the `DB_ENV->backup()` method. + +Alternatively, you can manually create a hot backup as follows: + +1. Set the `DB_HOTBACKUP_IN_PROGRESS` flag in your environment. For more information, see the DB_ENV->set_flags() API reference page. + +2. Copy all your database files to the backup location. Note that you can simply copy all of the database files, or you can determine which database files have been written during the lifetime of the current logs. To do this, use either the `DB_ENV->log_archive()` with the `DB_ARCH_DATA` option, or use the **db_archive** command with the `-s` option. + +3. Copy all logs to your backup location. + +4. Reset the `DB_HOTBACKUP_IN_PROGRESS` flag. + +### Note + +It is important to copy your database files *and then* your logs. In this way, you can complete or roll back any database operations that were only partially completed when you copied the databases. + +### Incremental Backups + +Once you have created a full backup (that is, either a offline or hot backup), you can create incremental backups. To do this, simply copy all of your currently existing log files to your backup location. + +Incremental backups do not require you to run a checkpoint or to cease database write operations. + +If your application uses the transactional bulk insert optimization, it is important to know that a database copy taken prior to a bulk loading event can no longer be used as the target of an incremental backup. This is true because bulk loading omits logging of some record insertions, so recovery cannot roll forward these insertions. It is recommended that a full backup be scheduled following a bulk loading event. + +For more information, see the description of the `DB_TXN_BULK` flag in the DB_ENV->txn_begin() API reference page. + +When you are working with incremental backups, remember that the greater the number of log files contained in your backup, the longer recovery will take. You should run full backups on some interval, and then do incremental backups on a shorter interval. How frequently you need to run a full backup is determined by the rate at which your databases change and how sensitive your application is to lengthy recoveries (should one be required). + +You can also shorten recovery time by running recovery against the backup as you take each incremental backup. Running recovery as you go means that there will be less work for DB to do if you should ever need to restore your environment from the backup. diff --git a/docs-src/guides/gsg_txn/blocking_deadlocks.md b/docs-src/guides/gsg_txn/blocking_deadlocks.md new file mode 100644 index 000000000..802704682 --- /dev/null +++ b/docs-src/guides/gsg_txn/blocking_deadlocks.md @@ -0,0 +1,160 @@ +--- +title: "Locks, Blocks, and Deadlocks" +api-name: "Locks, Blocks, and Deadlocks" +source: docs/gsg_txn/C/blocking_deadlocks.html +--- +## Locks, Blocks, and Deadlocks + + [Locks](blocking_deadlocks.md#locks) + + [Blocks](blocking_deadlocks.md#blocks) + + [Deadlocks](blocking_deadlocks.md#deadlocks) + +It is important to understand how locking works in a concurrent application before continuing with a description of the concurrency mechanisms DB makes available to you. Blocking and deadlocking have important performance implications for your application. Consequently, this section provides a fundamental description of these concepts, and how they affect DB operations. + +### Locks + +When one thread of control wants to obtain access to an object, it requests a *lock* for that object. This lock is what allows DB to provide your application with its transactional isolation guarantees by ensuring that: + +- no other thread of control can read that object (in the case of an exclusive lock), and + +- no other thread of control can modify that object (in the case of an exclusive or non-exclusive lock). + +#### Lock Resources + +When locking occurs, there are conceptually three resources in use: + +1. The locker. + + This is the thing that holds the lock. In a transactional application, the locker is a transaction handle. For non-transactional operations, the locker is a cursor or a DB handle. + +2. The lock. + + This is the actual data structure that locks the object. In DB, a locked object structure in the lock manager is representative of the object that is locked. + +3. The locked object. + + The thing that your application actually wants to lock. In a DB application, the locked object is usually a database page, which in turn contains multiple database entries (key and data). However, for Queue databases, individual database records are locked. + +You can configure how many total lockers, locks, and locked objects your application is allowed to support. See Configuring the Locking Subsystem for details. + +The following figure shows a transaction handle, `Txn A`, that is holding a lock on database page `002`. In this graphic, `Txn A` is the locker, and the locked object is page `002`. Only a single lock is in use in this operation. + +![](simplelock.jpg) + +#### Types of Locks + +DB applications support both exclusive and non-exclusive locks. *Exclusive locks* are granted when a locker wants to write to an object. For this reason, exclusive locks are also sometimes called *write locks*. + +An exclusive lock prevents any other locker from obtaining any sort of a lock on the object. This provides isolation by ensuring that no other locker can observe or modify an exclusively locked object until the locker is done writing to that object. + +*Non-exclusive locks* are granted for read-only access. For this reason, non-exclusive locks are also sometimes called *read locks*. Since multiple lockers can simultaneously hold read locks on the same object, read locks are also sometimes called *shared locks*. + +A non-exclusive lock prevents any other locker from modifying the locked object while the locker is still reading the object. This is how transactional cursors are able to achieve repeatable reads; by default, the cursor's transaction holds a read lock on any object that the cursor has examined until such a time as the transaction is committed or aborted. You can avoid these read locks by using snapshot isolation. See Using Snapshot Isolation for details. + +In the following figure, `Txn A` and `Txn B` are both holding read locks on page `002`, while `Txn C` is holding a write lock on page `003`: + +![](rwlocks1.jpg) + +#### Lock Lifetime + +A locker holds its locks until such a time as it does not need the lock any more. What this means is: + +1. A transaction holds any locks that it obtains until the transaction is committed or aborted. + +2. All non-transaction operations hold locks until such a time as the operation is completed. For cursor operations, the lock is held until the cursor is moved to a new position or closed. + +### Blocks + +Simply put, a thread of control is blocked when it attempts to obtain a lock, but that attempt is denied because some other thread of control holds a conflicting lock. Once blocked, the thread of control is temporarily unable to make any forward progress until the requested lock is obtained or the operation requesting the lock is abandoned. + +Be aware that when we talk about blocking, strictly speaking the thread is not what is attempting to obtain the lock. Rather, some object within the thread (such as a cursor) is attempting to obtain the lock. However, once a locker attempts to obtain a lock, the entire thread of control must pause until the lock request is in some way resolved. + +For example, if `Txn A` holds a write lock (an exclusive lock) on object 002, then if `Txn B` tries to obtain a read *or* write lock on that object, the thread of control in which `Txn B` is running is blocked: + +![](writeblock.jpg) + +However, if `Txn A` only holds a read lock (a shared lock) on object `002`, then only those handles that attempt to obtain a write lock on that object will block. + +![](readblock.jpg) + +### Note + +The previous description describes DB's default behavior when it cannot obtain a lock. It is possible to configure DB transactions so that they will not block. Instead, if a lock is unavailable, the application is immediately notified of a deadlock situation. See No Wait on Blocks for more information. + +#### Blocking and Application Performance + +Multi-threaded and multi-process applications typically perform better than simple single-threaded applications because the application can perform one part of its workload (updating a database record, for example) while it is waiting for some other lengthy operation to complete (performing disk or network I/O, for example). This performance improvement is particularly noticeable if you use hardware that offers multiple CPUs, because the threads and processes can run simultaneously. + +That said, concurrent applications can see reduced workload throughput if their threads of control are seeing a large amount of lock contention. That is, if threads are blocking on lock requests, then that represents a performance penalty for your application. + +Consider once again the previous diagram of a blocked write lock request. In that diagram, `Txn C` cannot obtain its requested write lock because `Txn A` and `Txn B` are both already holding read locks on the requested object. In this case, the thread in which `Txn C` is running will pause until such a time as `Txn C` either obtains its write lock, or the operation that is requesting the lock is abandoned. The fact that `Txn C`'s thread has temporarily halted all forward progress represents a performance penalty for your application. + +Moreover, any read locks that are requested while `Txn C` is waiting for its write lock will also block until such a time as `Txn C` has obtained and subsequently released its write lock. + +#### Avoiding Blocks + +Reducing lock contention is an important part of performance tuning your concurrent DB application. Applications that have multiple threads of control obtaining exclusive (write) locks are prone to contention issues. Moreover, as you increase the numbers of lockers and as you increase the time that a lock is held, you increase the chances of your application seeing lock contention. + +As you are designing your application, try to do the following in order to reduce lock contention: + +- Reduce the length of time your application holds locks. + + Shorter lived transactions will result in shorter lock lifetimes, which will in turn help to reduce lock contention. + + In addition, by default transactional cursors hold read locks until such a time as the transaction is completed. For this reason, try to minimize the time you keep transactional cursors opened, or reduce your isolation levels – see below. + +- If possible, access heavily accessed (read or write) items toward the end of the transaction. This reduces the amount of time that a heavily used page is locked by the transaction. + +- Reduce your application's isolation guarantees. + + By reducing your isolation guarantees, you reduce the situations in which a lock can block another lock. Try using uncommitted reads for your read operations in order to prevent a read lock being blocked by a write lock. + + In addition, for cursors you can use degree 2 (read committed) isolation, which causes the cursor to release its read locks as soon as it is done reading the record (as opposed to holding its read locks until the transaction ends). + + Be aware that reducing your isolation guarantees can have adverse consequences for your application. Before deciding to reduce your isolation, take care to examine your application's isolation requirements. For information on isolation levels, see Isolation. + +- Use snapshot isolation for read-only threads. + + Snapshot isolation causes the transaction to make a copy of the page on which it is holding a lock. When a reader makes a copy of a page, write locks can still be obtained for the original page. This eliminates entirely read-write contention. + + Snapshot isolation is described in Using Snapshot Isolation. + +- Consider your data access patterns. + + Depending on the nature of your application, this may be something that you can not do anything about. However, if it is possible to create your threads such that they operate only on non-overlapping portions of your database, then you can reduce lock contention because your threads will rarely (if ever) block on one another's locks. + +### Note + +It is possible to configure DB's transactions so that they never wait on blocked lock requests. Instead, if they are blocked on a lock request, they will notify the application of a deadlock (see the next section). + +You configure this behavior on a transaction by transaction basis. See No Wait on Blocks for more information. + +### Deadlocks + +A deadlock occurs when two or more threads of control are blocked, each waiting on a resource held by the other thread. When this happens, there is no possibility of the threads ever making forward progress unless some outside agent takes action to break the deadlock. + +For example, if `Txn A` is blocked by `Txn B` at the same time `Txn B` is blocked by `Txn A` then the threads of control containing `Txn A` and `Txn B` are deadlocked; neither thread can make any forward progress because neither thread will ever release the lock that is blocking the other thread. + +![](deadlock.jpg) + +When two threads of control deadlock, the only solution is to have a mechanism external to the two threads capable of recognizing the deadlock and notifying at least one thread that it is in a deadlock situation. Once notified, a thread of control must abandon the attempted operation in order to resolve the deadlock. DB's locking subsystem offers a deadlock notification mechanism. See Configuring Deadlock Detection for more information. + +Note that when one locker in a thread of control is blocked waiting on a lock held by another locker in that same thread of the control, the thread is said to be *self-deadlocked*. + +#### Deadlock Avoidance + +The things that you do to avoid lock contention also help to reduce deadlocks (see Avoiding Blocks). Beyond that, you can also do the following in order to avoid deadlocks: + +- Never have more than one active transaction at a time in a thread. A common cause of this is for a thread to be using auto-commit for one operation while an explicit transaction is in use in that thread at the same time. + +- Make sure all threads access data in the same order as all other threads. So long as threads lock database pages in the same basic order, there is no possibility of a deadlock (threads can still block, however). + + Be aware that if you are using secondary databases (indexes), it is not possible to obtain locks in a consistent order because you cannot predict the order in which locks are obtained in secondary databases. If you are writing a concurrent application and you are using secondary databases, you must be prepared to handle deadlocks. + +- If you are using BTrees in which you are constantly adding and then deleting data, turn Btree reverse split off. See Reverse BTree Splits for more information. + +- Declare a read/modify/write lock for those situations where you are reading a record in preparation of modifying and then writing the record. Doing this causes DB to give your read operation a write lock. This means that no other thread of control can share a read lock (which might cause contention), but it also means that the writer thread will not have to wait to obtain a write lock when it is ready to write the modified data back to the database. + + For information on declaring read/modify/write locks, see Read/Modify/Write. diff --git a/docs-src/guides/gsg_txn/enabletxn.md b/docs-src/guides/gsg_txn/enabletxn.md new file mode 100644 index 000000000..eb8b815cb --- /dev/null +++ b/docs-src/guides/gsg_txn/enabletxn.md @@ -0,0 +1,227 @@ +--- +title: "Chapter 2. Enabling Transactions" +api-name: "Chapter 2. Enabling Transactions" +source: docs/gsg_txn/C/enabletxn.html +--- +## Chapter 2. Enabling Transactions + +**Table of Contents** + + [Environments](enabletxn.md#environments) + + [File Naming](enabletxn.md#filenaming) + + [Error Support](enabletxn.md#errorsupport) + + [Shared Memory Regions](enabletxn.md#sharedmemory) + + [Security Considerations](enabletxn.md#security) + + [Opening a Transactional Environment and Database](envopen.md) + +In order to use transactions with your application, you must turn them on. To do this you must: + +- Use an environment (see Environments for details). + +- Turn on transactions for your environment. You do this by providing the `DB_INIT_TXN` flag to the `DB_ENV->open()` method. Note that initializing the transactional subsystem implies that the logging subsystem is also initialized. Also, note that if you do not initialize transactions when you first create your environment, then you cannot use transactions for that environment after that. This is because DB allocates certain structures needed for transactional locking that are not available if the environment is created without transactional support. + +- Initialize the in-memory cache by passing the `DB_INIT_MPOOL` flag to the `DB_ENV->open()` method. + +- Initialize the locking subsystem. This is what provides locking for concurrent applications. It also is used to perform deadlock detection. See Concurrency for more information. + + You initialize the locking subsystem by passing the `DB_INIT_LOCK` flag to the `DB_ENV->open()` method. + +- Initialize the logging subsystem. While this is enabled by default for transactional applications, we suggest that you explicitly initialize it anyway for the purposes of code readability. The logging subsystem is what provides your transactional application its durability guarantee, and it is required for recoverability purposes. See Managing DB Files for more information. + + You initialize the logging subsystem by passing the `DB_INIT_LOG` flag to the `DB_ENV->open()` method. + +- Transaction-enable your databases. If you are using the base API, transaction-enable your databases. You do this by encapsulating the database open in a transaction. Note that the common practice is for auto commit to be used to transaction-protect the database open. To use auto-commit, you must still enable transactions as described here, but you do not have to explicitly use a transaction when you open your database. An example of this is given in the next section. + +## Environments + + [File Naming](enabletxn.md#filenaming) + + [Error Support](enabletxn.md#errorsupport) + + [Shared Memory Regions](enabletxn.md#sharedmemory) + + [Security Considerations](enabletxn.md#security) + +For simple DB applications, environments are optional. However, in order to transaction protect your database operations, you must use an environment. + +An *environment*, represents an encapsulation of one or more databases and any associated log and region files. They are used to support multi-threaded and multi-process applications by allowing different threads of control to share the in-memory cache, the locking tables, the logging subsystem, and the file namespace. By sharing these things, your concurrent application is more efficient than if each thread of control had to manage these resources on its own. + +By default all DB databases are backed by files on disk. In addition to these files, transactional DB applications create logs that are also by default stored on disk (they can optionally be backed using shared memory). Finally, transactional DB applications also create and use shared-memory regions that are also typically backed by the filesystem. But like databases and logs, the regions can be maintained strictly in-memory if your application requires it. For an example of an application that manages all environment files in-memory, see In-Memory Transaction Example. + +### Warning + +Using environments with some journaling filesystems might result in log file corruption. This can occur if the operating system experiences an unclean shutdown when a log file is being created. Please see Using Recovery on Journaling Filesystems in the *Berkeley DB Programmer's Reference Guide* for more information. + +### File Naming + +In order to operate, your DB application must be able to locate its database files, log files, and region files. If these are stored in the filesystem, then you must tell DB where they are located (a number of mechanisms exist that allow you to identify the location of these files – see below). Otherwise, by default they are located in the current working directory. + +#### Specifying the Environment Home Directory + +The environment home directory is used to determine where DB files are located. Its location is identified using one of the following mechanisms, in the following order of priority: + +- If no information is given as to where to put the environment home, then the current working directory is used. + +- If a home directory is specified on the `DB_ENV->open()` method, then that location is always used for the environment home. + +- If a home directory is not supplied to `DB_ENV->open()`, then the directory identified by the `DB_HOME` environment variable is used *if* you specify either the `DB_USE_ENVIRON` or `DB_USE_ENVIRON_ROOT` flags to the `DB_ENV->open()` method. Both flags allow you to identify the path to the environment's home directory using the `DB_HOME` environment variable. However, `DB_USE_ENVIRON_ROOT` is honored only if the process is run with root or administrative privileges. + +#### Specifying File Locations + +By default, all DB files are created relative to the environment home directory. For example, suppose your environment home is in `/export/myAppHome`. Also suppose you name your database `data/myDatabase.db`. Then in this case, the database is placed in: `/export/myAppHome/data/myDatabase.db`. + +That said, DB always defers to absolute pathnames. This means that if you provide an absolute filename when you name your database, then that file is *not* placed relative to the environment home directory. Instead, it is placed in the exact location that you specified for the filename. + +On UNIX systems, an absolute pathname is a name that begins with a forward slash ('/'). On Windows systems, an absolute pathname is a name that begins with one of the following: + +- A backslash ('\\). + +- Any alphabetic letter, followed by a colon (':'), followed by a backslash ('\\). + +### Note + +Try not to use absolute path names for your environment's files. Under certain recovery scenarios, absolute path names can render your environment unrecoverable. This occurs if you are attempting to recover your environment on a system that does not support the absolute path name that you used. + +#### Identifying Specific File Locations + +As described in the previous sections, DB will place all its files in or relative to the environment home directory. You can also cause a specific database file to be placed in a particular location by using an absolute path name for its name. In this situation, the environment's home directory is not considered when naming the file. + +It is frequently desirable to place database, log, and region files on separate disk drives. By spreading I/O across multiple drives, you can increase parallelism and improve throughput. Additionally, by placing log files and database files on separate drives, you improve your application's reliability by providing your application with a greater chance of surviving a disk failure. + +You can cause DB's files to be placed in specific locations using the following mechanisms: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
File TypeTo Override
database files

You can cause database files to be created in a directory other than the environment home by using the DB_ENV->add_data_dir() method. The directory identified here must exist. If a relative path is provided, then the directory location is resolved relative to the environment's home directory.

+

This method modifies the directory used for database files created and managed by a single environment handle; it does not configure the entire environment. This method may not be called after the environment has been opened.

+

You can also set a default data location that is used by the entire environment by using the add_data_dir parameter in the environment's DB_CONFIG file. Note that the add_data_dir parameter overrides any value set by the DB_ENV->set_data_dir() method.

Log files

You can cause log files to be created in a directory other than the environment home directory by using the DB_ENV->set_lg_dir() method. The directory identified here must exist. If a relative path is provided, then the directory location is resolved relative to the environment's home directory.

+

This method modifies the directory used for database files created and managed by a single environment handle; it does not configure the entire environment. This method may not be called after the environment has been opened.

+

You can also set a default log file location that is used by the entire environment by using the set_lg_dir parameter in the environment's DB_CONFIG file. Note that the set_lg_dir parameter overrides any value set by the DB_ENV->set_lg_dir() method.

Temporary files

You can cause temporary files required by the environment to be created in a directory other than the environment home directory by using the DB_ENV->set_tmp_dir() method. The directory identified here must exist. If a relative path is provided, then the directory location is resolved relative to the environment's home directory.

+

You can also set a temporary file location by using the set_tmp_dir parameter in the environment's DB_CONFIG file. Note that the set_tmp_dir parameter overrides any value set by the DB_ENV->set_tmp_dir() method.

Metadata files

You can cause persistent metadata files required by the replicated applications to be created in a directory other than the environment home directory by using the DB_ENV->set_metadata_dir() method. The directory identified here must exist. If a relative path is provided, then the directory location is resolved relative to the environment's home directory.

+

You can also set a metadata directory location by using the set_metadata_dir parameter in the environment's DB_CONFIG file. Note that the set_metadata_dir parameter overrides any value set by the DB_ENV->set_metadata_dir() method.

Region filesIf backed by the filesystem, region files are always placed in the environment home directory.
+ +Note that the `DB_CONFIG` must reside in the environment home directory. Parameters are specified in it one parameter to a line. Each parameter is followed by a space, which is followed by the parameter value. For example: + +``` c + add_data_dir /export1/db/env_data_files +``` + +### Error Support + +To simplify error handling and to aid in application debugging, environments offer several useful methods. Note that many of these methods are identical to the error handling methods available for the DB structure. They are: + +- `set_errcall()` + + Defines the function that is called when an error message is issued by DB. The error prefix and message are passed to this callback. It is up to the application to display this information correctly. + + This is the recommended way to get error messages from DB. + +- `set_errfile()` + + Sets the C library `FILE *` to be used for displaying error messages issued by the DB library. + +- `set_errpfx()` + + Sets the prefix used to for any error messages issued by the DB library. + +- `err()` + + Issues an error message based upon a DB error code a message text that you supply. The error message is sent to the callback function as defined by `set_errcall()`. If that method has not been used, then the error message is sent to the file defined by `set_errfile()`. If none of these methods have been used, then the error message is sent to standard error. + + The error message consists of the prefix string (as defined by `set_errprefix()`), an optional `printf`-style formatted message, the DB error message associated with the supplied error code, and a trailing newline. + +- `errx()` + + Behaves identically to `err()` except that you do not provide the DB error code and so the DB message text is not displayed. + +In addition, you can use the `db_strerror()` function to directly return the error string that corresponds to a particular error number. For more information on the `db_strerror()` function, see the `Error Returns` section of the *Getting Started with Berkeley DB* guide. + +### Shared Memory Regions + +The subsystems that you enable for an environment (in our case, transaction, logging, locking, and the memory pool) are described by one or more regions. The regions contain all of the state information that needs to be shared among threads and/or processes using the environment. + +Regions may be backed by the file system, by heap memory, or by system shared memory. + +#### Regions Backed by Files + +By default, shared memory regions are created as files in the environment's home directory (*not* the environment's data directory). If it is available, the POSIX `mmap` interface is used to map these files into your application's address space. If `mmap` is not available, then the UNIX `shmget` interfaces are used instead (again, if they are available). + +In this default case, the region files are named `__db.###` (for example, `__db.001`, `__db.002`, and so on). + +#### Regions Backed by Heap Memory + +If heap memory is used to back your shared memory regions, then you can only open a single handle for the environment. This means that the environment cannot be accessed by multiple processes. In this case, the regions are managed only in memory, and they are not written to the filesystem. You indicate that heap memory is to be used for the region files by specifying `DB_PRIVATE` to the `DB_ENV->open()` method. + +Note that you can also set this flag by using the `set_open_flags` parameter in the `DB_CONFIG` file. See the *Berkeley DB C API Reference Guide* for more information. + +(For an example of an entirely in-memory transactional application, see In-Memory Transaction Example.) + +#### Regions Backed by System Memory + +Finally, you can cause system memory to be used for your regions instead of memory-mapped files. You do this by providing `DB_SYSTEM_MEM` to the `DB_ENV->open()` method. + +When region files are backed by system memory, DB creates a single file in the environment's home directory. This file contains information necessary to identify the system shared memory in use by the environment. By creating this file, DB enables multiple processes to share the environment. + +The system memory that is used is architecture-dependent. For example, on systems supporting X/Open-style shared memory interfaces, such as UNIX systems, the `shmget(2)` and related System V IPC interfaces are used. Additionally, VxWorks systems use system memory. In these cases, an initial segment ID must be specified by the application to ensure that applications do not overwrite each other's environments, so that the number of segments created does not grow without bounds. See the `DB_ENV->set_shm_key()` method for more information. + +On Windows platforms, the use of system memory for the region files is problematic because the operating system uses reference counting to clean up shared objects in the paging file automatically. In addition, the default access permissions for shared objects are different from files, which may cause problems when an environment is accessed by multiple processes running as different users. See Windows notes or more information. + +### Security Considerations + +When using environments, there are some security considerations to keep in mind: + +- Database environment permissions + + The directory used for the environment should have its permissions set to ensure that files in the environment are not accessible to users without appropriate permissions. Applications that add to the user's permissions (for example, UNIX `setuid` or `setgid` applications), must be carefully checked to not permit illegal use of those permissions such as general file access in the environment directory. + +- Environment variables + + Setting the `DB_USE_ENVIRON` or `DB_USE_ENVIRON_ROOT` flags so that environment variables can be used during file naming can be dangerous. Setting those flags in DB applications with additional permissions (for example, UNIX `setuid` or `setgid` applications) could potentially allow users to read and write databases to which they would not normally have access. + + For example, suppose you write a DB application that runs `setuid`. This means that when the application runs, it does so under a userid different than that of the application's caller. This is especially problematic if the application is granting stronger privileges to a user than the user might ordinarily have. + + Now, if the `DB_USE_ENVIRON` or `DB_USE_ENVIRON_ROOT` flags are set for the environment, then the environment that the application is using is modifiable using the `DB_HOME` environment variable. In this scenario, if the uid used by the application has sufficiently broad privileges, then the application's caller can read and/or write databases owned by another user simply by setting his `DB_HOME` environment variable to the environment used by that other user. + + Note that this scenario need not be malicious; the wrong environment could be used by the application simply by inadvertently specifying the wrong path to `DB_HOME`. + + As always, you should use `setuid` sparingly, if at all. But if you do use `setuid`, then you should refrain from specifying the `DB_USE_ENVIRON` or `DB_USE_ENVIRON_ROOT` flags for the environment open. And, of course, if you must use `setuid`, then make sure you use the weakest uid possible – preferably one that is used only by the application itself. + +- File permissions + + By default, DB always creates database and log files readable and writable by the owner and the group (that is, `S_IRUSR`, `S_IWUSR`, `S_IRGRP` and `S_IWGRP`; or octal mode 0660 on historic UNIX systems). The group ownership of created files is based on the system and directory defaults, and is not further specified by DB. + +- Temporary backing files + + If an unnamed database is created and the cache is too small to hold the database in memory, Berkeley DB will create a temporary physical file to enable it to page the database to disk as needed. In this case, environment variables such as `TMPDIR` may be used to specify the location of that temporary file. Although temporary backing files are created readable and writable by the owner only (`S_IRUSR` and `S_IWUSR`, or octal mode 0600 on historic UNIX systems), some filesystems may not sufficiently protect temporary files created in random directories from improper access. To be absolutely safe, applications storing sensitive data in unnamed databases should use the `DB_ENV->set_tmp_dir()` method to specify a temporary directory with known permissions. diff --git a/docs-src/guides/gsg_txn/envopen.md b/docs-src/guides/gsg_txn/envopen.md new file mode 100644 index 000000000..c316b259a --- /dev/null +++ b/docs-src/guides/gsg_txn/envopen.md @@ -0,0 +1,155 @@ +--- +title: "Opening a Transactional Environment and Database" +api-name: "Opening a Transactional Environment and Database" +source: docs/gsg_txn/C/envopen.html +--- +## Opening a Transactional Environment and Database + +To enable transactions for your environment, you must initialize the transactional subsystem. Note that doing this also initializes the logging subsystem. In addition, you must initialize the memory pool (in-memory cache). You must also initialize the locking subsystem. For example: + +Notice in the following example that you create your environment handle using the `db_env_create()` function before you open the environment: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t env_flags; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* Create the environment if it does + * not already exist. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_INIT_LOCK | /* Initialize locking. */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL; /* Initialize the in-memory cache. */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + +err: + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +You then create and open your database(s) as you would for a non-transactional system. The only difference is that you must pass the environment handle to the `db_create()` function, and you must open the database within a transaction. Typically auto commit is used for this purpose. To do so, pass `DB_AUTO_COMMIT` to the database open command. It is recommended that you close all your databases before you close your environment. For example: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* Create the environment if it does + * not already exist. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_INIT_LOCK | /* Initialize locking. */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL; /* Initialize the in-memory cache. */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + db_flags = DB_CREATE | DB_AUTO_COMMIT; + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + +err: + /* Close the database */ + if (dbp != NULL) { + ret_c = dbp->close(dbp, 0); + if (ret_c != 0) { + envp->err(envp, ret_c, "Database close failed."); + ret = ret_c; + } + } + + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +### Note + +Never close a database that has active transactions. Make sure all transactions are resolved (either committed or aborted) before closing the database. diff --git a/docs-src/guides/gsg_txn/exclusivelock.md b/docs-src/guides/gsg_txn/exclusivelock.md new file mode 100644 index 000000000..d3759d28d --- /dev/null +++ b/docs-src/guides/gsg_txn/exclusivelock.md @@ -0,0 +1,18 @@ +--- +title: "Exclusive Database Handles" +api-name: "Exclusive Database Handles" +source: docs/gsg_txn/C/exclusivelock.html +--- +## Exclusive Database Handles + +In some cases, concurrent applications can benefit from occasionally granting exclusive access to the entire database to a single database handle. This is desirable when a thread will perform an operation that touches all or most of the pages in a database. + +To configure a handle to have exclusive access to a database, you give it a single write lock to the entire database. This causes all other threads to block when they attempt to gain a read or write lock to any part of that database. + +The exclusive lock allows for improved throughput because the handle will not attempt to acquire any further locks once it has the exclusive write lock. It will also never be blocked waiting for a lock, and there is no possibility of a deadlock/retry cycle. + +Note that an exclusive database handle can only have one transaction active for it at a time. + +To configure a database handle with an exclusive lock, you use the `DB->set_lk_exclusive()` method before you open the database handle. Setting a value of `0` to this method means that the handle open operation will block until it can obtain the exclusive lock. A non-zero value means that if the method cannot obtain the exclusive lock immediately when the handle is opened, the open operation will exit with a `DB_LOCK_NOTGRANTED` error return. + +Once configured and opened, a handled configured with an exclusive database lock will hold that lock until the handle is closed. diff --git a/docs-src/guides/gsg_txn/filemanagement.md b/docs-src/guides/gsg_txn/filemanagement.md new file mode 100644 index 000000000..5ce21e6a5 --- /dev/null +++ b/docs-src/guides/gsg_txn/filemanagement.md @@ -0,0 +1,166 @@ +--- +title: "Chapter 5. Managing DB Files" +api-name: "Chapter 5. Managing DB Files" +source: docs/gsg_txn/C/filemanagement.html +--- +## Chapter 5. Managing DB Files + +**Table of Contents** + + [Checkpoints](filemanagement.md#checkpoints) + + [Backup Procedures](backuprestore.md) + + [About Unix Copy Utilities](backuprestore.md#copyutilities) + + [Offline Backups](backuprestore.md#standardbackup) + + [Hot Backup](backuprestore.md#hotbackup) + + [Incremental Backups](backuprestore.md#incrementalbackups) + + [Recovery Procedures](recovery.md) + + [Normal Recovery](recovery.md#normalrecovery) + + [Catastrophic Recovery](recovery.md#catastrophicrecovery) + + [Designing Your Application for Recovery](architectrecovery.md) + + [Recovery for Multi-Threaded Applications](architectrecovery.md#multithreadrecovery) + + [Recovery in Multi-Process Applications](architectrecovery.md#multiprocessrecovery) + + [Using Hot Failovers](hotfailover.md) + + [Removing Log Files](logfileremoval.md) + + [Configuring the Logging Subsystem](logconfig.md) + + [Setting the Log File Size](logconfig.md#logfilesize) + + [Configuring the Logging Region Size](logconfig.md#logregionsize) + + [Configuring In-Memory Logging](logconfig.md#inmemorylogging) + + [Setting the In-Memory Log Buffer Size](logconfig.md#logbuffer) + +DB is capable of storing several types of files on disk: + +- Data files, which contain the actual data in your database. + +- Log files, which contain information required to recover your database in the event of a system or application failure. + +- Region files, which contain information necessary for the overall operation of your application. + +- Temporary files, which are created only under certain special circumstances. These files never need to be backed up or otherwise managed and so they are not a consideration for the topics described in this chapter. See Security Considerations for more information on temporary files. + +Of these, you must manage your data and log files by ensuring that they are backed up. You should also pay attention to the amount of disk space your log files are consuming, and periodically remove any unneeded files. Finally, you can optionally tune your logging subsystem to best suit your application's needs and requirements. These topics are discussed in this chapter. + +## Checkpoints + +Before we can discuss DB file management, we need to describe checkpoints. When databases are modified (that is, a transaction is committed), the modifications are recorded in DB's logs, but they are *not* necessarily reflected in the actual database files on disk. + +This means that as time goes on, increasingly more data is contained in your log files that is not contained in your data files. As a result, you must keep more log files around than you might actually need. Also, any recovery run from your log files will take increasingly longer amounts of time, because there is more data in the log files that must be reflected back into the data files during the recovery process. + +You can reduce these problems by periodically running a checkpoint against your environment. The checkpoint: + +- Flushes dirty pages from the in-memory cache. This means that data modifications found in your in-memory cache are written to the database files on disk. Note that a checkpoint also causes data dirtied by an uncommitted transaction to also be written to your database files on disk. In this latter case, DB's normal recovery is used to remove any such modifications that were subsequently abandoned by your application using a transaction abort. + + Normal recovery is describe in Recovery Procedures. + +- Writes a checkpoint record. + +- Flushes the log. This causes all log data that has not yet been written to disk to be written. + +- Writes a list of open databases. + +There are several ways to run a checkpoint. One way is to use the **db_checkpoint** command line utility. (Note, however, that this command line utility cannot be used if your environment was opened using `DB_PRIVATE`.) + +You can also run a thread that periodically checkpoints your environment for you by calling the `DB_ENV->txn_checkpoint()` method. + +Note that you can prevent a checkpoint from occurring unless more than a specified amount of log data has been written since the last checkpoint. You can also prevent the checkpoint from running unless more than a specified amount of time has occurred since the last checkpoint. These conditions are particularly interesting if you have multiple threads or processes running checkpoints. + +For configuration information, see the DB_ENV->txn_checkpoint() API reference page. + +Note that running checkpoints can be quite expensive. DB must flush every dirty page to the backing database files. On the other hand, if you do not run checkpoints often enough, your recovery time can be unnecessarily long and you may be using more disk space than you really need. Also, you cannot remove log files until a checkpoint is run. Therefore, deciding how frequently to run a checkpoint is one of the most common tuning activity for DB applications. + +For example, to run a checkpoint from a separate thread of control: + +``` c +#include +#include +#include +#include "db.h" + +void *checkpoint_thread(void *); + +int +main(void) +{ + int ret; + u_int32_t env_flags; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + pthread_t ptid; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + /* Open the environment. */ + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Start a checkpoint thread. */ + if ((ret = pthread_create( + &ptid, NULL, checkpoint_thread, (void *)envp)) != 0) { + fprintf(stderr, + "txnapp: failed spawning checkpoint thread: %s\n", + strerror(ret)); + goto err; + } + + /* + * All other threads and application shutdown code + * omitted for brevity. + */ + + ... +} + +void * +checkpoint_thread(void *arg) { + DB_ENV *dbenv; + int ret; + + dbenv = arg; + + /* Checkpoint once a minute. */ + for (;; sleep(60)) + if ((ret = dbenv->txn_checkpoint(dbenv, 0, 0, 0)) != 0) { + dbenv->err(dbenv, ret, "checkpoint thread"); + exit (1); + } + + /* NOTREACHED */ +} +``` diff --git a/docs-src/guides/gsg_txn/hotfailover.md b/docs-src/guides/gsg_txn/hotfailover.md new file mode 100644 index 000000000..acc550b57 --- /dev/null +++ b/docs-src/guides/gsg_txn/hotfailover.md @@ -0,0 +1,50 @@ +--- +title: "Using Hot Failovers" +api-name: "Using Hot Failovers" +source: docs/gsg_txn/C/hotfailover.html +--- +## Using Hot Failovers + +You can maintain a backup that can be used for failover purposes. Hot failovers differ from the backup and restore procedures described previously in this chapter in that data used for traditional backups is typically copied to offline storage. Recovery time for a traditional backup is determined by: + +- How quickly you can retrieve that storage media. Typically storage media for critical backups is moved to a safe facility in a remote location, so this step can take a relatively long time. + +- How fast you can read the backup from the storage media to a local disk drive. If you have very large backups, or if your storage media is very slow, this can be a lengthy process. + +- How long it takes you to run catastrophic recovery against the newly restored backup. As described earlier in this chapter, this process can be lengthy because every log file must be examined during the recovery process. + +When you use a hot failover, the backup is maintained at a location that is reasonably fast to access. Usually, this is a second disk drive local to the machine. In this situation, recovery time is very quick because you only have to reopen your environment and database, using the failover environment for the environment open. + +Hot failovers obviously do not protect you from truly catastrophic disasters (such as a fire in your machine room) because the backup is still local to the machine. However, you can guard against more mundane problems (such as a broken disk drive) by keeping the backup on a second drive that is managed by an alternate disk controller. + +To maintain a hot failover: + +1. Copy all the active database files to the failover directory. Use the **db_archive** command line utility with the `-s` option to identify all the active database files. + +2. Identify all the inactive log files in your production environment and *move* these to the failover directory. Use the **db_archive** command with no command line options to obtain a list of these log files. + +3. Identify the active log files in your production environment, and *copy* these to the failover directory. Use the **db_archive** command with the `-l` option to obtain a list of these log files. + +4. Run catastrophic recovery against the failover directory. Use the **db_recover** command with the `-c` option to do this. + +5. Optionally copy the backup to an archival location. + +Once you have performed this procedure, you can maintain an active hot backup by repeating steps 2 - 5 as often as is required by your application. + +### Note + +If you perform step 1, steps 2-5 must follow in order to ensure consistency of your hot backup. + +### Note + +Rather than use the previous procedure, you can use the **db_hotbackup** command line utility to do the same thing. This utility will (optionally) run a checkpoint and then copy all necessary files to a target directory for you. + +To actually perform a failover, simply: + +1. Shut down all processes which are running against the original environment. + +2. If you have an archival copy of the backup environment, you can optionally try copying the remaining log files from the original environment and running catastrophic recovery against that backup environment. Do this *only* if you have a an archival copy of the backup environment. + + This step can allow you to recover data created or modified in the original environment, but which did not have a chance to be reflected in the hot backup environment. + +3. Reopen your environment and databases as normal, but use the backup environment instead of the production environment. diff --git a/docs-src/guides/gsg_txn/img/deadlock.jpg b/docs-src/guides/gsg_txn/img/deadlock.jpg new file mode 100644 index 000000000..0995a84d8 Binary files /dev/null and b/docs-src/guides/gsg_txn/img/deadlock.jpg differ diff --git a/docs-src/guides/gsg_txn/img/readblock.jpg b/docs-src/guides/gsg_txn/img/readblock.jpg new file mode 100644 index 000000000..16a511feb Binary files /dev/null and b/docs-src/guides/gsg_txn/img/readblock.jpg differ diff --git a/docs-src/guides/gsg_txn/img/rwlocks1-pdf.jpg b/docs-src/guides/gsg_txn/img/rwlocks1-pdf.jpg new file mode 100644 index 000000000..11346c0bb Binary files /dev/null and b/docs-src/guides/gsg_txn/img/rwlocks1-pdf.jpg differ diff --git a/docs-src/guides/gsg_txn/img/rwlocks1.jpg b/docs-src/guides/gsg_txn/img/rwlocks1.jpg new file mode 100644 index 000000000..0fc88fd31 Binary files /dev/null and b/docs-src/guides/gsg_txn/img/rwlocks1.jpg differ diff --git a/docs-src/guides/gsg_txn/img/simplelock-pdf.jpg b/docs-src/guides/gsg_txn/img/simplelock-pdf.jpg new file mode 100644 index 000000000..78f8321d9 Binary files /dev/null and b/docs-src/guides/gsg_txn/img/simplelock-pdf.jpg differ diff --git a/docs-src/guides/gsg_txn/img/simplelock.jpg b/docs-src/guides/gsg_txn/img/simplelock.jpg new file mode 100644 index 000000000..8dca4ad82 Binary files /dev/null and b/docs-src/guides/gsg_txn/img/simplelock.jpg differ diff --git a/docs-src/guides/gsg_txn/img/writeblock.jpg b/docs-src/guides/gsg_txn/img/writeblock.jpg new file mode 100644 index 000000000..4b382b82f Binary files /dev/null and b/docs-src/guides/gsg_txn/img/writeblock.jpg differ diff --git a/docs-src/guides/gsg_txn/index.md b/docs-src/guides/gsg_txn/index.md new file mode 100644 index 000000000..b190bb259 --- /dev/null +++ b/docs-src/guides/gsg_txn/index.md @@ -0,0 +1,168 @@ +--- +title: "Getting Started with Berkeley DB Transaction Processing" +api-name: "Getting Started with Berkeley DB Transaction Processing" +source: docs/gsg_txn/C/index.html +--- +# Getting Started with Berkeley DB Transaction Processing + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction](introduction.md) + + [Transaction Benefits](introduction.md#txnintro) + + [A Note on System Failure](sysfailure.md) + + [Application Requirements](apireq.md) + + [Multi-threaded and Multi-process Applications](multithread-intro.md) + + [Recoverability](recovery-intro.md) + + [Performance Tuning](perftune-intro.md) + + [2. Enabling Transactions](enabletxn.md) + + [Environments](enabletxn.md#environments) + + [File Naming](enabletxn.md#filenaming) + + [Error Support](enabletxn.md#errorsupport) + + [Shared Memory Regions](enabletxn.md#sharedmemory) + + [Security Considerations](enabletxn.md#security) + + [Opening a Transactional Environment and Database](envopen.md) + + [3. Transaction Basics](usingtxns.md) + + [Committing a Transaction](usingtxns.md#commitresults) + + [Non-Durable Transactions](nodurabletxn.md) + + [Aborting a Transaction](abortresults.md) + + [Auto Commit](autocommit.md) + + [Nested Transactions](nestedtxn.md) + + [Transactional Cursors](txncursor.md) + + [Secondary Indices with Transaction Applications](txnindices.md) + + [Configuring the Transaction Subsystem](maxtxns.md) + + [4. Concurrency](txnconcurrency.md) + + [Which DB Handles are Free-Threaded](txnconcurrency.md#concurrenthandles) + + [Locks, Blocks, and Deadlocks](blocking_deadlocks.md) + + [Locks](blocking_deadlocks.md#locks) + + [Blocks](blocking_deadlocks.md#blocks) + + [Deadlocks](blocking_deadlocks.md#deadlocks) + + [The Locking Subsystem](lockingsubsystem.md) + + [Configuring the Locking Subsystem](lockingsubsystem.md#configuringlock) + + [Configuring Deadlock Detection](lockingsubsystem.md#configdeadlkdetect) + + [Resolving Deadlocks](lockingsubsystem.md#deadlockresolve) + + [Setting Transaction Priorities](lockingsubsystem.md#setpriority) + + [Isolation](isolation.md) + + [Supported Degrees of Isolation](isolation.md#degreesofisolation) + + [Reading Uncommitted Data](isolation.md#dirtyreads) + + [Committed Reads](isolation.md#readcommitted) + + [Using Snapshot Isolation](isolation.md#snapshot_isolation) + + [Transactional Cursors and Concurrent Applications](txn_ccursor.md) + + [Using Cursors with Uncommitted Data](txn_ccursor.md#cursordirtyreads) + + [Exclusive Database Handles](exclusivelock.md) + + [Read/Modify/Write](readmodifywrite.md) + + [No Wait on Blocks](txnnowait.md) + + [Reverse BTree Splits](reversesplit.md) + + [5. Managing DB Files](filemanagement.md) + + [Checkpoints](filemanagement.md#checkpoints) + + [Backup Procedures](backuprestore.md) + + [About Unix Copy Utilities](backuprestore.md#copyutilities) + + [Offline Backups](backuprestore.md#standardbackup) + + [Hot Backup](backuprestore.md#hotbackup) + + [Incremental Backups](backuprestore.md#incrementalbackups) + + [Recovery Procedures](recovery.md) + + [Normal Recovery](recovery.md#normalrecovery) + + [Catastrophic Recovery](recovery.md#catastrophicrecovery) + + [Designing Your Application for Recovery](architectrecovery.md) + + [Recovery for Multi-Threaded Applications](architectrecovery.md#multithreadrecovery) + + [Recovery in Multi-Process Applications](architectrecovery.md#multiprocessrecovery) + + [Using Hot Failovers](hotfailover.md) + + [Removing Log Files](logfileremoval.md) + + [Configuring the Logging Subsystem](logconfig.md) + + [Setting the Log File Size](logconfig.md#logfilesize) + + [Configuring the Logging Region Size](logconfig.md#logregionsize) + + [Configuring In-Memory Logging](logconfig.md#inmemorylogging) + + [Setting the In-Memory Log Buffer Size](logconfig.md#logbuffer) + + [6. Summary and Examples](wrapup.md) + + [Anatomy of a Transactional Application](wrapup.md#anatomy) + + [Transaction Example](txnexample_c.md) + + [In-Memory Transaction Example](inmem_txnexample_c.md) diff --git a/docs-src/guides/gsg_txn/inmem_txnexample_c.md b/docs-src/guides/gsg_txn/inmem_txnexample_c.md new file mode 100644 index 000000000..942d9f83e --- /dev/null +++ b/docs-src/guides/gsg_txn/inmem_txnexample_c.md @@ -0,0 +1,461 @@ +--- +title: "In-Memory Transaction Example" +api-name: "In-Memory Transaction Example" +source: docs/gsg_txn/C/inmem_txnexample_c.html +--- +## In-Memory Transaction Example + +DB is sometimes used for applications that simply need to cache data retrieved from some other location (such as a remote database server). DB is also often used in embedded systems. + +In both cases, applications may want to use transactions for atomicity, consistency, and isolation guarantees, but they may also want to forgo the durability guarantee entirely. In doing so, they can keep their DB environment and databases entirely in-memory so as to avoid the performance impact of unneeded disk I/O. + +To do this: + +- Refrain from specifying a home directory when you open your environment. The exception to this is if you are using the `DB_CONFIG` configuration file — in that case you must identify the environment's home directory so that the configuration file can be found. + +- Configure your environment to back your regions from system memory instead of the filesystem. + +- Configure your logging subsystem such that log files are kept entirely in-memory. + +- Increase the size of your in-memory log buffer so that it is large enough to hold the largest set of concurrent write operations. + +- Increase the size of your in-memory cache so that it can hold your entire data set. You do not want your cache to page to disk. + +- Do not specify a file name when you open your database(s). + +As an example, this section takes the transaction example provided in Transaction Example and it updates that example so that the environment, database, log files, and regions are all kept entirely in-memory. + +For illustration purposes, we also modify this example so that uncommitted reads are no longer used to enable the `count_records()` function. Instead, we simply provide a transaction handle to `count_records()` so as to avoid the self-deadlock. Be aware that using a transaction handle here rather than uncommitted reads will work just as well as if we had continued to use uncommitted reads. However, the usage of the transaction handle here will probably cause more deadlocks than using read-uncommitted does, because more locking is being performed in this case. + +To begin, we simplify the beginning of our example a bit. Because we no longer need an environment home directory, we can remove all the code that we used to determine path delimiters and include the `getopt` function. We can also remove our `usage()` function because we no longer require any command line arguments. + +``` c +/* File: txn_guide_inmemory.c */ + +/* We assume an ANSI-compatible compiler */ +#include +#include +#include +#include +#include + +/* Run 5 writers threads at a time. */ +#define NUMWRITERS 5 + +/* + * Printing of pthread_t is implementation-specific, so we + * create our own thread IDs for reporting purposes. + */ +int global_thread_num; +pthread_mutex_t thread_num_lock; + +/* Forward declarations */ +int count_records(DB *, DB_TXN *); +int open_db(DB **, const char *, const char *, DB_ENV *, u_int32_t); +int writer_thread(void *); +``` + +Next, in our `main()`, we also eliminate some variables that this example no longer needs. In particular, we are able to remove the `db_home_dir` and `file_name` variables. We also remove all our `getopt` code. + +``` c +int +main(void) +{ + /* Initialize our handles */ + DB *dbp = NULL; + DB_ENV *envp = NULL; + + pthread_t writer_threads[NUMWRITERS]; + int i, ret, ret_t; + u_int32_t env_flags; + + /* Application name */ + const char *prog_name = "txn_guide_inmemory"; +``` + +Next we create our environment as always. However, we add `DB_PRIVATE` to our environment open flags. This flag causes our environment to back regions using our application's heap memory rather than by using the filesystem. This is the first important step to keeping our DB data entirely in-memory. + +We also remove the `DB_RECOVER` flag from the environment open flags. Because our databases, logs, and regions are maintained in-memory, there will never be anything to recover. + +Note that we show the additional code here in **`bold.`** + +``` c + /* Create the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + env_flags = + DB_CREATE | /* Create the environment if it does not exist */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_TXN | /* Initialize the transactional subsystem. This + * also turns on logging. */ + DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ + DB_PRIVATE | /* Region files are not backed by the filesystem. + * Instead, they are backed by heap memory. */ + DB_THREAD; /* Cause the environment to be free-threaded */ +``` + +Now we configure our environment to keep the log files in memory, increase the log buffer size to 10 MB, and increase our in-memory cache to 10 MB. These values should be more than enough for our application's workload. + +``` c + + /* Specify in-memory logging */ + ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); + if (ret != 0) { + fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Specify the size of the in-memory log buffer. + */ + ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); + if (ret != 0) { + fprintf(stderr, "Error increasing the log buffer size: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Specify the size of the in-memory cache. + */ + ret = envp->set_cachesize(envp, 0, + 10 * 1024 * 1024, 1); + if (ret != 0) { + fprintf(stderr, "Error increasing the cache size: %s\n", + db_strerror(ret)); + goto err; + } + + +``` + +Next, we open the environment and setup our lock detection. This is identical to how the example previously worked, except that we do not provide a location for the environment's home directory. + +``` c + /* + * Indicate that we want db to perform lock detection internally. + * Also indicate that the transaction with the fewest number of + * write locks will receive the deadlock notification in + * the event of a deadlock. + */ + ret = envp->set_lk_detect(envp, DB_LOCK_MINWRITE); + if (ret != 0) { + fprintf(stderr, "Error setting lock detect: %s\n", + db_strerror(ret)); + goto err; + } + + /* Now actually open the environment */ + ret = envp->open(envp, NULL, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } +``` + +When we call `open_db()`, which is what we use to open our database, we no not provide a database filename for the third parameter. When the filename is `NULL`, the database is not backed by the filesystem. + +``` c + /* + * If we had utility threads (for running checkpoints or + * deadlock detection, for example) we would spawn those + * here. However, for a simple example such as this, + * that is not required. + */ + + /* Open the database */ + ret = open_db(&dbp, prog_name, NULL, + envp, DB_DUPSORT); + if (ret != 0) + goto err; +``` + +After that, our `main()` function is unchanged, except that when we close the database, we change the error message string so as to not reference the database filename. + +``` c + /* Initialize a pthread mutex. Used to help provide thread ids. */ + (void)pthread_mutex_init(&thread_num_lock, NULL); + + /* Start the writer threads. */ + for (i = 0; i < NUMWRITERS; i++) + (void)pthread_create( + &writer_threads[i], NULL, (void *)writer_thread, (void *)dbp); + + /* Join the writers */ + for (i = 0; i < NUMWRITERS; i++) + (void)pthread_join(writer_threads[i], NULL); + +err: + /* Close our database handle, if it was opened. */ + if (dbp != NULL) { + ret_t = dbp->close(dbp, 0); + if (ret_t != 0) { + fprintf(stderr, "%s database close failed.\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Close our environment, if it was opened. */ + if (envp != NULL) { + ret_t = envp->close(envp, 0); + if (ret_t != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Final status message and return. */ + printf("I'm all done.\n"); + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +That completes `main()`. The bulk of our `writer_thread()` function implementation is unchanged from the initial transaction example, except that we no longer check for `DB_KEYEXISTS` in our `DB->put()` return code. Because we are configuring for a completely in-memory database, there is no possibility that we can run this code against an existing database. Therefore, there is no way that `DB_KEYEXISTS` will be returned by `DB->put()`. + +``` c +/* + * A function that performs a series of writes to a + * Berkeley DB database. The information written + * to the database is largely nonsensical, but the + * mechanism of transactional commit/abort and + * deadlock detection is illustrated here. + */ +int +writer_thread(void *args) +{ + DBT key, value; + DB_TXN *txn; + int i, j, payload, ret, thread_num; + int retry_count, max_retries = 20; /* Max retry on a deadlock */ + char *key_strings[] = {"key 1", "key 2", "key 3", "key 4", + "key 5", "key 6", "key 7", "key 8", + "key 9", "key 10"}; + + DB *dbp = (DB *)args; + DbEnv *envp = dbp->get_env(); + + /* Get the thread number */ + (void)pthread_mutex_lock(&thread_num_lock); + global_thread_num++; + thread_num = global_thread_num; + (void)pthread_mutex_unlock(&thread_num_lock); + + /* Initialize the random number generator */ + srand((u_int)pthread_self()); + + /* Write 50 times and then quit */ + for (i = 0; i < 50; i++) { + retry_count = 0; /* Used for deadlock retries */ + +retry: + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn_begin failed"); + return (EXIT_FAILURE); + } + for (j = 0; j < 10; j++) { + /* Set up our key and values DBTs */ + memset(&key, 0, sizeof(DBT)); + key.data = key_strings[j]; + key.size = (strlen(key_strings[j]) + 1) * sizeof(char); + + memset(&value, 0, sizeof(DBT)); + payload = rand() + i; + value.data = &payload; + value.size = sizeof(int); + + /* Perform the database put. */ + switch (ret = dbp->put(dbp, txn, &key, &value, 0)) { + case 0: + break; + + /* + * Here's where we perform deadlock detection. If + * DB_LOCK_DEADLOCK is returned by the put operation, + * then this thread has been chosen to break a deadlock. + * It must abort its operation, and optionally retry the + * put. + */ + case DB_LOCK_DEADLOCK: + /* + * First that we MUST do is abort the + * transaction. + */ + (void)txn->abort(txn); + /* + * Now we decide if we want to retry the operation. + * If we have retried less than max_retries, + * increment the retry count and goto retry. + */ + if (retry_count < max_retries) { + printf("Writer %i: Got DB_LOCK_DEADLOCK.\n", + thread_num); + printf("Writer %i: Retrying write operation.\n", + thread_num); + retry_count++; + goto retry; + } + /* + * Otherwise, just give up. + */ + printf("Writer %i: ", thread_num); + printf("Got DB_LOCK_DEADLOCK and out of retries.\n"); + printf("Writer %i: Giving up.\n", thread_num); + return (EXIT_FAILURE); + /* + * If a generic error occurs, we simply abort the + * transaction and exit the thread completely. + */ + default: + envp->err(envp, ret, "db put failed"); + ret = txn->abort(txn); + if (ret != 0) + envp->err(envp, ret, "txn abort failed"); + return (EXIT_FAILURE); + } /** End case statement **/ + + } /** End for loop **/ +``` + +The only other change to `writer_thread()` is that we pass `count_records()` a transaction handle, rather than configuring our entire application for uncommitted reads. Both mechanisms work well-enough for preventing a self-deadlock. However, the individual count in this example will tend to be lower than the counts seen in the previous transaction example, because `count_records()` can no longer see records created but not yet committed by other threads. + +``` c + /* + * print the number of records found in the database. + * See count_records() for usage information. + */ + printf("Thread %i. Record count: %i\n", thread_num, + count_records(dbp, txn)); + + /* + * If all goes well, we can commit the transaction and + * loop to the next transaction. + */ + ret = txn->commit(txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn commit failed"); + return (EXIT_FAILURE); + } + } + return (EXIT_SUCCESS); +} +``` + +Next we update `count_records()`. The only difference here is that we no longer specify `DB_READ_UNCOMMITTED` when we open our cursor. Note that even this minor change is not required. If we do not configure our database to support uncommitted reads, `DB_READ_UNCOMMITTED` on the cursor open will be silently ignored. However, we remove the flag anyway from the cursor open so as to avoid confusion. + +``` c +int +count_records(DB *dbp, DB_TXN *txn) +{ + DBT key, value; + DBC *cursorp; + int count, ret; + + cursorp = NULL; + count = 0; + + /* Get the cursor */ + ret = dbp->cursor(dbp, txn, &cursorp, 0); + if (ret != 0) { + dbp->err(dbp, ret, "count_records: cursor open failed."); + goto cursor_err; + } + + /* Get the key DBT used for the database read */ + memset(&key, 0, sizeof(DBT)); + memset(&value, 0, sizeof(DBT)); + do { + ret = cursorp->get(cursorp, &key, &value, DB_NEXT); + switch (ret) { + case 0: + count++; + break; + case DB_NOTFOUND: + break; + default: + dbp->err(dbp, ret, + "Count records unspecified error"); + goto cursor_err; + } + } while (ret == 0); + +cursor_err: + if (cursorp != NULL) { + ret = cursorp->close(cursorp); + if (ret != 0) { + dbp->err(dbp, ret, + "count_records: cursor close failed."); + } + } + + return (count); +} +``` + +Finally, we update `open_db()`. This involves removing `DB_READ_UNCOMMITTED` from the open flags. We are also careful to change our database open error message to no longer use the `file_name` variable because that value will always be `NULL` for this example. + +``` c +/* Open a Berkeley DB database */ +int +open_db(DB **dbpp, const char *progname, const char *file_name, + DB_ENV *envp, u_int32_t extra_flags) +{ + int ret; + u_int32_t open_flags; + DB *dbp; + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + fprintf(stderr, "%s: %s\n", progname, + db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* Point to the memory malloc'd by db_create() */ + *dbpp = dbp; + + if (extra_flags != 0) { + ret = dbp->set_flags(dbp, extra_flags); + if (ret != 0) { + dbp->err(dbp, ret, + "open_db: Attempt to set extra flags failed."); + return (EXIT_FAILURE); + } + } + + /* Now open the database */ + open_flags = DB_CREATE | /* Allow database creation */ + DB_THREAD | + DB_AUTO_COMMIT; /* Allow auto commit */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + open_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database open failed"); + return (EXIT_FAILURE); + } + return (EXIT_SUCCESS); +} +``` + +This completes our in-memory transactional example. If you would like to experiment with this code, you can find the example in the following location in your DB distribution: + +``` c +DB_INSTALL/examples_c/txn_guide +``` diff --git a/docs-src/guides/gsg_txn/introduction.md b/docs-src/guides/gsg_txn/introduction.md new file mode 100644 index 000000000..b1c692e5a --- /dev/null +++ b/docs-src/guides/gsg_txn/introduction.md @@ -0,0 +1,50 @@ +--- +title: "Chapter 1. Introduction" +api-name: "Chapter 1. Introduction" +source: docs/gsg_txn/C/introduction.html +--- +## Chapter 1. Introduction + +**Table of Contents** + + [Transaction Benefits](introduction.md#txnintro) + + [A Note on System Failure](sysfailure.md) + + [Application Requirements](apireq.md) + + [Multi-threaded and Multi-process Applications](multithread-intro.md) + + [Recoverability](recovery-intro.md) + + [Performance Tuning](perftune-intro.md) + +This book provides a thorough introduction and discussion on transactions as used with Berkeley DB (DB). It begins by offering a general overview to transactions, the guarantees they provide, and the general application infrastructure required to obtain full transactional protection for your data. + +This book also provides detailed examples on how to write a transactional application. Both single threaded and multi-threaded (as well as multi-process applications) are discussed. A detailed description of various backup and recovery strategies is included in this manual, as is a discussion on performance considerations for your transactional application. + +You should understand the concepts from the *Getting Started with Berkeley DB* guide before reading this book. + +## Transaction Benefits + +Transactions offer your application's data protection from application or system failures. That is, DB transactions offer your application full ACID support: + +- **A**tomicity + + Multiple database operations are treated as a single unit of work. Once committed, all write operations performed under the protection of the transaction are saved to your databases. Further, in the event that you abort a transaction, all write operations performed during the transaction are discarded. In this event, your database is left in the state it was in before the transaction began, regardless of the number or type of write operations you may have performed during the course of the transaction. + + Note that DB transactions can span one or more database handles. + +- **C**onsistency + + Your databases will never see a partially completed transaction. This is true even if your application fails while there are in-progress transactions. If the application or system fails, then either all of the database changes appear when the application next runs, or none of them appear. + + In other words, whatever consistency requirements your application has will never be violated by DB. If, for example, your application requires every record to include an employee ID, and your code faithfully adds that ID to its database records, then DB will never violate that consistency requirement. The ID will remain in the database records until such a time as your application chooses to delete it. + +- **I**solation + + While a transaction is in progress, your databases will appear to the transaction as if there are no other operations occurring outside of the transaction. That is, operations wrapped inside a transaction will always have a clean and consistent view of your databases. They never have to see updates currently in progress under the protection of another transaction. Note, however, that isolation guarantees can be relaxed from the default setting. See Isolation for more information. + +- **D**urability + + Once committed to your databases, your modifications will persist even in the event of an application or system failure. Note that like isolation, your durability guarantee can be relaxed. See Non-Durable Transactions for more information. diff --git a/docs-src/guides/gsg_txn/isolation.md b/docs-src/guides/gsg_txn/isolation.md new file mode 100644 index 000000000..1750c33a4 --- /dev/null +++ b/docs-src/guides/gsg_txn/isolation.md @@ -0,0 +1,395 @@ +--- +title: "Isolation" +api-name: "Isolation" +source: docs/gsg_txn/C/isolation.html +--- +## Isolation + + [Supported Degrees of Isolation](isolation.md#degreesofisolation) + + [Reading Uncommitted Data](isolation.md#dirtyreads) + + [Committed Reads](isolation.md#readcommitted) + + [Using Snapshot Isolation](isolation.md#snapshot_isolation) + +Isolation guarantees are an important aspect of transactional protection. Transactions ensure the data your transaction is working with will not be changed by some other transaction. Moreover, the modifications made by a transaction will never be viewable outside of that transaction until the changes have been committed. + +That said, there are different degrees of isolation, and you can choose to relax your isolation guarantees to one degree or another depending on your application's requirements. The primary reason why you might want to do this is because of performance; the more isolation you ask your transactions to provide, the more locking that your application must do. With more locking comes a greater chance of blocking, which in turn causes your threads to pause while waiting for a lock. Therefore, by relaxing your isolation guarantees, you can *potentially* improve your application's throughput. Whether you actually see any improvement depends, of course, on the nature of your application's data and transactions. + +### Supported Degrees of Isolation + +DB supports the following levels of isolation: + + + + + + + + + + + + + + + + + + + + + + + + + + +
DegreeANSI TermDefinition
1READ UNCOMMITTEDUncommitted reads means that one transaction will never overwrite another transaction's dirty data. Dirty data is data that a transaction has modified but not yet committed to the underlying data store. However, uncommitted reads allows a transaction to see data dirtied by another transaction. In addition, a transaction may read data dirtied by another transaction, but which subsequently is aborted by that other transaction. In this latter case, the reading transaction may be reading data that never really existed in the database.
2READ COMMITTED

Committed read isolation means that degree 1 is observed, except that dirty data is never read.

+

In addition, this isolation level guarantees that data will never change so long as it is addressed by the cursor, but the data may change before the reading cursor is closed. In the case of a transaction, data at the current cursor position will not change, but once the cursor moves, the previous referenced data can change. This means that readers release read locks before the cursor is closed, and therefore, before the transaction completes. Note that this level of isolation causes the cursor to operate in exactly the same way as it does in the absence of a transaction.

3SERIALIZABLE

Committed read is observed, plus the data read by a transaction, T, will never be dirtied by another transaction before T completes. This means that both read and write locks are not released until the transaction completes.

+

In addition, no transactions will see phantoms. Phantoms are records returned as a result of a search, but which were not seen by the same transaction when the identical search criteria was previously used.

+

This is DB's default isolation guarantee.

+ +By default, DB transactions and transactional cursors offer serializable isolation. You can optionally reduce your isolation level by configuring DB to use uncommitted read isolation. See Reading Uncommitted Data for more information. You can also configure DB to use committed read isolation. See Committed Reads for more information. + +Finally, in addition to DB's normal degrees of isolation, you can also use *snapshot isolation*. This allows you to avoid the read locks that serializable isolation requires. See Using Snapshot Isolation for details. + +### Reading Uncommitted Data + +Berkeley DB allows you to configure your application to read data that has been modified but not yet committed by another transaction; that is, dirty data. When you do this, you may see a performance benefit by allowing your application to not have to block waiting for write locks. On the other hand, the data that your application is reading may change before the transaction has completed. + +When used with transactions, uncommitted reads means that one transaction can see data modified but not yet committed by another transaction. When used with transactional cursors, uncommitted reads means that any database reader can see data modified by the cursor before the cursor's transaction has committed. + +Because of this, uncommitted reads allow a transaction to read data that may subsequently be aborted by another transaction. In this case, the reading transaction will have read data that never really existed in the database. + +To configure your application to read uncommitted data: + +1. Open your database such that it will allow uncommitted reads. You do this by specifying `DB_READ_UNCOMMITTED` when you open your database. + +2. Specify `DB_READ_UNCOMMITTED` when you create the transaction, open the cursor, or read a record from the database. + +For example, the following opens the database such that it supports uncommitted reads, and then creates a transaction that causes all reads performed within it to use uncommitted reads. Remember that simply opening the database to support uncommitted reads is not enough; you must also declare your read operations to be performed using uncommitted reads. + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DB_TXN *txn; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + const char *keystr ="thekey"; + const char *datastr = "thedata"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + db_flags = DB_CREATE | /* Create the db if it does not + * exist */ + DB_AUTO_COMMIT | /* Enable auto commit */ + DB_READ_UNCOMMITTED; /* Enable uncommitted reads */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + + /* Get the txn handle */ + txn = NULL; + ret = envp->txn_begin(envp, NULL, &txn, DB_READ_UNCOMMITTED); + if (ret != 0) { + envp->err(envp, ret, "Transaction begin failed."); + goto err; + } + + /* + * From here, you perform your database reads and writes as normal, + * committing and aborting the transactions as is necessary, and + * testing for deadlock exceptions as normal (omitted for brevity). + */ + + ... +``` + +### Committed Reads + +You can configure your transaction so that the data being read by a transactional cursor is consistent so long as it is being addressed by the cursor. However, once the cursor is done reading the record (that is, reading records from the page that it currently has locked), the cursor releases its lock on that record or page. This means that the data the cursor has read and released may change before the cursor's transaction has completed. + +For example, suppose you have two transactions, `Ta` and `Tb`. Suppose further that `Ta` has a cursor that reads `record R`, but does not modify it. Normally, `Tb` would then be unable to write `record R` because `Ta` would be holding a read lock on it. But when you configure your transaction for committed reads, `Tb` *can* modify `record R` before `Ta` completes, so long as the reading cursor is no longer addressing the record or page. + +When you configure your application for this level of isolation, you may see better performance throughput because there are fewer read locks being held by your transactions. Read committed isolation is most useful when you have a cursor that is reading and/or writing records in a single direction, and that does not ever have to go back to re-read those same records. In this case, you can allow DB to release read locks as it goes, rather than hold them for the life of the transaction. + +To configure your application to use committed reads, do one of the following: + +- Create your transaction such that it allows committed reads. You do this by specifying `DB_READ_COMMITTED` when you open the transaction. + +- Specify `DB_READ_COMMITTED` when you open the cursor. + +For example, the following creates a transaction that allows committed reads: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DB_TXN *txn; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + /* + * Notice that we do not have to specify any flags to the database to + * allow committed reads (this is as opposed to uncommitted reads + * where we DO have to specify a flag on the database open. + */ + db_flags = DB_CREATE | DB_AUTO_COMMIT; + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + + /* Get the txn handle */ + txn = NULL; + /* + * Open the transaction and enable committed reads. All cursors open + * with this transaction handle will use read committed isolation. + */ + ret = envp->txn_begin(envp, NULL, &txn, DB_READ_COMMITTED); + if (ret != 0) { + envp->err(envp, ret, "Transaction begin failed."); + goto err; + } + + /* + * From here, you perform your database reads and writes as normal, + * committing and aborting the transactions as is necessary, and + * testing for deadlock exceptions as normal (omitted for brevity). + * + * Using transactional cursors with concurrent applications is + * described in more detail in the following section. + */ + + ... +``` + +### Using Snapshot Isolation + +By default DB uses serializable isolation. An important side effect of this isolation level is that read operations obtain read locks on database pages, and then hold those locks until the read operation is completed. When you are using transactional cursors, this means that read locks are held until the transaction commits or aborts. In that case, over time a transactional cursor can gradually block all other transactions from writing to the database. + +You can avoid this by using snapshot isolation. Snapshot isolation uses *multiversion concurrency control* to guarantee repeatable reads. What this means is that every time a writer would take a read lock on a page, instead a copy of the page is made and the writer operates on that page copy. This frees other writers from blocking due to a read lock held on the page. + +### Note + +Snapshot isolation is strongly recommended for read-only threads when writer threads are also running, as this will eliminate read-write contention and greatly improve transaction throughput for your writer threads. However, in order for snapshot isolation to work for your reader-only threads, you must of course use transactions for your DB reads. + +#### Snapshot Isolation Cost + +Snapshot isolation does not come without a cost. Because pages are being duplicated before being operated upon, the cache will fill up faster. This means that you might need a larger cache in order to hold the entire working set in memory. + +If the cache becomes full of page copies before old copies can be discarded, additional I/O will occur as pages are written to temporary "freezer" files on disk. This can substantially reduce throughput, and should be avoided if possible by configuring a large cache and keeping snapshot isolation transactions short. + +You can estimate how large your cache should be by taking a checkpoint, followed by a call to the `DB_ENV->log_archive()` method. The amount of cache required is approximately double the size of the remaining log files (that is, the log files that cannot be archived). + +#### Snapshot Isolation Transactional Requirements + +In addition to an increased cache size, you may also need to increase the number of transactions that your application supports. (See Configuring the Transaction Subsystem for details on how to set this.) In the worst case scenario, you might need to configure your application for one more transaction for every page in the cache. This is because transactions are retained until the last page they created is evicted from the cache. + +#### When to Use Snapshot Isolation + +Snapshot isolation is best used when all or most of the following conditions are true: + +- You can have a large cache relative to your working data set size. + +- You require repeatable reads. + +- You will be using transactions that routinely work on the entire database, or more commonly, there is data in your database that will be very frequently written by more than one transaction. + +- Read/write contention is limiting your application's throughput, or the application is all or mostly read-only and contention for the lock manager mutex is limiting throughput. + +#### How to use Snapshot Isolation + +You use snapshot isolation by: + +- Opening the database with multiversion support. You can configure this either when you open your environment or when you open your database. Use the `DB_MULTIVERSION` flag to configure this support. + +- Configure your cursor or transaction to use snapshot isolation. + + To do this, pass the `DB_TXN_SNAPSHOT` flag when you open the cursor or create the transaction. If configured for the transaction, then this flag is not required when the cursor is opened. + +The simplest way to take advantage of snapshot isolation is for queries: keep update transactions using full read/write locking and use snapshot isolation on read-only transactions or cursors. This should minimize blocking of snapshot isolation transactions and will avoid deadlock errors. + +If the application has update transactions which read many items and only update a small set (for example, scanning until a desired record is found, then modifying it), throughput may be improved by running some updates at snapshot isolation as well. But doing this means that you must manage deadlock errors. See Resolving Deadlocks for details. + +The following code fragment turns on snapshot isolation for a transaction: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + /* Support snapshot isolation */ + envp->set_flags(envp, DB_MULTIVERSION, 1); + env_flags = DB_CREATE | /* Create the environment if it does + * not already exist. */ + + DB_INIT_LOCK | /* Initialize locking. */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL| /* Initialize the in-memory cache. */ + DB_INIT_TXN; /* Initialize transactions */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + /* + * Nothing needs to be supplied here to support snapshot isolation. + * The environment does, so its databases will too. + */ + db_flags = DB_CREATE | DB_AUTO_COMMIT; + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + + .... + + ret = envp->txn_begin(envp, NULL, &txn, DB_TXN_SNAPSHOT); + + /* remainder of the program omitted for brevity */ + + +``` diff --git a/docs-src/guides/gsg_txn/lockingsubsystem.md b/docs-src/guides/gsg_txn/lockingsubsystem.md new file mode 100644 index 000000000..b845f4fdf --- /dev/null +++ b/docs-src/guides/gsg_txn/lockingsubsystem.md @@ -0,0 +1,357 @@ +--- +title: "The Locking Subsystem" +api-name: "The Locking Subsystem" +source: docs/gsg_txn/C/lockingsubsystem.html +--- +## The Locking Subsystem + + [Configuring the Locking Subsystem](lockingsubsystem.md#configuringlock) + + [Configuring Deadlock Detection](lockingsubsystem.md#configdeadlkdetect) + + [Resolving Deadlocks](lockingsubsystem.md#deadlockresolve) + + [Setting Transaction Priorities](lockingsubsystem.md#setpriority) + +In order to allow concurrent operations, DB provides the locking subsystem. This subsystem provides inter- and intra- process concurrency mechanisms. It is extensively used by DB concurrent applications, but it can also be generally used for non-DB resources. + +This section describes the locking subsystem as it is used to protect DB resources. In particular, issues on configuration are examined here. For information on using the locking subsystem to manage non-DB resources, see the *Berkeley DB Programmer's Reference Guide*. + +### Configuring the Locking Subsystem + +You initialize the locking subsystem by specifying `DB_INIT_LOCK` to the `DB_ENV->open()` method. + +Before opening your environment, you can configure various values for your locking subsystem. Note that these limits can only be configured before the environment is opened. Also, these methods configure the entire environment, not just a specific environment handle. + +Finally, each bullet below identifies the `DB_CONFIG` file parameter that can be used to specify the specific locking limit. If used, these `DB_CONFIG` file parameters override any value that you might specify using the environment handle. + +The limits that you can configure are as follows: + +- The number of lockers supported by the environment. This value is used by the environment when it is opened to estimate the amount of space that it should allocate for various internal data structures. By default, 1,000 lockers are supported. + + To configure this value, use the `DB_ENV->set_memory_init()` method to configure the `DB_MEM_LOCKER` structure. + + As an alternative to this method, you can configure this value using the `DB_CONFIG` file's `set_lk_max_lockers` parameter. + +- The number of locks supported by the environment. By default, 1,000 locks are supported. + + To configure this value, use the `DB_ENV->set_memory_init()` method to configure the `DB_MEM_LOCK` structure. + + As an alternative to this method, you can configure this value using the `DB_CONFIG` file's `set_lk_max_locks` parameter. + +- The number of locked objects supported by the environment. By default, 1,000 objects can be locked. + + To configure this value, use the `DB_ENV->set_memory_init()` method to configure the `DB_MEM_LOCKOBJECT` structure. + + As an alternative to this method, you can configure this value using the `DB_CONFIG` file's `set_lk_max_objects` parameter. + +For a definition of lockers, locks, and locked objects, see Lock Resources. + +For example, to configure the number of locks that your environment can use: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t env_flags; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + /* Configure max locks */ + ret = envp->set_memory_init(envp, DB_MEM_LOCK, 5000); + if (ret != 0) { + fprintf(stderr, "Error configuring locks: %s\n", + db_strerror(ret)); + goto err; + } + + /* Open the environment. */ + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + +err: + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +### Configuring Deadlock Detection + +In order for DB to know that a deadlock has occurred, some mechanism must be used to perform deadlock detection. There are three ways that deadlock detection can occur: + +1. Allow DB to internally detect deadlocks as they occur. + + To do this, you use `DB_ENV->set_lk_detect()`. This method causes DB to walk its internal lock table looking for a deadlock whenever a lock request is blocked. This method also identifies how DB decides which lock requests are rejected when deadlocks are detected. For example, DB can decide to reject the lock request for the transaction that has the most number of locks, the least number of locks, holds the oldest lock, holds the most number of write locks, and so forth (see the API reference documentation for a complete list of the lock detection policies). + + You can call this method at any time during your application's lifetime, but typically it is used before you open your environment. + + Note that how you want DB to decide which thread of control should break a deadlock is extremely dependent on the nature of your application. It is not unusual for some performance testing to be required in order to make this determination. That said, a transaction that is holding the most number of locks is usually indicative of the transaction that has performed the most amount of work. Frequently you will not want a transaction that has performed a lot of work to abandon its efforts and start all over again. It is not therefore uncommon for application developers to initially select the transaction with the *minimum* number of write locks to break the deadlock. + + Using this mechanism for deadlock detection means that your application will never have to wait on a lock before discovering that a deadlock has occurred. However, walking the lock table every time a lock request is blocked can be expensive from a performance perspective. + +2. Use a dedicated thread or external process to perform deadlock detection. Note that this thread must be performing no other database operations beyond deadlock detection. + + To externally perform lock detection, you can use either the `DB_ENV->lock_detect()` method, or use the **db_deadlock** command line utility. This method (or command) causes DB to walk the lock table looking for deadlocks. + + Note that like `DB_ENV->set_lk_detect()`, you also use this method (or command line utility) to identify which lock requests are rejected in the event that a deadlock is detected. + + Applications that perform deadlock detection in this way typically run deadlock detection between every few seconds and a minute. This means that your application may have to wait to be notified of a deadlock, but you also save the overhead of walking the lock table every time a lock request is blocked. + +3. Lock timeouts. + + You can configure your locking subsystem such that it times out any lock that is not released within a specified amount of time. To do this, use the `DB_ENV->set_timeout()` method. Note that lock timeouts are only checked when a lock request is blocked or when deadlock detection is otherwise performed. Therefore, a lock can have timed out and still be held for some length of time until DB has a reason to examine its locking tables. + + Be aware that extremely long-lived transactions, or operations that hold locks for a long time, may be inappropriately timed out before the transaction or operation has a chance to complete. You should therefore use this mechanism only if you know your application will hold locks for very short periods of time. + +For example, to configure your application such that DB checks the lock table for deadlocks every time a lock request is blocked: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DB_TXN *txn; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + /* + * Configure db to perform deadlock detection internally, and to + * choose the transaction that has performed the least amount of + * writing to break the deadlock in the event that one is detected. + */ + ret = envp->set_lk_detect(envp, DB_LOCK_MINWRITE); + if (ret != 0) { + fprintf(stderr, "Error setting lk detect: %s\n", + db_strerror(ret)); + goto err; + } + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * From here, you open your databases, proceed with your + * database operations, and respond to deadlocks as + * is normal (omitted for brevity). + */ + + ... +``` + +Finally, the following command line call causes deadlock detection to be run against the environment contained in `/export/dbenv`. The transaction with the youngest lock is chosen to break the deadlock: + +``` c +> /usr/local/db_install/bin/db_deadlock -h /export/dbenv -a y +``` + +For more information, see the db_deadlock reference documentation. + +### Resolving Deadlocks + +When DB determines that a deadlock has occurred, it will select a thread of control to resolve the deadlock and then return `DB_LOCK_DEADLOCK` to that thread. If a deadlock is detected, the thread must: + +1. Cease all read and write operations. + +2. Close all open cursors. + +3. Abort the transaction. + +4. Optionally retry the operation. If your application retries deadlocked operations, the new attempt must be made using a new transaction. + +### Note + +If a thread has deadlocked, it may not make any additional database calls using the handle that has deadlocked. + +For example: + +``` c +retry: + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn_begin failed"); + return (EXIT_FAILURE); + } + ... + /* key and data are Dbts. Their usage is omitted for brevity. */ + ... + switch (ret = dbp->put(dbp, txn, &key, &data, 0)) { + case 0: + break; + /* Deadlock handling goes here */ + case DB_LOCK_DEADLOCK: + /* Abort the transaction */ + (void)txn->abort(txn); + + /* + * retry_count is a counter used to identify how many times + * we've retried this operation. To avoid the potential for + * endless looping, we won't retry more than + * MAX_DEADLOCK_RETRIES times. + */ + if (retry_count < MAX_DEADLOCK_RETRIES) { + printf("Got DB_LOCK_DEADLOCK.\n"); + printf("Retrying write operation.\n"); + retry_count++; + goto retry; + } + printf("Got DB_LOCK_DEADLOCK and out of retries."); + printf("Giving up.\n"); + return (EXIT_FAILURE); + default: + /* If some random database error occurs, we just give up */ + envp->err(envp, ret, "db put failed"); + ret = txn->abort(txn); + if (ret != 0) { + envp->err(envp, ret, "txn abort failed"); + return (EXIT_FAILURE); + } + } + /* If all goes well, commit the transaction */ + ret = txn->commit(txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn commit failed"); + return (EXIT_FAILURE); + } + + return (EXIT_SUCCESS); +``` + +### Setting Transaction Priorities + +Normally when a thread of control must be selected to resolve a deadlock, DB decides which thread will perform the resolution; you have no way of knowing in advance which thread will be selected to resolve the deadlock. + +However, there may be situations where you know it is better for one thread to resolve a deadlock over another thread. As an example, if you have a background thread running data management activities, and another thread responding to user requests, you might want deadlock resolution to occur in the background thread because you can better afford the throughput costs there. Under these circumstances, you can identify which thread of control will be selected for resolved deadlocks by setting a transaction priorities. + +When two transactions are deadlocked, DB will abort the transaction with the lowest priority. By default, every transaction is given a priority of 100. However, you can set a different priority on a transaction-by-transaction basis by using the `DB_TXN->set_priority()` method. + +When two or more transactions are tied for the lowest priority, the tie is broken based on the policy provided to the `DB_ENV->lock_detect()` method's `atype` parameter. + +A transaction's priority can be changed at any time after the transaction handle has been created and before the transaction has been resolved (committed or aborted). For example: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DBT key, data; + DB_TXN *txn; + + ... + // Open the environment and database as normal. + // Omitted for brevity + ... + + /* Get the txn handle */ + txn = NULL; + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction begin failed."); + goto err; + } + + ret = txn->set_priority(txn, 200); + if (ret != 0) { + envp->err(envp, ret, "Transaction set_priority failed."); + goto err; + } + + /* + * Perform the database write. If this fails, abort the transaction. + */ + ret = dbp->put(dbp, txn, &key, &data, 0); + if (ret != 0) { + envp->err(envp, ret, "Database put failed."); + txn->abort(txn); + goto err; + } + + /* + * Commit the transaction. + */ + ret = txn->commit(txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction commit failed."); + goto err; + } + +err: + ... + // Close the database and environment here, and exit the application. + // Omitted for brevity. +} +``` diff --git a/docs-src/guides/gsg_txn/logconfig.md b/docs-src/guides/gsg_txn/logconfig.md new file mode 100644 index 000000000..26810f081 --- /dev/null +++ b/docs-src/guides/gsg_txn/logconfig.md @@ -0,0 +1,186 @@ +--- +title: "Configuring the Logging Subsystem" +api-name: "Configuring the Logging Subsystem" +source: docs/gsg_txn/C/logconfig.html +--- +## Configuring the Logging Subsystem + + [Setting the Log File Size](logconfig.md#logfilesize) + + [Configuring the Logging Region Size](logconfig.md#logregionsize) + + [Configuring In-Memory Logging](logconfig.md#inmemorylogging) + + [Setting the In-Memory Log Buffer Size](logconfig.md#logbuffer) + +You can configure the following aspects of the logging subsystem: + +- Size of the log files. + +- Size of the logging subsystem's region. See Configuring the Logging Region Size. + +- Maintain logs entirely in-memory. See Configuring In-Memory Logging for more information. + +- Size of the log buffer in memory. See Setting the In-Memory Log Buffer Size. + +- On-disk location of your log files. See Identifying Specific File Locations. + +### Setting the Log File Size + +Whenever a pre-defined amount of data is written to a log file (10 MB by default), DB stops using the current log file and starts writing to a new file. You can change the maximum amount of data contained in each log file by using the `DB_ENV->set_lg_max()` method. Note that this method can be used at any time during an application's lifetime. + +Setting the log file size to something larger than its default value is largely a matter of convenience and a reflection of the application's preference in backup media and frequency. However, if you set the log file size too low relative to your application's traffic patterns, you can cause yourself trouble. + +From a performance perspective, setting the log file size to a low value can cause your active transactions to pause their writing activities more frequently than would occur with larger log file sizes. Whenever a transaction completes the log buffer is flushed to disk. Normally other transactions can continue to write to the log buffer while this flush is in progress. However, when one log file is being closed and another created, all transactions must cease writing to the log buffer until the switch over is completed. + +Beyond performance concerns, using smaller log files can cause you to use more physical files on disk. As a result, your application could run out of log sequence numbers, depending on how busy your application is. + +Every log file is identified with a 10 digit number. Moreover, the maximum number of log files that your application is allowed to create in its lifetime is 2,000,000,000. + +For example, if your application performs 6,000 transactions per second for 24 hours a day, and you are logging 500 bytes of data per transaction into 10 MB log files, then you will run out of log files in around 221 years: + +``` c + (10 * 2^20 * 2000000000) / (6000 * 500 * 365 * 60 *60 * 24) = 221 +``` + +However, if you were writing 2000 bytes of data per transaction, and using 1 MB log files, then the same formula shows you running out of log files in 5 years time. + +All of these time frames are quite long, to be sure, but if you do run out of log files after, say, 5 years of continuous operations, then you must reset your log sequence numbers. To do so: + +1. Backup your databases as if to prepare for catastrophic failure. See Backup Procedures for more information. + +2. Reset the log file's sequence number using the **db_load** utility's `-r` option. + +3. Remove all of the log files from your environment. Note that this is the only situation in which all of the log files are removed from an environment; in all other cases, at least a single log file is retained. + +4. Restart your application. + +### Configuring the Logging Region Size + +The logging subsystem's default region size is 60 KB. The logging region is used to store filenames, and so you may need to increase its size if a large number of files (that is, if you have a very large number of databases) will be opened and registered with DB's log manager. + +You can set the size of your logging region by using the `DB_ENV->set_lg_regionmax()` method. Note that this method can only be called before the first environment handle for your application is opened. + +### Configuring In-Memory Logging + +It is possible to configure your logging subsystem such that logs are maintained entirely in memory. When you do this, you give up your transactional durability guarantee. Without log files, you have no way to run recovery so any system or software failures that you might experience can corrupt your databases. + +However, by giving up your durability guarantees, you can greatly improve your application's throughput by avoiding the disk I/O necessary to write logging information to disk. In this case, you still retain your transactional atomicity, consistency, and isolation guarantees. + +To configure your logging subsystem to maintain your logs entirely in-memory: + +- Make sure your log buffer is capable of holding all log information that can accumulate during the longest running transaction. See Setting the In-Memory Log Buffer Size for details. + +- Do not run normal recovery when you open your environment. In this configuration, there are no log files available against which you can run recovery. As a result, if you specify recovery when you open your environment, it is ignored. + +- Specify `DB_LOG_IN_MEMORY` to the `DB_ENV->log_set_config()` method. Note that you must specify this before your application opens its first environment handle. + +For example: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + + envp = NULL; + + /* Create the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* + * Indicate that logging is to be performed only in memory. + * Doing this means that we give up our transactional durability + * guarantee. + */ + envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); + + /* + * Configure the size of our log memory buffer. This must be + * large enough to hold all the logging information likely + * to be created for our longest running transaction. The + * default size for the logging buffer is 1 MB when logging + * is performed in-memory. For this example, we arbitrarily + * set the logging buffer to 5 MB. + */ + ret = envp->set_lg_bsize(envp, 5 * 1024 * 1024); + if (ret != 0) { + fprintf(stderr, "Error setting log buffer size: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Set the normal flags for a transactional subsystem. Note that + * we DO NOT specify DB_RECOVER. Also, remember that the logging + * subsystem is automatically enabled when we initialize the + * transactional subsystem, so we do not explicitly enable + * logging here. + */ + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + /* Open the environment as normal */ + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * From here, you create transactions and perform database operations + * exactly as you would if you were logging to disk. This part is + * omitted for brevity. + */ + + ... + +err: + /* Close the databases (omitted) */ + + ... + + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +### Setting the In-Memory Log Buffer Size + +When your application is configured for on-disk logging (the default behavior for transactional applications), log information is stored in-memory until the storage space fills up, or a transaction commit forces the log information to be flushed to disk. + +It is possible to increase the amount of memory available to your file log buffer. Doing so improves throughput for long-running transactions, or for transactions that produce a large amount of data. + +When you have your logging subsystem configured to maintain your log entirely in memory (see Configuring In-Memory Logging), it is very important to configure your log buffer size because the log buffer must be capable of holding all log information that can accumulate during the longest running transaction. You must make sure that the in-memory log buffer size is large enough that no transaction will ever span the entire buffer. You must also avoid a state where the in-memory buffer is full and no space can be freed because a transaction that started the first log "file" is still active. + +When your logging subsystem is configured for on-disk logging, the default log buffer space is 32 KB. When in-memory logging is configured, the default log buffer space is 1 MB. + +You can increase your log buffer space using the `DB_ENV->set_lg_bsize()` method. Note that this method can only be called before the first environment handle for your application is opened. diff --git a/docs-src/guides/gsg_txn/logfileremoval.md b/docs-src/guides/gsg_txn/logfileremoval.md new file mode 100644 index 000000000..06c0ec56b --- /dev/null +++ b/docs-src/guides/gsg_txn/logfileremoval.md @@ -0,0 +1,42 @@ +--- +title: "Removing Log Files" +api-name: "Removing Log Files" +source: docs/gsg_txn/C/logfileremoval.html +--- +## Removing Log Files + +By default DB does not delete log files for you. For this reason, DB's log files will eventually grow to consume an unnecessarily large amount of disk space. To guard against this, you should periodically take administrative action to remove log files that are no longer in use by your application. + +You can remove a log file if all of the following are true: + +- the log file is not involved in an active transaction. + +- a checkpoint has been performed *after* the log file was created. + +- the log file is not the only log file in the environment. + +- the log file that you want to remove has already been included in an offline or hot backup. Failure to observe this last condition can cause your backups to be unusable. + +DB provides several mechanisms to remove log files that meet all but the last criteria (DB has no way to know which log files have already been included in a backup). The following mechanisms make it easy to remove unneeded log files, but can result in an unusable backup if the log files are not first saved to your archive location. All of the following mechanisms automatically delete unneeded log files for you: + +- Run the **db_archive** command line utility with the `-d` option. + +- From within your application, call the `DB_ENV->log_archive()` method with the `DB_ARCH_REMOVE` flag. + +- Call `DB_ENV->log_set_config()` method with the `DB_LOG_AUTO_REMOVE` flag. Note that this flag can be set at any point in the lifetime of your application. Setting this parameter affects all environment handles opened against the environment; not just the handle used to set the flag. + + Note that unlike the other log removal mechanisms identified here, this method actually causes log files to be removed on an on-going basis as they become unnecessary. This is extremely desirable behavior if what you want is to use the absolute minimum amount of disk space possible for your application. This mechanism *will* leave you with the log files that are required to run normal recovery. However, it is highly likely that this mechanism will prevent you from running catastrophic recovery. + + Do NOT use this mechanism if you want to be able to perform catastrophic recovery, or if you want to be able to maintain a hot backup. + +In order to safely remove log files and still be able to perform catastrophic recovery, use the **db_archive** command line utility as follows: + +1. Run either a normal or hot backup as described in Backup Procedures. Make sure that all of this data is safely stored to your backup media before continuing. + +2. If you have not already done so, perform a checkpoint. See Checkpoints for more information. + +3. If you are maintaining a hot backup, perform the hot backup procedure as described in Using Hot Failovers. + +4. Run the **db_archive** command line utility with the `-d` option against your production environment. + +5. Run the **db_archive** command line utility with the `-d` option against your failover environment, if you are maintaining one. diff --git a/docs-src/guides/gsg_txn/maxtxns.md b/docs-src/guides/gsg_txn/maxtxns.md new file mode 100644 index 000000000..c2c17a7d4 --- /dev/null +++ b/docs-src/guides/gsg_txn/maxtxns.md @@ -0,0 +1,106 @@ +--- +title: "Configuring the Transaction Subsystem" +api-name: "Configuring the Transaction Subsystem" +source: docs/gsg_txn/C/maxtxns.html +--- +## Configuring the Transaction Subsystem + +Most of the configuration activities that you need to perform for your transactional DB application will involve the locking and logging subsystems. See Concurrency and Managing DB Files for details. + +However, there are a couple of things that you can do to configure your transaction subsystem directly. These things are: + +- + + Configure the maximum number of simultaneous transactions needed by your application. In general, you should not need to do this unless you use deeply nested transactions or you have many threads all of which have active transactions. In addition, you may need to configure a higher maximum number of transactions if you are using snapshot isolation. See Snapshot Isolation Transactional Requirements for details. + + By default, your application can support 20 active transactions. + + You can set the maximum number of simultaneous transactions supported by your application using the `DB_ENV->set_tx_max()` method. Note that this method must be called before the environment has been opened. + + If your application has exceeded this maximum value, then any attempt to begin a new transaction will fail. + + This value can also be set using the `DB_CONFIG` file's `set_tx_max` parameter. Remember that the `DB_CONFIG` must reside in your environment home directory. + +- + + Configure the timeout value for your transactions. This value represents the longest period of time a transaction can be active. Note, however, that transaction timeouts are checked only when DB examines its lock tables for blocked locks (see Locks, Blocks, and Deadlocks for more information). Therefore, a transaction's timeout can have expired, but the application will not be notified until DB has a reason to examine its lock tables. + + Be aware that some transactions may be inappropriately timed out before the transaction has a chance to complete. You should therefore use this mechanism only if you know your application might have unacceptably long transactions and you want to make sure your application will not stall during their execution. (This might happen if, for example, your transaction blocks or requests too much data.) + + Note that by default transaction timeouts are set to 0 seconds, which means that they never time out. + + To set the maximum timeout value for your transactions, use the `DB_ENV->set_timeout()` method. This method configures the entire environment; not just the handle used to set the configuration. Further, this value may be set at any time during the application's lifetime. + + This value can also be set using the `DB_CONFIG` file's `set_txn_timeout` parameter. + +For example: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DB_TXN *txn; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN; /* Initialize transactions */ + + /* + * Configure a maximum transaction timeout of 1 second. + */ + ret = envp->set_timeout(envp, DB_SET_TXN_TIMEOUT, 1000000); + if (ret != 0) { + fprintf(stderr, "Error setting txn timeout: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * Configure 40 maximum transactions. + */ + ret = envp->set_tx_max(envp, 40); + if (ret != 0) { + fprintf(stderr, "Error setting max txns: %s\n", + db_strerror(ret)); + goto err; + } + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* + * From here, you open your databases, proceed with your + * database operations, and respond to deadlocks as + * is normal (omitted for brevity). + */ + ... +``` diff --git a/docs-src/guides/gsg_txn/moreinfo.md b/docs-src/guides/gsg_txn/moreinfo.md new file mode 100644 index 000000000..6fe6f07e9 --- /dev/null +++ b/docs-src/guides/gsg_txn/moreinfo.md @@ -0,0 +1,28 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/gsg_txn/C/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a transactional DB application: + +- Getting Started with Berkeley DB for C + +- Berkeley DB Getting Started with Replicated Applications for C + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB C API Reference Guide + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/gsg_txn/multithread-intro.md b/docs-src/guides/gsg_txn/multithread-intro.md new file mode 100644 index 000000000..96fb0c641 --- /dev/null +++ b/docs-src/guides/gsg_txn/multithread-intro.md @@ -0,0 +1,14 @@ +--- +title: "Multi-threaded and Multi-process Applications" +api-name: "Multi-threaded and Multi-process Applications" +source: docs/gsg_txn/C/multithread-intro.html +--- +## Multi-threaded and Multi-process Applications + +DB is designed to support multi-threaded and multi-process applications, but their usage means you must pay careful attention to issues of concurrency. Transactions help your application's concurrency by providing various levels of isolation for your threads of control. In addition, DB provides mechanisms that allow you to detect and respond to deadlocks. + +*Isolation* means that database modifications made by one transaction will not normally be seen by readers from another transaction until the first commits its changes. Different threads use different transaction handles, so this mechanism is normally used to provide isolation between database operations performed by different threads. + +Note that DB supports different isolation levels. For example, you can configure your application to see uncommitted reads, which means that one transaction can see data that has been modified but not yet committed by another transaction. Doing this might mean your transaction reads data "dirtied" by another transaction, but which subsequently might change before that other transaction commits its changes. On the other hand, lowering your isolation requirements means that your application can experience improved throughput due to reduced lock contention. + +For more information on concurrency, on managing isolation levels, and on deadlock detection, see Concurrency. diff --git a/docs-src/guides/gsg_txn/nestedtxn.md b/docs-src/guides/gsg_txn/nestedtxn.md new file mode 100644 index 000000000..5f4c39098 --- /dev/null +++ b/docs-src/guides/gsg_txn/nestedtxn.md @@ -0,0 +1,34 @@ +--- +title: "Nested Transactions" +api-name: "Nested Transactions" +source: docs/gsg_txn/C/nestedtxn.html +--- +## Nested Transactions + +A *nested transaction* is used to provide a transactional guarantee for a subset of operations performed within the scope of a larger transaction. Doing this allows you to commit and abort the subset of operations independently of the larger transaction. + +The rules to the usage of a nested transaction are as follows: + +- While the nested (child) transaction is active, the parent transaction may not perform any operations other than to commit or abort, or to create more child transactions. + +- Committing a nested transaction has no effect on the state of the parent transaction. The parent transaction is still uncommitted. However, the parent transaction can now see any modifications made by the child transaction. Those modifications, of course, are still hidden to all other transactions until the parent also commits. + +- Likewise, aborting the nested transaction has no effect on the state of the parent transaction. The only result of the abort is that neither the parent nor any other transactions will see any of the database modifications performed under the protection of the nested transaction. + +- If the parent transaction commits or aborts while it has active children, the child transactions are resolved in the same way as the parent. That is, if the parent aborts, then the child transactions abort as well. If the parent commits, then whatever modifications have been performed by the child transactions are also committed. + +- The locks held by a nested transaction are not released when that transaction commits. Rather, they are now held by the parent transaction until such a time as that parent commits. + +- Any database modifications performed by the nested transaction are not visible outside of the larger encompassing transaction until such a time as that parent transaction is committed. + +- The depth of the nesting that you can achieve with nested transaction is limited only by memory. + +To create a nested transaction, simply pass the parent transaction's handle when you created the nested transaction's handle. For example: + +``` c + /* parent transaction */ + DB_TXN *parent_txn, *child_txn; + ret = envp->txn_begin(envp, NULL, &parent_txn, 0); + /* child transaction */ + ret = envp->txn_begin(envp, parent_txn, &child_txn, 0); +``` diff --git a/docs-src/guides/gsg_txn/nodurabletxn.md b/docs-src/guides/gsg_txn/nodurabletxn.md new file mode 100644 index 000000000..57fd4e3c7 --- /dev/null +++ b/docs-src/guides/gsg_txn/nodurabletxn.md @@ -0,0 +1,28 @@ +--- +title: "Non-Durable Transactions" +api-name: "Non-Durable Transactions" +source: docs/gsg_txn/C/nodurabletxn.html +--- +## Non-Durable Transactions + +As previously noted, by default transaction commits are durable because they cause the modifications performed under the transaction to be synchronously recorded in your on-disk log files. However, it is possible to use non-durable transactions. + +You may want non-durable transactions for performance reasons. For example, you might be using transactions simply for the isolation guarantee. In this case, you might not want a durability guarantee and so you may want to prevent the disk I/O that normally accompanies a transaction commit. + +There are several ways to remove the durability guarantee for your transactions: + +- Specify `DB_TXN_NOSYNC` using the `DB_ENV->set_flags()` method. This causes DB to not synchronously force any log data to disk upon transaction commit. That is, the modifications are held entirely in the in-memory cache and the logging information is not forced to the filesystem for long-term storage. Note, however, that the logging data will eventually make it to the filesystem (assuming no application or OS crashes) as a part of DB's management of its logging buffers and/or cache. + + This form of a commit provides a weak durability guarantee because data loss can occur due to an application or OS crash. + + This behavior is specified on a per-environment handle basis. In order for your application to exhibit consistent behavior, you need to specify this flag for all of the environment handles used in your application. + + You can achieve this behavior on a transaction by transaction basis by specifying `DB_TXN_NOSYNC` to the `DB_TXN->commit()` method. + +- Specify `DB_TXN_WRITE_NOSYNC` using the `DB_ENV->set_flags()` method. This causes logging data to be synchronously written to the OS's file system buffers upon transaction commit. The data will eventually be written to disk, but this occurs when the operating system chooses to schedule the activity; the transaction commit can complete successfully before this disk I/O is performed by the OS. + + This form of commit protects you against application crashes, but not against OS crashes. This method offers less room for the possibility of data loss than does `DB_TXN_NOSYNC`. + + This behavior is specified on a per-environment handle basis. In order for your application to exhibit consistent behavior, you need to specify this flag for all of the environment handles used in your application. + +- Maintain your logs entirely in-memory. In this case, your logs are never written to disk. The result is that you lose all durability guarantees. See Configuring In-Memory Logging for more information. diff --git a/docs-src/guides/gsg_txn/perftune-intro.md b/docs-src/guides/gsg_txn/perftune-intro.md new file mode 100644 index 000000000..d36ba1a6e --- /dev/null +++ b/docs-src/guides/gsg_txn/perftune-intro.md @@ -0,0 +1,10 @@ +--- +title: "Performance Tuning" +api-name: "Performance Tuning" +source: docs/gsg_txn/C/perftune-intro.html +--- +## Performance Tuning + +From a performance perspective, the use of transactions is not free. Depending on how you configure them, transaction commits usually require your application to perform disk I/O that a non-transactional application does not perform. Also, for multi-threaded and multi-process applications, the use of transactions can result in increased lock contention due to extra locking requirements driven by transactional isolation guarantees. + +There is therefore a performance tuning component to transactional applications that is not applicable for non-transactional applications (although some tuning considerations do exist whether or not your application uses transactions). Where appropriate, these tuning considerations are introduced in the following chapters. However, for a more complete description of them, see the Transaction tuning and Transaction throughput sections of the *Berkeley DB Programmer's Reference Guide*. diff --git a/docs-src/guides/gsg_txn/preface.md b/docs-src/guides/gsg_txn/preface.md new file mode 100644 index 000000000..533a32de5 --- /dev/null +++ b/docs-src/guides/gsg_txn/preface.md @@ -0,0 +1,62 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/gsg_txn/C/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +This document describes how to use transactions with your Berkeley DB applications. It is intended to describe how to transaction protect your application's data. The APIs used to perform this task are described here, as are the environment infrastructure and administrative tasks required by a transactional application. This book also describes multi-threaded and multi-process DB applications and the requirements they have for deadlock detection. + +This book describes Berkeley DB 11*g* Release 2, which provides library version 11.2.5.3. + +This book is aimed at the software engineer responsible for writing a transactional DB application. + +This book assumes that you have already read and understood the concepts contained in the *Getting Started with Berkeley DB* guide. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Structure names are represented in `monospaced font`, as are `method names`. For example: "`DB->open()` is a method on a `DB` handle." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +In some situations, programming examples are updated from one chapter to the next. When this occurs, the new code is presented in **`monospaced bold`** font. For example: + +``` c +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + DB *itemname_sdbp; /* Index based on the item name index */ + char *db_home_dir; /* Directory containing the database files */ + char *itemname_db_name; /* Itemname secondary database */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of special interest are represented using a note block such as this. diff --git a/docs-src/guides/gsg_txn/readmodifywrite.md b/docs-src/guides/gsg_txn/readmodifywrite.md new file mode 100644 index 000000000..6161617b6 --- /dev/null +++ b/docs-src/guides/gsg_txn/readmodifywrite.md @@ -0,0 +1,42 @@ +--- +title: "Read/Modify/Write" +api-name: "Read/Modify/Write" +source: docs/gsg_txn/C/readmodifywrite.html +--- +## Read/Modify/Write + +If you are retrieving a record from the database for the purpose of modifying or deleting it, you should declare a read-modify-write cycle at the time that you read the record. Doing so causes DB to obtain write locks (instead of a read locks) at the time of the read. This helps to prevent deadlocks by preventing another transaction from acquiring a read lock on the same record while the read-modify-write cycle is in progress. + +Note that declaring a read-modify-write cycle may actually increase the amount of blocking that your application sees, because readers immediately obtain write locks and write locks cannot be shared. For this reason, you should use read-modify-write cycles only if you are seeing a large amount of deadlocking occurring in your application. + +In order to declare a read/modify/write cycle when you perform a read operation, pass the `DB_RMW` flag to the database or cursor get method. + +For example: + +``` c +retry: + /* Get the transaction */ + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn_begin failed"); + return (EXIT_FAILURE); + } + ... + /* key and data are Dbts. Their usage is omitted for brevity. */ + ... + /* Get the data. Declare the read/modify/write cycle here */ + ret = dbp->get(dbp, txn, &key, &data, DB_RMW); + + ... + /* Modify the key and data as is required (not shown here) */ + ... + + /* Put the data. Note that you do not have to provide any additional + * flags here due to the read/modify/write cycle. Simply put the + * data and perform your deadlock detection as normal. + */ + ret = dbp->put(dbp, txn, &key, &data, 0); + switch (ret) { + /* Deadlock detection omitted for brevity */ + .... +``` diff --git a/docs-src/guides/gsg_txn/recovery-intro.md b/docs-src/guides/gsg_txn/recovery-intro.md new file mode 100644 index 000000000..d73e93976 --- /dev/null +++ b/docs-src/guides/gsg_txn/recovery-intro.md @@ -0,0 +1,18 @@ +--- +title: "Recoverability" +api-name: "Recoverability" +source: docs/gsg_txn/C/recovery-intro.html +--- +## Recoverability + +An important part of DB's transactional guarantees is durability. *Durability* means that once a transaction has been committed, the database modifications performed under its protection will not be lost due to system failure. + +In order to provide the transactional durability guarantee, DB uses a write-ahead logging system. Every operation performed on your databases is described in a log before it is performed on your databases. This is done in order to ensure that an operation can be recovered in the event of an untimely application or system failure. + +Beyond logging, another important aspect of durability is recoverability. That is, backup and restore. DB supports a normal recovery that runs against a subset of your log files. This is a routine procedure used whenever your environment is first opened upon application startup, and it is intended to ensure that your database is in a consistent state. DB also supports archival backup and recovery in the case of catastrophic failure, such as the loss of a physical disk drive. + +This book describes several different backup procedures you can use to protect your on-disk data. These procedures range from simple offline backup strategies to hot failovers. Hot failovers provide not only a backup mechanism, but also a way to recover from a fatal hardware failure. + +This book also describes the recovery procedures you should use for each of the backup strategies that you might employ. + +For a detailed description of backup and restore procedures, see Managing DB Files. diff --git a/docs-src/guides/gsg_txn/recovery.md b/docs-src/guides/gsg_txn/recovery.md new file mode 100644 index 000000000..20d3a72c9 --- /dev/null +++ b/docs-src/guides/gsg_txn/recovery.md @@ -0,0 +1,166 @@ +--- +title: "Recovery Procedures" +api-name: "Recovery Procedures" +source: docs/gsg_txn/C/recovery.html +--- +## Recovery Procedures + + [Normal Recovery](recovery.md#normalrecovery) + + [Catastrophic Recovery](recovery.md#catastrophicrecovery) + +DB supports two types of recovery: + +- Normal recovery, which is run when your environment is opened upon application startup, examines only those log records needed to bring the databases to a consistent state since the last checkpoint. Normal recovery starts with any logs used by any transactions active at the time of the last checkpoint, and examines all logs from then to the current logs. + +- Catastrophic recovery, which is performed in the same way that normal recovery is except that it examines all available log files. You use catastrophic recovery to restore your databases from a previously created backup. + +Of these two, normal recovery should be considered a routine matter; in fact you should run normal recovery whenever you start up your application. + +Catastrophic recovery is run whenever you have lost or corrupted your database files and you want to restore from a backup. You also run catastrophic recovery when you create a hot backup (see Using Hot Failovers for more information). + +### Normal Recovery + +Normal recovery examines the contents of your environment's log files, and uses this information to ensure that your database files are consistent relative to the information contained in the log files. + +Normal recovery also recreates your environment's region files. This has the desired effect of clearing any unreleased locks that your application may have held at the time of an unclean application shutdown. + +Normal recovery is run only against those log files created since the time of your last checkpoint. For this reason, your recovery time is dependent on how much data has been written since the last checkpoint, and therefore on how much log file information there is to examine. If you run checkpoints infrequently, then normal recovery can take a relatively long time. + +### Note + +You should run normal recovery every time you perform application startup. + +To run normal recovery: + +- Make sure all your environment handles are closed. + +- Normal recovery *must be* single-threaded. + +- Provide the `DB_RECOVER` flag when you open your environment. + +You can also run recovery by pausing or shutting down your application and using the **db_recover** command line utility. + +For example: + +``` c +#include +#include +#include "db.h" + +int +main(void) +{ + int ret; + u_int32_t env_flags; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* Open the environment, specifying that recovery is to be run. */ + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_RECOVER; /* Run normal recovery */ + + /* Open the environment. */ + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + ... + /* + * All other operations are identical from here. Notice, however, + * that we have not created any other threads of control before + * recovery is complete. You want to run recovery for + * the first thread in your application that opens an environment, + * but not for any subsequent threads. + */ +``` + +### Catastrophic Recovery + +Use catastrophic recovery when you are recovering your databases from a previously created backup. Note that to restore your databases from a previous backup, you should copy the backup to a new environment directory, and then run catastrophic recovery. Failure to do so can lead to the internal database structures being out of sync with your log files. + +Catastrophic recovery must be run single-threaded. + +To run catastrophic recovery: + +- Shutdown all database operations. + +- Restore the backup to an empty directory. + +- Provide the `DB_RECOVER_FATAL` flag when you open your environment. This environment open must be single-threaded. + +You can also run recovery by pausing or shutting down your application and using the **db_recover** command line utility with the the `-c` option. + +Note that catastrophic recovery examines every available log file — not just those log files created since the last checkpoint as is the case for normal recovery. For this reason, catastrophic recovery is likely to take longer than does normal recovery. + +For example: + +``` c +#include +#include +#include "db.h" + +int +main(void) +{ + int ret; + u_int32_t env_flags; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* Open the environment, specifying that recovery is to be run. */ + env_flags = DB_CREATE | /* If the environment does not + * exist, create it. */ + DB_INIT_LOCK | /* Initialize locking */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL | /* Initialize the cache */ + DB_THREAD | /* Free-thread the env handle. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_RECOVER_FATAL; /* Run catastrophic recovery */ + + /* Open the environment. */ + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + ... + /* + * All other operations are identical from here. Notice, however, + * that we have not created any other threads of control before + * recovery is complete. You want to run recovery for + * the first thread in your application that opens an environment, + * but not for any subsequent threads. + */ +``` diff --git a/docs-src/guides/gsg_txn/reversesplit.md b/docs-src/guides/gsg_txn/reversesplit.md new file mode 100644 index 000000000..e4806bb8e --- /dev/null +++ b/docs-src/guides/gsg_txn/reversesplit.md @@ -0,0 +1,111 @@ +--- +title: "Reverse BTree Splits" +api-name: "Reverse BTree Splits" +source: docs/gsg_txn/C/reversesplit.html +--- +## Reverse BTree Splits + +If your application is using the Btree access method, and your application is repeatedly deleting then adding records to your database, then you might be able to reduce lock contention by turning off reverse Btree splits. + +As pages are emptied in a database, DB attempts to delete empty pages in order to keep the database as small as possible and minimize search time. Moreover, when a page in the database fills up, DB, of course, adds additional pages to make room for more data. + +Adding and deleting pages in the database requires that the writing thread lock the parent page. Consequently, as the number of pages in your database diminishes, your application will see increasingly more lock contention; the maximum level of concurrency in a database of two pages is far smaller than that in a database of 100 pages, because there are fewer pages that can be locked. + +Therefore, if you prevent the database from being reduced to a minimum number of pages, you can improve your application's concurrency throughput. Note, however, that you should do so only if your application tends to delete and then add the same data. If this is not the case, then preventing reverse Btree splits can harm your database search time. + +To turn off reverse Btree splits, provide the `DB_REVSPLITOFF` flag to the `DB->set_flags()` method. + +For example: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | + DB_INIT_LOCK | + DB_INIT_LOG | + DB_INIT_TXN | + DB_THREAD | + DB_INIT_MPOOL; + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + /* Set btree reverse split to off */ + ret = db->set_flags(&db, DB_REVSPLITOFF); + if (ret != 0) { + envp->err(envp, ret, "Turning off Btree reverse split failed"); + goto err; + } + + db_flags = DB_CREATE | DB_AUTO_COMMIT; + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + +err: + /* Close the database */ + if (dbp != NULL) { + ret_c = dbp->close(dbp, 0); + if (ret_c != 0) { + envp->err(envp, ret_c, "Database close failed."); + ret = ret_c + } + } + + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` diff --git a/docs-src/guides/gsg_txn/sysfailure.md b/docs-src/guides/gsg_txn/sysfailure.md new file mode 100644 index 000000000..e045aeec6 --- /dev/null +++ b/docs-src/guides/gsg_txn/sysfailure.md @@ -0,0 +1,18 @@ +--- +title: "A Note on System Failure" +api-name: "A Note on System Failure" +source: docs/gsg_txn/C/sysfailure.html +--- +## A Note on System Failure + +From time to time this manual mentions that transactions protect your data against 'system or application failure.' This is true up to a certain extent. However, not all failures are created equal and no data protection mechanism can protect you against every conceivable way a computing system can find to die. + +Generally, when this book talks about protection against failures, it means that transactions offer protection against the likeliest culprits for system and application crashes. So long as your data modifications have been committed to disk, those modifications should persist even if your application or OS subsequently fails. And, even if the application or OS fails in the middle of a transaction commit (or abort), the data on disk should be either in a consistent state, or there should be enough data available to bring your databases into a consistent state (via a recovery procedure, for example). You may, however, lose whatever data you were committing at the time of the failure, but your databases will be otherwise unaffected. + +### Note + +Be aware that many disks have a disk write cache and on some systems it is enabled by default. This means that a transaction can have committed, and to your application the data may appear to reside on disk, but the data may in fact reside only in the write cache at that time. This means that if the disk write cache is enabled and there is no battery backup for it, data can be lost after an OS crash even when maximum durability mode is in use. For maximum durability, disable the disk write cache or use a disk write cache with a battery backup. + +Of course, if your *disk* fails, then the transactional benefits described in this book are only as good as the backups you have taken. By spreading your data and log files across separate disks, you can minimize the risk of data loss due to a disk failure, but even in this case it is possible to conjure a scenario where even this protection is insufficient (a fire in the machine room, for example) and you must go to your backups for protection. + +Finally, by following the programming examples shown in this book, you can write your code so as to protect your data in the event that your code crashes. However, no programming API can protect you against logic failures in your own code; transactions cannot protect you from simply writing the wrong thing to your databases. diff --git a/docs-src/guides/gsg_txn/txn_ccursor.md b/docs-src/guides/gsg_txn/txn_ccursor.md new file mode 100644 index 000000000..becd585e2 --- /dev/null +++ b/docs-src/guides/gsg_txn/txn_ccursor.md @@ -0,0 +1,117 @@ +--- +title: "Transactional Cursors and Concurrent Applications" +api-name: "Transactional Cursors and Concurrent Applications" +source: docs/gsg_txn/C/txn_ccursor.html +--- +## Transactional Cursors and Concurrent Applications + + [Using Cursors with Uncommitted Data](txn_ccursor.md#cursordirtyreads) + +When you use transactional cursors with a concurrent application, remember that in the event of a deadlock you must make sure that you close your cursor before you abort and retry your transaction. + +Also, remember that when you are using the default isolation level, every time your cursor reads a record it locks that record until the encompassing transaction is resolved. This means that walking your database with a transactional cursor increases the chance of lock contention. + +For this reason, if you must routinely walk your database with a transactional cursor, consider using a reduced isolation level such as read committed. + +### Using Cursors with Uncommitted Data + +As described in Reading Uncommitted Data above, it is possible to relax your transaction's isolation level such that it can read data modified but not yet committed by another transaction. You can configure this when you create your transaction handle, and when you do so then all cursors opened inside that transaction will automatically use uncommitted reads. + +You can also do this when you create a cursor handle from within a serializable transaction. When you do this, only those cursors configured for uncommitted reads uses uncommitted reads. + +Either way, you must first configure your database handle to support uncommitted reads before you can configure your transactions or your cursors to use them. + +The following example shows how to configure an individual cursor handle to read uncommitted data from within a serializable (full isolation) transaction. For an example of configuring a transaction to perform uncommitted reads in general, see Reading Uncommitted Data. + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + DB *dbp; + DBC *cursorp; + DB_ENV *envp; + DB_TXN *txn; + int ret, c_ret; + char *replacementString = "new string"; + + dbp = NULL; + envp = NULL; + txn = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* Create the environment if it does + * not already exist. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_INIT_LOCK | /* Initialize locking. */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL; /* Initialize the in-memory cache. */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + db_flags = DB_CREATE | /* Create the db if it does not + * exist */ + DB_AUTO_COMMIT | /* Enable auto commit */ + DB_READ_UNCOMMITTED; /* Enable uncommitted reads */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + + /* Get the txn handle */ + /* Note that this is a degree 3 transaction */ + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction begin failed."); + goto err; + } + + /* Get the cursor, supply the txn handle at that time */ + /* Cause the cursor to perform uncommitted reads */ + ret = dbp->cursor(dbp, txn, &cursorp, DB_READ_UNCOMMITTED); + if (ret != 0) { + envp->err(envp, ret, "Cursor open failed."); + txn->abort(txn); + goto err; + } + + /* + * From here, you perform your cursor reads and writes as normal, + * committing and aborting the transactions as is necessary, and + * testing for deadlock exceptions as normal (omitted for brevity). + */ + + ... +``` diff --git a/docs-src/guides/gsg_txn/txnconcurrency.md b/docs-src/guides/gsg_txn/txnconcurrency.md new file mode 100644 index 000000000..2f4208d8e --- /dev/null +++ b/docs-src/guides/gsg_txn/txnconcurrency.md @@ -0,0 +1,104 @@ +--- +title: "Chapter 4. Concurrency" +api-name: "Chapter 4. Concurrency" +source: docs/gsg_txn/C/txnconcurrency.html +--- +## Chapter 4. Concurrency + +**Table of Contents** + + [Which DB Handles are Free-Threaded](txnconcurrency.md#concurrenthandles) + + [Locks, Blocks, and Deadlocks](blocking_deadlocks.md) + + [Locks](blocking_deadlocks.md#locks) + + [Blocks](blocking_deadlocks.md#blocks) + + [Deadlocks](blocking_deadlocks.md#deadlocks) + + [The Locking Subsystem](lockingsubsystem.md) + + [Configuring the Locking Subsystem](lockingsubsystem.md#configuringlock) + + [Configuring Deadlock Detection](lockingsubsystem.md#configdeadlkdetect) + + [Resolving Deadlocks](lockingsubsystem.md#deadlockresolve) + + [Setting Transaction Priorities](lockingsubsystem.md#setpriority) + + [Isolation](isolation.md) + + [Supported Degrees of Isolation](isolation.md#degreesofisolation) + + [Reading Uncommitted Data](isolation.md#dirtyreads) + + [Committed Reads](isolation.md#readcommitted) + + [Using Snapshot Isolation](isolation.md#snapshot_isolation) + + [Transactional Cursors and Concurrent Applications](txn_ccursor.md) + + [Using Cursors with Uncommitted Data](txn_ccursor.md#cursordirtyreads) + + [Exclusive Database Handles](exclusivelock.md) + + [Read/Modify/Write](readmodifywrite.md) + + [No Wait on Blocks](txnnowait.md) + + [Reverse BTree Splits](reversesplit.md) + +DB offers a great deal of support for multi-threaded and multi-process applications even when transactions are not in use. Many of DB's handles are thread-safe, or can be made thread-safe by providing the appropriate flag at handle creation time, and DB provides a flexible locking subsystem for managing databases in a concurrent application. Further, DB provides a robust mechanism for detecting and responding to deadlocks . All of these concepts are explored in this chapter. + +Before continuing, it is useful to define a few terms that will appear throughout this chapter: + +- *Thread of control* + + Refers to a thread that is performing work in your application. Typically, in this book that thread will be performing DB operations. + + Note that this term can also be taken to mean a separate process that is performing work — DB supports multi-process operations on your databases. + + Also, DB is agnostic with regard to the type or style of threads in use in your application. So if you are using multiple threads (as opposed to multiple processes) to perform concurrent database access, you are free to use whatever thread package is best for your platform and application. That said, this manual will use pthreads for its threading examples because those have the best chance of being supported across a large range of platforms. + +- *Locking* + + When a thread of control obtains access to a shared resource, it is said to be *locking* that resource. Note that DB supports both exclusive and non-exclusive locks. See Locks for more information. + +- *Free-threaded* + + Data structures and objects are free-threaded if they can be shared across threads of control without any explicit locking on the part of the application. Some books, libraries, and programming languages may use the term *thread-safe* for data structures or objects that have this characteristic. The two terms mean the same thing. + + For a description of free-threaded DB objects, see Which DB Handles are Free-Threaded. + +- *Blocked* + + When a thread cannot obtain a lock because some other thread already holds a lock on that object, the lock attempt is said to be *blocked*. See Blocks for more information. + +- *Deadlock* + + Occurs when two or more threads of control attempt to access conflicting resource in such a way as none of the threads can any longer make further progress. + + For example, if Thread A is blocked waiting for a resource held by Thread B, while at the same time Thread B is blocked waiting for a resource held by Thread A, then neither thread can make any forward progress. In this situation, Thread A and Thread B are said to be *deadlocked.* + + For more information, see Deadlocks. + +## Which DB Handles are Free-Threaded + +The following describes to what extent and under what conditions individual handles are free-threaded. + +- `DB_ENV` + + Free-threaded so long as the `DB_THREAD` flag is provided to the environment `open()` method. + +- `DB` + + Free-threaded so long as the `DB_THREAD` flag is provided to the database `open()` method, or if the database is opened using a free-threaded environment handle. + +- `DBC` + + Cursors are not free-threaded. However, they can be used by multiple threads of control so long as the application serializes access to the handle. + +- `DB_TXN` + + Access must be serialized by the application across threads of control. diff --git a/docs-src/guides/gsg_txn/txncursor.md b/docs-src/guides/gsg_txn/txncursor.md new file mode 100644 index 000000000..54c069afa --- /dev/null +++ b/docs-src/guides/gsg_txn/txncursor.md @@ -0,0 +1,99 @@ +--- +title: "Transactional Cursors" +api-name: "Transactional Cursors" +source: docs/gsg_txn/C/txncursor.html +--- +## Transactional Cursors + +You can transaction-protect your cursor operations by specifying a transaction handle at the time that you create your cursor. Beyond that, you do not ever provide a transaction handle directly to a cursor method. + +Note that if you transaction-protect a cursor, then you must make sure that the cursor is closed before you either commit or abort the transaction. For example: + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + DBT key, data; + DBC *cursorp; + DB_TXN *txn = NULL; + int ret, c_ret; + char *replacementString = "new string"; + + ... + /* environment and db handle creation omitted */ + ... + + /* Get the txn handle */ + txn = NULL; + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction begin failed."); + goto err; + } + + /* Get the cursor, supply the txn handle at that time */ + ret = dbp->cursor(dbp, txn, &cursorp, 0); + if (ret != 0) { + envp->err(envp, ret, "Cursor open failed."); + txn->abort(txn); + goto err; + } + + /* + * Now use the cursor. Note that we do not supply any txn handles to + * these methods. + */ + /* Prepare the DBTs */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + while (cursor->get(&key, &data, DB_NEXT) == 0) { + data->data = (void *)replacementString; + data->size = (strlen(replacementString) + 1) * sizeof(char); + c_ret = cursor->put(cursor, &key, &data, DB_CURRENT); + if (c_ret != 0) { + /* abort the transaction and goto error */ + envp->err(envp, ret, "Cursor put failed."); + cursorp->close(cursorp); + cursorp = NULL; + txn->abort(txn); + goto err; + } + } + + /* + * Commit the transaction. Note that the transaction handle + * can no longer be used. + */ + ret = cursorp->close(cursorp); + if (ret != 0) { + envp->err(envp, ret, "Cursor close failed."); + txn->abort(txn); + goto err; + } + ret = txn->commit(txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction commit failed."); + goto err; + } + +err: + /* Close the cursor (if the handle is not NULL) + * and perform whatever other cleanup is required */ + + /* Close the database */ + + /* Close the environment */ + + ... + + if (c_ret != 0) + ret = c_ret; + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` diff --git a/docs-src/guides/gsg_txn/txnexample_c.md b/docs-src/guides/gsg_txn/txnexample_c.md new file mode 100644 index 000000000..d02a470c9 --- /dev/null +++ b/docs-src/guides/gsg_txn/txnexample_c.md @@ -0,0 +1,533 @@ +--- +title: "Transaction Example" +api-name: "Transaction Example" +source: docs/gsg_txn/C/txnexample_c.html +--- +## Transaction Example + +The following code provides a fully functional example of a multi-threaded transactional DB application. For improved portability across platforms, this examples uses pthreads to provide threading support. + +The example opens an environment and database and then creates 5 threads, each of which writes 500 records to the database. The keys used for these writes are pre-determined strings, while the data is a random value. This means that the actual data is arbitrary and therefore uninteresting; we picked it only because it requires minimum code to implement and therefore will stay out of the way of the main points of this example. + +Each thread writes 10 records under a single transaction before committing and writing another 10 (this is repeated 50 times). At the end of each transaction, but before committing, each thread calls a function that uses a cursor to read every record in the database. We do this in order to make some points about database reads in a transactional environment. + +Of course, each writer thread performs deadlock detection as described in this manual. In addition, normal recovery is performed when the environment is opened. + +We start with our normal `include` directives: + +``` c +/* File: txn_guide.c */ + +/* We assume an ANSI-compatible compiler */ +#include +#include +#include +#include +#include + +#ifdef _WIN32 +extern int getopt(int, char * const *, const char *); +#else +#include +#endif +``` + +We also need a directive that we use to identify how many threads we want our program to create: + +``` c +/* Run 5 writers threads at a time. */ +#define NUMWRITERS 5 +``` + +Next we declare a couple of global variables (used by our threads), and we provide our forward declarations for the functions used by this example. + +``` c +/* + * Printing of pthread_t is implementation-specific, so we + * create our own thread IDs for reporting purposes. + */ +int global_thread_num; +pthread_mutex_t thread_num_lock; + +/* Forward declarations */ +int count_records(DB *, DB_TXN *); +int open_db(DB **, const char *, const char *, DB_ENV *, u_int32_t); +int usage(void); +void *writer_thread(void *); +``` + +We now implement our usage function, which identifies our only command line parameter: + +``` c +/* Usage function */ +int +usage() +{ + fprintf(stderr, " [-h ]\n"); + return (EXIT_FAILURE); +} +``` + +With that, we have finished up our program's housekeeping, and we can now move on to the main part of our program. As usual, we begin with `main()`. First we declare all our variables, and then we initialize our DB handles. + +``` c +int +main(int argc, char *argv[]) +{ + /* Initialize our handles */ + DB *dbp = NULL; + DB_ENV *envp = NULL; + + pthread_t writer_threads[NUMWRITERS]; + int ch, i, ret, ret_t; + u_int32_t env_flags; + char *db_home_dir; + /* Application name */ + const char *prog_name = "txn_guide"; + /* Database file name */ + const char *file_name = "mydb.db"; +``` + +Now we need to parse our command line. In this case, all we want is to know where our environment directory is. If the `-h` option is not provided when this example is run, the current working directory is used instead. + +``` c + /* Parse the command line arguments */ +#ifdef _WIN32 + db_home_dir = ".\\"; +#else + db_home_dir = "./"; +#endif + while ((ch = getopt(argc, argv, "h:")) != EOF) + switch (ch) { + case 'h': + db_home_dir = optarg; + break; + case '?': + default: + return (usage()); + } +``` + +Next we create our database handle, and we define our environment open flags. There are a few things to notice here: + +- We specify `DB_RECOVER`, which means that normal recovery is run every time we start the application. This is highly desirable and recommended for most applications. + +- We also specify `DB_THREAD`, which means our environment handle will be free-threaded. This is very important because we will be sharing the environment handle across threads. + +``` c + /* Create the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + env_flags = + DB_CREATE | /* Create the environment if it does not exist */ + DB_RECOVER | /* Run normal recovery. */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_TXN | /* Initialize the transactional subsystem. This + * also turns on logging. */ + DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ + DB_THREAD; /* Cause the environment to be free-threaded */ +``` + +Now we configure how we want deadlock detection performed. In our case, we will cause DB to perform deadlock detection by walking its internal lock tables looking for a block every time a lock is requested. Further, in the event of a deadlock, the thread that holds the youngest lock will receive the deadlock notification. + +### Note + +You will notice that every database operation checks the operation's status return code, and if an error (non-zero) status is returned, we log the error and then go to a `err` label in our program. Unlike object-oriented programs such as C++ or Java, we do not have `try` blocks in C. Therefore, this is the best way for us to implement cascading error handling for this example. + +``` c + /* + * Indicate that we want db to perform lock detection internally. + * Also indicate that the transaction with the fewest number of + * write locks will receive the deadlock notification in + * the event of a deadlock. + */ + ret = envp->set_lk_detect(envp, DB_LOCK_MINWRITE); + if (ret != 0) { + fprintf(stderr, "Error setting lock detect: %s\n", + db_strerror(ret)); + goto err; + } +``` + +Now we open our environment. + +``` c + /* + * If we had utility threads (for running checkpoints or + * deadlock detection, for example) we would spawn those + * here. However, for a simple example such as this, + * that is not required. + */ + + /* Now actually open the environment */ + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } +``` + +Now we call the function that will open our database for us. This is not very interesting, except that you will notice that we are specifying `DB_DUPSORT`. This is required purely by the data that we are writing to the database, and it is only necessary if you run the application more than once without first deleting the environment. + +Also, we do not provide any error logging here because the `open_db()` function does that for us. (The implementation of `open_db()` is described later in this section.) + +``` c + /* Open the database */ + ret = open_db(&dbp, prog_name, file_name, envp, DB_DUPSORT); + if (ret != 0) + goto err; +``` + +Now we create our threads. In this example we are using pthreads for our threading package. A description of threading (beyond how it impacts DB usage) is beyond the scope of this manual. However, the things that we are doing here should be familiar to anyone who has prior experience with any threading package. We are simply initializing a mutex, creating our threads, and then joining our threads, which causes our program to wait until the joined threads have completed before continuing operations in the main thread. + +``` c + /* Initialize a pthread mutex. Used to help provide thread ids. */ + (void)pthread_mutex_init(&thread_num_lock, NULL); + + /* Start the writer threads. */ + for (i = 0; i < NUMWRITERS; i++) + (void)pthread_create(&writer_threads[i], NULL, + writer_thread, (void *)dbp); + + /* Join the writers */ + for (i = 0; i < NUMWRITERS; i++) + (void)pthread_join(writer_threads[i], NULL); +``` + +Finally, to wrap up `main()`, we close out our database and environment handle, as is normal for any DB application. Notice that this is where our `err` label is placed in our application. If any database operation prior to this point in the program returns an error status, the program simply jumps to this point and closes our handles if necessary before exiting the application completely. + +``` c +err: + /* Close our database handle, if it was opened. */ + if (dbp != NULL) { + ret_t = dbp->close(dbp, 0); + if (ret_t != 0) { + fprintf(stderr, "%s database close failed: %s\n", + file_name, db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Close our environment, if it was opened. */ + if (envp != NULL) { + ret_t = envp->close(envp, 0); + if (ret_t != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Final status message and return. */ + printf("I'm all done.\n"); + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +Now that we have completed `main()`, we need to implement the function that our writer threads will actually run. This is where the bulk of our transactional code resides. + +We start as usual with variable declarations and initialization. + +``` c +/* + * A function that performs a series of writes to a + * Berkeley DB database. The information written + * to the database is largely nonsensical, but the + * mechanisms of transactional commit/abort and + * deadlock detection are illustrated here. + */ +void * +writer_thread(void *args) +{ + DBT key, value; + DB_TXN *txn; + int i, j, payload, ret, thread_num; + int retry_count, max_retries = 20; /* Max retry on a deadlock */ + char *key_strings[] = {"key 1", "key 2", "key 3", "key 4", + "key 5", "key 6", "key 7", "key 8", + "key 9", "key 10"}; + + DB *dbp = (DB *)args; + DB_ENV *envp = dbp->get_env(dbp); +``` + +Now we want a thread number for reporting purposes. It is possible to use the `pthread_t` value directly for this purpose, but how that is done unfortunately differs depending on the pthread implementation you are using. So instead we use a mutex-protected global variable to obtain a simple integer for our reporting purposes. + +Note that we are also use this thread id for initializing a random number generator, which we do here. We use this random number generator for data generation. + +``` c + /* Get the thread number */ + (void)pthread_mutex_lock(&thread_num_lock); + global_thread_num++; + thread_num = global_thread_num; + (void)pthread_mutex_unlock(&thread_num_lock); + + /* Initialize the random number generator */ + srand((u_int)pthread_self()); +``` + +Now we begin the loop that we use to write data to the database. Notice that at the beginning of the top loop, we begin a new transaction. We will actually use 50 transactions per writer thread, although we will only ever have one active transaction per thread at a time. Within each transaction, we will perform 10 database writes. + +By combining multiple writes together under a single transaction, we increase the likelihood that a deadlock will occur. Normally, you want to reduce the potential for a deadlock and in this case the way to do that is to perform a single write per transaction. To avoid deadlocks, we could be using auto commit to write to our database for this workload. + +However, we want to show deadlock handling and by performing multiple writes per transaction we can actually observe deadlocks occurring. We also want to underscore the idea that you can combing multiple database operations together in a single atomic unit of work in order to improve the efficiency of your writes. + +Finally, on an issue of style, you will notice the `retry` label that we place immediately before our transaction begin code. We use this to loop in the event that a deadlock is detected and the write operation has to be performed. A great many people dislike looping with `goto` statements, and we certainly could have written this code to avoid it. However, we find that using the `goto` in this case greatly helps to clarify the code, so we ignore the bias against `goto` programming in order to clearly support looping in the event of what is really an error condition. + +``` c + /* Write 50 times and then quit */ + for (i = 0; i < 50; i++) { + retry_count = 0; /* Used for deadlock retries */ + +retry: + /* Begin our transaction. */ + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn_begin failed"); + return ((void *)EXIT_FAILURE); + } +``` + +Now we begin the inner loop that we use to actually perform the write. Notice that we use a `case` statement to examine the return code from the database put. This case statement is what we use to determine whether we need to abort (or abort/retry in the case of a deadlock) our current transaction. + +``` c + for (j = 0; j < 10; j++) { + /* Set up our key and values DBTs */ + memset(&key, 0, sizeof(DBT)); + key.data = key_strings[j]; + key.size = (strlen(key_strings[j]) + 1) * sizeof(char); + + memset(&value, 0, sizeof(DBT)); + payload = rand() + i; + value.data = &payload; + value.size = sizeof(int); + + /* Perform the database put. */ + switch (ret = dbp->put(dbp, txn, &key, &value, 0)) { + case 0: + break; + /* + * Our database is configured for sorted duplicates, + * so there is a potential for a KEYEXIST error return. + * If we get one, simply ignore it and continue on. + * + * Note that you will see KEYEXIST errors only after you + * have run this program at least once. + */ + case DB_KEYEXIST: + printf("Got keyexists.\n"); + break; + /* + * Here's where we perform deadlock detection. If + * DB_LOCK_DEADLOCK is returned by the put operation, + * then this thread has been chosen to break a deadlock. + * It must abort its operation, and optionally retry the + * put. + */ + case DB_LOCK_DEADLOCK: + /* + * First thing we MUST do is abort the + * transaction. + */ + (void)txn->abort(txn); + /* + * Now we decide if we want to retry the operation. + * If we have retried less than max_retries, + * increment the retry count and goto retry. + */ + if (retry_count < max_retries) { + printf("Writer %i: Got DB_LOCK_DEADLOCK.\n", + thread_num); + printf("Writer %i: Retrying write operation.\n", + thread_num); + retry_count++; + goto retry; + } + /* + * Otherwise, just give up. + */ + printf("Writer %i: ", thread_num); + printf("Got DB_LOCK_DEADLOCK and out of retries.\n"); + printf("Writer %i: Giving up.\n", thread_num); + return ((void *)EXIT_FAILURE); + /* + * If a generic error occurs, we simply abort the + * transaction and exit the thread completely. + */ + default: + envp->err(envp, ret, "db put failed"); + ret = txn->abort(txn); + if (ret != 0) + envp->err(envp, ret, "txn abort failed"); + return ((void *)EXIT_FAILURE); + } /** End case statement **/ + + } /** End for loop **/ +``` + +Having completed the inner database write loop, we could simply commit the transaction and continue on to the next block of 10 writes. However, we want to first illustrate a few points about transactional processing so instead we call our `count_records()` function before calling the transaction commit. `count_records()` uses a cursor to read every record in the database and return a count of the number of records that it found. + +``` c + /* + * print the number of records found in the database. + * See count_records() for usage information. + */ + printf("Thread %i. Record count: %i\n", thread_num, + count_records(dbp, NULL)); + + /* + * If all goes well, we can commit the transaction and + * loop to the next transaction. + */ + ret = txn->commit(txn, 0); + if (ret != 0) { + envp->err(envp, ret, "txn commit failed"); + return ((void *)EXIT_FAILURE); + } + } + return ((void *)EXIT_SUCCESS); +} +``` + +If you look at the `count_records()` function prototype at the beginning of this example, you will see that the function's second parameter takes a transaction handle. However, our usage of the function here does not pass a transaction handle through to the function. + +Because `count_records()` reads every record in the database, if used incorrectly the thread will self-deadlock. The writer thread has just written 500 records to the database, but because the transaction used for that write has not yet been committed, each of those 500 records are still locked by the thread's transaction. If we then simply run a non-transactional cursor over the database from within the same thread that has locked those 500 records, the cursor will block when it tries to read one of those transactional protected records. The thread immediately stops operation at that point while the cursor waits for the read lock it has requested. Because that read lock will never be released (the thread can never make any forward progress), this represents a self-deadlock for the the thread. + +There are three ways to prevent this self-deadlock: + +1. We can move the call to `count_records()` to a point after the thread's transaction has committed. + +2. We can allow `count_records()` to operate under the same transaction as all of the writes were performed (this is what the transaction parameter for the function is for). + +3. We can reduce our isolation guarantee for the application by allowing uncommitted reads. + +For this example, we choose to use option 3 (uncommitted reads) to avoid the deadlock. This means that we have to open our database such that it supports uncommitted reads, and we have to open our cursor handle so that it knows to perform uncommitted reads. + +Note that in In-Memory Transaction Example, we simply perform the cursor operation using the same transaction as is used for the thread's writes. + +The following is the `count_records()` implementation. There is not anything particularly interesting about this function other than specifying uncommitted reads when we open the cursor handle, but we include the function here anyway for the sake of completeness. + +``` c +/* + * This simply counts the number of records contained in the + * database and returns the result. + * + * Note that this function exists only for illustrative purposes. + * A more straight-forward way to count the number of records in + * a database is to use DB->stat() or DB->stat_print(). + */ + +int +count_records(DB *dbp, DB_TXN *txn) +{ + DBT key, value; + DBC *cursorp; + int count, ret; + + cursorp = NULL; + count = 0; + + /* Get the cursor */ + ret = dbp->cursor(dbp, txn, &cursorp, DB_READ_UNCOMMITTED); + if (ret != 0) { + dbp->err(dbp, ret, "count_records: cursor open failed."); + goto cursor_err; + } + + /* Get the key DBT used for the database read */ + memset(&key, 0, sizeof(DBT)); + memset(&value, 0, sizeof(DBT)); + do { + ret = cursorp->get(cursorp, &key, &value, DB_NEXT); + switch (ret) { + case 0: + count++; + break; + case DB_NOTFOUND: + break; + default: + dbp->err(dbp, ret, "Count records unspecified error"); + goto cursor_err; + } + } while (ret == 0); + +cursor_err: + if (cursorp != NULL) { + ret = cursorp->close(cursorp); + if (ret != 0) { + dbp->err(dbp, ret, + "count_records: cursor close failed."); + } + } + + return (count); +} +``` + +Finally, we provide the implementation of our `open_db()` function. This function should hold no surprises for you. Note, however, that we do specify uncommitted reads when we open the database. If we did not do this, then our `count_records()` function would cause our thread to self-deadlock because the cursor could not be opened to support uncommitted reads (that flag on the cursor open would, in fact, be silently ignored by DB). + +``` c +/* Open a Berkeley DB database */ +int +open_db(DB **dbpp, const char *progname, const char *file_name, + DB_ENV *envp, u_int32_t extra_flags) +{ + int ret; + u_int32_t open_flags; + DB *dbp; + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + fprintf(stderr, "%s: %s\n", progname, + db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* Point to the memory malloc'd by db_create() */ + *dbpp = dbp; + + if (extra_flags != 0) { + ret = dbp->set_flags(dbp, extra_flags); + if (ret != 0) { + dbp->err(dbp, ret, + "open_db: Attempt to set extra flags failed."); + return (EXIT_FAILURE); + } + } + + /* Now open the database */ + open_flags = DB_CREATE | /* Allow database creation */ + DB_READ_UNCOMMITTED | /* Allow uncommitted reads */ + DB_AUTO_COMMIT | /* Allow auto commit */ + DB_THREAD; /* Cause the database to be + free-threaded */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + open_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed", + file_name); + return (EXIT_FAILURE); + } + return (EXIT_SUCCESS); +} +``` + +This completes our transactional example. If you would like to experiment with this code, you can find the example in the following location in your DB distribution: + +``` c +DB_INSTALL/examples_c/txn_guide +``` diff --git a/docs-src/guides/gsg_txn/txnindices.md b/docs-src/guides/gsg_txn/txnindices.md new file mode 100644 index 000000000..1a262d682 --- /dev/null +++ b/docs-src/guides/gsg_txn/txnindices.md @@ -0,0 +1,54 @@ +--- +title: "Secondary Indices with Transaction Applications" +api-name: "Secondary Indices with Transaction Applications" +source: docs/gsg_txn/C/txnindices.html +--- +## Secondary Indices with Transaction Applications + +You can use transactions with your secondary indices so long as you open the secondary index so that it supports transactions (that is, you wrap the database open in a transaction, or use auto commit, in the same way as when you open a primary transactional database). In addition, you must make sure that when you associate the secondary index with the primary database, the association is performed using a transaction. The easiest thing to do here is to simply specify `DB_AUTO_COMMIT` when you perform the association. + +All other aspects of using secondary indices with transactions are identical to using secondary indices without transactions. In addition, transaction-protecting cursors opened against secondary indices is performed in exactly the same way as when you use transactional cursors against a primary database. See Transactional Cursors for details. + +Note that when you use transactions to protect your database writes, your secondary indices are protected from corruption because updates to the primary and the secondaries are performed in a single atomic transaction. + +For example: + +``` c +#include + +... + +DB_ENV *envp; /* Environment pointer */ +DB *dbp, *sdbp; /* Primary and secondary DB handles */ +u_int32_t flags; /* Primary database open flags */ +int ret; /* Function return value */ + +/* Environment and primary database opens omitted */ + +/* Secondary */ +ret = db_create(&sdbp, envp, 0); +if (ret != 0) { + /* Error handling goes here */ +} +/* open the secondary database */ +ret = sdbp->open(sdbp, /* DB structure pointer */ + NULL, /* Transaction pointer */ + "my_secdb.db", /* On-disk file that holds the + * database. */ + NULL, /* Optional logical database name */ + DB_BTREE, /* Database access method */ + DB_AUTO_COMMIT, /* Open flags */ + 0); /* File mode (using defaults) */ +if (ret != 0) { + /* Error handling goes here */ +} + +/* Now associate the secondary to the primary */ +dbp->associate(dbp, /* Primary database */ + NULL, /* TXN id */ + sdbp, /* Secondary database */ + get_sales_rep, /* Callback used for key creation. This + * is described in the Getting Started + * guide. */ + DB_AUTO_COMMIT);/* Flags */ +``` diff --git a/docs-src/guides/gsg_txn/txnnowait.md b/docs-src/guides/gsg_txn/txnnowait.md new file mode 100644 index 000000000..5f438bf69 --- /dev/null +++ b/docs-src/guides/gsg_txn/txnnowait.md @@ -0,0 +1,24 @@ +--- +title: "No Wait on Blocks" +api-name: "No Wait on Blocks" +source: docs/gsg_txn/C/txnnowait.html +--- +## No Wait on Blocks + +Normally when a DB transaction is blocked on a lock request, it must wait until the requested lock becomes available before its thread-of-control can proceed. However, it is possible to configure a transaction handle such that it will report a deadlock rather than wait for the block to clear. + +You do this on a transaction by transaction basis by specifying `DB_TXN_NOWAIT` to the `DB_ENV->txn_begin()` method. + +For example: + +``` c + /* Get the transaction */ + DB_TXN *txn = NULL; + ret = envp->txn_begin(envp, NULL, &txn, DB_TXN_NOWAIT); + if (ret != 0) { + envp->err(envp, ret, "txn_begin failed"); + return (EXIT_FAILURE); + } + .... + /* Deadlock detection and exception handling omitted for brevity. */ +``` diff --git a/docs-src/guides/gsg_txn/usingtxns.md b/docs-src/guides/gsg_txn/usingtxns.md new file mode 100644 index 000000000..416b80355 --- /dev/null +++ b/docs-src/guides/gsg_txn/usingtxns.md @@ -0,0 +1,203 @@ +--- +title: "Chapter 3. Transaction Basics" +api-name: "Chapter 3. Transaction Basics" +source: docs/gsg_txn/C/usingtxns.html +--- +## Chapter 3. Transaction Basics + +**Table of Contents** + + [Committing a Transaction](usingtxns.md#commitresults) + + [Non-Durable Transactions](nodurabletxn.md) + + [Aborting a Transaction](abortresults.md) + + [Auto Commit](autocommit.md) + + [Nested Transactions](nestedtxn.md) + + [Transactional Cursors](txncursor.md) + + [Secondary Indices with Transaction Applications](txnindices.md) + + [Configuring the Transaction Subsystem](maxtxns.md) + +Once you have enabled transactions for your environment and your databases, you can use them to protect your database operations. You do this by acquiring a transaction handle and then using that handle for any database operation that you want to participate in that transaction. + +You obtain a transaction handle using the `DB_ENV->txn_begin()` method. + +Once you have completed all of the operations that you want to include in the transaction, you must commit the transaction using the `DB_TXN->commit()` method. + +If, for any reason, you want to abandon the transaction, you abort it using `DB_TXN->abort()`. + +Any transaction handle that has been committed or aborted can no longer be used by your application. + +Finally, you must make sure that all transaction handles are either committed or aborted before closing your databases and environment. + +### Note + +If you only want to transaction protect a single database write operation, you can use auto commit to perform the transaction administration. When you use auto commit, you do not need an explicit transaction handle. See Auto Commit for more information. + +For example, the following example opens a transactional-enabled environment and database, obtains a transaction handle, and then performs a write operation under its protection. In the event of any failure in the write operation, the transaction is aborted and the database is left in a state as if no operations had ever been attempted in the first place. + +``` c +#include +#include + +#include "db.h" + +int +main(void) +{ + int ret, ret_c; + u_int32_t db_flags, env_flags; + DB *dbp; + DB_ENV *envp; + DBT key, data; + DB_TXN *txn; + const char *db_home_dir = "/tmp/myEnvironment"; + const char *file_name = "mydb.db"; + const char *keystr ="thekey"; + const char *datastr = "thedata"; + + dbp = NULL; + envp = NULL; + + /* Open the environment */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + return (EXIT_FAILURE); + } + + env_flags = DB_CREATE | /* Create the environment if it does + * not already exist. */ + DB_INIT_TXN | /* Initialize transactions */ + DB_INIT_LOCK | /* Initialize locking. */ + DB_INIT_LOG | /* Initialize logging */ + DB_INIT_MPOOL; /* Initialize the in-memory cache. */ + + ret = envp->open(envp, db_home_dir, env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + envp->err(envp, ret, "Database creation failed"); + goto err; + } + + db_flags = DB_CREATE | DB_AUTO_COMMIT; + /* + * Open the database. Note that we are using auto commit for the open, + * so the database is able to support transactions. + */ + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + envp->err(envp, ret, "Database '%s' open failed", + file_name); + goto err; + } + + /* Prepare the DBTs */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + + key.data = &keystr; + key.size = strlen(keystr) + 1; + data.data = &datastr; + data.size = strlen(datastr) + 1; + + /* Get the txn handle */ + txn = NULL; + ret = envp->txn_begin(envp, NULL, &txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction begin failed."); + goto err; + } + + /* + * Perform the database write. If this fails, abort the transaction. + */ + ret = dbp->put(dbp, txn, &key, &data, 0); + if (ret != 0) { + envp->err(envp, ret, "Database put failed."); + txn->abort(txn); + goto err; + } + + /* + * Commit the transaction. Note that the transaction handle + * can no longer be used. + */ + ret = txn->commit(txn, 0); + if (ret != 0) { + envp->err(envp, ret, "Transaction commit failed."); + goto err; + } + +err: + /* Close the database */ + if (dbp != NULL) { + ret_c = dbp->close(dbp, 0); + if (ret_c != 0) { + envp->err(envp, ret_c, "Database close failed."); + ret = ret_c + } + } + + /* Close the environment */ + if (envp != NULL) { + ret_c = envp->close(envp, 0); + if (ret_c != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_c)); + ret = ret_c; + } + } + + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +## Committing a Transaction + +In order to fully understand what is happening when you commit a transaction, you must first understand a little about what DB is doing with the logging subsystem. Logging causes all database write operations to be identified in logs, and by default these logs are backed by files on disk. These logs are used to restore your databases in the event of a system or application failure, so by performing logging, DB ensures the integrity of your data. + +Moreover, DB performs *write-ahead* logging. This means that information is written to the logs *before* the actual database is changed. This means that all write activity performed under the protection of the transaction is noted in the log before the transaction is committed. Be aware, however, that database maintains logs in-memory. If you are backing your logs on disk, the log information will eventually be written to the log files, but while the transaction is on-going the log data may be held only in memory. + +When you commit a transaction, the following occurs: + +- A commit record is written to the log. This indicates that the modifications made by the transaction are now permanent. By default, this write is performed synchronously to disk so the commit record arrives in the log files before any other actions are taken. + +- Any log information held in memory is (by default) synchronously written to disk. Note that this requirement can be relaxed, depending on the type of commit you perform. See Non-Durable Transactions for more information. Also, if you are maintaining your logs entirely in-memory, then this step will of course not be taken. To configure your logging system for in-memory usage, see Configuring In-Memory Logging. + +- All locks held by the transaction are released. This means that read operations performed by other transactions or threads of control can now see the modifications without resorting to uncommitted reads (see Reading Uncommitted Data for more information). + +To commit a transaction, you simply call `DB_TXN->commit()`. + +Notice that committing a transaction does not necessarily cause data modified in your memory cache to be written to the files backing your databases on disk. Dirtied database pages are written for a number of reasons, but a transactional commit is not one of them. The following are the things that can cause a dirtied database page to be written to the backing database file: + +- Checkpoints. + + Checkpoints cause all dirtied pages currently existing in the cache to be written to disk, and a checkpoint record is then written to the logs. You can run checkpoints explicitly. For more information on checkpoints, see Checkpoints. + +- Cache is full. + + If the in-memory cache fills up, then dirtied pages might be written to disk in order to free up space for other pages that your application needs to use. Note that if dirtied pages are written to the database files, then any log records that describe how those pages were dirtied are written to disk before the database pages are written. + +Be aware that because your transaction commit caused database modifications recorded in your logs to be forced to disk, your modifications are by default "persistent" in that they can be recovered in the event of an application or system failure. However, recovery time is gated by how much data has been modified since the last checkpoint, so for applications that perform a lot of writes, you may want to run a checkpoint with some frequency. + +Note that once you have committed a transaction, the transaction handle that you used for the transaction is no longer valid. To perform database activities under the control of a new transaction, you must obtain a fresh transaction handle. diff --git a/docs-src/guides/gsg_txn/wrapup.md b/docs-src/guides/gsg_txn/wrapup.md new file mode 100644 index 000000000..71c9cc4ff --- /dev/null +++ b/docs-src/guides/gsg_txn/wrapup.md @@ -0,0 +1,74 @@ +--- +title: "Chapter 6. Summary and Examples" +api-name: "Chapter 6. Summary and Examples" +source: docs/gsg_txn/C/wrapup.html +--- +## Chapter 6. Summary and Examples + +**Table of Contents** + + [Anatomy of a Transactional Application](wrapup.md#anatomy) + + [Transaction Example](txnexample_c.md) + + [In-Memory Transaction Example](inmem_txnexample_c.md) + +Throughout this manual we have presented the concepts and mechanisms that you need to provide transactional protection for your application. In this chapter, we summarize these mechanisms, and we provide a complete example of a multi-threaded transactional DB application. + +## Anatomy of a Transactional Application + +Transactional applications are characterized by performing the following activities: + +1. Create your environment handle. + +2. Open your environment, specifying that the following subsystems be used: + + - Transactional Subsystem (this also initializes the logging subsystem). + + - Memory pool (the in-memory cache). + + - Logging subsystem. + + - Locking subsystem (if your application is multi-process or multi-threaded). + + It is also highly recommended that you run normal recovery upon first environment open. Normal recovery examines only those logs required to ensure your database files are consistent relative to the information found in your log files. + +3. Optionally spawn off any utility threads that you might need. Utility threads can be used to run checkpoints periodically, or to periodically run a deadlock detector if you do not want to use DB's built-in deadlock detector. + +4. Open whatever database handles that you need. + +5. Spawn off worker threads. How many of these you need and how they split their DB workload is entirely up to your application's requirements. However, any worker threads that perform write operations will do the following: + + 1. Begin a transaction. + + 2. Perform one or more read and write operations. + + 3. Commit the transaction if all goes well. + + 4. Abort and retry the operation if a deadlock is detected. + + 5. Abort the transaction for most other errors. + +6. On application shutdown: + + 1. Make sure there are no opened cursors. + + 2. Make sure there are no active transactions. Either abort or commit all transactions before shutting down. + + 3. Close your databases. + + 4. Close your environment. + +### Note + +Robust DB applications should monitor their worker threads to make sure they have not died unexpectedly. If a thread does terminate abnormally, you must shutdown all your worker threads and then run normal recovery (you will have to reopen your environment to do this). This is the only way to clear any resources (such as a lock or a mutex) that the abnormally exiting worker thread might have been holding at the time that it died. + +Failure to perform this recovery can cause your still-functioning worker threads to eventually block forever while waiting for a lock that will never be released. + +In addition to these activities, which are all entirely handled by code within your application, there are some administrative activities that you should perform: + +- Periodically checkpoint your application. Checkpoints will reduce the time to run recovery in the event that one is required. See Checkpoints for details. + +- Periodically back up your database and log files. This is required in order to fully obtain the durability guarantee made by DB's transaction ACID support. See Backup Procedures for more information. + +- You may want to maintain a hot failover if 24x7 processing with rapid restart in the face of a disk hit is important to you. See Using Hot Failovers for more information. diff --git a/docs-src/guides/installation/_meta.toml b/docs-src/guides/installation/_meta.toml new file mode 100644 index 000000000..5be6e7008 --- /dev/null +++ b/docs-src/guides/installation/_meta.toml @@ -0,0 +1,108 @@ +# Nav/index metadata for the installation guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Berkeley DB Installation and Build Guide" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "introduction", + "ch01s02", + "install", + "install_multiple", + "debug", + "debug_compile", + "debug_runtime", + "debug_printlog", + "build_android_intro", + "build_android_jdbc", + "build_android_config", + "build_win", + "win_build64", + "win_build_cygwin", + "win_build_cxx", + "win_build_stl", + "build_win_java", + "build_win_csharp", + "build_win_sql", + "build_win_tcl", + "win_build_dist_dll", + "win_additional_options", + "build_win_small", + "build_win_test", + "build_win_notes", + "build_win_faq", + "build_unix", + "build_unix_conf", + "build_unix_sql", + "build_unix_small", + "build_unix_flags", + "cross_compile_unix", + "build_unix_install", + "build_unix_shlib", + "build_unix_test", + "build_unix_notes", + "build_unix_aix", + "build_unix_freebsd", + "build_unix_iphone", + "build_unix_irix", + "build_unix_linux", + "build_unix_macosx", + "build_unix_qnx", + "build_unix_sco", + "build_unix_solaris", + "build_unix_sunos", + "upgrade_53_toc", + "upgrade_11gr2_53_build_windows", + "upgrade_11gr2_53_conn_status", + "upgrade_11gr2_53_excl", + "upgrade_11gr2_53_heap_regionsize", + "upgrade_11gr2_53_hotbackup", + "upgrade_11gr2_53_jdbc", + "upgrade_11gr2_53_meta_dir", + "upgrade_11gr2_53_sql_build", + "upgrade_11gr2_53_sql_pragma", + "upgrade_11gr2_53_sql_rep", + "upgrade_11gr2_53_xa_mvcc", + "changelog_5_3", + "upgrade_52_toc", + "upgrade_11gr2_52_sqlite_ver", + "upgrade_11gr2_52_bit_cmp_win", + "upgrade_11gr2_52_rep_dbt_readonly", + "upgrade_11gr2_52_dyn_env", + "upgrade_11gr2_52_excl_txn_sql", + "upgrade_11gr2_52_grp_mbr", + "upgrade_11gr2_52_heap", + "upgrade_11gr2_52_mvcc_sql", + "upgrade_11gr2_52_rep_2site_strict", + "upgrade_11gr2_52_rep_sql", + "upgrade_11gr2_52_repmgr_channels", + "upgrade_11gr2_52_seq_sql", + "upgrade_11gr2_52_xa", + "upgrade_11gr2_52_hot_backup", + "changelog_5_2", + "upgrade_51_toc", + "upgrade_11gr2_51_dpl_recompile", + "upgrade_11gr2_51_src_reorg", + "upgrade_11gr2_51_sqlite_ver", + "upgrade_11gr2_51_mod_db4_unsupp", + "changelog_5_1", + "upgrade_11gr2_toc", + "upgrade_11gr2_dbsqlcodegen", + "upgrade_11gr2_autoinit", + "upgrade_11gr2_repmgr", + "build_unix_encrypt", + "build_unix_db_nosync", + "upgrade_11gr2_remsupp", + "build_unix_stacksize", + "changelog_5_0", + "upgrade_4_8_toc", + "upgrade_4_8_dpl", + "upgrade_4_8_mpool", + "upgrade_4_8_fcntl", + "upgrade_4_8_disk", + "changelog_4_8", + "test", + "test_faq", +] diff --git a/docs-src/guides/installation/build_android_config.md b/docs-src/guides/installation/build_android_config.md new file mode 100644 index 000000000..025c70a51 --- /dev/null +++ b/docs-src/guides/installation/build_android_config.md @@ -0,0 +1,31 @@ +--- +title: "Android Configuration Options" +api-name: "Android Configuration Options" +source: docs/installation/build_android_config.html +--- +## Android Configuration Options + +There are several configuration options you can specify in `LOCAL_CFLAGS` located in the `Android.mk` file. + +- BDBSQL_CONVERT_SQLITE + + This option enables to convert SQLite database to BDB SQL database format. See Migrating from SQLite to Berkeley DB for more information. + +- BDBSQL_SHARE_PRIVATE + + This flag is enabled by default and keeps all the region files in memory instead of the disk. This flag also implements database-level locking. + +- SQLITE_DEFAULT_CACHE_SIZE + + SQLite provides an in-memory cache which you size according to the maximum number of database pages that you want to hold in memory at any given time. See Changing Compile Options. + +- SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT + + For SQLite, this pragma identifies the maximum size that the journal file is allowed to be. Berkeley DB does not have a journal file, but it writes and uses log files. A new log file is created when the current log file has reached the defined maximum size. This flag defines this maximum size for a log file. Default value is 10 MB for Berkeley DB SQL interface. + +Hard-coded numbers in the build can be adjusted using the following SQLite PRAGMA commands: + +- PRAGMA cache_size +- PRAGMA journal_size_limit + +You can configure most aspects of your Berkeley DB environment by using the DB_CONFIG file. diff --git a/docs-src/guides/installation/build_android_intro.md b/docs-src/guides/installation/build_android_intro.md new file mode 100644 index 000000000..b2956df34 --- /dev/null +++ b/docs-src/guides/installation/build_android_intro.md @@ -0,0 +1,142 @@ +--- +title: "Chapter 4. Building Berkeley DB for Android" +api-name: "Chapter 4. Building Berkeley DB for Android" +source: docs/installation/build_android_intro.html +--- +## Chapter 4. Building Berkeley DB for Android + +**Table of Contents** + + [Building the Drop-In Replacement for Android](build_android_intro.md#build_android) + + [Migrating from SQLite to Berkeley DB](build_android_intro.md#build_android_migrate) + + [Building the Android JDBC Driver](build_android_jdbc.md) + + [Android Configuration Options](build_android_config.md) + +Berkeley DB provides support for the Android platform enabling you to develop and deploy a wide range of mobile applications and services. Android provides SQLite as the default database for developing applications that need database support. Berkeley DB SQL API is fully compatible with SQLite and can be used as a replacement. The `build_android` directory in the Berkeley DB distribution contains a makefile, `Android.mk`, for building a drop-in replacement for SQLite. + +Oracle offers two different solutions for building the BDB SQL API for Android. The first creates a library that can be used as a drop-in replacement for SQLite on Android. The second creates a JDBC driver for Android. + +## Building the Drop-In Replacement for Android + + [Migrating from SQLite to Berkeley DB](build_android_intro.md#build_android_migrate) + +This section describes how to build a library that can be used as a drop-in replacement for SQLite on Android. + +1. Download and compile the Android source tree. + + The compiling process takes time but is a one time activity. For information on downloading and compiling the Android source code, see http://source.android.com/source/download.html. + +2. Copy the Berkeley DB code into the Android build tree. + + ``` c + $ cd /external/sqlite/dist + $ tar zxf ${DB_PATH} + ``` + + where \ is the root of the Android source tree and \${DB_PATH} is the path where you saved the `db-xx.tar.gz` version of the Berkeley DB distribution. + +3. Update the Android build file to identify Berkeley DB. + + Replace the `Android.mk` file with the one from the Berkeley DB source tree by doing the following: + + ``` c + $ cd /external/sqlite/dist + $ mv Android.mk Android.mk.sqlite + $ cp ${DB_INSTALL}/build_android/Android.mk ./ + ``` + + where \${DB_INSTALL} is the directory into which you installed the Berkeley DB library. + +4. Tuning parameters. + + The configuration options for performance tuning can be added/edited in the `Android.mk` file by modifying `LOCAL_CFLAGS` located in the `build libsqlite replacement` section. For more information, see Android Configuration Options. + + It is also possible to change these settings using PRAGMA commands or through the DB_CONFIG file. + +5. Build the new Android image. + + To build the Android image with Berkeley DB SQL included, do the following: + + ``` c + $ cd + $ . build/envsetup.sh + $ make clean-libsqlite + $ mmm -B external/sqlite/dist + $ make snod + ``` + + You can locate the new image in `/out/target/product/generic`. + +### Migrating from SQLite to Berkeley DB + +This section describes how to enable automatic conversion of SQLite format databases to Berkeley DB SQL when they are opened. To do this, you must first make sure that the `-DBDBSQL_CONVERT_SQLITE` option is added to `LOCAL_CFLAGS` when you configure your Berkeley DB database build. + +1. Build a static SQLite shell for Android platform. + + Create a script, build_sqlite3_shell.sh, in the \/external/sqlite/dist directory. + + ``` c + #!/bin/bash + # This script shows how to use built-in toolchain to build + # sqlite3 shell, which is required by Berkeley DB SQL + # on-the-fly migration feature. + + # Note: these variables should be set per active Android source tree + # We assume $PWD=$ROOT/external/sqlite/dist + ROOT=${PWD}/../../.. + TOOLCHAIN=${ROOT}/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0 + CC=${TOOLCHAIN}/bin/arm-eabi-gcc + LIB="${ROOT}/out/target/product/generic/obj/lib" + INCLUDE="${ROOT}/ndk/build/platforms/android-8/arch-arm/usr/include" + + # CFLAGS should be set per Android.mk.sqlite (the original + # version of SQLite's Android.mk) + CFLAGS="-DHAVE_USLEEP=1 -DSQLITE_THREADSAFE=1 -DNDEBUG=1 \ + -DSQLITE_DEFAULT_JOURNAL_SIZE_LIMIT=1048576 \ + -DSQLITE_ENABLE_MEMORY_MANAGEMENT=1 \ + -DSQLITE_DEFAULT_AUTOVACUUM=1 \ + -DSQLITE_TEMP_STORE=3 -DSQLITE_ENABLE_FTS3 \ + -DSQLITE_ENABLE_FTS3_BACKWARDS -DTHREADSAFE=1" + CFLAGS="${CFLAGS} -I${INCLUDE}" + + LDFLAGS="-ldl -nostdlib -Wl,--gc-sections -lc -llog -lgcc \ + -Wl,--no-undefined,-z,nocopyreloc ${LIB}/crtend_android.o \ + ${LIB}/crtbegin_dynamic.o -L${LIB} -Wl,-rpath,${LIB}" + + ${CC} -DANDROID -DOS_ANDROID --sysroot="${SYSROOT}" -mandroid \ + -fvisibility=hidden -ffunction-sections -fdata-sections \ + -fPIC ${LDFLAGS} ${CFLAGS} \ + sqlite3.c shell.c -o sqlite3orig + ``` + + Ensure you adjust the variables as per your actual Android environment. This script is suited for Android 2.2. + +2. Execute the build_sqlite3_shell.sh script and to get the static sqlite3 shell utility - sqlite3orig. + +3. Change the system image file. + + Use the `xyaffs2` utiltiy to decompress the `system.img` and get the directory system. + + ``` c + $ xyaffs2 ./system.img system + ``` + + Add static sqlite3 shell utility. + + ``` c + $ cp /external/sqlite/dist/sqlite3orig \ + system/xbin/sqlite3orig + ``` + + Use the `mkyaffs2image` utility to rebuild `system.img` from the changed directory system. + + ``` c + $ mkyaffs2image -f $PWD/system system.img + ``` + + ### Note + + To open the database in the `SQLite` format use the `sqlite3orig` command. diff --git a/docs-src/guides/installation/build_android_jdbc.md b/docs-src/guides/installation/build_android_jdbc.md new file mode 100644 index 000000000..785e20e2e --- /dev/null +++ b/docs-src/guides/installation/build_android_jdbc.md @@ -0,0 +1,239 @@ +--- +title: "Building the Android JDBC Driver" +api-name: "Building the Android JDBC Driver" +source: docs/installation/build_android_jdbc.html +--- +## Building the Android JDBC Driver + +This section describes how to build and use the BDB JDBC driver for Android. Note that the BDB JDBC driver cannot currently be built on a Windows platform. + +1. Download and install the Android SDK. The installation instructions can be found here: + + http://developer.android.com/sdk/installing.html + +2. Download and install the Android NDK. It can be found here: + + http://developer.android.com/sdk/ndk/index.html + +3. Build the BDB JDBC libraries. + + 1. If you do not already have it, download the Berkeley DB package from here: + + http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html + + Note that you must be using a 5.3.x or higher version of the product in order for these instructions to work. Once you have the package, unpack it: + + ``` c + $ tar zxvf db-x.y.z.tar.gz + $ cd db-x.y.z + ``` + + Where `x.y.z` the major, minor, and point release version of the Berkeley DB distribution which you are using. + + Also, note that in the following instructions, the directory denoted by `db-x.y.z`, above, is referred to as ``. + + 2. Build an x86/x64 JDBC package. This is required because the building process will generate target files which are required to build Android NDK. Also, the built JAR file can be imported by eclipse, which will then convert it to the Android Dalvik JAR format. + + To do this, edit `/lang/sql/jdbc/SQLit/Database.java` and replace all instances of `System.loadLibrary("sqlite_jni")` with `System.loadLibrary("oracle-jdbc")`. + + Once you have done this, configure and make the library. The following example shows the minimum configuration options that you need to use in order to configure the Berkeley DB JDBC driver. For your particular installation, other configuration options might be interesting to you. See Configuring Berkeley DB and Android Configuration Options for more information. + + ``` c + cd /build_unix + ../dist/configure --enable-jdbc && make + ``` + +4. Build the Android NDK: + + ``` c + $ cd /build_android/jdbc/jni + $ /ndk-build + ``` + + This results in the following required files: + + | | + |----------------------------------------------------------| + | \/build_android/jdbc/libs/armeabi/liboracle-jdbc.so | + | \/build_android/jdbc/libs/armeabi/dbsql | + | \/build_unix/jdbc/sqlite.jar | + +Having built the JDBC driver, you can now use it with your project. You can do this using Eclipse and the ADT plugin, which you can get from here: + +http://developer.android.com/sdk/eclipse-adt.html + +To make sure everything is working: + +1. Start Eclipse and create an Android project. Use: + + - `test_jdbc` as the Android project name. + + - Create it as an Android 3.2 project. + + - For the package name, use `example.jdbc`. + +2. This results in an empty code file. Copy and paste the following example code into that file: + + ``` c + package example.testjdbc; + + import SQLite.*; + import java.io.*; + + import android.app.Activity; + import android.os.Bundle; + import android.widget.TextView; + import java.sql.*; + + public class Test_jdbcActivity extends Activity { + + /* + * This is the main entrance/body of our sample program. This + * example illustrates all of the major API usages. + */ + public void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + + TextView tv = new TextView(this); + tv.setText("App Started"); + setContentView(tv); + + System.out.println("Appstart: "); + + String url = + "jdbc:sqlite://data/data/example.testjdbc/example.db"; + Connection con; + String dropString = "drop table if exists COFFEES"; + String createString; + createString = "create table COFFEES " + + "(COF_NAME varchar(32), " + + "SUP_ID int, " + "PRICE float, " + "SALES int, " + + "TOTAL int)"; + String insertString = "drop table COFFEES if exisits"; + String query = "select COF_NAME, PRICE from COFFEES"; + Statement stmt; + + try { + Class.forName("SQLite.JDBCDriver"); + + } catch (java.lang.ClassNotFoundException e) { + System.err.print("ClassNotFoundException: "); + System.err.println(e.getMessage()); + } + + try { + con = + DriverManager.getConnection(url, "myLogin", "myPW"); + + stmt = con.createStatement(); + stmt.executeUpdate(dropString); + stmt.executeUpdate(createString); + stmt.close(); + + stmt = con.createStatement(); + stmt.executeUpdate("insert into COFFEES " + + "values('Colombian', 00101, 7.99, 0, 0)"); + + stmt.executeUpdate("insert into COFFEES " + + "values('French_Roast', 00049, 8.99, 0, 0)"); + + stmt.executeUpdate("insert into COFFEES " + + "values('Espresso', 00150, 9.99, 0, 0)"); + + stmt.executeUpdate("insert into COFFEES " + + "values('Colombian_Decaf', 00101, 8.99, 0, 0)"); + + stmt.executeUpdate("insert into COFFEES " + + "values('French_Roast_Decaf', 00049, 9.99, 0, 0)"); + + ResultSet rs = stmt.executeQuery(query); + + System.out.println("Coffee Break Coffees and Prices:"); + while (rs.next()) { + String s = rs.getString("COF_NAME"); + float f = rs.getFloat("PRICE"); + System.out.println(s + " " + f); + } + stmt.close(); + con.close(); + + } catch (SQLException ex) { + System.err.println("SQLException: " + ex.getMessage()); + } + } + } + ``` + +3. Copy the following files into place: + + ``` c + $ cd /test_jdbc + $ mkdir -p libs/armeabi + $ cp -r /build_android/jdbc/libs/armeabi/liboracle-jdbc.so \ + libs/armeabi + $ cp -r /build_unix/jdbc/sqlite.jar libs + ``` + +4. Back in Eclipse, right click the project name, and select the `refresh` option to reload the project from the directory. The two new files that were copied into place in the previous step are now included in the project view. + +5. Convert the JAR file to the Android Dalvik format: + + 1. Right-click on your project. + + 2. Choose `Build Path -> Configure Build Path` + + 3. Click the `Libraries` tab. + + 4. Click `Add JARS`. + +6. Run the project: + + 1. Choose `Property -> Android` and select any one of the usable build targets. + + 2. Right click the project. Choose `Run As -> Android` + +7. Verify your installation. After a short pause (depending on the speed of your system), the application logo is displayed. Use the Android adb command line application to make sure the application is running as expected: + + ``` c + $ cd /platform-tools + $ ./adb logcat + I/System.out( 539): Appstart: + I/System.out( 539): Coffee Break Coffees and Prices: + I/System.out( 539): Colombian 7.99 + I/System.out( 539): French_Roast 8.99 + I/System.out( 539): Espresso 9.99 + I/System.out( 539): Colombian_Decaf 8.99 + I/System.out( 539): French_Roast_Decaf 9.99 + ``` + + You can also check if the database (`example.db`) exists in the emulator: + + ``` c + $ ./adb shell ls /data/data/example.testjdbc + example.db + example.db-journal + lib + ``` + + Finally, check the database using the BDB SQL shell: + + ``` c + $ ./adb push /build_android/jdbc/libs/armeabi/dbsql \ + /data/data/example.testjdbc + 326 KB/s (1293760 bytes in 3.865s) + $ ./adb shell + root@android:/ # cd /data/data/example.testjdbc + root@android:/data/data/example.testjdbc # ./dbsql example.db + Berkeley DB 11g Release 2, library version 11.2.5.2.36 + Enter ".help" for instructions + Enter SQL statements terminated with a ";" + dbsql> .tables + COFFEES + dbsql> select * from COFFEES; + Colombian|101|7.99|0|0 + French_Roast|49|8.99|0|0 + Espresso|150|9.99|0|0 + Colombian_Decaf|101|8.99|0|0 + French_Roast_Decaf|49|9.99|0|0 + dbsql> .quit + ``` diff --git a/docs-src/guides/installation/build_unix.md b/docs-src/guides/installation/build_unix.md new file mode 100644 index 000000000..10b0c1357 --- /dev/null +++ b/docs-src/guides/installation/build_unix.md @@ -0,0 +1,132 @@ +--- +title: "Chapter 7.  Building Berkeley DB for UNIX/POSIX" +api-name: "Chapter 7.  Building Berkeley DB for UNIX/POSIX" +source: docs/installation/build_unix.html +--- +## Chapter 7.  Building Berkeley DB for UNIX/POSIX + +**Table of Contents** + + [Building for UNIX/POSIX](build_unix.md#build_unix_intro) + + [Building the Berkeley DB SQL Interface](build_unix.md#build_unix_sqlinter) + + [Configuring Berkeley DB](build_unix_conf.md) + + [Configuring the SQL Interface](build_unix_sql.md) + + [Changing Compile Options](build_unix_sql.md#config_sql) + + [Enabling Extensions](build_unix_sql.md#idp500824) + + [Building the JDBC Driver](build_unix_sql.md#build_unix_jdbc) + + [Using the JDBC Driver](build_unix_sql.md#idp571856) + + [Building the ODBC Driver](build_unix_sql.md#idp593744) + + [Building the BFILE extension](build_unix_sql.md#bfile) + + [Building a small memory footprint library](build_unix_small.md) + + [Changing compile or load options](build_unix_flags.md) + + [Cross-Compiling on Unix](cross_compile_unix.md) + + [Installing Berkeley DB](build_unix_install.md) + + [Dynamic shared libraries](build_unix_shlib.md) + + [Running the test suite under UNIX](build_unix_test.md) + + [Building SQL Test Suite on Unix](build_unix_test.md#build_unix_test_sql) + + [Architecture independent FAQ](build_unix_notes.md) + + [AIX](build_unix_aix.md) + + [FreeBSD](build_unix_freebsd.md) + + [Apple iOS (iPhone OS)](build_unix_iphone.md) + + [IRIX](build_unix_irix.md) + + [Linux](build_unix_linux.md) + + [Mac OS X](build_unix_macosx.md) + + [QNX](build_unix_qnx.md) + + [SCO](build_unix_sco.md) + + [Solaris](build_unix_solaris.md) + + [SunOS](build_unix_sunos.md) + +## Building for UNIX/POSIX + + [Building the Berkeley DB SQL Interface](build_unix.md#build_unix_sqlinter) + +The Berkeley DB distribution builds up to four separate libraries: the base C API Berkeley DB library and the optional C++, Java, and Tcl API libraries. For portability reasons, each library is standalone and contains the full Berkeley DB support necessary to build applications; that is, the C++ API Berkeley DB library does not require any other Berkeley DB libraries to build and run C++ applications. + +Building for Linux, Apple iOS (known as iPhone OS previously), Mac OS X or the QNX Neutrino release is the same as building for a conventional UNIX platform. + +The Berkeley DB distribution uses the Free Software Foundation's autoconf and libtool tools to build on UNIX platforms. In general, the standard configuration and installation options for these tools apply to the Berkeley DB distribution. + +To perform a standard UNIX build of Berkeley DB, change to the **build_unix** directory and then enter the following two commands: + +``` c +../dist/configure +make +``` + +This will build the Berkeley DB library. + +To install the Berkeley DB library, enter the following command: + +``` c +make install +``` + +To rebuild Berkeley DB, enter: + +``` c +make clean +make +``` + +If you change your mind about how Berkeley DB is to be configured, you must start from scratch by entering the following command: + +``` c +make realclean +../dist/configure +make +``` + +To uninstall Berkeley DB, enter: + +``` c +make uninstall +``` + +To build multiple UNIX versions of Berkeley DB in the same source tree, create a new directory at the same level as the build_unix directory, and then configure and build in that directory as described previously. + +### Building the Berkeley DB SQL Interface + +To perform a standard UNIX build of the Berkeley DB SQL interface, go to the **build_unix** directory and then enter the following two commands: + +``` c +../dist/configure --enable-sql +make +``` + +This creates a library, `libdb_sql`, and a command line tool, `dbsql`. You can create and manipulate SQL databases using the `dbsql` shell. + +You can optionally provide the `--enable-sql_compat` argument to the `configure` script. In addition to creating `libdb_sql` and `dbsql` this causes a thin wrapper library called `libsqlite3` and a command line tool called `sqlite3` to be built. This library can be used as a drop-in replacement for SQLite. The `sqlite3` command line tool is identical to the `dbsql` executable but is named so that existing scripts for SQLite can easily work with Berkeley DB. + +``` c +../dist/configure --enable-sql_compat +make +``` + +There are several arguments you can specify when configuring the Berkeley DB SQL Interface. See Configuring the SQL Interface for more information. diff --git a/docs-src/guides/installation/build_unix_aix.md b/docs-src/guides/installation/build_unix_aix.md new file mode 100644 index 000000000..2693c6a25 --- /dev/null +++ b/docs-src/guides/installation/build_unix_aix.md @@ -0,0 +1,52 @@ +--- +title: "AIX" +api-name: "AIX" +source: docs/installation/build_unix_aix.html +--- +## AIX + +1. **I can't compile and run multithreaded applications.** + + Special compile-time flags are required when compiling threaded applications on AIX. If you are compiling a threaded application, you must compile with the \_THREAD_SAFE flag and load with specific libraries; for example, "-lc_r". Specifying the compiler name with a trailing "\_r" usually performs the right actions for the system. + + ``` c + xlc_r ... + cc -D_THREAD_SAFE -lc_r ... + ``` + + The Berkeley DB library will automatically build with the correct options. + +2. **I can't run using the DB_SYSTEM_MEM option to DB_ENV->open().** + + AIX 4.1 allows applications to map only 10 system shared memory segments. In AIX 4.3, this has been raised to 256K segments, but only if you set the environment variable "export EXTSHM=ON". + +3. **On AIX 4.3.2 (or before) I see duplicate symbol warnings when building the C++ shared library and when linking applications.** + + We are aware of some duplicate symbol warnings with this platform, but they do not appear to affect the correct operation of applications. + +4. **On AIX 4.3.3 I see undefined symbols for DbEnv::set_error_stream, Db::set_error_stream or DbEnv::verify when linking C++ applications. (These undefined symbols also appear when building the Berkeley DB C++ example applications).** + + By default, Berkeley DB is built with \_LARGE_FILES set to 1 to support the creation of "large" database files. However, this also affects how standard classes, like iostream, are named internally. When building your application, use a "-D_LARGE_FILES=1" compilation option, or insert "#define \_LARGE_FILES 1" before any \#include statements. + +5. **I can't create database files larger than 1GB on AIX.** + + If you're running on AIX 4.1 or earlier, try changing the source code for `os/os_open.c` to always specify the **O_LARGEFILE** flag to the `open`(2) system call, and recompile Berkeley DB from scratch. + + Also, the documentation for the IBM Visual Age compiler states that it does not not support the 64-bit filesystem APIs necessary for creating large files; the ibmcxx product must be used instead. We have not heard whether the GNU gcc compiler supports the 64-bit APIs or not. + + Finally, to create large files under AIX, the filesystem has to be configured to support large files and the system wide user hard-limit for file sizes has to be greater than 1GB. + +6. **I see errors about "open64" when building Berkeley DB applications.** + + System include files (most commonly fcntl.h) in some releases of AIX and Solaris redefine "open" when large-file support is enabled for applications. This causes problems when compiling applications because "open" is a method in the Berkeley DB APIs. To work around this problem: + + 1. Avoid including the problematical system include files in source code files which also include Berkeley DB include files and call into the Berkeley DB API. + 2. Before building Berkeley DB, modify the generated include file db.h to itself include the problematical system include files. + 3. Turn off Berkeley DB large-file support by specifying the --disable-largefile configuration option and rebuilding. + +7. **I see the error "Redeclaration of lseek64" when building Berkeley DB with the --enable-sql and --enable-test options.** + + In some releases of AIX, the system include files (most commonly `unistd.h`) redefine `lseek` to `lseek64` when large-file support is enabled even though `lseek` may have already been defined when the `_LARGE_FILE_API` macro is on. To work around this problem, do either one of the following: + + 1. Disable large-file support in Berkeley DB by specifying the `--disable-largefile` configuration option and rebuilding. + 2. Edit `db.h` manually after running the configure command, and remove the line that includes `unistd.h`. diff --git a/docs-src/guides/installation/build_unix_conf.md b/docs-src/guides/installation/build_unix_conf.md new file mode 100644 index 000000000..ce17438df --- /dev/null +++ b/docs-src/guides/installation/build_unix_conf.md @@ -0,0 +1,177 @@ +--- +title: "Configuring Berkeley DB" +api-name: "Configuring Berkeley DB" +source: docs/installation/build_unix_conf.html +--- +## Configuring Berkeley DB + +There are several arguments you can specify when configuring Berkeley DB. Although only the Berkeley DB-specific ones are described here, most of the standard GNU autoconf arguments are available and supported. To see a complete list of possible arguments, specify the --help flag to the configure program. + +The Berkeley DB specific arguments are as follows: + +- **--disable-largefile** + + Some systems, notably versions of Solaris, require special compile-time options in order to create files larger than 2^32 bytes. These options are automatically enabled when Berkeley DB is compiled. For this reason, binaries built on current versions of these systems may not run on earlier versions of the system because the library and system calls necessary for large files are not available. To disable building with these compile-time options, enter --disable-largefile as an argument to configure. + +- **--disable-shared, --disable-static** + + On systems supporting shared libraries, Berkeley DB builds both static and shared libraries by default. (Shared libraries are built using the GNU Project's Libtool distribution, which supports shared library builds on many (although not all) systems.) To not build shared libraries, configure using the --disable-shared argument. To not build static libraries, configure using the --disable-static argument. + +- **--disable-heap** + + Disables the Heap access method so that it cannot be used by Berkeley DB applications. + +- **--enable-compat185** + + To compile or load Berkeley DB 1.85 applications against this release of the Berkeley DB library, enter --enable-compat185 as an argument to configure. This will include Berkeley DB 1.85 API compatibility code in the library. + +- **--enable-cxx** + + To build the Berkeley DB C++ API, enter --enable-cxx as an argument to configure. + +- **--enable-debug** + + To build Berkeley DB with **-g** as a compiler flag and with **DEBUG** \#defined during compilation, enter --enable-debug as an argument to configure. This will create a Berkeley DB library and utilities with debugging symbols, as well as load various routines that can be called from a debugger to display pages, cursor queues, and so forth. If installed, the utilities will not be stripped. This argument should not be specified when configuring to build production binaries. + +- **--enable-debug_rop** + + To build Berkeley DB to output log records for read operations, enter --enable-debug_rop as an argument to configure. This argument should not be specified when configuring to build production binaries. + +- **--enable-debug_wop** + + To build Berkeley DB to output log records for write operations, enter --enable-debug_wop as an argument to configure. This argument should not be specified when configuring to build production binaries. + +- **--enable-diagnostic** + + To build Berkeley DB with run-time debugging checks, enter --enable-diagnostic as an argument to configure. This causes a number of additional checks to be performed when Berkeley DB is running, and also causes some failures to trigger process abort rather than returning errors to the application. Applications built using this argument should not share database environments with applications built without this argument. This argument should not be specified when configuring to build production binaries. + +- **--enable-dump185** + + To convert Berkeley DB 1.85 (or earlier) databases to this release of Berkeley DB, enter --enable-dump185 as an argument to configure. This will build the db_dump185 utility, which can dump Berkeley DB 1.85 and 1.86 databases in a format readable by the Berkeley DB db_load utility. + + The system libraries with which you are loading the db_dump185 utility must already contain the Berkeley DB 1.85 library routines for this to work because the Berkeley DB distribution does not include them. If you are using a non-standard library for the Berkeley DB 1.85 library routines, you will have to change the Makefile that the configuration step creates to load the db_dump185 utility with that library. + +- **--enable-java** + + To build the Berkeley DB Java API, enter --enable-java as an argument to configure. To build Java, you must also build with shared libraries. Before configuring, you must set your PATH environment variable to include javac. Note that it is not sufficient to include a symbolic link to javac in your PATH because the configuration process uses the location of javac to determine the location of the Java include files (for example, jni.h). On some systems, additional include directories may be needed to process jni.h; see Changing compile or load options for more information. + +- **--enable-posixmutexes** + + To force Berkeley DB to use the POSIX pthread mutex interfaces for underlying mutex support, enter --enable-posixmutexes as an argument to configure. This is rarely necessary: POSIX mutexes will be selected automatically on systems where they are the preferred implementation. + + The --enable-posixmutexes configuration argument is normally used in two ways: First, when there are multiple mutex implementations available and the POSIX mutex implementation is not the preferred one (for example, on Solaris where the LWP mutexes are used by default). Second, by default the Berkeley DB library will only select the POSIX mutex implementation if it supports mutexes shared between multiple processes, as described for the pthread_condattr_setpshared and pthread_mutexattr_setpshared interfaces. The --enable-posixmutexes configuration argument can be used to force the selection of POSIX mutexes in this case, which can improve application performance significantly when the alternative mutex implementation is a non-blocking one (for example test-and-set assembly instructions). However, configuring to use POSIX mutexes when the implementation does not have inter-process support will only allow the creation of private database environments, that is, environments where the DB_PRIVATE flag is specified to the DB_ENV->open() method. + + Specifying the --enable-posixmutexes configuration argument may require that applications and Berkeley DB be linked with the -lpthread library. + +- **--enable-pthread_api** + + To configure Berkeley DB for a POSIX pthreads application (with the exception that POSIX pthread mutexes may not be selected as the underlying mutex implementation for the build), enter --enable-pthread_api as an argument to configure. The build will include the Berkeley DB replication manager interfaces and will use the POSIX standard pthread_self and pthread_yield functions to identify threads of control and yield the processor. The --enable-pthread_api argument requires POSIX pthread support already be installed on your system. + + Specifying the --enable-pthread_api configuration argument may require that applications and Berkeley DB be linked with the -lpthread library. + +- **--enable-sql** + + To build the command tool dbsql, enter --enable-sql as an argument to configure. The dbsql utility provides access to the Berkeley DB SQL interface. See Configuring the SQL Interface for more information. + +- **--enable-sql_compat** + + To build the command tool sqlite3, enter --enable-sql_compat as an argument to configure. Sqlite3 is a command line tool that enables you to manually enter and execute SQL commands. It is identical to the dbsql executable but named so that existing scripts for SQLite can easily work with Berkeley DB. See Configuring the SQL Interface for more information. + +- **--enable-sql_codegen** + + To build the command line tool db_sql_codegen, enter --enable-sql_codegen as an argument to configure. The db_sql_codegen utility translates a schema description written in a SQL Data Definition Language dialect into C code that implements the schema using Berkeley DB. + +- **--enable-smallbuild** + + To build a small memory footprint version of the Berkeley DB library, enter --enable-smallbuild as an argument to configure. The --enable-smallbuild argument is equivalent to individually specifying --with-cryptography=no, --disable-hash, --disable-queue, --disable-replication, --disable-statistics and --disable-verify, turning off cryptography support, the Hash and Queue access methods, database environment replication support and database and log verification support. See Building a small memory footprint library for more information. + +- **--enable-stl** + + To build the Berkeley DB C++ STL API, enter --enable-stl as an argument to configure. Setting this argument implies that --enable-cxx is set, and the Berkeley DB C++ API will be built too. + + There will be a libdb_stl-X.X.a and libdb_stl-X.X.so built, which are the static and shared library you should link your application with in order to make use of Berkeley DB via its STL API. + + If your compiler is not ISO C++ compliant, the configure may fail with this argument specified because the STL API requires standard C++ template features. In this case, you will need a standard C++ compiler. So far gcc is the best choice, we have tested and found that gcc-3.4.4 and all its newer versions can build the Berkeley DB C++ STL API successfully. + + For information on db_stl supported compilers, see the Portability section in the *Programmer's Reference Guide*. + + And you need to include the STL API header files in your application code. If you are using the Berkeley DB source tree, the header files are in \/stl directory; If you are using the installed version, these header files are in \< Berkeley DB Installed Directory\>/include, as well as the db.h and db_cxx.h header files. + +- **--enable-tcl** + + To build the Berkeley DB Tcl API, enter --enable-tcl as an argument to configure. This configuration argument expects to find Tcl's tclConfig.sh file in the `/usr/local/lib` directory. See the --with-tcl argument for instructions on specifying a non-standard location for the Tcl installation. See Loading Berkeley DB with Tcl for information on sites from which you can download Tcl and which Tcl versions are compatible with Berkeley DB. To build Tcl, you must also build with shared libraries. + +- **--enable-test** + + To build the Berkeley DB test suite, enter --enable-test as an argument to configure. To run the Berkeley DB test suite, you must also build the Tcl API. This argument should not be specified when configuring to build production binaries. + +- **--enable-uimutexes** + + To force Berkeley DB to use the UNIX International (UI) mutex interfaces for underlying mutex support, enter --enable-uimutexes as an argument to configure. This is rarely necessary: UI mutexes will be selected automatically on systems where they are the preferred implementation. + + The --enable-uimutexes configuration argument is normally used when there are multiple mutex implementations available and the UI mutex implementation is not the preferred one (for example, on Solaris where the LWP mutexes are used by default). + + Specifying the --enable-uimutexes configuration argument may require that applications and Berkeley DB be linked with the -lthread library. + +- **--enable-umrw** + + Rational Software's Purify product and other run-time tools complain about uninitialized reads/writes of structure fields whose only purpose is padding, as well as when heap memory that was never initialized is written to disk. Specify the --enable-umrw argument during configuration to mask these errors. This argument should not be specified when configuring to build production binaries. + +- **--enable-dtrace** \[**--enable-perfmon-statistics**\] + + To build Berkeley DB with performance event monitoring probes add --enable-dtrace to the configuration options. Both native DTrace (on Solaris and Mac OS X) and the Statically Defined Tracing compatibility layer in Linux SystemTap version 1.1 or better are supported. That compatibility package may be called systemtap-sdt-devel; it includes `sys/sdt.h`. + + If --enable-perfmon-statistics is combined with --enable-dtrace then additional probes are defined for the tracking variables from which DB's statistics are obtained. They allow DTrace and SystemTap access to these values when they are updated, are the basis of the statistics as displayed db_stat and the API functions that return statistics. + + The --enable-dtrace option may not be specified at the same time as --disable-statistics. + + For information on Berkeley DB Performance Event Monitoring, see the Performance Event Monitoring section in the *Programmer's Reference Guide*. + +- **--enable-localization** + + Enable localized error message text, if available. This option should not be used when `--enable-stripped_messages` is in use. + +- **--enable-stripped_messages** + + Causes all error messages to be stripped of their textual information. Instead, only error return codes are used. This option should not be used when `--enable-localization` is in use. Use of this build option can reduce your library foot print by up to 44KB (.so) or 50KB (.a). + + If you use this configuration option, you can get an idea of what text should be issued for a given error message by using the Message Reference for Stripped Libraries guide. + +- **--with-cryptography** + + To build Berkeley DB with support for cryptography, enter --with-cryptography=yes as an argument to configure. + + To build Berkeley DB without support for cryptography, enter --with-cryptography=no as an argument to configure. + + To build Berkeley DB with support for cryptography using Intel's Performance Primitive (IPP) library, enter --with-cryptography=ipp as an argument to configure. Additionally, set the following arguments: + + -L/path/to/ipp/sharedlib to LDFLAGS + + -I/path/to/ipp/include to CPPFLAGS + + -lippcpem64t -lpthread to LIBS + + An example configuration command for IPP encryption is as follows: + + ``` c + ../dist/configure -with-cryptography=ipp + CPPFLAGS="-I/opt/intel/ipp/6.1.3.055/em64t/include" + LDFLAGS="-L/opt/intel/ipp/6.1.3.055/em64t/sharedlib" + LIBS="-lippcpem64t -lpthread" + ``` + + See the Intel Documenation for specific instructions on configuring environment variables. + + Note: The --with-cryptography=ipp argument works only on Linux. + +- **--with-mutex=MUTEX** + + To force Berkeley DB to use a specific mutex implementation, configure with --with-mutex=MUTEX, where MUTEX is the mutex implementation you want. For example, --with-mutex=x86/gcc-assembly will configure Berkeley DB to use the x86 GNU gcc compiler based test-and-set assembly mutexes. This is rarely necessary and should be done only when the default configuration selects the wrong mutex implementation. A list of available mutex implementations can be found in the distribution file `dist/aclocal/mutex.m4`. + +- **--with-tcl=DIR** + + To build the Berkeley DB Tcl API, enter --with-tcl=DIR, replacing DIR with the directory in which the Tcl tclConfig.sh file may be found. See Loading Berkeley DB with Tcl for information on sites from which you can download Tcl and which Tcl versions are compatible with Berkeley DB. To build Tcl, you must also build with shared libraries. + +- **--with-uniquename=NAME** + + To build Berkeley DB with unique symbol names (in order to avoid conflicts with other application modules or libraries), enter --with-uniquename=NAME, replacing NAME with a string that to be appended to every Berkeley DB symbol. If "=NAME" is not specified, a default value of "\_MAJORMINOR" is used, where MAJORMINOR is the major and minor release numbers of the Berkeley DB release. See Building with multiple versions of Berkeley DB for more information. diff --git a/docs-src/guides/installation/build_unix_db_nosync.md b/docs-src/guides/installation/build_unix_db_nosync.md new file mode 100644 index 000000000..29b3ee564 --- /dev/null +++ b/docs-src/guides/installation/build_unix_db_nosync.md @@ -0,0 +1,10 @@ +--- +title: "DB_NOSYNC Flag to Flush Files" +api-name: "DB_NOSYNC Flag to Flush Files" +source: docs/installation/build_unix_db_nosync.html +--- +## DB_NOSYNC Flag to Flush Files + +Applications must now pass the DB_NOSYNC flag to the methods - `DB->remove`, `DB->rename`, `DB_ENV->dbremove`, and `DB_ENV->dbrename`, to avoid a multi-database file to be flushed from cache. This flag is applicable if you have created the database handle in a non-transactional environment. + +By default, all non-transactional database remove/rename operations cause data to be synced to disk. This can now be overridden using the DB_NOSYNC flag so that files can be accessed outside the environment after the database handles are closed. diff --git a/docs-src/guides/installation/build_unix_encrypt.md b/docs-src/guides/installation/build_unix_encrypt.md new file mode 100644 index 000000000..83e5eb773 --- /dev/null +++ b/docs-src/guides/installation/build_unix_encrypt.md @@ -0,0 +1,16 @@ +--- +title: "Cryptography Support" +api-name: "Cryptography Support" +source: docs/installation/build_unix_encrypt.html +--- +## Cryptography Support + +In this release, the configuration options, --disable-cryptography and --enable-cryptography are deprecated. --disable-cryptography is replaced by --with-cryptography=no and --enable-cryptography is replaced by --with-cryptography=yes. + +To build Berkeley DB with support for cryptography, enter --with-cryptography=yes as an argument to configure instead of --enable-cryptography. + +To build Berkeley DB without support for cryptography, enter --with-cryptography=no as an argument to configure instead of --disable-cryptography. + +Berkeley DB now supports encryption using Intel's Performance Primitive (IPP) on Linux. To build Berkeley DB with support for cryptography using Intel's Performance Primitive (IPP) library, enter --with-cryptography=ipp as an argument to configure. + +Note: The --with-cryptography=ipp argument works only on Linux. diff --git a/docs-src/guides/installation/build_unix_flags.md b/docs-src/guides/installation/build_unix_flags.md new file mode 100644 index 000000000..dab6d6529 --- /dev/null +++ b/docs-src/guides/installation/build_unix_flags.md @@ -0,0 +1,61 @@ +--- +title: "Changing compile or load options" +api-name: "Changing compile or load options" +source: docs/installation/build_unix_flags.html +--- +## Changing compile or load options + +You can specify compiler and/or compile and load time flags by using environment variables during Berkeley DB configuration. For example, if you want to use a specific compiler, specify the CC environment variable before running configure: + +``` c +prompt: env CC=gcc ../dist/configure +``` + +Using anything other than the native compiler will almost certainly mean that you'll want to check the flags specified to the compiler and loader, too. + +To specify debugging and optimization options for the C compiler, use the CFLAGS environment variable: + +``` c +prompt: env CFLAGS=-O2 ../dist/configure +``` + +To specify header file search directories and other miscellaneous options for the C preprocessor and compiler, use the CPPFLAGS environment variable: + +``` c +prompt: env CPPFLAGS=-I/usr/contrib/include ../dist/configure +``` + +To specify debugging and optimization options for the C++ compiler, use the CXXFLAGS environment variable: + +``` c +prompt: env CXXFLAGS=-Woverloaded-virtual ../dist/configure +``` + +To specify miscellaneous options or additional library directories for the linker, use the LDFLAGS environment variable: + +``` c +prompt: env LDFLAGS="-N32 -L/usr/local/lib" ../dist/configure +``` + +If you want to specify additional libraries, set the LIBS environment variable before running configure. For example, the following would specify two additional libraries to load, "posix" and "socket": + +``` c +prompt: env LIBS="-lposix -lsocket" ../dist/configure +``` + +Make sure that you prepend -L to any library directory names and that you prepend -I to any include file directory names! Also, if the arguments you specify contain blank or tab characters, be sure to quote them as shown previously; that is with single or double quotes around the values you are specifying for LIBS. + +The env command, which is available on most systems, simply sets one or more environment variables before running a command. If the env command is not available to you, you can set the environment variables in your shell before running configure. For example, in sh or ksh, you could do the following: + +``` c +prompt: LIBS="-lposix -lsocket" ../dist/configure +``` + +In csh or tcsh, you could do the following: + +``` c +prompt: setenv LIBS "-lposix -lsocket" +prompt: ../dist/configure +``` + +See your command shell's manual page for further information. diff --git a/docs-src/guides/installation/build_unix_freebsd.md b/docs-src/guides/installation/build_unix_freebsd.md new file mode 100644 index 000000000..58f895daa --- /dev/null +++ b/docs-src/guides/installation/build_unix_freebsd.md @@ -0,0 +1,20 @@ +--- +title: "FreeBSD" +api-name: "FreeBSD" +source: docs/installation/build_unix_freebsd.html +--- +## FreeBSD + +1. **I can't compile and run multithreaded applications.** + + Special compile-time flags are required when compiling threaded applications on FreeBSD. If you are compiling a threaded application, you must compile with the \_THREAD_SAFE and -pthread flags: + + ``` c + cc -D_THREAD_SAFE -pthread ... + ``` + + The Berkeley DB library will automatically build with the correct options. + +2. **I see fsync and close system call failures when accessing databases or log files on NFS-mounted filesystems.** + + Some FreeBSD releases are known to return ENOLCK from fsync and close calls on NFS-mounted filesystems, even though the call has succeeded. The Berkeley DB code should be modified to ignore ENOLCK errors, or no Berkeley DB files should be placed on NFS-mounted filesystems on these systems. diff --git a/docs-src/guides/installation/build_unix_install.md b/docs-src/guides/installation/build_unix_install.md new file mode 100644 index 000000000..fad173028 --- /dev/null +++ b/docs-src/guides/installation/build_unix_install.md @@ -0,0 +1,48 @@ +--- +title: "Installing Berkeley DB" +api-name: "Installing Berkeley DB" +source: docs/installation/build_unix_install.html +--- +## Installing Berkeley DB + +Berkeley DB installs the following files into the following locations, with the following default values: + +| Configuration Variables | Default value | +|-------------------------|-------------------------------------------| +| --prefix | /usr/local/BerkeleyDB.**Major**.**Minor** | +| --exec_prefix | \$(prefix) | +| --bindir | \$(exec_prefix)/bin | +| --includedir | \$(prefix)/include | +| --libdir | \$(exec_prefix)/lib | +| docdir | \$(prefix)/docs | + +| Files | Default location | +|---------------|------------------| +| include files | \$(includedir) | +| libraries | \$(libdir) | +| utilities | \$(bindir) | +| documentation | \$(docdir) | + +With one exception, this follows the GNU Autoconf and GNU Coding Standards installation guidelines; please see that documentation for more information and rationale. + +The single exception is the Berkeley DB documentation. The Berkeley DB documentation is provided in HTML format, not in UNIX-style man or GNU info format. For this reason, Berkeley DB configuration does not support **--infodir** or **--mandir**. To change the default installation location for the Berkeley DB documentation, modify the Makefile variable, **docdir**. + +When installing Berkeley DB on filesystems shared by machines of different architectures, please note that although Berkeley DB include files are installed based on the value of \$(prefix), rather than \$(exec_prefix), the Berkeley DB include files are not always architecture independent. + +To move the entire installation tree to somewhere besides **/usr/local**, change the value of **prefix**. + +To move the binaries and libraries to a different location, change the value of **exec_prefix**. The values of **includedir** and **libdir** may be similarly changed. + +Any of these values except for **docdir** may be set as part of the configuration: + +``` c +prompt: ../dist/configure --bindir=/usr/local/bin +``` + +Any of these values, including **docdir**, may be changed when doing the install itself: + +``` c +prompt: make prefix=/usr/contrib/bdb install +``` + +The Berkeley DB installation process will attempt to create any directories that do not already exist on the system. diff --git a/docs-src/guides/installation/build_unix_iphone.md b/docs-src/guides/installation/build_unix_iphone.md new file mode 100644 index 000000000..37e8106f1 --- /dev/null +++ b/docs-src/guides/installation/build_unix_iphone.md @@ -0,0 +1,46 @@ +--- +title: "Apple iOS (iPhone OS)" +api-name: "Apple iOS (iPhone OS)" +source: docs/installation/build_unix_iphone.html +--- +## Apple iOS (iPhone OS) + +Building Berkeley DB in Apple iOS (known as iPhone OS previously) is the same as building for a conventional UNIX platform. This section lists the commands for building Berkeley DB in both the iPhone simulator (a software simulator included in the iPhone SDK that you can use to test your application without using the iPhone/iPod Touch) and the iPhone device. + +Prior to building BDB in an iPhone simulator/iPhone device, set the required environment variables for iOS (iPhone OS). For a simulator/iPhone version 4.2 or older, set the LDFLAGS variable as follows: + +``` c +export LDFLAGS="-L$SDKROOT/usr/lib/" +``` + +Otherwise, set LDFLAGS as follows: + +``` c +export LDFLAGS="-L$SDKROOT/usr/lib/system/" +``` + +The steps to build BDB in an iPhone simulator are as follows: + +``` c +export CFLAGS="-arch i386 -pipe -no-cpp-precomp --sysroot=$SDKROOT" +export CXXFLAGS="-arch i386 -pipe -no-cpp-precomp --sysroot=$SDKROOT" +cd $BDB_HOME/build_unix +../dist/configure --host=i386-apple-darwin\ + --prefix=$SDKROOT ... +make +``` + +The steps to build BDB in an iPhone device are as follows: + +``` c +export CFLAGS="-arch armv6 -pipe -Os -gdwarf-2\ + -no-cpp-precomp -mthumb -isysroot $SDKROOT " +export CXXFLAGS="-arch armv6 -pipe -Os -gdwarf-2\ + -no-cpp-precomp -mthumb -isysroot $SDKROOT " +cd $BDB_HOME/build_unix +../dist/configure --host=arm-apple-darwin9\ + --prefix=$SDKROOT ... +make +``` + +Both sets of commands create the BDB dynamic library - libdb-5.3.dylib. To build the static library, libdb-5.3.a, add the `--enable-shared=no` option while configuring. diff --git a/docs-src/guides/installation/build_unix_irix.md b/docs-src/guides/installation/build_unix_irix.md new file mode 100644 index 000000000..d1f3de528 --- /dev/null +++ b/docs-src/guides/installation/build_unix_irix.md @@ -0,0 +1,16 @@ +--- +title: "IRIX" +api-name: "IRIX" +source: docs/installation/build_unix_irix.html +--- +## IRIX + +1. **I can't compile and run multithreaded applications.** + + Special compile-time flags are required when compiling threaded applications on IRIX. If you are compiling a threaded application, you must compile with the \_SGI_MP_SOURCE flag: + + ``` c + cc -D_SGI_MP_SOURCE ... + ``` + + The Berkeley DB library will automatically build with the correct options. diff --git a/docs-src/guides/installation/build_unix_linux.md b/docs-src/guides/installation/build_unix_linux.md new file mode 100644 index 000000000..a1b173c74 --- /dev/null +++ b/docs-src/guides/installation/build_unix_linux.md @@ -0,0 +1,24 @@ +--- +title: "Linux" +api-name: "Linux" +source: docs/installation/build_unix_linux.html +--- +## Linux + +1. **I can't compile and run multithreaded applications.** + + Special compile-time flags are required when compiling threaded applications on Linux. If you are compiling a threaded application, you must compile with the \_REENTRANT flag: + + ``` c + cc -D_REENTRANT ... + ``` + + The Berkeley DB library will automatically build with the correct options. + +2. **I see database corruption when accessing databases.** + + Some Linux filesystems do not support POSIX filesystem semantics. Specifically, ext2 and early releases of ReiserFS, and ext3 in some configurations, do not support "ordered data mode" and may insert random data into database or log files when systems crash. Berkeley DB files should not be placed on a filesystem that does not support, or is not configured to support, POSIX semantics. + +3. **What scheduler should I use?** + + In some Linux kernels you can select schedulers, and the default is the "anticipatory" scheduler. We recommend not using the "anticipatory" scheduler for transaction processing workloads. diff --git a/docs-src/guides/installation/build_unix_macosx.md b/docs-src/guides/installation/build_unix_macosx.md new file mode 100644 index 000000000..5e6b94577 --- /dev/null +++ b/docs-src/guides/installation/build_unix_macosx.md @@ -0,0 +1,46 @@ +--- +title: "Mac OS X" +api-name: "Mac OS X" +source: docs/installation/build_unix_macosx.html +--- +## Mac OS X + +1. **When trying to link multiple Berkeley DB language interfaces (for example, Tcl, C++, Java, Python) into a single process, I get "multiple definitions" errors from dyld.** + + To fix this problem, set the environment variable MACOSX_DEPLOYMENT_TARGET to 10.3 (or your current version of OS X), and reconfigure and rebuild Berkeley DB from scratch. See the OS X ld(1) and dyld(1) man pages for information about how OS X handles symbol namespaces, as well as undefined and multiply-defined symbols. + +2. **When trying to use system-backed shared memory on OS X I see failures about "too many open files".** + + The default number of shared memory segments on OS X is too low. To fix this problem, edit the file /etc/rc, changing the kern.sysv.shmmax and kern.sysv.shmseg values as follows: + + ``` c + *** /etc/rc.orig Fri Dec 19 09:34:09 2003 + --- /etc/rc Fri Dec 19 09:33:53 2003 + *************** + *** 84,93 **** + # System tuning + sysctl -w kern.maxvnodes=$(echo $(sysctl -n hw.physmem) '33554432 / + 512 * 1024 +p'|dc) + ! sysctl -w kern.sysv.shmmax=4194304 + sysctl -w kern.sysv.shmmin=1 + sysctl -w kern.sysv.shmmni=32 + ! sysctl -w kern.sysv.shmseg=8 + sysctl -w kern.sysv.shmall=1024 + if [ -f /etc/sysctl-macosxserver.conf ]; then + awk '{ if (!-1 && -1) print $1 }' < + /etc/sysctl-macosxserver.conf | while read + --- 84,93 ---- + # System tuning + sysctl -w kern.maxvnodes=$(echo $(sysctl -n hw.physmem) '33554432 / + 512 * 1024 +p'|dc) + ! sysctl -w kern.sysv.shmmax=134217728 + sysctl -w kern.sysv.shmmin=1 + sysctl -w kern.sysv.shmmni=32 + ! sysctl -w kern.sysv.shmseg=32 + sysctl -w kern.sysv.shmall=1024 + if [ -f /etc/sysctl-macosxserver.conf ]; then + awk '{ if (!-1 && -1) print $1 }' < + /etc/sysctl-macosxserver.conf | while read + ``` + + and then reboot the system. diff --git a/docs-src/guides/installation/build_unix_notes.md b/docs-src/guides/installation/build_unix_notes.md new file mode 100644 index 000000000..3fbea4d06 --- /dev/null +++ b/docs-src/guides/installation/build_unix_notes.md @@ -0,0 +1,107 @@ +--- +title: "Architecture independent FAQ" +api-name: "Architecture independent FAQ" +source: docs/installation/build_unix_notes.html +--- +## Architecture independent FAQ + +1. **I have gcc installed, but configure fails to find it.** + + Berkeley DB defaults to using the native C compiler if none is specified. That is usually "cc", but some platforms require a different compiler to build multithreaded code. To configure Berkeley DB to build with gcc, run configure as follows: + + ``` c + env CC=gcc ../dist/configure ... + ``` + +2. **When compiling with gcc, I get unreferenced symbols; for example the following:** + + ``` c + symbol __muldi3: referenced symbol not found + symbol __cmpdi2: referenced symbol not found + ``` + + Berkeley DB often uses 64-bit integral types on systems supporting large files, and gcc performs operations on those types by calling library functions. These unreferenced symbol errors are usually caused by linking an application by calling "ld" rather than by calling "gcc": gcc will link in libgcc.a and will resolve the symbols. If that does not help, another possible workaround is to reconfigure Berkeley DB using the --disable-largefile configuration option and then rebuild. + +3. **My C++ program traps during a failure in a DB call on my gcc-based system.** + + We believe there are some severe bugs in the implementation of exceptions for some gcc compilers. Exceptions require some interaction between compiler, assembler, and runtime libraries. We're not sure exactly what is at fault, but one failing combination is gcc 2.7.2.3 running on SuSE Linux 6.0. The problem on this system can be seen with a rather simple test case of an exception thrown from a shared library and caught in the main program. + + A variation of this problem seems to occur on AIX, although we believe it does not necessarily involve shared libraries on that platform. + + If you see a trap that occurs when an exception might be thrown by the Berkeley DB runtime, we suggest that you use static libraries instead of shared libraries. See the documentation for configuration. If this doesn't work and you have a choice of compilers, try using a more recent gcc- or a non-gcc based compiler to build Berkeley DB. + + Finally, you can disable the use of exceptions in the C++ runtime for Berkeley DB by using the DB_CXX_NO_EXCEPTIONS flag with the DbEnv or Db constructors. When this flag is on, all C++ methods fail by returning an error code rather than throwing an exception. + +4. **I get unexpected results and database corruption when running threaded programs.** + + **I get error messages that mutex (for example, pthread_mutex_XXX or mutex_XXX) functions are undefined when linking applications with Berkeley DB.** + + On some architectures, the Berkeley DB library uses the ISO POSIX standard pthreads and UNIX International (UI) threads interfaces for underlying mutex support; Solaris is an example. You can specify compilers or compiler flags, or link with the appropriate thread library when loading your application to resolve the undefined references: + + ``` c + cc ... -lpthread ... + cc ... -lthread ... + xlc_r ... + cc ... -mt ... + ``` + + See the appropriate architecture-specific Reference Guide pages for more information. + + On systems where more than one type of mutex is available, it may be necessary for applications to use the same threads package from which Berkeley DB draws its mutexes. For example, if Berkeley DB was built to use the POSIX pthreads mutex calls for mutex support, the application may need to be written to use the POSIX pthreads interfaces for its threading model. This is only conjecture at this time, and although we know of no systems that actually have this requirement, it's not unlikely that some exist. + + In a few cases, Berkeley DB can be configured to use specific underlying mutex interfaces. You can use the --enable-posixmutexes and --enable-uimutexes configuration options to specify the POSIX and Unix International (UI) threads packages. This should not, however, be necessary in most cases. + + In some cases, it is vitally important to make sure that you load the correct library. For example, on Solaris systems, there are POSIX pthread interfaces in the C library, so applications can link Berkeley DB using only C library and not see any undefined symbols. However, the C library POSIX pthread mutex support is insufficient for Berkeley DB, and Berkeley DB cannot detect that fact. Similar errors can arise when applications (for example, tclsh) use dlopen to dynamically load Berkeley DB as a library. + + If you are seeing problems in this area after you confirm that you're linking with the correct libraries, there are two other things you can try. First, if your platform supports interlibrary dependencies, we recommend that you change the Berkeley DB Makefile to specify the appropriate threads library when creating the Berkeley DB shared library, as an interlibrary dependency. Second, if your application is using dlopen to dynamically load Berkeley DB, specify the appropriate thread library on the link line when you load the application itself. + +5. **I get core dumps when running programs that fork children.** + + Berkeley DB handles should not be shared across process forks, each forked child should acquire its own Berkeley DB handles. + +6. **I get reports of uninitialized memory reads and writes when running software analysis tools (for example, Rational Software Corp.'s Purify tool).** + + For performance reasons, Berkeley DB does not write the unused portions of database pages or fill in unused structure fields. To turn off these errors when running software analysis tools, build with the --enable-umrw configuration option. + +7. **Berkeley DB programs or the test suite fail unexpectedly.** + + The Berkeley DB architecture does not support placing the shared memory regions on remote filesystems -- for example, the Network File System (NFS) or the Andrew File System (AFS). For this reason, the shared memory regions (normally located in the database home directory) must reside on a local filesystem. See Shared memory region for more information. + + With respect to running the test suite, always check to make sure that TESTDIR is not on a remote mounted filesystem. + +8. **The db_dump utility fails to build.** + + The db_dump185 utility is the utility that supports the conversion of Berkeley DB 1.85 and earlier databases to current database formats. If the build errors look something like the following, it means the db.h include file being loaded is not a Berkeley DB 1.85 version include file: + + ``` c + db_dump185.c: In function `main': + db_dump185.c:210: warning: assignment makes pointer from integer + without a cast + db_dump185.c:212: warning: assignment makes pointer from integer + without a cast + db_dump185.c:227: structure has no member named `seq' + db_dump185.c:227: `R_NEXT' undeclared (first use in this function) + ``` + + If the build errors look something like the following, it means that the Berkeley DB 1.85 code was not found in the standard libraries: + + ``` c + cc -o db_dump185 db_dump185.o + ld: + Unresolved: + dbopen + ``` + + To build the db_dump185 utility, the Berkeley DB version 1.85 code must already been built and available on the system. If the Berkeley DB 1.85 header file is not found in a standard place, or if the library is not part of the standard libraries used for loading, you will need to edit your Makefile, and change the following lines: + + ``` c + DB185INC= + DB185LIB= + ``` + + So that the system Berkeley DB 1.85 header file and library are found; for example: + + ``` c + DB185INC=/usr/local/include + DB185LIB=-ldb185 + ``` diff --git a/docs-src/guides/installation/build_unix_qnx.md b/docs-src/guides/installation/build_unix_qnx.md new file mode 100644 index 000000000..b611332b0 --- /dev/null +++ b/docs-src/guides/installation/build_unix_qnx.md @@ -0,0 +1,34 @@ +--- +title: "QNX" +api-name: "QNX" +source: docs/installation/build_unix_qnx.html +--- +## QNX + +1. **To what versions of QNX has DB been ported?** + + Berkeley DB has been ported to the QNX Neutrino technology which is commonly referred to as QNX RTP (Real-Time Platform). Berkeley DB has not been ported to earlier versions of QNX, such as QNX 4.25. + +2. **Building Berkeley DB shared libraries fails.** + + The `/bin/sh` utility distributed with some QNX releases drops core when running the GNU libtool script (which is used to build Berkeley DB shared libraries). There are two workarounds for this problem: First, only build static libraries. You can disable building shared libraries by specifying the configuration flag when configuring Berkeley DB. + + Second, build Berkeley DB using an alternate shell. QNX distributions include an accessories disk with additional tools. One of the included tools is the GNU bash shell, which is able to run the libtool script. To build Berkeley DB using an alternate shell, move `/bin/sh` aside, link or copy the alternate shell into that location, configure, build and install Berkeley DB, and then replace the original shell utility. + +3. **Are there any QNX filesystem issues?** + + Berkeley DB generates temporary files for use in transactionally protected file system operations. Due to the filename length limit of 48 characters in the QNX filesystem, applications that are using transactions should specify a database name that is at most 43 characters. + +4. **What are the implications of QNX's requirement to use `shm_open`(2) in order to use `mmap`(2)?** + + QNX requires that files mapped with `mmap`(2) be opened using `shm_open`(2). There are other places in addition to the environment shared memory regions, where Berkeley DB tries to memory map files if it can. + + The memory pool subsystem normally attempts to use `mmap`(2) even when using private memory, as indicated by the DB_PRIVATE flag to DB_ENV->open(). In the case of QNX, if an application is using private memory, Berkeley DB will not attempt to map the memory and will instead use the local cache. + +5. **What are the implications of QNX's mutex implementation using microkernel resources?** + + On QNX, the primitives implementing mutexes consume system resources. Therefore, if an application unexpectedly fails, those resources could leak. Berkeley DB solves this problem by always allocating mutexes in the persistent shared memory regions. Then, if an application fails, running recovery or explicitly removing the database environment by calling the DB_ENV->remove() method will allow Berkeley DB to release those previously held mutex resources. If an application specifies the DB_PRIVATE flag (choosing not to use persistent shared memory), and then fails, mutexes allocated in that private memory may leak their underlying system resources. Therefore, the DB_PRIVATE flag should be used with caution on QNX. + +6. **The make clean command fails to execute when building the Berkeley DB SQL interface.** + + Remove the build directory manually to clean up and proceed. diff --git a/docs-src/guides/installation/build_unix_sco.md b/docs-src/guides/installation/build_unix_sco.md new file mode 100644 index 000000000..974007471 --- /dev/null +++ b/docs-src/guides/installation/build_unix_sco.md @@ -0,0 +1,10 @@ +--- +title: "SCO" +api-name: "SCO" +source: docs/installation/build_unix_sco.html +--- +## SCO + +1. **If I build with gcc, programs such as db_dump and db_stat core dump immediately when invoked.** + + We suspect gcc or the runtime loader may have a bug, but we haven't tracked it down. If you want to use gcc, we suggest building static libraries. diff --git a/docs-src/guides/installation/build_unix_shlib.md b/docs-src/guides/installation/build_unix_shlib.md new file mode 100644 index 000000000..c8dadeae7 --- /dev/null +++ b/docs-src/guides/installation/build_unix_shlib.md @@ -0,0 +1,47 @@ +--- +title: "Dynamic shared libraries" +api-name: "Dynamic shared libraries" +source: docs/installation/build_unix_shlib.html +--- +## Dynamic shared libraries + +**Warning**: the following information is intended to be generic and is likely to be correct for most UNIX systems. Unfortunately, dynamic shared libraries are not standard between UNIX systems, so there may be information here that is not correct for your system. If you have problems, consult your compiler and linker manual pages, or your system administrator. + +The Berkeley DB dynamic shared libraries are created with the name libdb-**major**.**minor**.so, where **major** is the major version number and **minor** is the minor version number. Other shared libraries are created if Java and Tcl support are enabled: specifically, libdb_java-**major**.**minor**.so and libdb_tcl-**major**.**minor**.so. + +On most UNIX systems, when any shared library is created, the linker stamps it with a "SONAME". In the case of Berkeley DB, the SONAME is libdb-**major**.**minor**.so. It is important to realize that applications linked against a shared library remember the SONAMEs of the libraries they use and not the underlying names in the filesystem. + +When the Berkeley DB shared library is installed, links are created in the install lib directory so that libdb-**major**.**minor**.so, libdb-**major**.so, and libdb.so all refer to the same library. This library will have an SONAME of libdb-**major**.**minor**.so. + +Any previous versions of the Berkeley DB libraries that are present in the install directory (such as libdb-2.7.so or libdb-2.so) are left unchanged. (Removing or moving old shared libraries is one drastic way to identify applications that have been linked against those vintage releases.) + +Once you have installed the Berkeley DB libraries, unless they are installed in a directory where the linker normally looks for shared libraries, you will need to specify the installation directory as part of compiling and linking against Berkeley DB. Consult your system manuals or system administrator for ways to specify a shared library directory when compiling and linking applications with the Berkeley DB libraries. Many systems support environment variables (for example, LD_LIBRARY_PATH or LD_RUN_PATH), or system configuration files (for example, /etc/ld.so.conf) for this purpose. + +**Warning**: some UNIX installations may have an already existing `/usr/lib/libdb.so`, and this library may be an incompatible version of Berkeley DB. + +We recommend that applications link against libdb.so (for example, using -ldb). Even though the linker uses the file named libdb.so, the executable file for the application remembers the library's SONAME (libdb-**major**.**minor**.so). This has the effect of marking the applications with the versions they need at link time. Because applications locate their needed SONAMEs when they are executed, all previously linked applications will continue to run using the library they were linked with, even when a new version of Berkeley DB is installed and the file `libdb.so` is replaced with a new version. + +Applications that know they are using features specific to a particular Berkeley DB release can be linked to that release. For example, an application wanting to link to Berkeley DB major release "3" can link using -ldb-3, and applications that know about a particular minor release number can specify both major and minor release numbers; for example, -ldb-3.5. + +If you want to link with Berkeley DB before performing library installation, the "make" command will have created a shared library object in the `.libs` subdirectory of the build directory, such as `build_unix/.libs/libdb-major.minor.so`. If you want to link a file against this library, with, for example, a major number of "3" and a minor number of "5", you should be able to do something like the following: + +``` c +cc -L BUILD_DIRECTORY/.libs -o testprog testprog.o -ldb-3.5 +env LD_LIBRARY_PATH="BUILD_DIRECTORY/.libs:$LD_LIBRARY_PATH" ./testprog +``` + +where **BUILD_DIRECTORY** is the full directory path to the directory where you built Berkeley DB. + +The libtool program (which is configured in the build directory) can be used to set the shared library path and run a program. For example, the following runs the gdb debugger on the db_dump utility after setting the appropriate paths: + +``` c +libtool gdb db_dump +``` + +Libtool may not know what to do with arbitrary commands (it is hardwired to recognize "gdb" and some other commands). If it complains the mode argument will usually resolve the problem: + +``` c +libtool --mode=execute my_debugger db_dump +``` + +On most systems, using libtool in this way is exactly equivalent to setting the LD_LIBRARY_PATH environment variable and then executing the program. On other systems, using libtool has the virtue of knowing about any other details on systems that don't behave in this typical way. diff --git a/docs-src/guides/installation/build_unix_small.md b/docs-src/guides/installation/build_unix_small.md new file mode 100644 index 000000000..7d283c890 --- /dev/null +++ b/docs-src/guides/installation/build_unix_small.md @@ -0,0 +1,61 @@ +--- +title: "Building a small memory footprint library" +api-name: "Building a small memory footprint library" +source: docs/installation/build_unix_small.html +--- +## Building a small memory footprint library + +There are a set of configuration options to assist you in building a small memory footprint library. These configuration options turn off specific functionality in the Berkeley DB library, reducing the code size. These configuration options include: + + `--enable-smallbuild` +Equivalent to individually specifying all of the following configuration options. In addition, when compiling building with the GNU gcc compiler, this option uses the `-Os` compiler build flag instead of the default `-O3`. + + `--with-cryptography=no` +Builds Berkeley DB without support for cryptography. + + `--disable-hash` +Builds Berkeley DB without support for the Hash access method. + + `--disable-heap` +Builds Berkeley DB without support for the Heap access method. + + `--disable-queue` +Builds Berkeley DB without support for the Queue access method. + + `--disable-replication` +Builds Berkeley DB without support for the database environment replication. + + `--disable-statistics` +Builds Berkeley DB without support for the statistics interfaces. + + `--disable-verify` +Builds Berkeley DB without support for database verification. + + `--enable-stripped_messages` +Strips message text from the error messages issued by Berkeley DB. This can reduce the size of the library by roughly another 22KB. + +If your library has stripped messages, you can get an idea of what text should be issued for a given error message by using the Message Reference for Stripped Libraries guide. + +### Note + +`--disable-cryptography` and `--enable-cryptography` are deprecated in the Berkeley DB 11gR2 release. Use `--with-cryptography=no` and `--with-cryptography=yes` instead. + +The following configuration options will increase the size of the Berkeley DB library dramatically and are only useful when debugging applications: + + --enable-debug +Build Berkeley DB with symbols for debugging. + + --enable-debug_rop +Build Berkeley DB with read-operation logging. + + --enable-debug_wop +Build Berkeley DB with write-operation logging. + + --enable-diagnostic +Build Berkeley DB with run-time debugging checks. + +In addition, static libraries are usually smaller than shared libraries. By default Berkeley DB will build both shared and static libraries. To build only a static library, configure Berkeley DB with the Configuring Berkeley DB option. + +The size of the Berkeley DB library varies depending on the compiler, machine architecture, and configuration options. As an estimate, production Berkeley DB libraries built with GNU gcc version 4.X compilers have footprints in the range of 600KB to 1.4MB on 32-bit x86 architectures, and in the range of 700KB to 1.6MB on 64-bit x86 architectures. + +For assistance in further reducing the size of the Berkeley DB library, or in building small memory footprint libraries on other systems, please contact Berkeley DB support. diff --git a/docs-src/guides/installation/build_unix_solaris.md b/docs-src/guides/installation/build_unix_solaris.md new file mode 100644 index 000000000..7fafec1c4 --- /dev/null +++ b/docs-src/guides/installation/build_unix_solaris.md @@ -0,0 +1,89 @@ +--- +title: "Solaris" +api-name: "Solaris" +source: docs/installation/build_unix_solaris.html +--- +## Solaris + +1. **I can't compile and run multithreaded applications.** + + Special compile-time flags and additional libraries are required when compiling threaded applications on Solaris. If you are compiling a threaded application, you must compile with the D_REENTRANT flag and link with the libpthread.a or libthread.a libraries: + + ``` c + cc -mt ... + cc -D_REENTRANT ... -lthread + cc -D_REENTRANT ... -lpthread + ``` + + The Berkeley DB library will automatically build with the correct options. + +2. **I've installed gcc on my Solaris system, but configuration fails because the compiler doesn't work.** + + On some versions of Solaris, there is a cc executable in the user's path, but all it does is display an error message and fail: + + ``` c + % which cc + /usr/ucb/cc + % cc + /usr/ucb/cc: language optional software package not installed + ``` + + Because Berkeley DB always uses the native compiler in preference to gcc, this is a fatal error. If the error message you are seeing is the following, then this may be the problem: + + ``` c + checking whether the C compiler (cc -O) works... no + configure: error: installation or configuration problem: C compiler + cannot create executables. + ``` + + The simplest workaround is to set your CC environment variable to the system compiler and reconfigure; for example: + + ``` c + env CC=gcc ../dist/configure + ``` + + If you are using the --configure-cxx option, you may also want to specify a C++ compiler, for example the following: + + ``` c + env CC=gcc CCC=g++ ../dist/configure + ``` + +3. **I see the error "libc internal error: \_rmutex_unlock: rmutex not held", followed by a core dump when running threaded or JAVA programs.** + + This is a known bug in Solaris 2.5 and it is fixed by Sun patch 103187-25. + +4. **I see error reports of nonexistent files, corrupted metadata pages and core dumps.** + + Solaris 7 contains a bug in the threading libraries (-lpthread, -lthread), which causes the wrong version of the pwrite routine to be linked into the application if the thread library is linked in after the C library. The result will be that the pwrite function is called rather than the pwrite64. To work around the problem, use an explicit link order when creating your application. + + Sun Microsystems is tracking this problem with Bug Id's 4291109 and 4267207, and patch 106980-09 to Solaris 7 fixes the problem: + + ``` c + Bug Id: 4291109 + Duplicate of: 4267207 + Category: library + Subcategory: libthread + State: closed + Synopsis: pwrite64 mapped to pwrite + Description: + When libthread is linked after libc, there is a table of functions in + libthread that gets "wired into" libc via _libc_threads_interface(). + The table in libthread is wrong in both Solaris 7 and on28_35 for the + TI_PWRITE64 row (see near the end). + ``` + +5. **I see corrupted databases when doing hot backups or creating a hot failover archive.** + + The Solaris cp utility is implemented using the mmap system call, and so writes are not blocked when it reads database pages. See Berkeley DB recoverability for more information. + +6. **Performance is slow and the application is doing a lot of I/O to the disk on which the database environment's files are stored.** + + By default, Solaris periodically flushes dirty blocks from memory-mapped files to the backing filesystem. This includes the Berkeley DB database environment's shared memory regions and can affect Berkeley DB performance. Workarounds include creating the shared regions in system shared memory (DB_SYSTEM_MEM) or application private memory (DB_PRIVATE), or configuring Solaris to not flush memory-mapped pages. For more information, see the "Solaris Tunable Parameters Reference Manual: fsflush and Related Tunables". + +7. **I see errors about "open64" when building Berkeley DB applications.** + + System include files (most commonly fcntl.h) in some releases of AIX and Solaris redefine "open" when large-file support is enabled for applications. This causes problems when compiling applications because "open" is a method in the Berkeley DB APIs. To work around this problem: + + 1. Avoid including the problematical system include files in source code files which also include Berkeley DB include files and call into the Berkeley DB API. + 2. Before building Berkeley DB, modify the generated include file db.h to itself include the problematical system include files. + 3. Turn off Berkeley DB large-file support by specifying the --disable-largefile configuration option and rebuilding. diff --git a/docs-src/guides/installation/build_unix_sql.md b/docs-src/guides/installation/build_unix_sql.md new file mode 100644 index 000000000..c0940828b --- /dev/null +++ b/docs-src/guides/installation/build_unix_sql.md @@ -0,0 +1,263 @@ +--- +title: "Configuring the SQL Interface" +api-name: "Configuring the SQL Interface" +source: docs/installation/build_unix_sql.html +--- +## Configuring the SQL Interface + + [Changing Compile Options](build_unix_sql.md#config_sql) + + [Enabling Extensions](build_unix_sql.md#idp500824) + + [Building the JDBC Driver](build_unix_sql.md#build_unix_jdbc) + + [Using the JDBC Driver](build_unix_sql.md#idp571856) + + [Building the ODBC Driver](build_unix_sql.md#idp593744) + + [Building the BFILE extension](build_unix_sql.md#bfile) + +There are a set of options you can provide to **configure** in order to control how the Berkeley DB SQL interface is built. These configuration options include: + +--disable-log-checksum +Disables checksums in log records. This provides a boost to performance at the risk of log files having undetectable corruption that could prevent proper data recovery in case of database corruption. + +Note that while this option is meant for use with the SQL interface, it will also disable checksum for the non-SQL interfaces. + +--enable-sql +Causes the **dbsql** command line interpreter to be built. Along with **dbsql**, this argument also builds the libdb_sqlXX.{so\|la} library, a C API library that mirrors the SQLite C API. + +--enable-sql_compat +Causes the **sqlite3** command line tool to be built. This tool is identical to the **dbsql** command line tool, except that it has the same name as the command line tool that comes with standard SQLite. + +In addition, the libsqlite3.{so\|la} C API library is built if this option is specified. This library is identical to the libdb_sqlXX.{so\|la} library that is normally built for Berkeley DB's sql interface, except that it has the same name as the library which is built for standard SQLite. + +### Warning + +Use this compatibility option with *extreme* care. Standard SQLite is used by many programs and utilities on many different platforms. Some platforms, such as Mac OS X, come with standard SQLite built in because default applications for the platform use that library. + +**Use of this option on platforms where standard SQLite is in production use can cause unexpected runtime errors either for your own application, or for applications and utilities commonly found on the platform, depending on which library is found first in the platform's library search path.** + +Use this option *only* if you know exactly what you are doing. + +This option is provided so that there is an easy upgrade path for legacy SQLite tools and scripts that want to use BDB SQL without rewriting the tool or script. However, data contained in standard SQLite databases must be manually migrated from the old database to your BDB SQL database even if you use this option. See the *Berkeley DB Getting Started with the SQL APIs* guide for information on migrating data from standard SQLite to BDB SQL databases. + +Note that in addition to the renamed command line tool and library, this option also causes versions of the command line tool and library to be built that use the normal BDB SQLite names (**dbsql** and libdb_sqlXX.{so\|la}). + +--enable-test +Cause the Berkeley DB SQL interface test suite to be built. This argument can also be used with either `--enable-sql` or `--enable-sql_compat` to build the SQLite Tcl test runner. + +--enable-jdbc +Causes the JDBC driver to be built. Setting this option implies that `--enable-sql` is set, which means that the Berkeley DB SQL API will be built too. + +The following configuration options are useful when debugging applications: + + --enable-debug +Builds the Berkeley DB SQL interface with debug symbols. + + --enable-diagnostic +Builds the Berkeley DB SQL interface with run-time debugging checks. + +Any arguments that you can provide to the standard SQLite configure script can also be supplied when configuring Berkeley DB SQL interface. + +### Changing Compile Options + +There are several configuration options you can specify as an argument to the configure script using the standard environment variable, CFLAGS. + +BDBSQL_DEFAULT_PAGE_SIZE +To set the default page size when you create a database, specify the BDBSQL_DEFAULT_PAGE_SIZE flag. The value assigned must be a 0, 512, 1024, 2048, 4096, 8192 16384, 32768, or 65536. The default value is 4096. If the value is set to zero, Berkeley DB queries the file system to determine the best page size, and the value of SQLITE_DEFAULT_PAGE_SIZE is used to calculate the cache size, as the cache size is specified as a number of pages. + +BDBSQL_FILE_PER_TABLE +To generate each table in a separate file, rather than as subdatabases in a single file, specify the BDBSQL_FILE_PER_TABLE flag. When this option is enabled, the SQL database name is used as a directory name. This directory contains one file for the metadata and one file each for every table created by the SQL API. Note that adding or deleting files from the database directory may corrupt your database. To backup the metadata (schema), make a copy of the `metadata` and `table00001` files from the database directory. Make a new copy whenever the schema is changed. + +BDBSQL_LOG_REGIONMAX +To configure the log region size for the underlying storage engine, specify the BDBSQL_LOG_REGIONMAX flag. For more information, see DB_ENV->get_lg_regionmax(). + +BDBSQL_OMIT_LEAKCHECK +For Berkeley DB to use the default system allocation routines rather than the SQLite allocation routines, specify the BDBSQL_OMIT_LEAKCHECK flag. + +BDBSQL_OMIT_LOG_REMOVE +Berkeley DB automatically removes log files that are not required any more, that is, files that are older than the most recent checkpoint. To disable this functionality, specify the BDBSQL_OMIT_LOG_REMOVE flag. It is necessary to provide this flag if you are using replication with Berkeley DB SQL. + +BDBSQL_OMIT_SHARING +To create a private environment rather than a shared environment, specify the BDBSQL_OMIT_SHARING flag. That is, the cache and other region files will be created in memory rather than using file backed shared memory. For more information, see the DB_PRIVATE flag of DB_ENV->open(). + +BDBSQL_SINGLE_THREAD +To disable locking and thread safe connections, specify the BDBSQL_SINGLE_THREAD flag. If an application is going to use Berkeley DB from a single thread and a single process, enabling this flag can deliver significant performance advantages. + +SQLITE_DEFAULT_CACHE_SIZE +SQLite provides an in-memory cache which you size according to the maximum number of database pages that you want to hold in memory at any given time. Berkeley DB's in-memory cache feature performs the same function as SQLite. To specify the suggested maximum number of pages of disk cache that will be allocated per open database file specify the SQLITE_DEFAULT_CACHE_SIZE flag. Default value is 2000 pages. For more information, see the SQLite documentation on PRAGMA default_cache_size. + +SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT +For SQLite, this pragma identifies the maximum size that the journal file is allowed to be. Berkeley DB does not have a journal file, but it writes and uses log files. A new log file is created when the current log file has reached the defined maximum size. To define this maximum size for a log file, specify the SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT flag. Default value is 10 MB for the Berkeley DB SQL interface. + +### Enabling Extensions + +The Berkeley DB SQL API provides extensions such as full text search and R-Tree index. By default, these extensions are disabled. To enable an extension in the Berkeley DB SQL interface, specify the related option as an argument to the configure script using the standard environment variable, CPPFLAGS. + +SQLITE_ENABLE_FTS3 +Enable building the Berkeley DB full text search layer + +SQLITE_ENABLE_RTREE +Enables the Berkeley DB R-Tree layer. + +See the SQLite Documentation for more information on full text search and R-Tree. + +### Building the JDBC Driver + +This section describes how to build the JDBC driver code using `autoconf`, which is the only method supported and tested by the Berkeley DB team. + +To build the JDBC driver, you must have Sun Java Development Kit 1.1 or above installed. + +``` c +cd build_unix + ../dist/configure --enable-jdbc --prefix= + make install +``` + +You can test the build by entering the following commands from the `build_unix/jdbc` directory: + +| | +|------------------------------------------------------------------| +| javac -classpath ./sqlite.jar test3.java | +| java -Djava.library.path=./.libs -classpath ./sqlite.jar:. test3 | + +### Using the JDBC Driver + +This section describes how to download, build, and run sample programs using the built JDBC driver. + +#### Downloading JDBC Sample Code + +The download link for JDBC sample code is available on the Oracle Technology Network (OTN) page. You can identify the link by the "JDBC programming examples from all three editions (ZIP format)" text beside it. + +#### Modifying Sample Code + +Before running the example code, do the following: + +1. Unzip the file containing the sample code to a new directory (for example, jdbc_ex). + +2. Substitute `jdbc:sqlite:/` for the generic JDBC URL that appears in the code. That is, put `jdbc:sqlite:/` between the quotation marks in the line: + + `String url = "jdbc:mySubprotocol:myDataSource";` + + Note: The \ can either be an absolute path name like `"/jdbc_ex_db/myDataSource"`, or a relative path-file-name like `"../jdbc_ex_db/myDataSource"`, or a file name, like `"myDataSource"`, in which case the database file will be stored at the current directory. + +3. Substitute `SQLite.JDBCDriver` for `myDriver.ClassName` in the line: `Class.forName("myDriver.ClassName");` + +4. Optionally substitute the username and password you use for your database in the following: `"myLogin", "myPassword"`. + +5. If your JDK version is above 1.5, change the variable name `enum` in `OutputApplet.java` to some other variable name because, as of JDK release 5 `enum` is a keyword and can not be used as an identifier. + +#### Building and Running the JDBC Sample code + +See Building the JDBC Driver for instructions on building the JDBC driver. + +To build and run the JDBC examples do the following: + +1. Copy `build_unix/jdbc/sqlite.jar` and `build_unix/jdbc/.libs/libsqlite_jni.so` to the `jdbc_ex` directory. + +2. In the `jdbc_ex` directory, run the following commands: + + ``` c + $ javac -classpath ./sqlite.jar *.java + $ java -classpath .:sqlite.jar -Djava.library.path=. \ + + + ``` + +3. After you run the CreateCoffees example, use the `dbsql` executable to open the `myDataSource` database file and check if the table `COFFEES` has been successfully created in the database. + + ``` c + $ dbsql myDataSourcedbsql> .tables + COFFEES + dbsql> .dump + PRAGMA foreign_keys=OFF; + BEGIN TRANSACTION; + CREATE TABLE COFFEES (COF_NAME varchar(32),\ + SUP_ID int, PRICE float, SALES int, TOTAL int); + COMMIT; + dbsql> + + ``` + +4. Repeat step 3 to run other examples. + + Note: Some examples, such as AutoGenKeys, are not yet supported by BDB JDBC driver. The `SQLFeatureNotSupportedException` is displayed for those unsupported examples. + +### Building the ODBC Driver + +This section describes how to build the ODBC driver. + +#### Configuring Your System + +To configure your system prior to building the ODBC driver, do the following: + +1. Download and install the latest unixODBC if ODBC is not already installed on your system. +2. Configure the ODBC server to work with SQLite databases. Follow these instructions from Christian Werner. + +#### Building the Library + +To build the library, do the following: + +``` c + $ cd db-5.3.XX/build_unix + $ CFLAGS="-fPIC" ../dist/configure --enable-sql_compat --disable-shared + $ make + $ cd ../lang/sql/odbc + $ CFLAGS="-DHAVE_ERRNO_H -I../../../build_unix -I../../../src/dbinc \ + -I../sqlite/src" LDFLAGS="../../../build_unix/libdb-5.3.a" \ + ./configure --with-sqlite3=../generated + $ make + +``` + +The `libsqlite3odbc.so` library containing a statically linked version of Berkeley DB SQL is now built. + +NOTE: The final `make` command above is known to generate a warning when using GCC. The warning states: `Warning: Linking the shared library libsqlite3odbc.la against the static library ../../build_unix/libdb-5.3.a is not portable!`. It is generally safe to ignore the warning when using the generated library. + +#### Testing the ODBC Driver + +The steps to verify that the installed driver works are as follows: + +1. Alter the `/etc/odbcinst.ini` and `~/.odbc.ini` configuration files to refer to the libsqlite3odbc.so file built above. + +2. Create a data source, and launch a data source viewer application by doing the following: + + ``` c + $ mkdir ~/databases + $ cd ~/databases + $ /path/to/Berkeley DB/build_unix/sqlite3 mytest.db + dbsql> CREATE TABLE t1(x); + dbsql> .quit; + $ DataManager + ``` + + The final step opens a GUI application that displays ODBC data sources on a system. You should be able to find the `mytest.db` data source just created. + +### Building the BFILE extension + +The BFILE extension allows you to store binary files outside of the database, but still operate upon them as if they were stored within the database. To enable this extension, use the `--enable-load-extension` configuration flag. For example: + +``` c +$ cd /build_unix +$ export DBSQL_DIR=$PWD/../install +$ ../dist/configure --enable-sql --enable-load-extension \ + --prefix=$DBSQL_DIR && make && make install +$ cd ../lang/sql/sqlite/ext/bfile/build +$ make && make install +``` + +BFILE extensions are only supported for Unix platforms. + +Note that the extension support has two interfaces: SQL expressions and a C-functions API. By default, the SQL expressions are built when you use `--enable-load_extension`. To use the C-functions API, edit `/lang/sql/ext/bfile/build/Makefile` and set `ENABLE_BFILE_CAPI` to `1`. + +Once you have enabled the extension and built the library, you can run the included example: + +``` c +$ cd lang/sql/sqlite/ext/bfile/build +$ export LD_LIBRARY_PATH=$PWD:$DBSQL_DIR/lib +$ ./bfile_example_sql # for SQL expressions interface +$ ./bfile_example_capi # for C-functions API +``` + +For more information on using the BFILE extension, see the *Berkeley DB Getting Started with the SQL APIs* guide. diff --git a/docs-src/guides/installation/build_unix_stacksize.md b/docs-src/guides/installation/build_unix_stacksize.md new file mode 100644 index 000000000..f45bb08ae --- /dev/null +++ b/docs-src/guides/installation/build_unix_stacksize.md @@ -0,0 +1,8 @@ +--- +title: "Changing Stack Size" +api-name: "Changing Stack Size" +source: docs/installation/build_unix_stacksize.html +--- +## Changing Stack Size + +Prior to the 11gR2 release, Berkeley DB limited the stack size for threads it created using the POSIX thread API to 128 KB for 32-bit platforms and 256 KB for 64-bit platforms. In this release, the system default stack size is used unless you run the Berkeley DB configure script with the `--with-stacksize=SIZE` argument to override the default. diff --git a/docs-src/guides/installation/build_unix_sunos.md b/docs-src/guides/installation/build_unix_sunos.md new file mode 100644 index 000000000..ac298dee9 --- /dev/null +++ b/docs-src/guides/installation/build_unix_sunos.md @@ -0,0 +1,10 @@ +--- +title: "SunOS" +api-name: "SunOS" +source: docs/installation/build_unix_sunos.html +--- +## SunOS + +1. **I can't specify the DB_SYSTEM_MEM flag to DB_ENV->open().** + + The `shmget`(2) interfaces are not used on SunOS releases prior to 5.0, even though they apparently exist, because the distributed include files did not allow them to be compiled. For this reason, it will not be possible to specify the DB_SYSTEM_MEM flag to those versions of SunOS. diff --git a/docs-src/guides/installation/build_unix_test.md b/docs-src/guides/installation/build_unix_test.md new file mode 100644 index 000000000..36ff896b5 --- /dev/null +++ b/docs-src/guides/installation/build_unix_test.md @@ -0,0 +1,46 @@ +--- +title: "Running the test suite under UNIX" +api-name: "Running the test suite under UNIX" +source: docs/installation/build_unix_test.html +--- +## Running the test suite under UNIX + + [Building SQL Test Suite on Unix](build_unix_test.md#build_unix_test_sql) + +The Berkeley DB test suite is built if you specify --enable-test as an argument when configuring Berkeley DB. The test suite also requires that you configure and build the Tcl interface to the library. + +Before running the tests for the first time, you may need to edit the `include.tcl` file in your build directory. The Berkeley DB configuration assumes that you intend to use the version of the tclsh utility included in the Tcl installation with which Berkeley DB was configured to run the test suite, and further assumes that the test suite will be run with the libraries prebuilt in the Berkeley DB build directory. If either of these assumptions are incorrect, you will need to edit the `include.tcl` file and change the following line to correctly specify the full path to the version of tclsh with which you are going to run the test suite: + +``` c +set tclsh_path ... +``` + +You may also need to change the following line to correctly specify the path from the directory where you are running the test suite to the location of the Berkeley DB Tcl library you built: + +``` c +set test_path ... +``` + +It may not be necessary that this be a full path if you have configured your system's shared library mechanisms to search the directory where you built or installed the Tcl library. + +All Berkeley DB tests are run from within **tclsh**. After starting tclsh, you must source the file `test.tcl` in the test directory. For example, if you built in the `build_unix` directory of the distribution, this would be done using the following command: + +``` c +% source ../test/tcl/test.tcl +``` + +If no errors occur, you should get a "%" prompt. + +You are now ready to run tests in the test suite; see Running the test suite for more information. + +### Building SQL Test Suite on Unix + +The Berkeley DB SQL interface test suite is built if you specify --enable-test and --enable-sql as arguments, when configuring Berkeley DB. The test suite also requires that you build the Berkeley DB Tcl API. + +``` c +../dist/configure --enable-sql --enable-test --with-tcl=/usr/lib +``` + +This builds the *testfixture* project in `../build_unix/sql`. + +To enable extensions like full text search layer and R-Tree layer in the SQL test suite, configure with --enable-amalgamation. diff --git a/docs-src/guides/installation/build_win.md b/docs-src/guides/installation/build_win.md new file mode 100644 index 000000000..0d4a68540 --- /dev/null +++ b/docs-src/guides/installation/build_win.md @@ -0,0 +1,129 @@ +--- +title: "Chapter 5.  Building Berkeley DB for Windows" +api-name: "Chapter 5.  Building Berkeley DB for Windows" +source: docs/installation/build_win.html +--- +## Chapter 5.  Building Berkeley DB for Windows + +**Table of Contents** + + [Building Berkeley DB for 32 bit Windows](build_win.md#win_build32) + + [Visual C++ .NET 2010](build_win.md#idp242512) + + [Visual C++ .NET 2008](build_win.md#idp249264) + + [Visual C++ .NET 2005](build_win.md#idp220616) + + [Build results](build_win.md#idp205672) + + [Building Berkeley DB for 64-bit Windows](win_build64.md) + + [x64 build with Visual Studio 2005 or newer](win_build64.md#idp259672) + + [Building Berkeley DB with Cygwin](win_build_cygwin.md) + + [Building the C++ API](win_build_cxx.md) + + [Building the C++ STL API](win_build_stl.md) + + [Building the Java API](build_win_java.md) + + [Building the C# API](build_win_csharp.md) + + [Building the SQL API](build_win_sql.md) + + [Binary Compatibility With SQLite](build_win_sql.md#idp290248) + + [Setting Preprocessor Flags](build_win_sql.md#idp276576) + + [Enabling Extensions](build_win_sql.md#idp288280) + + [Disabling Log Checksums](build_win_sql.md#win-disablechecksums) + + [Building the JDBC Driver](build_win_sql.md#build_jdbc) + + [Using the JDBC Driver](build_win_sql.md#idp266616) + + [Building the ODBC Driver](build_win_sql.md#idp305704) + + [Using the ADO.NET Driver](build_win_sql.md#idp320888) + + [Building the Tcl API](build_win_tcl.md) + + [Distributing DLLs](win_build_dist_dll.md) + + [Additional build options](win_additional_options.md) + + [Building a small memory footprint library](build_win_small.md) + + [Running the test suite under Windows](build_win_test.md) + + [Building the software needed by the tests](build_win_test.md#idp368040) + + [Running the test suite under Windows](build_win_test.md#idp379184) + + [Building the software needed by the SQL tests](build_win_test.md#build_win_test_sql) + + [Windows notes](build_win_notes.md) + + [Windows FAQ](build_win_faq.md) + +This chapter contains general instructions on building Berkeley DB for specific windows platforms using specific compilers. The Windows FAQ also contains helpful information. + +The `build_windows` directory in the Berkeley DB distribution contains project files for Microsoft Visual Studio: + +| Project File | Description | +|:----------------------:|:----------------------------------:| +| Berkeley_DB.sln | Visual Studio 2005 (8.0) workspace | +| \*.vcproj | Visual Studio 2005 (8.0) projects | +| Berkeley_DB_vs2010.sln | Visual Studio 2010 workspace | +| \*.vcxproj | Visual Studio 2010 projects | + +These project files can be used to build Berkeley DB for the following platforms: Windows NT/2K/XP/2003/Vista and Windows7; and 64-bit Windows XP/2003/Vista and Windows7. + +## Building Berkeley DB for 32 bit Windows + + [Visual C++ .NET 2010](build_win.md#idp242512) + + [Visual C++ .NET 2008](build_win.md#idp249264) + + [Visual C++ .NET 2005](build_win.md#idp220616) + + [Build results](build_win.md#idp205672) + +### Visual C++ .NET 2010 + +1. Choose *File -\> Open -\> Project/Solution...*. In the `build_windows` directory, select `Berkeley_DB_vs2010.sln` and click Open. +2. Choose the desired project configuration from the drop-down menu on the tool bar (either Debug or Release). +3. Choose the desired platform configuration from the drop-down menu on the tool bar (usually Win32 or x64). +4. To build, right-click on the `Berkeley_DB_vs2010` solution and select Build Solution. + +### Visual C++ .NET 2008 + +1. Choose *File -\> Open -\> Project/Solution...*. In the `build_windows` directory, select `Berkeley_DB.sln` and click Open. +2. The *Visual Studio Conversion Wizard* will open automatically. Click the *Finish* button. +3. On the next screen click the *Close* button. +4. Choose the desired project configuration from the drop-down menu on the tool bar (either Debug or Release). +5. Choose the desired platform configuration from the drop-down menu on the tool bar (usually Win32 or x64). +6. To build, right-click on the Berkeley_DB solution and select Build Solution. + +### Visual C++ .NET 2005 + +1. Choose *File -\> Open -\> Project/Solution...*. In the `build_windows` directory, select `Berkeley_DB.sln` and click Open +2. Choose the desired project configuration from the drop-down menu on the tool bar (either Debug or Release). +3. Choose the desired platform configuration from the drop-down menu on the tool bar (usually Win32 or x64). +4. To build, right-click on the Berkeley_DB solution and select Build Solution. + +### Build results + +The results of your build will be placed in one of the following Berkeley DB subdirectories, depending on the configuration that you chose: + +| | +|--------------------------------------| +| `build_windows\Win32\Debug` | +| `build_windows\Win32\Release` | +| `build_windows\Win32\Debug_static` | +| `build_windows\Win32\Release_static` | + +When building your application during development, you should normally use compile options "Debug Multithreaded DLL" and link against `build_windows\Debug\libdb53d.lib`. You can also build using a release version of the Berkeley DB libraries and tools, which will be placed in `build_windows\Win32\Release\libdb53.lib`. When linking against the release build, you should compile your code with the "Release Multithreaded DLL" compile option. You will also need to add the `build_windows` directory to the list of include directories of your application's project, or copy the Berkeley DB include files to another location. diff --git a/docs-src/guides/installation/build_win_csharp.md b/docs-src/guides/installation/build_win_csharp.md new file mode 100644 index 000000000..9b8c8eb98 --- /dev/null +++ b/docs-src/guides/installation/build_win_csharp.md @@ -0,0 +1,28 @@ +--- +title: "Building the C# API" +api-name: "Building the C# API" +source: docs/installation/build_win_csharp.html +--- +## Building the C# API + +The C# support is built by a separate Visual Studio solution and requires version 2.0 (or higher) of the .NET platform. If the Berkeley DB utilities are required, build Berkeley DB first following the instructions under Building Berkeley DB for 32 bit Windows or Building Berkeley DB for 64-bit Windows. + +To build the C# API in Visual Studio 2005/Visual Studio 2008, the solution is `build_windows\BDB_dotnet.sln`; in Visual Studio 2010, the solution is `build_windows\BDB_dotnet_vs2010.sln`. + +By default, the solution will build the native libraries, the managed assembly and all example programs. The NUnit tests need to be built explicitly because of their dependence upon the NUnit assembly. The native libraries will be placed in one of the following subdirectories, depending upon the chosen configuration: + +| | +|-------------------------------| +| `build_windows\Win32\Debug` | +| `build_windows\Win32\Release` | +| `build_windows\x64\Debug` | +| `build_windows\x64\Release` | + +The managed assembly and all C# example programs will be placed in one of the following subdirectories, depending upon the chosen configuration: + +| | +|--------------------------------| +| `build_windows\AnyCPU\Debug` | +| `build_windows\AnyCPU\Release` | + +The native libraries need to be locatable by the .NET platform, meaning they must be copied into an application's directory, the Windows or System directory, or their location must be added to the PATH environment variable. The example programs demonstrate how to programmatically edit the PATH variable. diff --git a/docs-src/guides/installation/build_win_faq.md b/docs-src/guides/installation/build_win_faq.md new file mode 100644 index 000000000..1cb610f76 --- /dev/null +++ b/docs-src/guides/installation/build_win_faq.md @@ -0,0 +1,38 @@ +--- +title: "Windows FAQ" +api-name: "Windows FAQ" +source: docs/installation/build_win_faq.html +--- +## Windows FAQ + +1. **My Win\* C/C++ application crashes in the Berkeley DB library when Berkeley DB calls fprintf (or some other standard C library function).** + + You should be using the "Debug Multithreaded DLL" compiler option in your application when you link with the build_windows\Debug\libdb48d.lib library (this .lib file is actually a stub for libdb48d.DLL). To check this setting in Visual C++, choose the *Project/Settings* menu item and select *Code Generation* under the tab marked *C/C++*; and see the box marked *Use runtime library*. This should be set to *Debug Multithreaded DLL*. If your application is linked against the static library, build_windows\Debug\libdb48sd.lib; then, you will want to set *Use runtime library* to *Debug Multithreaded*. + + Setting this option incorrectly can cause multiple versions of the standard libraries to be linked into your application (one on behalf of your application, and one on behalf of the Berkeley DB library). That violates assumptions made by these libraries, and traps can result. + + Also, using different Visual Studio compilers in the application and libraries can lead to a crash. So rebuild the application with the same Visual C++ version as that of the library. + +2. **Why are the build options for DB_DLL marked as "Use MFC in a Shared DLL"? Does Berkeley DB use MFC?** + + Berkeley DB does not use MFC at all. It does however, call malloc and free and other facilities provided by the Microsoft C runtime library. We found in our work that many applications and libraries are built assuming MFC, and specifying this for Berkeley DB solves various interoperation issues, and guarantees that the right runtime libraries are selected. Note that because we do not use MFC facilities, the MFC library DLL is not marked as a dependency for libdb.dll, but the appropriate Microsoft C runtime is. + +3. **How can I build Berkeley DB for MinGW?** + + Follow the instructions in Building for UNIX/POSIX, and specify the --enable-mingw option to the configuration script. This configuration option currently only builds static versions of the library, it does not yet build a DLL version of the library, and file sizes are limited to 2GB (2^32 bytes.) + +4. **How can I build a Berkeley DB for Windows 98/ME?** + + Windows 98/ME is no longer supported by Berkeley DB. The following is therefore only of interest to historical users of Berkeley DB. + + By default on Windows, Berkeley DB supports internationalized filenames by treating all directory paths and filenames passed to Berkeley DB methods as UTF-8 encoded strings. All paths are internally converted to wide character strings and passed to the wide character variants of Windows system calls. + + This allows applications to create and open databases with names that cannot be represented with ASCII names while maintaining compatibility with applications that work purely with ASCII paths. + + Windows 98 and ME do not support Unicode paths directly. To build for those versions of Windows, either: + + - Follow the instructions at Microsoft's web site. + + - Open the workspace or solution file with Visual Studio. Then open the Project properties/settings section for the project you need to build (at least db_dll). In the *C/C++-\>Preprocessor-\>Preprocessor Definitions* section, remove *\_UNICODE* and *UNICODE* entries. Add in an entry of *\_MBCS*. Build the project as normal. + + The ASCII builds will also work on Windows NT/2K/XP/2003 and Windows7, but will not translate paths to wide character strings. diff --git a/docs-src/guides/installation/build_win_java.md b/docs-src/guides/installation/build_win_java.md new file mode 100644 index 000000000..4e1fdd1df --- /dev/null +++ b/docs-src/guides/installation/build_win_java.md @@ -0,0 +1,29 @@ +--- +title: "Building the Java API" +api-name: "Building the Java API" +source: docs/installation/build_win_java.html +--- +## Building the Java API + +Java support is not built automatically. The following instructions assume that you have installed the Sun Java Development Kit in `d:\java`. Of course, if you installed elsewhere or have different Java software, you will need to adjust the pathnames accordingly. + +1. Set your include directories. + - In Visual Studio 2005/Visual Studio 2008 - Choose *Tools -\> Options -\> Projects -\> VC++ Directories*. Under the "Show directories for" pull-down, select "Include files". Add the full pathnames for the `d:\java\include` and `d:\java\include\win32` directories. Then click OK. + - In Visual Studio 2010 - Right-click db_java project, choose *Properties-\>Configuration Properties-\> VC++ Directories-\>Include Directories*. Add the full pathnames for the `d:\java\include` and `d:\java\include\win32` directories. Then click OK. + + These are the directories needed when including jni.h. + +2. Set the executable files directories. + - In Visual Studio 2005/Visual Studio 2008 - Choose *Tools -\> Options -\> Projects -\> VC++ Directories*. Under the "Show directories for" pull-down, select "Executable files". Add the full pathname for the `d:\java\bin` directory, then click OK. + - In Visual Studio 2010 - Right-click db_java project, choose *Properties-\>Configuration Properties-\> VC++ Directories-\>Executable Directories*. Add the full pathnames for the `d:\java\bin` directories. Then click OK. + + This is the directory needed to find javac. + +3. Set the build type to Release or Debug in the drop-down on the tool bar. + +4. To build, right-click on db_java and select Build. This builds the Java support library for Berkeley DB and compiles all the java files, placing the resulting `db.jar` and `dbexamples.jar` files in one of the following Berkeley DB subdirectories, depending on the configuration that you chose: + + | | + |-------------------------------| + | `build_windows\Win32\Debug` | + | `build_windows\Win32\Release` | diff --git a/docs-src/guides/installation/build_win_notes.md b/docs-src/guides/installation/build_win_notes.md new file mode 100644 index 000000000..dd2f41f71 --- /dev/null +++ b/docs-src/guides/installation/build_win_notes.md @@ -0,0 +1,22 @@ +--- +title: "Windows notes" +api-name: "Windows notes" +source: docs/installation/build_win_notes.html +--- +## Windows notes + +If a system memory environment is closed by all processes, subsequent attempts to open it will return an error. To successfully open a transactional environment in this state, recovery must be run by the next process to open the environment. For non-transactional environments, applications should remove the existing environment and then create a new database environment. + +1. Berkeley DB does not support the Windows/95, Windows/98 or Windows/ME platforms. + +2. On Windows, system paging file memory is freed on last close. For this reason, multiple processes sharing a database environment created using the DB_SYSTEM_MEM flag must arrange for at least one process to always have the environment open, or alternatively that any process joining the environment be prepared to re-create it. + +3. When using the DB_SYSTEM_MEM flag, Berkeley DB shared regions are created without ACLs, which means that the regions are only accessible to a single user. If wider sharing is appropriate (for example, both user applications and Windows/NT service applications need to access the Berkeley DB regions), the Berkeley DB code will need to be modified to create the shared regions with the correct ACLs. Alternatively, by not specifying the DB_SYSTEM_MEM flag, filesystem-backed regions will be created instead, and the permissions on those files may be directly specified through the DB_ENV->open() method. + +4. Applications that operate on wide character strings can use the Windows function WideCharToMultiByte with the code page CP_UTF8 to convert paths to the form expected by Berkeley DB. Internally, Berkeley DB calls MultiByteToWideChar on paths before calling Windows functions. + +5. Various Berkeley DB methods take a **mode** argument, which is intended to specify the underlying file permissions for created files. Berkeley DB currently ignores this argument on Windows systems. + + It would be possible to construct a set of security attributes to pass to **CreateFile** that accurately represents the mode. In the worst case, this would involve looking up user and all group names, and creating an entry for each. Alternatively, we could call the **\_chmod** (partial emulation) function after file creation, although this leaves us with an obvious race. + + Practically speaking, however, these efforts would be largely meaningless on a FAT file system, which only has a "readable" and "writable" flag, applying to all users. diff --git a/docs-src/guides/installation/build_win_small.md b/docs-src/guides/installation/build_win_small.md new file mode 100644 index 000000000..6d4f60fe5 --- /dev/null +++ b/docs-src/guides/installation/build_win_small.md @@ -0,0 +1,18 @@ +--- +title: "Building a small memory footprint library" +api-name: "Building a small memory footprint library" +source: docs/installation/build_win_small.html +--- +## Building a small memory footprint library + +For applications that don't require all of the functionality of the full Berkeley DB library, an option is provided to build a static library with certain functionality disabled. In particular, cryptography, hash and queue access methods, replication and verification are all turned off. In addition, all message text is stripped from the library. This can reduce the memory footprint of Berkeley DB significantly. + +### Note + +If your library has stripped messages, you can get an idea of what text should be issued for a given error message by using the Message Reference for Stripped Libraries guide. + +In general on Windows systems, you will want to evaluate the size of the final application, not the library build. The Microsoft LIB file format (like UNIX archives) includes copies of all of the object files and additional information. The linker rearranges symbols and strips out the overhead, and the resulting application is much smaller than the library. There is also a Visual C++ optimization to "Minimize size" that will reduce the library size by a few percent. + +A Visual C++ project file called `db_small` is provided for this small memory configuration. During a build, static libraries are created in `Release` or `Debug`, respectively. The library name is `libdb_small48sd.lib` for the debug build, or `libdb_small48s.lib` for the release build. + +For assistance in further reducing the size of the Berkeley DB library, or in building small memory footprint libraries on other systems, please contact Berkeley DB support. diff --git a/docs-src/guides/installation/build_win_sql.md b/docs-src/guides/installation/build_win_sql.md new file mode 100644 index 000000000..8df2c1c71 --- /dev/null +++ b/docs-src/guides/installation/build_win_sql.md @@ -0,0 +1,205 @@ +--- +title: "Building the SQL API" +api-name: "Building the SQL API" +source: docs/installation/build_win_sql.html +--- +## Building the SQL API + + [Binary Compatibility With SQLite](build_win_sql.md#idp290248) + + [Setting Preprocessor Flags](build_win_sql.md#idp276576) + + [Enabling Extensions](build_win_sql.md#idp288280) + + [Disabling Log Checksums](build_win_sql.md#win-disablechecksums) + + [Building the JDBC Driver](build_win_sql.md#build_jdbc) + + [Using the JDBC Driver](build_win_sql.md#idp266616) + + [Building the ODBC Driver](build_win_sql.md#idp305704) + + [Using the ADO.NET Driver](build_win_sql.md#idp320888) + +SQL support is built as part of the default build on Windows. For information on the build instructions, see Building Berkeley DB for Windows . + +The SQL library is built as `libdb_sql53.dll` in the Release mode or `libdb_sql53d.dll` in the Debug mode. An SQL command line interpreter called `dbsql.exe` is also built. + +### Binary Compatibility With SQLite + +`libdb_sql53.dll` is compatible with `sqlite3.dll`. You can copy `libdb_sql53.dll` to `sqlite3.dll` and `dbsql.exe` to `sqlite3.exe`, and use these applications as a replacement for the standard SQLite binaries with the same names. However, if you want to do this, then any legacy data in use by those tools must be migrated from the standard SQLite database to a Berkeley DB SQL database *before* you replace the standard SQLite dll and executable with the Berkeley DB equivalent. For information on migrating data from standard SQLite databases to a Berkeley DB SQL database, see the *Berkeley DB Getting Started with the SQL APIs* guide. + +### Warning + +Rename your dlls and executables to the standard SQLite names with *extreme* care. Doing this will cause all existing tools to break that currently have data stored in a standard SQLite database. + + *For best results, rename your dlls and command line tool to use the standard SQLite names only if you know there are no other tools on your production platform that rely on standard SQLite.* + +### Setting Preprocessor Flags + +By default, Berkeley DB SQL generates each table as a subdatabase in a single file. To generate each table in a separate file, specify *BDBSQL_FILE_PER_TABLE* in *Preprocessor Definitions* of the `db_sql` project. + +When this option is enabled, the SQL database name is used as a directory name. This directory contains one file for the metadata and one file each for every table created by the SQL API. Do not add or delete files from the database directory. Adding or deleting files may corrupt the database. To backup just the metadata (schema), make a copy of the `metadata` and `table00001` files from the database directory. Make a new copy whenever the schema is changed. + +### Enabling Extensions + +The Berkeley DB SQL API provides extensions such as full text search and R-Tree index. To enable these extensions, do the following: + +1. Open the Berkeley DB solution in Visual Studio. +2. Specify *SQLITE_ENABLE_FTS3* or *SQLITE_ENABLE_RTREE* in *Preprocessor Definitions* of the `db_sql` project. +3. Re-build the `db_sql` project. + +See the SQLite Documentation for more information on full text search and R-Tree. + +### Disabling Log Checksums + +You can disable checksums in log records so as to provide a boost to database performance. However, this comes at the risk of having undetectable log file corruption that could prevent data recovery in the event of database corruption. + +### Note + +Note that disabling log record checksums is meant to only be used with the SQL interface. However, disabling checksums for the SQL interface also disables checksums for the non-SQL interfaces. + +To disable log checksums, before you build the library edit the `build_windows/db_config.h` file, and delete the following line: + +``` c +#define HAVE_LOG_CHECKSUM 1 +``` + +### Building the JDBC Driver + +This section describes the steps to build the JDBC driver. + +1. Configure your build environment. For information on how to configure to build Java applications, see Building the Java API. + +2. Build the SQL project in Debug mode. + +3. Open Berkeley_DB.sln or Berkeley_DB_vs2010.sln in Visual Studio. If you are using Java 1.6, do the following: + + - In the Solution Explorer, right-click the `db_sql_jdbc` project and select *properties*. + + - In the *Configuration Properties -\> Build Events -\> Pre-Build Event* section, alter the command to refer to `JDBC2z` instead of `JDBC2x`. + + If you are using Java 1.7, do the following: + + - In the Solution Explorer, right-click the `db_sql_jdbc` project and select *properties*. + + - In the *Configuration Properties -\> Build Events -\> Pre-Build Event* section, alter the command to refer to `JDBC2z1` instead of `JDBC2x`. Also, remove the option of "-target 1.5". + + - Go to `db\lang\sql\jdbc\SQLite`, and replace `JDBCDriver.java` with `JDBCDriver.java17`, and replace `JDBCDataSource.java` with `JDBCDataSource.java17`. + +4. Build the `db_sql_jdbc` project in Visual Studio. + +You can test the build by entering the following commands from the `db\build_windows\Win32\Debug` directory: + +| | +|----------------------------------------------------------| +| javac -cp ".;jdbc.jar" -d . ..\\.\\.\sql\jdbc\test3.java | +| java -cp ".;jdbc.jar" test3 | + +### Using the JDBC Driver + +This section describes the steps to download, build, and run sample programs using the built JDBC driver. + +#### Downloading JDBC Sample Code + +The download link for JDBC sample code is available on the Oracle Technology Network (OTN) . You can identify the link by the "JDBC programming examples from all three editions (ZIP format)" text beside it. + +#### Modifying Sample Code + +Before running the sample code, do the following: + +1. Unzip the file containing the sample code to a new directory (for example, jdbc_ex). + +2. Substitute `jdbc:sqlite:/` for the generic JDBC URL that appears in the code. That is, put `jdbc:sqlite:/` between the quotation marks in the line: + + ` String url = "jdbc:mySubprotocol:myDataSource";` + + Note: The \ can either be an absolute path name like `"D:\\jdbc_ex_db\\myDataSource"`, or a relative path-file-name like `"..\\jdbc_ex_db\myDataSource"`, or a file name, like `"myDataSource"`, in which the database file will be stored at the current directory. + +3. Substitute `SQLite.JDBCDriver` for `myDriver.ClassName` in the line: `Class.forName("myDriver.ClassName");` + +4. Substitute the username and password you use for your database in the following: `"myLogin", "myPassword"`. + + This is optional. + +5. If your JDK version is above 1.5, change the variable name `enum` in `OutputApplet.java` to some other variable name because, as of JDK release 5 `enum` is a keyword and can not be used as an identifier. + +#### Building and Running the JDBC Sample code + +See Building the JDBC Driver for instructions about building JDBC driver. + +To build and run the JDBC examples do the following: + +1. In the `db\build_windows\Win32\Debug` directory, run following commands: + + ``` c + $ javac -classpath ".;jdbc.jar" -d . \path\to\jdbc_ex\*.java + $ java -classpath ".;jdbc.jar" + + ``` + +2. After you run the CreateCoffees example, use the `dbsql` executable to open the `myDataSource` database file and check if the table `COFFEES` has been successfully created in the database. + + ``` c + $ dbsql myDataSourcedbsql> .tables + COFFEES + dbsql> .dump + PRAGMA foreign_keys=OFF; + BEGIN TRANSACTION; + CREATE TABLE COFFEES (COF_NAME varchar(32),/ + SUP_ID int, PRICE float, SALES int, TOTAL int); + COMMIT; + dbsql> + + ``` + +3. Repeat step 2 to run other examples. + + Note: Some examples like AutoGenKeys are not yet supported by BDB JDBC driver. The `SQLFeatureNotSupportedException` is displayed for those unsupported examples. + +### Building the ODBC Driver + +This section describes the steps required to build the ODBC driver. + +#### Configuring Your System + +To configure your system prior to building the ODBC driver, do the following: + +1. Download and install the latest SQLite ODBC driver Windows installer package for 32 bit Windows or 64 bit Windows. +2. Download and install the latest Microsoft Data Access Components (MDAC) SDK . The MDAC SDK is only required for testing the installation. + +#### Building the Library + +1. Build the SQL project in Release mode. See Building the SQL API. +2. Open Visual Studio. +3. Load the Berkeley_DB solution file into Visual Studio. +4. Set the build target to *Release* +5. Build the solution. +6. Select *File* -\> *Add* -\> *Existing Project*. +7. Select `build_windows`. +8. Select the appropriate directory for your compiler: `VS8` or `VS10`. +9. Select `db_sql_odbc.vcproj` and add it to the Berkeley_DB solution. This adds the `db_sql_odbc` Visual Studio project to the Berkeley_DB solution file. +10. Build the `db_sql_odbc` project. This can be done by right-clicking the `db_sql_odbc` project in the project explorer panel, and selecting `build`. + +The `sqlite3odbc.dll`, `libdb_sql53.dll` and `libdb53.dll` files are now built. + +#### Installing the Library + +Copy the dll files built in the *Building the Library* section to the Windows system folder. + +The Windows system folder is different on different systems, but is often `C:\WINDOWS\System32`. + +#### Testing the ODBC Install + +The steps to verify that the installed driver works are as follows: + +1. Open the Unicode ODBCTest application. On Windows XP: *Windows start* -\> *Microsoft Data Access SDK 2.8* -\> *ODBCTest (Unicode, x86).* +2. Select the *Conn* -\> *Full Connect...* menu item. +3. Select `SQLite3 Datasource` and click `OK`. +4. Select the *Stmt* -\> *SQLExecDirect...* menu item. +5. Enter `CREATE TABLE t1(x);` in the `Statement` text box and click `OK`. +6. Verify that no error messages were output to the error window. + +### Using the ADO.NET Driver + +Go to the Oracle Berkeley DB download page, and download the ADO.NET package. Build the package by following the instructions included in the package. diff --git a/docs-src/guides/installation/build_win_tcl.md b/docs-src/guides/installation/build_win_tcl.md new file mode 100644 index 000000000..3d4502af9 --- /dev/null +++ b/docs-src/guides/installation/build_win_tcl.md @@ -0,0 +1,33 @@ +--- +title: "Building the Tcl API" +api-name: "Building the Tcl API" +source: docs/installation/build_win_tcl.html +--- +## Building the Tcl API + +Tcl support is not built automatically. See Loading Berkeley DB with Tcl for information on sites from which you can download Tcl and which Tcl versions are compatible with Berkeley DB. These notes assume that Tcl is installed as `d:\tcl`, but you can change that if you want. + +The Tcl library must be built as the same build type as the Berkeley DB library (both Release or both Debug). We found that the binary release of Tcl can be used with the Release configuration of Berkeley DB, but you will need to build Tcl from sources for the Debug configuration. Before building Tcl, you will need to modify its makefile to make sure that you are building a debug version, including thread support. This is because the set of DLLs linked into the Tcl executable must match the corresponding set of DLLs used by Berkeley DB. + +1. Set the include directories. + - In Visual Studio 2005/Visual Studio 2008 - Choose *Tools -\> Options -\> Projects -\> VC++ Directories*. Under the "Show directories for" pull-down, select "Include files". Add the full pathname for `d:\tcl\include`, then click OK. + - In Visual Studio 2010 - Right-click db_tcl project, choose *Properties-\>Configuration Properties-\> VC++ Directories-\>Include Directories*. Add the full pathnames for `d:\tcl\include`, then click OK. + + This is the directory that contains `tcl.h`. + +2. Set the library files directory. + - In Visual Studio 2005/Visual Studio 2008 - Choose *Tools -\> Options -\> Projects -\> VC++ Directories*. Under the "Show directories for" pull-down, select "Library files". Add the full pathname for the `d:\tcl\lib` directory, then click OK. + - In Visual Studio 2010 - Right-click db_tcl project, choose *Properties-\>Configuration Properties-\> VC++ Directories-\>Library Directories*. Add the full pathname for the `d:\tcl\lib` directory, then click OK. + + This is the directory needed to find `tcl85g.lib` (or whatever the library is named in your distribution). + +3. Set the build type to Release or Debug in the drop-down on the tool bar. + +4. To build, right-click on db_tcl and select Build. This builds the Tcl support library for Berkeley DB, placing the result into one of the following Berkeley DB subdirectories, depending upon the configuration that you chose: + + | | + |-----------------------------------------------| + | `build_windows\Win32\Debug\libdb_tcl53d.dll` | + | `build_windows\Win32\Release\libdb_tcl53.dll` | + +If you use a version different from Tcl 8.5.x you will need to change the name of the Tcl library used in the build (for example, `tcl85g.lib`) to the appropriate name. To do this, right click on *db_tcl*, go to *Properties -\> Linker -\> Input -\> Additional dependencies* and change `tcl85g.lib` to match the Tcl version you are using. diff --git a/docs-src/guides/installation/build_win_test.md b/docs-src/guides/installation/build_win_test.md new file mode 100644 index 000000000..33d70475d --- /dev/null +++ b/docs-src/guides/installation/build_win_test.md @@ -0,0 +1,91 @@ +--- +title: "Running the test suite under Windows" +api-name: "Running the test suite under Windows" +source: docs/installation/build_win_test.html +--- +## Running the test suite under Windows + + [Building the software needed by the tests](build_win_test.md#idp368040) + + [Running the test suite under Windows](build_win_test.md#idp379184) + + [Building the software needed by the SQL tests](build_win_test.md#build_win_test_sql) + +To build the test suite on Windows platforms, you will need to configure Tcl support. You will also need sufficient main memory (at least 64MB), and disk (around 250MB of disk will be sufficient). + +### Building the software needed by the tests + +The test suite must be run against a Debug version of Berkeley DB, so you will need a Debug version of the Tcl libraries. This involves building Tcl from its source. See the Tcl sources for more information. Then build the Tcl API - see Building the Tcl API for details. + +#### Visual Studio 2005 or newer + +To build for testing, perform the following steps: + +1. Open the Berkeley DB solution. +2. Ensure that the target configuration is Debug +3. Right click the *db_tcl* project in the Solution Explorer, and select *Build*. +4. Right click the *db_test* project in the Solution Explorer, and select *Build*. + +### Running the test suite under Windows + +Before running the tests for the first time, you must edit the file `include.tcl` in your build directory and change the line that reads: + +``` c +set tclsh_path SET_YOUR_TCLSH_PATH +``` + +You will want to use the location of the `tclsh` program (be sure to include the name of the executable). For example, if Tcl is installed in `d:\tcl`, this line should be the following: + +``` c +set tclsh_path d:\tcl\bin\tclsh85g.exe +``` + +If your path includes spaces be sure to enclose it in quotes: + +``` c +set tclsh_path "c:\Program Files\tcl\bin\tclsh85g.exe" +``` + +Make sure that the path to Berkeley DB's tcl library is in your current path. On Windows NT/2000/XP, edit your PATH using the My Computer -\> Properties -\> Advanced -\> Environment Variables dialog. On earlier versions of Windows, you may find it convenient to add a line to c:\AUTOEXEC.BAT: + +``` c +SET PATH=%PATH%;c:\db\build_windows +``` + +Then, in a shell of your choice enter the following commands: + +1. cd build_windows + +2. run `d:\tcl\bin\tclsh85g.exe`, or the equivalent name of the Tcl shell for your system. + + You should get a "%" prompt. + +3. % source ../test/tcl/test.tcl + + If no errors occur, you should get a "%" prompt. + +You are now ready to run tests in the test suite; see Running the test suite for more information. + +### Building the software needed by the SQL tests + +The SQL test suite must be run against a Debug version of Berkeley DB, so you need a Debug version of the Tcl libraries. This involves building Tcl from its source. See the Tcl sources for more information. Then build the Tcl API - see Building the Tcl API for details. + +Before building for SQL tests, build the db_tcl and db_sql_testfixture projects. This requires Tcl 8.5 or above. If you are using a later version of Tcl, edit the Tcl library that db_tcl and db_sql_testfixture link to. + +To do this right click the *db_tcl*`/`*db_sql_testfixture* project, select *Properties-\>Configuration Properties-\>Linker-\>Input-\>Additional Dependencies* and edit the Tcl library, *tcl85g.lib*, to match the version you are using. + +Building the db_sql_testfixture project builds the testfixture.exe program in `../build_windows/Win32/Debug`. It also builds the projects db and db_sql, on which it depends. + +#### Visual Studio 2005 or newer + +To build for testing, perform the following steps: + +1. Open the Berkeley DB solution. +2. Ensure that the target configuration is Debug. +3. Right click the *db_tcl* project in the Solution Explorer, and select *Build*. +4. Right click the *db_sql_testfixture* project in the Solution Explorer, and select *Build*. + +To test extensions, specify the following in the *Preprocessor Definitions* of the *db_sql_testfixture* project: + +- `SQLITE_ENABLE_FTS3` to enable the full text search layer +- `SQLITE_ENABLE_RTREE` to enable the R-Tree layer diff --git a/docs-src/guides/installation/ch01s02.md b/docs-src/guides/installation/ch01s02.md new file mode 100644 index 000000000..8526ee0dc --- /dev/null +++ b/docs-src/guides/installation/ch01s02.md @@ -0,0 +1,46 @@ +--- +title: "Supported Platforms" +api-name: "Supported Platforms" +source: docs/installation/ch01s02.html +--- +## Supported Platforms + +You can install Berkeley DB on the following platforms: + +- Most versions of Linux (x86-64 and x86) including: + + - Oracle Linux 4, 5, and 6 + + - Red Hat + + - Ubuntu + + - Wind River + + - MontaVista Embedded Linux version 6.0 + +- Oracle Solaris versions 9 and 10 on x86_64, x86, and SPARC. + +- FreeBSD + +- Microsoft Windows (x86-64 and x86). + + - XP (SP2, SP3) + - Vista + - Windows 7 + - Server 2008 + - Windows Mobile (6.x) + +- Apple Mac OS X 10.5 and 10.6. + +- IBM AIX version 5 and 6. + +- VxWorks 6.x + +- QNX Neutrino/POSIX version 6 + +- Android + +- Apple iOS (previously known as iPhone OS) + +Apart from those mentioned in the list above, you can install Berkeley DB on most other systems which are POSIX-compliant. When there is a need to run Berkeley DB on a platform that is currently not supported, DB is distributed in source code form that you can use as base source to port Berkeley DB to that platform. For more information on porting to other platforms, see the Berkeley DB Porting Guide. diff --git a/docs-src/guides/installation/changelog_4_8.md b/docs-src/guides/installation/changelog_4_8.md new file mode 100644 index 000000000..d6c35d20a --- /dev/null +++ b/docs-src/guides/installation/changelog_4_8.md @@ -0,0 +1,504 @@ +--- +title: "Berkeley DB 4.8.28 Change Log" +api-name: "Berkeley DB 4.8.28 Change Log" +source: docs/installation/changelog_4_8.html +--- +## Berkeley DB 4.8.28 Change Log + + [Changes between 4.8.26 and 4.8.28:](changelog_4_8.md#idp1162104) + + [Known bugs in 4.8](changelog_4_8.md#idp1184264) + + [Changes between 4.8.24 and 4.8.26:](changelog_4_8.md#idp1139288) + + [Changes between 4.8.21 and 4.8.24:](changelog_4_8.md#idp1091200) + + [Changes between 4.7 and 4.8.21:](changelog_4_8.md#idp1199520) + + [Database or Log File On-Disk Format Changes:](changelog_4_8.md#idp1200208) + + [New Features:](changelog_4_8.md#idp981712) + + [Database Environment Changes:](changelog_4_8.md#idp1130224) + + [Concurrent Data Store Changes:](changelog_4_8.md#idp1209320) + + [General Access Method Changes:](changelog_4_8.md#idp1209720) + + [Btree Access Method Changes:](changelog_4_8.md#idp1218064) + + [Hash Access Method Changes:](changelog_4_8.md#idp1215560) + + [Queue Access Method Changes:](changelog_4_8.md#idp1226120) + + [Recno Access Method Changes:](changelog_4_8.md#idp1163928) + + [C-specific API Changes:](changelog_4_8.md#idp1138904) + + [C++-specific API Changes:](changelog_4_8.md#idp1218344) + + [Java-specific API Changes:](changelog_4_8.md#idp1238856) + + [Direct Persistence Layer (DPL), Bindings and Collections API:](changelog_4_8.md#idp1232112) + + [Tcl-specific API Changes:](changelog_4_8.md#idp1232384) + + [RPC-specific Client/Server Changes:](changelog_4_8.md#idp1244368) + + [Replication Changes:](changelog_4_8.md#idp1245896) + + [XA Resource Manager Changes:](changelog_4_8.md#idp1242240) + + [Locking Subsystem Changes:](changelog_4_8.md#idp1247728) + + [Logging Subsystem Changes:](changelog_4_8.md#idp1241128) + + [Memory Pool Subsystem Changes:](changelog_4_8.md#idp1258328) + + [Mutex Subsystem Changes:](changelog_4_8.md#idp1258720) + + [Test Suite Changes](changelog_4_8.md#idp1240832) + + [Transaction Subsystem Changes:](changelog_4_8.md#idp1249776) + + [Utility Changes:](changelog_4_8.md#idp1271664) + + [Configuration, Documentation, Sample Application, Portability and Build Changes:](changelog_4_8.md#idp1274104) + +### Changes between 4.8.26 and 4.8.28: + +1. Limit the size of a log record generated by freeing pages from a database so it fits in the log file size. \[#17313\] + +2. Fix a bug that could cause a file to be removed if it was both the source and target of two renames within a transaction. \[#18069\] + +3. Modified how we go about selecting a usable buffer in the cache. Place more emphasis on single version and obsolete buffers. \[#18114\] + +### Known bugs in 4.8 + +1. Sharing logs across mixed-endian systems does not work.\[#18032\] + +### Changes between 4.8.24 and 4.8.26: + +1. Fixed a bug where the truncate log record could be too large when freeing too many pages during a compact. \[#17313\] + +2. Fixed a bug where the deadlock detector might not run properly. \[#17555\] + +3. Fixed three bugs related to properly detecting thread local storage for DbStl. \[#17609\] \[#18001\] \[#18038\] + +4. Fixed a bug that prevented some of our example code from running correctly in a Windows environment. \[#17627\] + +5. Fixed a bug where a "unable to allocate space from buffer cache" error was improperly generated. \[#17630\] + +6. Fixed a bug where DB-\>exists() did not accept the DB_AUTO_COMMIT flag. \[#17687\] + +7. Fixed a bug where DB_TXN_SNAPSHOT was not getting ignored when DB_MULTIVERSION not set. \[#17706\] + +8. Fixed a bug that prevented callback based partitioning through the Java API. \[#17735\] + +9. Fixed a replication bug where log files were not automatically removed from the client side. \[#17899\] + +10. Fixed a bug where code generated from db_sql stored the key in both the data and key DBTs. \[#17925\] + +11. Fixed a bug that prevented a sequence from closing properly after the EntityStore closed. \[#17951\] + +12. Fixed a bug where gets fail if the DB_GET_BOTH_FLAG is specified in a hash, sorted duplicates database.\[#17997\] + +### Changes between 4.8.21 and 4.8.24: + +1. Fixed a bug in the C# API where applications in a 64-bit environment could hang. \[#17461\] + +2. Fixed a bug in MVCC where an exclusive latch was not removed when we couldn't obtain a buffer. \[#17479\] + +3. Fixed a bug where a lock wasn't removed on a non-transactional locker. \[#17509\] + +4. Fixed a bug which could trigger an assertion when performing a B-tree page split and running out of log space or with MVCC enabled. \[#17531\] + +5. Fixed a bug in the repquote example that could cause the application to crash. \[#17547\] + +6. Fixed a couple of bugs when using the GCC 4.4 compiler to build the examples and the dbstl API. \[#17504\] \[#17476\] + +7. Fixed an incorrect representation of log system configuration info. \[#17532\] + +### Changes between 4.7 and 4.8.21: + +### Database or Log File On-Disk Format Changes: + +1. The log file format changed in 4.8. + +### New Features: + +1. Improved scalability and throughput when using BTree databases especially when running with multiple threads that equal or exceed the number of available CPUs. + +2. Berkeley DB has added support for C#. In addition to the new C# api, C# specific tests and sample applications were also added. \[#16137\] + +3. Berkeley DB has added an STL API, which is compatible with and very similar to C++ Standard Template Library (STL). Tests and sample applications and documentation were also added. \[#16217\] + +4. Berkeley DB has added database partitioning. BTree or Hash databases may now be partitioned across multiple directories. Partitioned databases can be used to increase concurrency and to improve performance by spreading access across disk subsystems. \[#15862\] + +5. Berkeley DB now supports bulk insertion and deletion of data. Similar to the bulk get interface, the bulk put and bulk delete allow the developer to populate a buffer of key-value pairs and then pass it to the BDB library with a single API call. + +6. Berkeley DB now supports compression when using BTree. + +7. Berkeley DB introduces a new utility named db_sql which replaces db_codegen. Similar to db_codegen, db_sql accepts an input file with DDL statements and generates a Berkeley DB application using the C API that creates and performs CRUD operations on the defined tables. The developer can then use that code as a basis for further application development. + +8. The Replication Manager now supports shared access to the Master database environment from multiple processes. In earlier versions, multiple process support on the Master required use of the Base Replication API. \[#15982\] + +9. Foreign Key Support has been added to Berkeley DB. + +10. Several enhancements were made to DB_REGISTER & DB_ENV-\>failchk(). + +11. Berkeley now supports 100% in-memory replication. + +12. Berkeley DB now has the ability to compare two cursors for equality. \[#16811\] + +### Database Environment Changes: + +1. Fixed a bug that could cause an allocation error while trying to allocate thread tracking information for the DB_ENV-\>failcheck system. \[#16300\] + +2. Fixed a bug that could cause a trap if an environment open failed and failchk thread tracking was enabled.\[#16770\] + +### Concurrent Data Store Changes: + +None. + +### General Access Method Changes: + +1. Fixed a bug where doing an insert with secondary indices and the NOOVERWRITE flag could corrupt the secondary index. \[#15912\] + +2. Fixed a possible file handle leak that occurred while aborting the create of a database whose metadata page was not initialized. \[#16359\] + +3. Fixed a bug so that we now realloc the filename buffer only if we need it to grow. \[#16385\] \[#16219\] + +4. Fixed a race freeing a transaction object when using MVCC. \[#16381\] + +5. Added missing get methods for the DB and DB_ENV classes where there already was a corresponding set method. \[#16505\] + +6. Fixed a bug to now ensure that DB_STAT_SUBSYSTEM is distinct from other stat flags. \[#16798\] + +7. Fixed a bug related to updating multiple secondary keys (using DB_MULTIPLE). \[#16885\] + +8. Fixed a bug so that verify (db-\>verify, db_verify) will now report when it cannot read a page rather than just saying the database is bad. \[#16916\] + +9. Fixed a bug that could cause memory corruption if a transaction allocating a page aborted while DB-\>compact was running on that database. \[#16862\] + +10. Fixed a bug where logging was occurring during remove of an in-memory database when the DB_TXN_NOT_DURABLE flag was set. \[#16571\] + +11. Fixed a bug to remove a race condition during database/file create. \[#17020\] + +12. Fixed a bug where a call to DB-\>verify and specifying DB_SALVAGE could leak memory when the call returned. \[#17161\] + +13. Fixed a bug to avoid accessing freed memory during puts on primaries with custom comparators. \[#17189\] + +14. Fixed a bug that could cause old versions of pages to be written over new versions if an existing database is opened with the DB_TRUNCATE flag. \[#17191\] + +### Btree Access Method Changes: + +1. Fixed a bug which could cause DB-\>compact to fail with DB_NOTFOUND or DB_PAGE_NOTFOUND if the height of the tree was reduced by another thread while compact was active. The bug could also cause a page split to trigger splitting of internal nodes which did not need to be split. \[#16192\] + +2. Fixed a bug that caused Db-\>compact to loop if run on an empty RECNO database when there were pages in the free list. \[#16778\] + +3. Added a new flag, DB_OVERWRITE_DUP, to DB-\>put and DBC-\>put. This flag is equivalent to DB_KEYLAST in almost all cases: the exception is that with sorted duplicates, if a matching key/data pair exists, we overwrite it rather than returning DB_KEYEXIST. \[#16803\] + +### Hash Access Method Changes: + +1. Fixed a bug to now force a group allocation that rolls forward to reinit all the pages. Otherwise a previous aborted allocation may change the header. \[#15414\] + +2. Fixed a bug to now return the expected buffer size on a DB_BUFFER_SMALL condition. \[#16881\] + +### Queue Access Method Changes: + +1. Fixed a bug that would cause the LSN reset functionality to not process queue extents. \[#16213\] + +2. Fixed a bug that prevented a partial put on a queue database with secondaries configured. \[#16460\] + +3. Fixed a bug to now prevent an unpinned page to be returned if a delete from a HASH database deadlocked. \[#16371\] + +4. Fixed a bug that could cause a queue extent to be recreated if an application deleted a record that was already deleted in that extent. \[#17004\] + +5. Added the DB_CONSUME flag to DB-\>del and DBC-\>del to force adjustment of the head of the queue. \[#17004\] + +### Recno Access Method Changes: + +1. Fixed a bug which could cause DB-\>compact of a RECNO database to loop if the number of pages on the free list was reduced by another thread while compact was active. \[#16199\] + +2. Fixed a bug that occurs when deleting from a Recno database and using DB_READ_UNCOMMITTED where we could try to downgrade a lock twice. \[#16347\] + +3. Fixed a bug to now disallow passing DB_DUP and DB_RECNUM together to \_\_db_set_flags. \[#16585\] + +### C-specific API Changes: + +1. Add get functions for each set functions of DB and DB_ENV structures which didn't have one.\[#16505\] + +### C++-specific API Changes: + +1. The get and set_lk_partitions methods are now available. + +2. Add get functions for each set functions of Db and DbEnv classes which didn't have one.\[#16505\] + +3. Fixed a memory leak when using nested transactions.\[#16956\] + +### Java-specific API Changes: + +1. Fixed a bug where the replication finer-grained verbose flags were not available in the Java API. \[#15419\] + +2. Fixed a bug in the BTree prefix compression API when called from the Java API. DBTs were not properly initialized. \[#16417\] + +3. Fixed a bug so that LogCursor will work correctly from the Java API. \[#16827\] + +4. Fixed a bug so that position(), limit() and capacity() of ByteBuffers are obeyed by DatabaseEntry objects. \[#16982\] + +### Direct Persistence Layer (DPL), Bindings and Collections API: + +1. The StoredMap class now implements the standard java.util.concurrent.ConcurrentMap interface. \[#15382\] + +2. Report a meaningful IllegalArgumentException when @Persistent is incorrectly declared on an enum class. Before, the confusing message Persistent class has non-persistent superclass: java.lang.Enum was reported. \[#15623\] + +3. Report a meaningful IllegalArgumentException when @Persistent is incorrectly declared on an interface. Before, a NullPointerException was reported. \[#15841\] + +4. Several validation checks have been added or corrected having to do with entity subclasses, which are @Persistent classes that extend an @Entity class. \[#16077\] + +5. Optimized marshaling for large numbers of embedded objects improving performance. \[#16198\] + +6. The StoredMap class now implements the Java 1.5 ConcurrentMap interface. \[#16218\] + +7. Fix a DPL bug that caused exceptions when using a class Converter for an instance containing non-simple fields. \[#16233\] + +8. Add EntityCursor.setCacheMode and getCacheMode. See the com.sleepycat.je.CacheMode class for more information. \[#16239\] + +9. Fix a bug that prevents evolution of @SecondaryKey information in an entity subclass (a class that extends an @Entity class). \[#16253\] + +10. Report a meaningful IllegalArgumentException when @Persistent or @Entity is incorrectly used on an inner class (a non-static nested class). Before, the confusing message No default constructor was reported. \[#16279\] + +11. Improved the reliability of Entity subclasses that define secondary keys by requiring that they be registered prior to storing an instance of the class. \[#16399\] + +12. Fix a bug that under certain circumstances causes "IllegalArgumentException: Not a key class" when calling EntityStore.getSubclassIndex, EntityStore.getPrimaryConfig, EntityStore.getSecondaryConfig, or PrimaryIndex.put, and a composite key class is used. \[#16407\] + +13. Fixed a bug so that one can now compile DPL in the Java API on Windows. \[#16570\] + +14. The com.sleepycat.collections.TransactionRunner.handleException method has been added to allow overriding the default transaction retry policy. See the javadoc for this method for more information. \[#16574\] + +15. Fix a bug that causes an assertion to fire or a NullPointerException (when assertions are disabled) from the EntityStore constructor. The problem occurs only when the previously created EntityStore contains an entity with a secondary key definition in which the key name has been overridden and is different than the field name. \[#16819\] + +16. Key cursors have been optimized to significantly reduce I/O when the READ_UNCOMMITTED isolation mode is used. See EntityIndex.keys for more information. \[#16859\] + +17. Report a meaningful IllegalArgumentException when NULLIFY is used with a @SecondaryKey and the field is a primitive type. Before, the confusing message Key field object may not be null was reported. \[#17011\] + +18. Enum fields may now be used as DPL keys, including primary keys, secondary keys, and fields of composite key classes. Comparators are supported for composite key classes containing enum fields. \[#17140\] + +19. Fix a bug that prevented the use of custom key comparisons (composite key classes that implement Comparable) for secondary keys defined as ONE_TO_MANY or MANY_TO_MANY.\[#17207\] + +20. The db.jar file now contains a Premain class which enables bytecode enhancement using the JVM instrumentation commands. The built-in proxy classes are also now enhanced in the db.jar file, which enables off-line bytecode enhancement. For more information on DPL bytecode enhancement and how to use both instrumentation and off-line enhancement, please see the com.sleepycat.persist.model.ClassEnhancer javadoc. \[#17233\] + +### Tcl-specific API Changes: + +1. The mutex API is now available when using Tcl. \[#16342\] + +### RPC-specific Client/Server Changes: + +- RPC support has been removed from Berkeley DB. \[#16785\] + +### Replication Changes: + +1. Improved testing of initial conditions for rep and repmgr APIs and added heartbeat timeouts to rep_get_timeout.\[#14977\] + +2. Added DB_REP_CONF_INMEM replication configuration flag to store replication information exclusively in-memory without creating any files on-disk. \[#15257\] + +3. Added repmgr support for multi-process shared env \[#15982\] + +4. Fixed a bug where opening a cursor from a database handle failed to check whether the database handle was still fresh. If the database handle had been invalidated by a replication client synchronizing with a new master, it could point to invalid information. \[#15990\] + +5. Fixed a bug so that if LOG_REQ gets an archived LSN, replication sends VERIFY_FAIL. \[#16004\] + +6. Added timestamp and process/thread id to replication verbose messages. \[#16098\] + +7. Fixed a bug where, in very rare circumstances, two repmgr sites could connect to each other at the exact same time, the connection attempts "collide" and fail, and the same collision repeats in time synchronization indefinitely. \[#16114\] + +8. Fixed a bug where a missing database file (FILE_FAIL error condition) can interrupt a client synchronization without restarting it. \[#16130\] + +9. Fixed a bug by adding REP_F_INREPSTART flag to prevent racing threads in rep_start. \[#16247\] + +10. Fixed a bug to not return HOLDELECTION if we are already in the middle of an election. Updated the egen so the election thread will notice. \[#16270\] + +11. Fixed a bug in buffer space computation, which could have led to memory corruption in rare circumstances, when using bulk transfer. \[#16357\] + +12. Fixed a bug that prevented replication clients from opening a sequence. The sequence is opened for read operations only. \[#16406\] + +13. Fixed a bug by removing an assertion about priority in elections. It is not correct because it could have changed by then. Remove unused recover_gen field. \[#16412\] + +14. Fixed a bug to now ignore a message from client if it is an LSN not recognized in a LOG_REQ. \[#16444\] + +15. Fixed a bug so that on POSIX systems, repmgr no longer restores default SIGPIPE action upon env close, if it was necessary to change it during start-up. This allows remaining repmgr environments within the same process, if any, to continue operating after one of them is closed. \[#16454\] + +16. After a replication client restarts with recovery, any named in-memory databases are now re-materialized from the rest of the replication group upon synchronization with the master. \[#16495\] + +17. Fixed a bug by adding missing rep_get_config flags. \[#16527\] + +18. Instead of sleeping if the bulk buffer is in transmission, return so that we can send as a singleton. \[#16537\] + +19. Fixed a bug by changing \_\_env_refresh to not hit assert on -private -rep env with an in-memory database. \[#16546\] + +20. Fixed a bug in the Windows implementation of repmgr where a large number of commit threads concurrently awaiting acknowledgments could result in memory corruption, and leaking Win32 Event Objects. \[#16548\] + +21. Fixed a bug by changing repmgr to count a dropped connection when noticing a lacking heartbeat; fixed hearbeat test to check for election, rather than connection drop count, and more reasonable time limit; fixed test to poll until desired result, rather than always sleeping max possible time. \[#16550\] + +22. Fixed "master changes" stat to count when local site becomes master too. \[#16562\] + +23. Fixed a bug where a c2c client would send UPDATE_REQ to another client \[#16592\] + +24. Removed code to proactively expire leases when we don't get acks. Leases maintain their own LSNs to know. \[#16494\] + +25. Fixed a bug where a client may not sync pages during internal init. \[#16671\] + +26. Fixed a bug where a client that received and skipped a log record from the master during an election, then won the election, could then try to request a copy of the skipped log record. The result was an attempt to send a request to the local site, which is invalid: this could confuse a replication Base API application, or cause the Replication Manager to crash. \[#16700\] + +27. Fixed a bug which could have caused data loss or corruption (at the client only) if a replication client rolled back existing transactions in order to synchronize with a new master, and then crashed/recovered before a subsequent checkpoint operation had been replicated from the master. \[#16732\] + +28. Fixed a bug so that replication now retries on DB_LOCK_NOTGRANTED. \[#16741\] + +29. Fixed a potential deadlock in rep_verify_fail. \[#16779\] + +30. Fixed a bug so that an application will no longer segv if nsites given was smaller than number of sites that actually exists. \[#16825\] + +### XA Resource Manager Changes: + +1. The XA Resource Manager has been removed from Berkeley DB. \[#6459\] + +### Locking Subsystem Changes: + +1. Fixed a bug to prevent unlocking a mutex twice if we ran out of transactional locks. \[#16285\] + +2. Fixed a bug to prevent a segmentation trap in \_\_lock_open if there were an error during the opening of an environment. \[#16307\] + +3. Fixed a bug to now avoid a deadlock if user defined locks are used only one lock partition is defined.\[#16415\] + +4. Fixed concurrency problems in \_\_dd_build, \_\_dd_abort by adding LOCK_SYSTEM_LOCK() calls to \_\_dd_build and \_\_dd_abort. \[16489\] + +5. Fixed a bug that could cause a panic if a transaction which updated a database that was supporting READ_UNCOMMITED readers aborted and it hit a race with a thread running the deadlock detector. \[#16490\] + +6. Fixed a race condition in deadlock detection that could overwrite heap. \[#16541\] + +7. Fixed a bug so that DB_STAT_CLEAR now restores the value of st_partitions. \[#16701\] + +### Logging Subsystem Changes: + +1. Fixed a bug so that the header checksum is only ignored when the log is from a previous version \[#16281\] + +2. Fixed a bug by removing a possible race condition with logc_get(DB_FIRST) and log archiving. \[#16387\] + +3. Fixed a bug that could cause a recovery failure of a create of a database that was aborted. \[#16824\] + +4. An in-memory database creation has an intermediate phase where we have a semi-open DBP. If we crash in that state, then recovery was failing because it tried to use a partically open database handle. This fix checks for that case, and avoids trying to undo page writes for databases in that interim step. \[#17203\] + +### Memory Pool Subsystem Changes: + +1. Fixed a bug that occurred after all open handles on a file are closed. Needed to clear the TXN_NOT_DURABLE flag (if set) and mark the file as DURABLE_UNKNOWN in the memory pool. \[#16091\] + +2. Fixed a possible race condition between dirtying and freeing a buffer that could result in a panic or corruption. \[#16530\] + +3. Fixed a memory leak where allocated space for temporary file names are not released. \[#16956\] + +### Mutex Subsystem Changes: + +1. Fixed a bug when using mutexes for SMP MIPS/Linux systems. \[#15914\] + +2. POSIX mutexes are now the default on Solaris. \[#16066\] + +3. Fixed a bug in mutex allocation with multiple cache regions. \[#16178\] + +4. Fixed MIPS/Linux mutexes in 4.7. \[#16209\] + +5. Fixed a bug that would cause a mutex to be unlocked a second time if we ran out of space while tracking pinned pages. \[#16228\] + +6. Fixed a bug Sparc/GCC when using test-and-set mutexes. They are now aligned on an 8-byte boundary. \[#16243\] + +7. Fixed a bug to now prevent a thread calling DB_ENV-\>failcheck to hang on a mutex held by a dead thread. \[#16446\] + +8. Fixed a bug so that \_\_db_pthread_mutex_unlock() now handles the failchk case of finding a busy mutex which was owned by a now-dead process. \[#16557\] + +9. Removed support for the mutex implementation based on the "fcntl" system call. Anyone configuring Berkeley DB to use this type of mutex in an earlier release will need to either switch to a different mutex type or contact Oracle for support. \[#17470\] + +### Test Suite Changes + +1. Fixed a bug when using failchk(), where a mutex was not released. \[#15982\] + +2. Added a set of basic repmgr tests to run_std and run_all. \[#16092\] + +3. Added control wrapper for db_reptest to test suite. \[#16161\] + +4. Fixed a bug to now skip tests if db_reptest is not configured. \[#16161\] + +5. Changed name of run_db_in_mem to run_inmem_db, and run_inmem to run_inmem_log and made the arg orders consistent. \[#16358\] + +6. Fixed a bug to now clean up stray handles when rep_verify doesn't work. \[#16390\] + +7. Fixed a bug to avoid db_reptest passing the wrong flag to repmgr_start when there is already a master. \[#16475\] + +8. Added new tests for abbreviated internal init. Fixed test not to expect in-memory database to survive recovery. \[#16495\] + +9. Fix a bug, to add page size for txn014 if the default page size is too small. Move files instead of renaming directory for env015 on QNX. \[#16627\] + +10. Added new rep088 test for log truncation integrity. \[#16732\] + +11. Fixed a bug by adding a checkpoint in rep061 to make sure we have messages to process. Otherwise we could hang with client stuck in internal init, and no incoming messages to trigger rerequest. \[#16781\] + +### Transaction Subsystem Changes: + +1. Fixed a bug to no longer generate an error if DB_ENV-\>set_flags (DB_TXN_NOSYNC) was called after the environment was opened. \[#16492\] + +2. Fixed a bug to remove a potential hang condition in replication os_yield loops when DB_REGISTER used with replication by adding PANIC_CHECKS. \[#16502\] + +3. Fix a bug to now release mutex obtained before special condition returns in \_\_db_cursor_int and \_\_txn_record_fname. \[#16665\] + +4. Fixed a leak in the transaction region when a snapshot update transaction accesses more than 4 databases. \[#16734\] + +5. Enabled setting of set_thread_count via the DB_CONFIG file. \[#16878\] + +6. Fixed a mutex leak in some corner cases. \[#16665\] + +### Utility Changes: + +1. The db_stat utility with the -RA flags will now print a list of known remote replication flags when using repmgr. \[#15484\] + +2. Restructured DB salvage to walk known leaf pages prior to looping over all db pages. \[#16219\] + +3. Fixed a problem with upgrades to 4.7 on big endian machines. \[#16411\] + +4. Fixed a bug so that now db_load consistently returns \>1 on failure. \[#16765\] + +5. The db_dump utility now accepts a "-m" flag to dump information from a named in-memory database. \[#16896\] + +6. Fixed a bug that would cause db_hotbackup to fail if a database file was removed while it was running. \[#17234\] + +### Configuration, Documentation, Sample Application, Portability and Build Changes: + +1. Fixed a bug to now use the correct Perl include path. \[#16058\] + +2. Updated the version of the Microsoft runtime libraries shipped. \[#16058\] + +3. Upgraded the Visual Studio build files to be based on Visual Studio 8 (2005+). The build is now simplified. Users can still upgrade the Visual Studio 6.0 project files, if they want to use Visual Studio .NET (7.1) \[#16108\] + +4. Expanded the ex_rep example with checkpoint and log archive threads, deadlock detection, new options for acknowledgment policy and bulk transfer, and use of additional replication features and events. \[#16109\] + +5. Fixed a bug so that optimizations on AIX are re-enabled, avoiding incorrect code generation. \[#16141\] + +6. Removed a few compiler warnings and three type redefinitons when using vxworks and the GNU compiler. \[#16341\] + +7. Fixed a bug on Sparc v9 so that MUTEX_MEMBAR() now uses membar_enter() to get a \#storeload barrier rather than just stbar's \#storestor. \[#16468\] + +8. Berkeley DB no longer supports Win9X and Windows Me (Millenium edition). + +9. Fixed lock_get and lock_vec examples from the Java (and C#) API. Updated the Java lock example. \[#16506\] + +10. Fixed a bug to correctly handle the TPC-B history record on 64-bit systems. \[#16709\] + +11. Add STL API to Linux build. Can be enabled via the --enable-stl flag. \[#16786\] + +12. Add STL API to Windows build, by building the db_stl project in the solution. There are also stl's test and examples projects in this solution. \[#16786\] + +13. Add support to build dll projects for WinCE, in order to enable users to build DB into a dll in addition to a static library.\[#16625\] + +14. Fixed a weakness where several malloc/realloc return values are not checked before use.\[#16664\] + +15. Enabled DB-\>compact for WinCE.\[#15897\] + +16. HP-UX 10 is no longer supported. diff --git a/docs-src/guides/installation/changelog_5_0.md b/docs-src/guides/installation/changelog_5_0.md new file mode 100644 index 000000000..eb48a0b62 --- /dev/null +++ b/docs-src/guides/installation/changelog_5_0.md @@ -0,0 +1,362 @@ +--- +title: "Berkeley DB 11g Release 2 Change Log" +api-name: "Berkeley DB 11g Release 2 Change Log" +source: docs/installation/changelog_5_0.html +--- +## Berkeley DB 11g Release 2 Change Log + + [Changes between 11.2.5.0.26 and 11.2.5.0.32](changelog_5_0.md#idp1125968) + + [Changes between 11.2.5.0.21 and 11.2.5.0.26](changelog_5_0.md#idp1126872) + + [Changes between 4.8 and 11.2.5.0.21](changelog_5_0.md#idp1125192) + + [Known Bugs](changelog_5_0.md#idp1131672) + +### Changes between 11.2.5.0.26 and 11.2.5.0.32 + +1. Added Visual Studio 2010 support. Users can find Visual Studio 2010 solutions and projects on build_windows. \[#18889\] + +2. Fixed a leak of log file ids when a database is closed before the end of a transaction that references it. \[#15957\] + +3. Fixed a race condition that was causing an "unable to allocate space from the buffer cache" error. The error can only be triggered when multiple mpool regions are used and there is a periodic gathering and clearing of statistics. This also fixes a second bug where if you compile without statistics and explicitly set the mpool default pagesize, other environment handles to that environment would not see the correct mpool default pagesize. \[#18386\] + +4. Fix failure to flush pages to disk. \[#18760\] + +5. Fix a general I/O problem on Windows where system doesn't always return ENOENT when file is missing. \[#18762\] + +6. Fixed locking bugs: \[#18789\] + + - Db-\>compact of BTREE with MVCC could return an unpinned page. + - RECNO would fail to lock the next page when splitting a leaf page. + +7. Don't await ack if message not sent due to queue limit exceeded. \[#18682\] + +8. Fixed a bug that could cause data to not be returned in a HASH database that was one of multiple databases in a file and it was opened prior to running DB-\>compact on that database in another thread of control \[#18824\] + +9. Return HANDLE_DEAD on cursor creation that names a specific txn after client callback. \[#18862\] + +10. Remove parting_shot rep_start(CLIENT) in election thread because it can occasionally conflict with rep_start(MASTER) in another thread. \[#18946\] + +11. Fixed a bug that would cause handle locks to be left referencing the wrong metadata page if DB-\>compact moved the metadata page of a sub-database. \[#18944\] + +12. Fixed a bug that might cause an update to a HASH database to fail with an "unpinned page returned" error if it first gets an I/O error while logging. \[#18985\] + +13. Fixed a bug that failed to dirty a page when DB-\>compact moved records within a hash bucket \[#18994\] + +14. Fixed a bug in page allocation where if a non-transactional update was being done, then we release the metadata page lock too early possibly leading to the corruption of the in memory page list used by DB-\>compact. \[#19036\] + +15. A log write failure on a replication master will now cause a panic since the transaction may be committed on some clients. \[#19054\] + +16. Removed the possibility that checkpoints will overlap in the log, decreasing the time to recover \[#19062\] + +17. Fixed a bug that could leave a hash bucket overflow page not linked to the bucket if the unlink of that page aborted. \[#19001\] + +18. Fixed a bug that would leave the next page pointer of a hash bucket that was removed pointing to an invalid page. \[#19004\] + +19. Fixed several bugs that could cause an update running with MVCC to get the wrong version of a page or improperly update the metadata last page number. \[#19063\] + +20. Fixed a bug where an error during an update to a hash database with DB_NOOVERWRITE set could return DB_KEYEXIST rather than the correct error. \[#19077\] + +21. Fixed a bug where an updater supporting DB_READ_UNCOMMITED might downgrade its lock too soon if there was an error during the update \[#19155\] + +22. Fixed a bug that could cause the wrong page number to be on a root or metadata page if DB-\>compact moved the page and the operation was later rolled forward \[#19167\] + +23. Fixed a bug that could cause the close of a secondary index database to fail if the transaction doing the open aborted \[#19169\] + +24. The database open code will no longer log the open and close of the master database in a file when opening a sub database in that file \[#19071\] + +### Changes between 11.2.5.0.21 and 11.2.5.0.26 + +1. Fixed a bug that might cause recovery to fail if processed part of the log that had previously been recovered and a database which was not present was opened in the log and not closed. \[#18459\] + +2. Fixed a bug which could occur when using bulk transfer with Replication Manager. When closing a DB_ENV handle, any remaining bulk buffer contents are flushed, and Replication Manager could have tried to send the resulting messages even though its connections had already been closed, leading in rare circumstances to spurious EBADF error reports, or possibly even arbitrary memory corruption. \[#18469\] + +3. Fixed a bug in C# HasMultiple() that this function always throws exceptions when there are multiple databases in a single db file. \[#18483\] + +4. Fixed the '--enable-dbm' argument to configure. \[#18497\] + +5. Fixed a bug in the Java API where populating a SecondaryDatabase on open could lead to an OutOfMemoryException. \[#18529\] + +6. Fixed a bug where DB SQL reports "The database disk image is malformed" in "group by" operations. \[#18531\] + +7. Fixed a bug that prevented the same process from reconnecting to the database when DB_REGISTER is being used. \[#18535\] + +8. Fix a race between opening and closing SQL databases from multiple threads that could lead to the error "DB_REGISTER limits processes to one open DB_ENV handle per environment". \[#18538\] + +9. Fixed some bugs that could cause a panic or a DB_RUN_RECOVERY error if the sync of the transaction log failed. \[#18588\] + +10. Fixed a bug which would occur when recovery checkpoint was not written because the cache ran out of space attempting to flush the mpool cache. The environment was recovered and all database where made available, but some databases were incorrectly closed. This would cause a subsequent recovery to fail on its backward pass with the error "PANIC: No such file or directory". \[#18590\] + +11. Fixed a bug that segementation fault would occur if DB-\>set_partition_dirs was called before DB-\>set_partition. \[#18591\] + +12. Fixed a bug that the error of "unknown path" would occur if putting duplicate records to duplicated sorted hash database with DB_OVERWRITE_DUP.\[#18607\] + +13. Fixed a bug where DatabaseConfig.getUnsortedDuplicates() returned true when the datbase had been configured for sorted duplicates. \[#18612\] + +14. Fixed a bug that could cause recovery to fail with the error "DB_LOGC-\>get: log record LSN %u/%u: checksum mismatch" if the last log file was nearly full and ended with a partially written log record which was smaller than a checkpoint record. It now erases the invalid partial record before switching to the new log file. \[#18651\] + +15. Initialize DatabaseConfig.pageSize so that it can be queried from Java. \[#18691\] + +16. Fixed a bug that might cause an aborting transaction to fail if it aborted while a DB-\>compact of the same HASH database was compacting the dynamic hash table \[#18695\] + +### Changes between 4.8 and 11.2.5.0.21 + +#### Database or Log File On-Disk Format Changes + +1. The log file format changed in 11.2.5.0.21 + +#### New Features + +1. Replication Manager sites can specify one or more possible client-to-client peers. \[#14776\] + +2. Added resource management feature in all Berkeley DB APIs to automatically manage cursor and database handles by closing them when they are not required, if they are not yet closed.\[#16188\] + +3. Added a SQL interface to the Berkeley DB library. The interface is based on - and a drop-in-replacement for - the SQLite API. It can be accessed via a command line utility, a C API, or existing APIs built for SQLite. \[#16809\] + +4. Added hash databases support to the DB-\>compact interface. \[#16936\] + +5. Renamed the "db_sql" utility to "db_sql_codegen". This utility is not built by default. To build this utility, enter --enable-sql_codegen as an argument to configure. \[#18265\] + +6. Added transactional support in db_sql_codegen utility. Specify TRANSACTIONAL or NONTRANSACTIONAL in hint comments in SQL statement, db_sql_codegen enable/disable transaction in generated code accordingly. \[#17237\] + +7. Added the feature read-your-writes consistency that allows client application to check, or wait for a specific transaction to be replicated from the master before reading database. \[#17323\] + +8. Added DB log verification feature, accessible via the API and a new utility. This feature can help debugging and analysis. \[#17420\] + +9. Added support for applications to assign master/client role explicitly at any time. Replication Manager can now be configured not to initiate elections. \[#17484\] + +10. Enhanced the DB-\>compact method so that it can reassign metadata and root pages from subdatabases to lower numbered pages while compacting a database file that contains multiple databases. This feature helps to free the higher numbered pages and truncate the file. \[#17554\] + +11. Added system diagnostic messages that are ON by default. \[#17561\] + +12. Added the feature to assign a priority level to transactions. When resolving a deadlock: + + - if the transactions have differing priority, the lowest priority transaction is aborted + - if all transactions have the same priority, the same poilcy that existed before priorities were introduced is used \[#17604\] + +13. Added a feature in which log_archive uses group-wide information for archiving purposes if Replication Manager is in use. \[#17664\] + +14. Added a feature by which the Replication Manager application clients now automatically request any missing information, even when there is no master transaction activity. \[#17665\] + +15. Added support for sharing logs across mixed-endian systems. \[#18032\] + +16. Added an option to specify the first and last pages to the db_dump utility. You can do this by providing -F and -L flags to the db_dump -d option. \[#18072\] + +17. Added Intel Performance Primitive (IPP) AES encryption support. \[#18110\] + +18. Removed support for the configuration option --with-mutex=UNIX/fcntl as of version 4.8. If Berkeley DB was configured to use this type of mutex in an earlier release, switch to a different mutex type or contact Oracle for support. \[#18361\] + +#### Database Environment Changes + +1. Fixed a bug to reflect the correct configuration of the logging subsystem when the DB_ENV-\>log_set_config method is called with the DB_LOG_ZERO flag in a situation where a DB_ENV handle is open and an environment exists. \[#17532\] + +2. Fixed a bug to prevent memory leak caused when the environment is closed by the named in-memory database in a private database environment which has open named in-memory databases. \[#17816\] + +3. Fixd a race condition in an internal directory-scanning function that returns the ENOENT ("No such file or directory") error, if a file is removed just before a call to stat() or its eqivalent. \[#17850\] + +#### Access Method Changes + +1. Fixed a bug to prevent a page in the hash database from carrying the wrong header information when a group allocation is rolled forward by recovery. \[#15414\] + +2. Improved the sort function for bulk put operations. \[#17440\] + +3. Fixed a bug in the DB-\>compact method to ensure locking of leaf pages when merging higher level interior nodes or when freeing interior nodes when descending to find a non-zero length key. \[#17485\]\[#16466\] + +4. Fixed a bug to prevent a trap if a cursor is opened or closed when another thread is adjusting cursors due to an update in the same database. \[#17602\] + +5. Fixed a bug that incorrectly lead to the error message "library build did not include support for the Hash access method" \[#17672\] + +6. Fixed a bug to ensure that the DB-\>exists method accepts the DB_AUTO_COMMIT flag. \[#17687\] + +7. In the past, removing a database from a multi-database file that was opened in an environment always caused dirty pages in the file to be flushed from the cache. In this release, there is no implicit flush as part of a DB-\>remove for handles opened in an environment. Applications that expect the database file to be flushed will need to add an explicit flush. \[#17775\] + +8. Fixed a bug so that the code does not loop if a DB-\>compact operation processed a 3 or more level non-sorted off page duplicate tree. \[#17831\] + +9. Fixed a bug that could leave pages pinned in the cache if an allocation failed during a DB-\>compact operation. \[#17845\] + +10. Fixed a bug to ensure sequences are closed when an EntityStore is closed. \[#17951\] + +11. Fixed a bug that prevented retrieval of a non-duplicate record with DB_GET_BOTH_RANGE in hash sorted duplicate db. In a database configured with sorted duplicate support, when the DBcursor-\>get method is passed the DB_GET_BOTH_RANGE flag, the data item should be retrieved that is the smallest value greater than or equal to the value provided by the data parameter (as determined by the comparison function). \[#17997\] + +12. Fixed a bug that causes the wrong file to be removed if multiple cascading renames are done in the same transaction. \[#18069\] + +13. Fixed a bug to prevent the DB-\>compact method specified with the DB_AUTO_COMMIT flag from acquiring too many locks. \[#18072\] + +14. Fixed a bug that might cause DB-\>compact on a DB_BTREE database to get a spurious read lock on the metadata page. If the database was opened non-transactionally the lock would get left behind. \[#18257\] + +15. Fixed a bug that could lead to btree structure corruption if the DB-\>compact method ran out of locks \[#18361\] + +16. Fixed a bug that would generate an error if a non-BDB file was used to create a database and the DB_TRUNCATE flag was specified. \[#18373\] + +17. Fixed a bug that might cause a trap reading unitialized memory when backing out a merge of a duplicate tree leaf page during DB-\>compact. \[#18461\] + +#### Locking Subsystem Changes + +1. Fixed a bug to ensure deadlock detection works even when there are blocked transactions, configured with and without timeouts. \[#17555\] + +2. Fixed a bug to ensure a call to the DB-\>key_range method from outside a transaction does not lock pages. \[#17930\] + +3. Fixed a bug that could cause a segmentation fault if the lock manager ran out of mutexes \[#18428\] + +#### Logging Subsystem Changes + +1. Limited the size of a log record generated by freeing pages from a database, so that it fits in the log file size. \[#17313\] + +#### Memory Pool Subsystem Changes + +1. Fixed a bug to ensure mulitple versions of a buffer are not created when MVCC is not set. \[#17495\] + +2. Fixed a bug to detect if cache size is being set when the cache is not configured. \[#17556\] + +3. Fixed a bug to ensure the error message "unable to allocate space from the buffer cache" generated when there is still some space available, can be cleared by running recovery.\[#17630\] + +4. Fixed a race condition that causes an operation to return EPERM when the buffer cache is nearly filled with pages belonging to recently closed queue extents. \[#17840\] + +5. Fixed a bug that could cause a page needed by a snapshot reader to be overwritten rather than copied when it was freed. \[#17973\] + +6. Enabled set_mp_pagesize to be specified in the DB_CONFIG file. \[#18015\] + +7. Fixed a bug to ensure single-version or obsolete buffers were selected over any intermediate version. \[#18114\] + +#### Mutex Subsystem Changes + +1. Fixed a bug on HP-UX when specifying --with-mutex=HP/msem_init during configure. It would generate the error "TAS: mutex not appropriately aligned" at runtime, when initializing the first mutex. \[#17489\] + +2. Fixed a race condition which could cause unnecessary retrying of btree searches when several threads simulatenously attempted to get a shared latch. \[#18078\] + +3. Exclusive Transactions have been implemented for the SQL API. See the documentation for details on the behavior of this feature. \[#17822\] + +#### Tcl-specific API Changes + +1. Fixed a bug in Tcl API to prevent a segmentation fault from occurring when the get_dbname method is called to get the db name and the db handle is opened without providing either the filename or dbname. \[#18037\] + +#### C#-specific API Changes + +1. Fixed a bug in C# to prevent a System.AccessViolationException from occurring on Windows7 when trying to open new database. \[#18422\] + +2. Fixed a bug in the C# API to make DB_COMPACT consistent with \_\_db_compact in teh C API. \[#18246\] + +#### API Changes + +1. Added the dbstl_thread_exit method to release thread specific resouces on thread exit. \[#17595\] + +2. Fixed the parser to allow configuration API flags set in the DB_CONFIG file to accept an optional ON/OFF string. The DB_REP_CONF_NOAUTOINIT flag has been removed. It is replaced by DB_REP_CONF_AUTOINIT. However, replication's default behavior remains the same. \[#17795\] + +#### Replication Changes + +1. Fixed bug where a not-in-sync client could service a peer request. \[#18279\] + +2. Fixed bug where page gaps, once filled, would not immediately request the next page gap it finds. This was already fixed for logs. \[#18219\] + +3. Fixed a bug so that only one thread waits for the meta-page lock during internal initialization and broadcasts out the information rather than all threads waiting. Removed the former retry code. \[#17871\] + +4. Added a feature by which the DB-\>open method now allows the DB_CREATE flag on a replication client. It is ignored, but this allows a replication application to make one call that can work on either master or client. It fixes a possible race that could develop in a Replication Manager application if a call to DB-\>open is made around the same time as a master/client role change. \[#15167\] + +5. The DB_ENV-\>repmgr_site_list method now returns an indication on whether the site is a client-to-client peer. \[#16113\] + +6. Fixed a bug that could occasionally lead to elections failing to complete. \[#17105\] + +7. Fixed a bug that could cause DB_ENV-\>txn_stat to trap. \[#17198\] + +8. Added a new JOIN_FAILURE event to notify Replication Manager applications which refuse auto-initialization. \[#17319\] + +9. Fixed a bug where a failed master lease check at a client site causes an ASSERT when processing a master lease grant at the master site. \[#17869\] + +10. Fixed a bug to ensure a second simultaneous call to the DB_ENV-\>rep_elect method does not incorrectly clear a bit flag. \[#17875\] + +11. Fixed a bug in client-side autoremoval of log files. \[#17899\] + +12. Removed the likelihood of dual data streams to enhance network traffic. \[#17955\] + +13. Fixed a bug such that non-txn dup cursors are accounted for in the replication API lockout. \[#18080\] + +14. Fixed a bug to ensure checking for other sync states for the rerequest thread. \[#18126\] + +15. Fixed a bug to avoid getting stuck in an election forever. \[#18151\] + +16. Fixed a bug where using client-to-client synchronization with Master Leases could have resulted in failure of a new master to get initial lease grants from sufficient number of clients, resulting in a master environment panic. \[#18254\] + +17. Fixed a bug which had prevented Replication Manager socket operations from working on HP/UX systems. \[#18382\] + +18. Fixed a bug where starting as a client in multiple threads after receiving dupmaster messages could have resulted in a failure to find a new log file, resulting in a panic. \[#18388\] + +19. The default thread stack size is no longer overridden by default for Berkeley DB threads. \[#18383\] + +#### Transaction Subsystem Changes + +1. Fixed a bug that caused transactions to deadlock on the mutex in the sequence object. \[#17731\] + +2. Fixed a bug to ensure that the failure checking mechanism reconstructs child transactions correctly when a process dies with active sub-transactions. \[#18154\] + +3. Removed a memory leak during recovery related to a deleted database \[#18273\] + +#### Utility Changes + +1. Fixed compiler warnings in the db_sql_codegen utility. \[#17503\] + +2. Enhanced the db_recover -v utility to display the message, "No log files found", if no logs are present. \[#17504\] + +3. Modified the db_verify utility to verify all files instead of aborting on the first failure. \[#17513\] + +4. Modified the db_verify utility to display a message after verification is completed. \[#17545\] + +5. Fixed a bug in the db_sql_codegen utility where the primary key is stored in both key and data fields. Removed it from the data field. \[#17925\] + +#### Example Changes + +1. Fixed a bug that causes the ex_txn C# example to hang. \[#17376\] + +2. Fixed Solaris alignment issues in Stl port test code. \[#17459\] + +3. Added GCC 4.4 compatibility support for all examples. \[#17584\] + +4. Added new command line arguments(-h and -d) to the env examples. \[#17624\] + +5. Fixed configuration problems related to running java API tests. \[#17625\] + +6. Updated the bench_001 example to include bulk testing examples. \[#17766\] + +7. Added a new Stl example to demo advanced feature usage. The Stl test cases referred earlier are replaced by these new examples in the Stl reference document. \[#18175\] + +#### Deprecated Features + +1. The configuration options --disable-cryptography and --enable-cryptoraphy are being deprecated. \[#18110\] + +#### Configuration, Documentation, Sample Apps, Portability and Build Changes + +1. Remove build files for Windows Visual Studio 6.0. \[#16848\] + +2. Added an API, DBENV-\>db_full_version, to return db full version. + +3. Berkeley DB no longer supports Win9X, Windows Me (Millenium edition) and NT 4.0. The minimum supported windows platform is Win 2k. + +4. Berkeley DB no longer supports Visual Studio 6.0. The earliest version supported is Visual Studio 2005. + +5. Added "+u1" to CFLAGS for HP ANSI C Compiler on HP-UX(IA64) to fix the alignment issue found with the allocation functions DB-\>set-alloc and DB_ENV-\>set_alloc. \[#17257\] + +6. Fixed a bug such that the thread local storage (TLS) definition modifier is correctly deduced from the m4 script on all platforms. \[#17609\]\[#17713\] + +7. Fixed a bug such that TLS key is not initialized on platforms which do not support thread local storage (TLS) keywords, such as MAC OSX, and where TLS is implemented using pthread API. \[#18001\] + +8. Fixed a bug to ensure that when using Intel C++ compiler (icpc), the TLS code builds successfully. A stricter criteria is adopted to deduce the TLS keyword, and hence pthread API is more likely to be used to implement TLS. \[#18038\] + +9. Adding new configuration option, --with-cryptography={yes\|no\|ipp}. Using --with-cryptography=yes, will give equivalent behavior to the old --enable-cryptography option. Using --with-cryptography=no, will give equivalent behavior to the old --disable-cryptography option. Using --with-cryptograhy=ipp will enable Intel's Performance Primitive (IPP) encryption on linux. \[#18110\] + +### Known Bugs + +1. The configure option --with-uniquename may cause macro redefinition warnings on platforms where BDB implements parts of the standard C library. These warnings (e.g., '"db_int_def.h", line 586: warning: macro redefined: strsep') may occur when functions in the "clib" directory are included during configuration. This cosmetic affect does not affect the correct operation of the library. \[#17172\] + +2. A multithreaded application using a private environment and multi-version concurrency control could, on very rare occasions, generate an illegal pointer access error during the final steps of a clean environment shutdown. \[#17507\] + +3. Although rare, it is possible for a partial log record header at the end of a transaction log to be erroneously accepted as if it were valid, causing the error "Illegal record type 0 in log" during recovery. \[#17851\] + +4. It is possible to get the error "unable to allocate space from the buffer cache" when there are disk errors on the freezer files used by multi-version concurrency control . \[#17902\] + +5. Java API does not support partitioning by keys and the C# API doesn't support partitioning. \[#18350\] + +6. If a database is removed from an environment and it was still opened transactionally and recovery is run, then a future recovery that must process that part of the log may fail. \[#18459\] + +7. Replication "bulk transfer" does not work if Berkeley DB is unable to determine, at environment open time, whether the Replication Manager will be used. To work around this problem, an application using the Replication Manager should call DB_ENV-\>repmgr_set_local_site() before opening the environment. An application using the replication Base API should call DB_ENV-\>rep_set_transport() before opening the environment. \[#18476\] + +8. The BTree prefix comparison function behaves slightly differently in the C API vs the C# API. In the C# API it returns a signed int and in the C API it returns an unsigned int. This can be a problem if the application needs to save more than 2^31 bytes. diff --git a/docs-src/guides/installation/changelog_5_1.md b/docs-src/guides/installation/changelog_5_1.md new file mode 100644 index 000000000..a43a64309 --- /dev/null +++ b/docs-src/guides/installation/changelog_5_1.md @@ -0,0 +1,327 @@ +--- +title: "Berkeley DB Library Version 11.2.5.1 Change Log" +api-name: "Berkeley DB Library Version 11.2.5.1 Change Log" +source: docs/installation/changelog_5_1.html +--- +## Berkeley DB Library Version 11.2.5.1 Change Log + + [Database or Log File On-Disk Format Changes](changelog_5_1.md#idp1052992) + + [New Features](changelog_5_1.md#idp953176) + + [Database Environment Changes](changelog_5_1.md#idp1045336) + + [Concurrent Data Store Changes](changelog_5_1.md#idp1059760) + + [Access Method Changes](changelog_5_1.md#idp981016) + + [API Changes](changelog_5_1.md#idp1049008) + + [SQL-Specific API Changes](changelog_5_1.md#idp1055592) + + [Tcl-Specific API Changes](changelog_5_1.md#idp1056952) + + [Java-Specific API Changes](changelog_5_1.md#idp1052280) + + [C#-Specific API Changes](changelog_5_1.md#idp987592) + + [Direct Persistence Layer (DPL), Bindings and Collections API](changelog_5_1.md#idp1060648) + + [Replication Changes](changelog_5_1.md#idp1070000) + + [Locking Subsystem Changes](changelog_5_1.md#idp1080936) + + [Logging Subsystem Changes](changelog_5_1.md#idp1092608) + + [Memory Pool Subsystem Changes](changelog_5_1.md#idp1076376) + + [Mutex Subsystem Changes](changelog_5_1.md#idp1080752) + + [Transaction Subsystem Changes](changelog_5_1.md#idp1089584) + + [Test Suite Changes](changelog_5_1.md#idp1067160) + + [Utility Changes](changelog_5_1.md#idp1088000) + + [Configuration, Documentation, Sample Apps, Portability, and Build Changes](changelog_5_1.md#idp1091312) + + [Example Changes](changelog_5_1.md#idp1081576) + + [Miscellaneous Bug Fixes](changelog_5_1.md#idp1102152) + + [Deprecated Features](changelog_5_1.md#idp1100024) + + [Known Bugs](changelog_5_1.md#idp1100672) + +This is the changelog for Berkeley DB 11*g* Release 2 (library version 11.2.5.1). + +### Database or Log File On-Disk Format Changes + +1. The database file format was unchanged in 11gR2 library version 11.2.5.1. + +2. The log file format was unchanged in 11gR2 library version 11.2.5.1. + +### New Features + +1. Added Performance event monitoring support for DTrace and SystemTap which can be enabled during configuration. Static probes have been defined where statistics values are updated, where mutex or transactional consistency lock waits occur, and where some other potentially lengthy operations may be initiated. \[#15605\] + +2. Added a new acknowledge policy - DB_REPMGR_ACKS_ALL_AVAILABLE. \[#16762\] + +3. Added transactional bulk loading optimization for non-nested transactions. \[#17669\] + +4. Added exclusive transaction support for the SQL API. \[#17822\] + +5. Added support for bulk update and delete in C# API. \[#18011\] + +6. Added a db_replicate utility. \[#18326\] + +7. Added an implementation of the Online Backup API. \[#18500\] + +8. Added support in Berkeley DB SQL for the vacuum and incremental vacuum pragmas \[#18545\] + +9. Added an option to automatically convert SQLite databases to Berkeley DB on opening. \[#18531\] + +10. Added BDBSQL_SHARE_PRIVATE, an option to enable inter-process sharing of DB_PRIVATE environments using multiple-reader. \[#18533\] + +11. Added database-level locking to optimize single-threaded operations and remove locking limitations for database load operations. \[#18549\] + +12. Added support for DB_INIT_REP, DB_PRIVATE and DB_THREAD in DB_CONFIG file.\[#18555\] + +13. Added support for the BDBSQL_DEFAULT_PAGE_SIZE pragma to override Berkeley DB's choice of page size depending on the filesystem. Use SQLITE_DEFAULT_PAGE_SIZE rather than a hard-coded default. \[#18577\] + +14. Added an extension that allows access to binary files stored outside of the database. What is stored in the database is a pointer to the binary file. \[#18635\] + +15. Added .stat command to dbsql shell to print environment, table, and index statistics. \[#18640\] + +16. Added enhancements to reduce the size of indexes in the SQL API by allowing duplicates in the index database and moving the rowid from the index key into the index data. \[#18653\] + +17. Added a compile time flag BDBSQL_FILE_PER_TABLE that causes each table to be created in a separate file. This flag replaces the BDBSQL_SPLIT_META_TABLE flag. \[#18664\] + +18. Added the handling of read only and read write open of the same database in BDB SQL \[#18672\] + +19. Added an encryption implementation to the SQL API \[#18683\] + +### Database Environment Changes + +1. Fixed failchk behavior on QNX. \[#17403\] + +2. Fixed a bug that prevented the same process from connecting to the database after recovery is performed. \[#18535\] + +3. Fixed a bug which would occur when recovery checkpoint was not written because the cache ran out of space attempting to flush the memory pool cache. The environment would be recovered and all database where made available, but some databases would incorrectly closed. This would cause a subsequent recovery to fail on its backward pass with the error "PANIC: No such file or directory". \[#18590\] + +4. Fixed a bug that could cause recovery to fail with the error "DB_LOGC-\>get: log record LSN %u/%u: checksum mismatch" if the last log file was nearly full and ended with a partially written log record which was smaller than a checkpoint record. It now erases the invalid partial record before switching to the new log file. \[#18651\] + +### Concurrent Data Store Changes + +1. None + +### Access Method Changes + +1. Fixed a bug such that segementation fault does not occur if DB-\>set_partition_dirs is called before DB-\>set_partition. \[#18591\] + +2. Fixed a bug such that the error "unknown path" does not occur if you put duplicate records into a duplicated sorted HASH database with DB_OVERWRITE_DUP option. \[#18607\] + +3. Added the ability to specify that data should not be logged when removing pages from a database. This can be used if the ability to recover the data is not required after the database has been removed. \[#18666\] + +4. Fixed a bug that caused an aborting transaction to fail if it aborted while a DB-\>compact of the same HASH database was compacting the dynamic hash table \[#18695\] + +5. Fixed a bug that could cause DB-\>compact to loop on a DB_RECNO database or a database with an multilevel unsorted off page duplicate tree. \[#18722\] + +6. Fixed a bug that could cause an illegal page type error when using a HASH database with MVCC and the HASH table was contracted and then extended. \[#18785\] + +7. Fixed locking bugs: \[#18789\] The Db-\>compact method of BTREE with MVCC would return an unpinned page. The RECNO option would fail to lock the next page when splitting a leaf page. + +8. Fixed a bug that could cause data to not be returned in a HASH database that was one of multiple databases in a file and was opened prior to running DB-\>compact method on that database in another thread of control \[#18824\] + +9. Fixed a bug where doing a bulk insert with secondaries could return an error incorrectly. \[#18878\] + +10. Fixed a bug that would return DB_NOTFOUND instead of DB_BUFFER_SMALL when the first item in a HASH database is larger than the user supplied buffer. \[#18829\] + +### API Changes + +1. Fixed various items uncovered by extending DB_CONFIG support: \[#18720\] - Added missing set_cache_max method, and fixed name of log_set_config (was set_log_config). - Added new DB_ENV-\>repmgr_get_local_site method. - Fixed a bug which could fail to allocate enough mutexes when specifying a maximum cache size. - Fixed a bug that could allocate multiple caches when a small cache size was specified. + +### SQL-Specific API Changes + +1. Allowed SQL applications to attach to the same database multiple times unless shared cache mode is explicitly requested. \[#18340\] + +2. Fixed a bug where auto-removal of log files after writing a checkpoint was not functioning correctly. \[#18413\] + +3. Fixed a race between opening and closing SQL databases from multiple threads that could lead to the error "DB_REGISTER limits processes to one open DB_ENV handle per environment". \[#18538\] + +4. Optimized the SQL adapter for joins. Reduce the number of Berkeley DB operations in a join by caching the maximum key in the primary. \[#18566\] + +5. A SQLITE_LOCKED or SQLITE_BUSY error returned by a statement in an explict transaction will no longer invalidate the entire transaction, but just the statement that returned the error. \[#18582\] + +6. Changed how multiple connections to the same database are detected. Used a fileid so that different paths can be used without error. \[#18646\] + +7. Fixed a bug where the journal (environment) directory was being created prior to the actual environment. \[#18656\] + +8. Added a new PRAGMA to allow tuning of when checkpoints are run. \[#18657\] + +9. Fixed spurious "column \ not unique" error messages.\[#18667\] + +10. Fixed a segmentation fault that could happen when memory could not be allocated for the index key in the SQL API.\[#18783\] + +11. Fixed a bug causing a segfault when releasing a savepoint that was already released. \[#18784\] + +### Tcl-Specific API Changes + +1. Changed to link tcl8.5 by default on Windows\[#18244\] + +### Java-Specific API Changes + +1. Fixed a bug where getAllowPopulate and getImmutableSecondaryKey method always returned false for SecondaryConfig objects returned by SecondaryDatabase.getSecondaryConfig method. \[#16018\] + +2. Fixed a bug which made it impossible to (re)set VerboseConfig.REPLICATION_SYSTEM on the Java API. \[#17561\] + +3. Fixed a bug where populating a SecondaryDatabase on open could lead to an OutOfMemoryException. \[#18529\] + +4. Fixed a bug such that segementation fault does not occur when putting records into callback-partitioned database. \[#18596\] + +5. Fixed a bug where DatabaseConfig.getUnsortedDuplicates method returned true when the datbase had been configured for sorted duplicates. \[#18612\] + +6. Initialized DatabaseConfig.pageSize so that it can be queried. \[#18691\] + +7. Fixed a bug by opening a write cursor for Direct Persistent Layer(DPL) entity's put operation in the Concurrent Data Store product. \[#18692\] + +8. Synchronized Java persistence code and tests from Java Edition to Berkeley DB. \[#18711\] + +9. Introduced the EnvironmentConfig.setReplicationInMemory method as a way to configure in-memory internal replication files before opening the Environment handle on the Java API. \[#18719\] + +10. Fixed a bug in the bulk DatabaseEntry class, where it was possible to overflow the buffer. \[#18850\] + +11. Added LEASE_TIMEOUT field to the ReplicationTimeoutType class that enables configuring the amount of time a client grants its Master Lease to a master. \[#18867\] + +### C#-Specific API Changes + +1. Fixed a bug in BTree prefix comparison method such that there is no problem when the application needs to save a number larger than or equal to 2^31. The BTree prefix comparison function now returns an unsigned int instead of a signed int. \[#18481\] + +2. Fixed a bug which caused the HasMultiple method to throw an exception when there were multiple databases in a single database file. \[#18483\] + +3. Fixed a bug to ensure the CachePriority is set for Database and Cursor objects. \[#18716\] + +4. Fixed a bug that use leading to the error: "Transaction that opened the DB handle is still active" when applications used different transactional handles in the associate and open methods in a secondary database. \[#18873\] + +### Direct Persistence Layer (DPL), Bindings and Collections API + +1. All setter methods in the DPL \|StoreConfig\| and \|EvolveConfig\| now return \|this\| rather than having a \|void\| return type. This change requires that applications using the DPL be recompiled. \[#17021\] + +2. Improve performance of \|StoredCollection.removeAll\|. This method no longer iterates over all records in the stored collection. \[#17727\] + +3. Several new tuple formats and binding classes have been added in the \|com.sleepycat.bind.tuple\| package: + + - Packed integer formats have been added that support default natural sorting. These are intended to replace the old unsorted packed integer formats. + - Two new \|BigDecimal\| formats have been added. One format supports default natural sorting. The other format is unsorted, but has other advantages: trailing zeros after the decimal place are preserved, and a more compact, faster serialization format is used. See the \|com.sleepycat.bind.tuple\| package description for an overview of the new bindings and a comparative description of all tuple bindings. \[#18379\] + +4. The following classes are now certified to be serializable. \[#18738\] + + - com.sleepycat.persist.IndexNotAvailableException + - com.sleepycat.persist.StoreExistsException + - com.sleepycat.persist.StoreNotFoundException + - com.sleepycat.persist.evolve.DeletedClassException + - com.sleepycat.persist.evolve.IncompatibleClassException + +### Replication Changes + +1. Replication Manager now uses the standard system implementation of getaddrinfo() when running on Windows, which means that it can support IPv6 addresses if support is present and configured in the operating system. \[#18263\] + +2. Fixed a bug which caused a "full election" to fail if a majority of sites were not ready when the election started. \[#18456\] + +3. Fixed a bug which could occur when using bulk transfer with Replication Manager. When closing a DB_ENV handle, any remaining bulk buffer contents are flushed, and Replication Manager could have tried to send the resulting messages even though its connections had already been closed, leading in rare circumstances to spurious EBADF error reports, or possibly even arbitrary memory corruption. \[#18469\] + +4. Fixed a bug which caused Replication Manager to wait for acknowledgement from client, even if it had failed to send a log record, due to "queue limit exceeded". Replication Manager now returns immediately, with a PERM_FAILED indication, to avoid a pointless delay to the commit() operation. \[#18682\] + +5. Fixed a bug where changes made in one process to Replication Manager configuration values (such as ack policy or ack timeout) were not observed in other processes sharing the same database environment. \[#18839\] + +6. Fixed bugs that could prevent client synchronization from completing due to a failure to request missing log records. \[#18849\] + +7. Fixed a bug where a client that had rolled back transactions to synchronize with a new master, failed to invalidate existing database handles later used for cursor operations based on an explicitly provided transaction. \[#18862\] + +8. Fixed a bug where Replication Manager called for an election after a DUPMASTER event, even when using Master Leases. In such a case it now simply accepts the new (remote) master. \[#18864\] + +9. Fixed a bug which would cause failure if client env attempted to perform sync-up recovery to a point in the log that happened to fall exactly on a log file boundary. \[#18907\] + +### Locking Subsystem Changes + +1. Moved the wait mutex from the lock structure to the locker structure, reducing the number of mutexes required in the system. \[#18685\] + +### Logging Subsystem Changes + +1. None. + +### Memory Pool Subsystem Changes + +1. Fixed a race condition that was causing the error: "Unable to allocate space from the buffer cache". The error can only be triggered when multiple memory pool regions are used and there is a periodic gathering and clearing of statistics. This also fixes a second bug where if you compile without statistics and explicitly set the memoru pool default pagesize, other environment handles to that environment would not see the correct memory pool default pagesize. \[#18386\] + +2. Fixed a bug where the get_cachesize method and the mpool_stat method returned the initial cache size, even if the cache size had been changed. \[#18706\] + +3. Changed memory pool allocation so that the EIO error is returned rather than the ENOMEM error when the memory cannot be allocated because dirty pages cannot be written. \[#18740\] + +### Mutex Subsystem Changes + +1. Fixed problems with the printed statistics for DB_MUTEX_SHARED latches. The DB_STAT_CLEAR flag (as specified by db_stat -Z) did not clear the counts of the number of times a shared latch either had to wait to get access or was able to get the latch without waiting. Also, the ownership state of a test-and-set latch (not a hybrid one) was always displayed as not owned, even when it was held. \[#17585\] \[#18743\] + +### Transaction Subsystem Changes + +1. Fixed bugs that could caused PANIC or DB_RUNRECOVERY errors when the synchronization of the transaction log failed. \[#18588\] + +2. Fix javadoc to note the exception to the rule that a transaction handle may not be accessed after commit() operaton (getCommitToken() is allowed). \[#18730\] + +### Test Suite Changes + +1. None. \[#18831\] + +### Utility Changes + +1. Modified the db_printlog and db_dump -da so that they use the same formatting. The db_logprint utility now uses the message stream. Both db_dump -da and db_printlog accept a -D flag to indicate the numer of bytes of data items to display. You can set this value when calling the DB_ENV-\>set_data_len method or in the DB_CONFIG. \[#18365\] + +2. Fixed a bug that caused a segmentation violation when using the db_printlog utility. \[#18694\] + +3. Fixed a bug in db_hotbackup that would cause a trap if the -D flag is used and the DB_CONFIG file does not specify a log directory. \[#18841\] + +### Configuration, Documentation, Sample Apps, Portability, and Build Changes + +1. Added support and documentation for iPhone OS. \[#18223\] + +2. Fix a bug to make the configuration option --enable-debug work when CFLAGS is set. \[#18432\] + +3. Updated Visual Studio project files to enable ADO.NET support. \[#18448\] + +4. Enhanced the source tree layout making it easier to navigate. \[#18492\] + +5. Fixed the --enable-dbm argument to configure. \[#18497\] + +6. Fixed Visual Studio project files so that they can load into Visual Studio 2010. \[#18505\] + +7. Updated Windows CE build files to be consistent with desktop Windows build files. Added Windows Mobile 6.5.3 Professional as a target platform. \[#18516\] + +8. Added a fix so that an error message is displayed when the 'ar' utility is missing when configured. \[#18619\] + +9. Added tighter integration of JDBC on POSIX/autoconf by including an argument --enable-jdbc to configure. \[#18621\] + +10. Fix build conflicts in log verify with other configurations. \[#18658\] + +11. Upgraded Berkeley DB SQL to SQLite version 3.7.0 \[#18857\] + +### Example Changes + +1. Renamed examples/c/bench_001 to examples/c/ex_bulk. \[#18537\] + +### Miscellaneous Bug Fixes + +1. Provided a functionality on the Windows platform to choose a default page size based on the underlying file system sector size. \[#16538\] + +2. Changed DB_NOSYNC from an "operation" constant to a flag value. \[#17775\] + +3. Changed the default permissions for files in the release tree to allow write access. \[#17974\] + +4. Fixed a bug which caused database verification to hang when verifying a database in a Concurrent Data Store environment that performs locking on an environment-wide basis (DB_CDB_ALLDB.) \[#18571\] + +### Deprecated Features + +1. \[#18871\] Removed the mod_db4 PHP/Apache wrapper. It only supported Apache 1.3 and has not been actively supported. Use php_db4 instead. + +### Known Bugs + +1. None diff --git a/docs-src/guides/installation/changelog_5_2.md b/docs-src/guides/installation/changelog_5_2.md new file mode 100644 index 000000000..8dc66d375 --- /dev/null +++ b/docs-src/guides/installation/changelog_5_2.md @@ -0,0 +1,328 @@ +--- +title: "Berkeley DB Library Version 11.2.5.2 Change Log" +api-name: "Berkeley DB Library Version 11.2.5.2 Change Log" +source: docs/installation/changelog_5_2.html +--- +## Berkeley DB Library Version 11.2.5.2 Change Log + + [Database or Log File On-Disk Format Changes](changelog_5_2.md#idp972456) + + [New Features](changelog_5_2.md#idp978720) + + [Database Environment Changes](changelog_5_2.md#idp984720) + + [Concurrent Data Store Changes](changelog_5_2.md#idp995752) + + [Access Method Changes](changelog_5_2.md#idp989160) + + [SQL API Changes](changelog_5_2.md#idp989544) + + [C API Changes](changelog_5_2.md#idp971912) + + [Tcl-specific API Changes](changelog_5_2.md#idp996528) + + [C#-specific API Changes](changelog_5_2.md#idp972000) + + [Replication Changes](changelog_5_2.md#idp994456) + + [Locking Subsystem Changes](changelog_5_2.md#idp996912) + + [Logging Subsystem Changes](changelog_5_2.md#idp1010640) + + [Memory Pool Subsystem Changes](changelog_5_2.md#idp992728) + + [Mutex Subsystem Changes](changelog_5_2.md#idp1018872) + + [Transaction Subsystem Changes](changelog_5_2.md#idp1011056) + + [Test Suite Changes](changelog_5_2.md#idp1003424) + + [Utility Changes](changelog_5_2.md#idp1029752) + + [Configuration, Documentation, Sample Apps, Portability and Build Changes](changelog_5_2.md#idp1031368) + + [Example Changes](changelog_5_2.md#idp1003200) + + [Miscellaneous Bug Fixes](changelog_5_2.md#idp1034280) + + [Deprecated Features](changelog_5_2.md#idp1035816) + + [Known Bugs](changelog_5_2.md#idp1037736) + +This is the changelog for Berkeley DB 11*g* Release 2 (library version 11.2.5.2). + +### Database or Log File On-Disk Format Changes + +1. Existing database file formats were unchanged in library version 11.2.5.2. However, a new database file format, "heap", was introduced. + +2. The log file format changed in library version 11.2.5.2. + +### New Features + +1. Replication Manager now manages Group Membership. This allows sites to be added to and removed from the replication group dynamically. Replication Manager also now automatically keeps track of the group size (nsites). \[#14778\] + +2. Initial allocations for various non-pagebuffer (mpool) system resources may now be specified, as well as a total maximum of memory to use, rather than specifying a maximum value for each resource. \[#16334\] + +3. Implemented Berkeley DB globalization support architecture to enable localized and stripped error and output messages. \[#16863\] + +4. Added a new access method, DB_HEAP. Heap aims for efficient use (and re-use) of disk space. Keys in a heap database are automatically generated by BDB, it is recommended that one or more secondary indexes be used with a heap database. For full details on DB_HEAP, see the Programmer's Reference Guide. \[#17627\] + +5. Added a compatible mode for 32bit and 64bit Windows environment. \[#18225\] + +6. For the SQL API, concurrency between read and write transactions can now be enabled using "PRAGMA multiversion". Added several pragmas that can be used to configure the Berkeley DB datastore. \[#18521\] + +7. Add several new pragmas to provide in-process support for replication in the SQL API. \[#18528\] + +8. The Berkeley DB X/open compliant XA resource manager has been restored, including support for multi-threaded servers. \[#18701\] + +9. Improved the ability to recover from an application crash on connections through the SQL API. Berkeley DB will try to automatically clean up locks, mutexes and transactions from the failed process. \[#18713\] + +10. Add support for sequence usage in the SQL API using SQLite custom functions. \[#19007\] + +11. Add a pragma in the SQL API to allow execution of a cache trickle command. \[#19202\] + +12. Add a pragma in the SQL API to allow configuration of DB_SYSTEM_MEM environments. \[#19249\] + +13. The new db_env_set_win_security(SECURITY_ATTRIBUTES \*) function allows an application to specify the particular Microsoft Windows security attributes to be used by Berkeley DB. This helps support applications which reduce their privileges after opening the environment. \[#19529\] + +### Database Environment Changes + +1. None + +### Concurrent Data Store Changes + +1. None + +### Access Method Changes + +1. Modified the queue access method so that it only uses latches on the metadata page rather than a latch and a lock. This was done to improve performance. \[#18749\] + +2. Fixed several bugs that could cause an update running with MVCC to get the wrong version of a page or improperly update the metadata last page number. \[#19063\] + +3. The database open code will no longer log the open and close of the master database in a file when opening a sub database in that file. \[#19071\] + +4. Fixed a bug where an error during an update to a hash database with DB_NOOVERWRITE set could return DB_KEYEXIST rather than the correct error. \[#19077\] + +5. Fixed a bug that could cause the wrong page number to be on a root or metadata page if DB-\>compact moved the page and the operation was later rolled forward. \[#19167\] + +6. Fixed a bug that could cause the close of a secondary index database to fail if the transaction doing the open aborted. \[#19169\] + +7. Fixed a bug that could prevent an update to a primary recno or queue database with DB_NOOVERWITE set. \[#19230\] + +8. Fixed a bug when an update to a database with DB_NOOVERWRITE set could incorrectly return DB_KEYEXIST rather than the correct error (e.g., DB_LOCK_DEADLOCK). \[#19345\] + +9. Fixed a bug preventing the use of the set_re_len and set_re_pad methods with a RECNO database when configuring with --disable-queue. \[#19367\] + +10. Fixed a bug in DB-\>compact on BTREE databases that did not check if the last page in the database could be moved to a lower numbered page. \[#19394\] + +11. Fixed a bug that could cause a Log Sequence Error when recovering the deallocation of a multiple page overflow chain. \[#19474\] + +12. Fixed a bug that could cause a diagnostic assertion if MVCC was in use and multiple levels of a btree needed to be split. \[#19481\] + +13. Fixed a few error paths that could cause a Panic with an "unpinned page returned" error. \[#19493\] + +14. Fixed a bug that closed a race condition that under heavy mult-threaded appending to a queue database could cause some records to be lost. \[#19498\] + +15. Fixed a bug that might cause DB-\>compact to mis-estimate the size of an overflow record when merging two pages. This may cause the page to have more data than desired. \[#19562\] + +16. Fixed a bug in DB_ENV-\>fileid_reset that did not update the fileid's on the metadata pages of subdatabases if the database file was not in native byte order. \[#19608\] + +17. Fixed a bug that caused the first directory specified in the create of a partitioned database to get too many partitions. \[#20041\] + +### SQL API Changes + +1. Fixed a race condition that would cause a corruption error in one process when two processes created the same SQL database. \[#18929\] + +2. Fixed a bug that would cause a constraint violation when updating the primary key with the same value. \[#18976\] + +3. Overwriting an old backup with a new backup using the SQL online backup API will no longer double the size of the database. \[#19021\] + +4. Implemented index optimizations for indexes on large values. \[#19094\] + +5. Fixed a bug that could cause an undetected deadlock between a thread which moved a metadata or root page via a DB-\>compact operation and another thread trying to open the database if the old page was being removed from the file. \[#19186\] + +6. Fix a bug in the BDBSQL_FILE_PER_TABLE option, to allow absolute path names. \[#19190\] + +7. Add a pragma to allow configuration of DB_SYSTEM_MEM environments. \[#19249\] + +8. Exclusive transactions will now block new transactions and will prevent existing transactions from making forward progress. \[#19256\] + +9. Fixed a bug that would cause assert error when opening an in-memory hash database with thread count configured when compiled with --enable-diagnostic. \[#19357\] + +10. Upgrade the bundled version of SQLite to 3.7.6.2 \[#19376\] + +11. Fixed a performance bug with the cache victim selection algorithm when there were multiple cache regions. \[#19385\] + +12. Fixed a bug which could cause two SQL threads to have an undetected deadlock when opening or closing tables. \[#19386\] + +13. Fix a bug that could cause a hang when deleting a table if there are multiple connections to a database from different processes. \[#19419\] + +14. Fixed a bug which could cause multiple threads performing DB-\>compact on the same database file to overrun the in-memory freelist, which could potentially lead to memory corruption. \[#19571\] + +15. Fixed a bug in DB-\>compact that could cause a loop if an attempt to move a sub-database meta data page deadlocked. \[#20028\] + +### C API Changes + +1. Fixed a bug where encryption could not be enabled for individual databases in an encrypted environment. \[#18891\] + +2. Removed two unused error codes, DB_NOSERVER_HOME and DB_NOSERVER_ID. \[#18978\] + +3. Added a DB_DBT_READONLY flag so that users can pass in a non-usermem key (DB_DBT_USERMEM) for get operations. \[#19360\] + +4. Fixed a bug in DB/DBC-\>get/pget that the partial flags are silently ignored with positional flags and return inconsistent DBT. \[#19540\] + +5. Fixed a bug which prevented items from being deleted on a secondary database. \[#19573\] + +6. Fixed a bug to correctly handle the DB_BUFFER_SMALL case on delete operations when compression is enabled. \[#19660\] + +### Tcl-specific API Changes + +1. None. + +### C#-specific API Changes + +1. Added support for partial put/get in the C# API. \[#18795\] + +2. Fixed a bug in compare delegate for secondary db. \[#18935\] + +### Replication Changes + +1. Replication Manager now allows differing ack policies at different sites throughout the group, and supports dynamic changes to the ack policy. (The ack policy in force is determined by the current master.) \[#14993\] + +2. Replication Manager "channels" feature allows applications to share repmgr's communication facilities. \[#17228\] + +3. Add example program for RepMgr "channels" feature: ex_rep_chan. \[#17387\] + +4. Replication Manager now allows dynamic changes to a site's "electability" (changes between zero and non-zero priority). This feature should be used with care, because electability changes can in boundary cases invalidate durability guarantees granted for previous transactions. \[#17497\] + +5. Changed election criteria so that later group transactions won't get overwritten by earlier generations with more log. \[#17815\] + +6. Added changes to master lease checks that result in improved performance when using master leases. \[#18960\] + +7. A log write failure on a replication master will now cause a panic since the transaction may be committed on some clients. \[#19054\] + +8. Fixed a few memory leak conditions on error paths. \[#19131\] + +9. Change lease code so that zero priority sites do not count in lease guarantees since they cannot be elected. \[#19154\] + +10. Repmgr rerequest processing is moved from a dedicated thread to heartbeat messages. Repmgr clients using heartbeats can now detect and rerequest missing final master log records without master activity. \[#19197\] + +11. Repmgr statistics are now included in full statistics output for an environment. \[#19198\] + +12. Fix an inefficiency in mixed version elections. We now check if an election is won via the EID instead of priority. \[#19254\] + +13. Changed election LSNs to use the last txn commit LSN instead of the end of the log. \[#19278\] + +14. Create replication internal database files in the environment home directory rather than the data directory so that they are in the same location as the other internal replication files. \[#19403\] + +15. Fix a bug that was preventing repmgr from calling an election when starting a site with the DB_REP_ELECTION flag. \[#19546\] + +16. Fixed a bug which could cause a segfault at a replication master if a named in-memory database was being created around the same time as a client site were synchronizing (in "internal init") with the master. \[#19583\] + +17. Adjust lease code to consider timeout length when retrying. \[#19705\] + +18. Fixed a bug that could cause a crash in replication groups of more than 10 sites, with multiple processes sharing each DB environment concurrently. \[#19818\] + +19. Fix a bug where an assertion failure could happen if pages in a database were deallocated during a client internal initialization.\[#19851\] + +20. Fix a bug where an internal initialization of a queue database with non-contiguous extent files could return an error. \[#19925\] + +21. The 2SITE_STRICT replication configuration parameter is now turned on by default. It can be turned off via a call to DB_ENV-\>rep_set_config(). \[#19937\] + +22. Repmgr heartbeats can now help detect a duplicate master without the need for application activity. \[#19950\] + +### Locking Subsystem Changes + +1. Fixed a bug where an updater supporting DB_READ_UNCOMMITED might downgrade its lock too soon if there was an error during the update. \[#19155\] + +2. Fixed a bug where transaction timeouts could have been specified in a database environment where the locking subsystem was disabled. \[#19582\] + +3. Fixed a bug in a diagnostic assertion that was improperly triggered by the removal of a sub-database. \[#19683\] + +4. Fixed a bug that would cause DB_ENV-\>failcheck to free locks for a locker associated with a database handle after the thread that opened the handle exited. \[#19881\] + +### Logging Subsystem Changes + +1. Enhanced recovery so that it will not output extra checkpoint or transaction id recycle log records if there was no activity since the last checkpoint. \[#15330\] + +2. Log checksums can now be disabled using the compile argument --disable-log-checksum. This will give a performance increase at the risk of undetectable corruption in the log records, which would make recovery impossible. \[#19143\] + +3. Fixed a bug that could cause a page that should have been removed from the end of a file still be in the copy of the file in a hot backup. \[#19996\] + +### Memory Pool Subsystem Changes + +1. Fixed a bug in MPOOLFILE-\>get that did not permit the DB_MPOOL_DIRTY flag to be used with other flags. \[#19421\] + +### Mutex Subsystem Changes + +1. Fixed a bug when the mutex region needs to be larger than 4GB, the region size was incorrectly adjusted to be slightly too small to fit the mutexes. \[#18968\] + +2. Fixed a performance problem with hybrid shared latches in which a request for exclusive access would busy-wait (rather than put itself to sleep) if the latch were held by a shared reader. This also fixed the timeout handling of hybrid mutexes. In some cases the timeout would not be honored, resulting in delays for the replication "read your writes" feature which were longer than requested. \[#18982\] + +3. Fixed the timeout handling of the pthreads mutexes used by the replication "read your writes" feature. When a timeout occurred there was a race condition which might result in a hang. \[#19047\] + +### Transaction Subsystem Changes + +1. Fixed a leak of log file ids when a database is closed before the end of a transaction that references it. \[#15957\] + +2. Fixed a bug that would cause a panic if a child transaction performed a database rename, then aborted, and then the parent transaction committed. \[#18069\] + +3. Fixed a bug where we released the metadata page lock too early if a non-transactional update was being done. \[#19036\] + +4. Removed the possibility that checkpoints will overlap in the log, decreasing the time to recover. \[#19062\] + +### Test Suite Changes + +1. Require Tcl 8.5 or greater. + +### Utility Changes + +1. Added a new utility, db_tuner, which analyzes the data in a btree database, and suggests a reasonable pagesize. \[#18910\] + +2. Fixed some bugs in log_verify when there are in-memory database logs and subdb logs. \[#19157\] + +3. Modified db_hotbackup to not read from the file system as required on non-UNIX systems. Also provided the db_copy function for this purpose. \[#19863\] + +4. Fixed db_hotbackup so that when -d/-l or -D is not specified, DB_CONFIG is used to determine the locations of the databases and logs in the source environment. \[#19994\] + +### Configuration, Documentation, Sample Apps, Portability and Build Changes + +1. Changed SQL API library built on \*nix to link with libpthreads when necessary. \[#19098\] + +2. Added CPPFLAGS into our --enable-jdbc configuration. \[#19234\] + +3. Added encryption support into the Windows CE build project for SQL API. \[#19632\] + +4. Fixed a bug in the STAT_INC_VERB() dtrace probe that was causing compiler warnings. \[#19707\] + +5. Fixed a bug that could cause a trap in db_dump using salvage mode if a page was found that was not associated with any database in the file. \[#19974\] + +6. On Cygwin, circumvented a bug in libtool that is exposed when building the BDB SQL API in a directory path containing whitespace characters. \[#19812\] + +### Example Changes + +1. Update repmgr C, C#, C++, Java examples(ex_rep_mgr, ex_rep_gsg_repmgr, ex_rep_chan, excs_repquote, excxx_repquote, excxx_epquote_gsg, repquote, repquote_gsg) with their related API changes for group membership. \[#19586\]\[#19622\] + +2. Port ex_rep_chan, ex_rep_gsg_repmgr,ex_rep_gsg_simple, excxx_repquote_gsg_repmgr, excxx_repquote_gsg_simple to Window.\[#19890\] + +### Miscellaneous Bug Fixes + +1. Fixed a bug where memory copied from the Java API could leak if flags were not correctly configured. \[#19152\] + +### Deprecated Features + +1. None + +### Known Bugs + +1. The SQL API has a known issue when using a blob field with a lot of content and multiple concurrent connections to the database. \[#19945\] + +2. Rollback of a dropped table in the SQL layer contains a mutex leak, which can consume all mutex resources if enough rollbacks of table drops are performed. \[#20077\] + +3. The DB_CONFIG configuration parameters which specify path names currently do not support names containing any whitespace characters. \[#20158\] + +4. The BFile module has a known crash issue when using BFile handle for SQL expressions interface on 64bit platforms. \[#20193\] + +5. On systems without FTRUNCATE, db_verify will return an error for truncated heap databases. This is a bug in db_verify, the database has been truncated correctly and can be used in the future. \[#20195\] + +6. An application using queue extents which is append mostly could see a decrease in the buffer pool hit rate due to the failure to remove pages from closed extents from the buffer pool. \[#20217\] diff --git a/docs-src/guides/installation/changelog_5_3.md b/docs-src/guides/installation/changelog_5_3.md new file mode 100644 index 000000000..e4dbdac4c --- /dev/null +++ b/docs-src/guides/installation/changelog_5_3.md @@ -0,0 +1,306 @@ +--- +title: "Berkeley DB Library Version 11.2.5.3 Change Log" +api-name: "Berkeley DB Library Version 11.2.5.3 Change Log" +source: docs/installation/changelog_5_3.html +--- +## Berkeley DB Library Version 11.2.5.3 Change Log + + [Changes between 11.2.5.3.21 and 11.2.5.3.28](changelog_5_3.md#idp839120) + + [Changes between 11.2.5.3.15 and 11.2.5.3.21](changelog_5_3.md#idp845408) + + [Database or Log File On-Disk Format Changes](changelog_5_3.md#idp636088) + + [New Features](changelog_5_3.md#idp856040) + + [Database Environment Changes](changelog_5_3.md#idp853696) + + [Access Method Changes](changelog_5_3.md#idp844240) + + [SQL API Changes](changelog_5_3.md#idp838728) + + [Java-specific API changes](changelog_5_3.md#idp863240) + + [Replication Changes](changelog_5_3.md#idp867984) + + [Locking Subsystem Changes](changelog_5_3.md#idp853912) + + [Logging Subsystem Changes](changelog_5_3.md#idp844888) + + [Memory Pool Subsystem Changes](changelog_5_3.md#idp868368) + + [Mutex Subsystem Changes](changelog_5_3.md#idp883216) + + [Transaction Subsystem Changes](changelog_5_3.md#idp875448) + + [Utility Changes](changelog_5_3.md#idp889064) + + [Configuration, Documentation, Sample Apps, Portability and Build Changes](changelog_5_3.md#idp892136) + + [Known Bugs](changelog_5_3.md#idp892656) + +This is the changelog for Berkeley DB 11*g* Release 2 (library version 11.2.5.3). + +### Changes between 11.2.5.3.21 and 11.2.5.3.28 + +1. Fixed tcl library linking for AIX 7. \[#17109\] + +2. Fixed a bug that could cause a trap if a lock timeout occurred while opening a database. \[#21098\] + +3. Fixed missing encryption support for the Android JDBC driver. \[#21129\] + +4. Fixed an incorrect message being displayed when the -l option was specified to the db_hotbackup utility. \[#21313\] + +5. Fixed a bug that DB_ENV-\>log_get_config did not work correctly before DB_ENV-\>open. \[#21359\] + +6. Fixed a bug that could cause a SQL build failure when FTS3 is enabled. \[#21382\] + +7. Fixed a bug that prevented in-memory SQL database from being created properly. They can now be created without the use of the SQLITE_OPEN_CREATE flag. \[#21456\] + +8. Fixed a memory leak in SQL online backup. \[#21460\] + +9. Added additional examples for C++ \[#21477\] + +10. Fixed a bug that could make odbc fail to build with sqlite source code. \[#21490\] + +11. Fixed bugs in compaction of large keys in the upper levels of btrees. \[#21569\] + +12. The db utilities (db_xxxx) no longer operate on replication clients that are being automatically initialized. The DB_REP_LOCKOUT error is now returned. \[#21593\] + +13. Using DB_TXN_SNAPSHOT on an HA client will now result in an error. \[#21601\] + +14. Fixed a bug that prevented a sub-database from being created under the directory identified in DB-\>set_create_dir. \[#21603\] + +15. Fixed a race condition on a cursor when using a multi-threaded application with the SQL API. \[#21714\] + +16. Fixed a race condition in the failchk code when cleaning up mutexes. \[#21796\] + +17. Fixed a resource leak in the db-\>verify() function for btrees. The bug would slow down verification and possibly cause it to run out of memory. \[#21917\] + +18. Removed a potential hang when compacting databases with many duplicates. \[#21975\] + +19. Fixed a bug that caused a crash when reentering dbsql and specifying replication=on for a SQL database where replication was already enabled. \[#22116\] + +20. Fixed an incorrect recursive call dealing with joins. \[#22398\] + +21. Fix build failures in ado.net \[#22405\] + +22. Fixed a bug that could cause a JDBC build failure on recent versions of Visual Studio. \[#22497\] + +23. Correct build problems when building dbstl with gcc-4.7.3 \[#22615\] + +### Changes between 11.2.5.3.15 and 11.2.5.3.21 + +1. Fixed incompatibility problems of Java DPL with JDK7, so DPL will now work with JDK7. \[#20586\] + +2. Added a flag to allow database locking to be disabled from the SQL API. \[#20928\] + +3. Fixed a bug that could allocate a heap data page in a region after the region creation has been undone. \[#20939\] + +4. Redundant whitespaces are now ignored in DB_CONFIG lines pertaining to directories, e.g. set_data_dir. \[#20158\] + +5. Fixed a rare race condition that could cause a crash if two processes opened the same database at the same time. \[#21041\] + +6. Fixed a bug that caused DB_ENV-\>backup to stop early if DB_BACKUP_FILES was not set and a non-DB file was in the data directory. \[#21076\] + +7. Fixed missing cross compiling capability for the JDBC driver. \[#21101\] + +8. Allow the same system/machine to host both a master and a replica database through the use of relative pathnames. \[#21105\] + +9. Fixed a bug in the Java API where EnvironmentConfig.setCreateDir would fail to configure the environment. \[#21127\] + +10. Fixed an assert failure in btreeCompare when allocating memory in the wrong thread was causing a memory leak. \[#21232\] + +11. Fixed a bug in the Java API where concurrent operations that change the database schema could lead to a hang. \[#21265\] + +12. Added JDBC code to the code base and updated the windows build files to include the JDBC solution. \[#21294\] + +13. Fixed a bug where the heap's region size was not getting swapped correctly in mixed-endian environments. \[#21295\] + +14. Fixed a bug in the db_sql_jdbc project file for vs2010 that was preventing it from building correctly. \[#21332\] + +### Database or Log File On-Disk Format Changes + +1. Existing database file formats were unchanged in library version 11.2.5.3. + +2. The log file format changed in library version 11.2.5.3. + +### New Features + +1. Added support for verifying named in-memory dbs. \[#16941\] + +2. Added an integer key comparison function to improve performance through the SQL API. \[#19609\] + +3. Support build on the platforms where pthread_t is a struct. \[#19876\] + +4. Added an API call so the user can specify the size of the region in a heap db. \[#19914\] + +5. Improved Replication Manager's ability to recover from the (perhaps rare) phenomenon of two sites trying to connect to each other simultaneously, which used to result in loss of both connections, requiring a retry after the CONNECTION_RETRY timeout period. \[#19980\] + +6. Enhanced the interface for copying databases for a hot backup. Added configure support for --enable-atomicfileread. \[#20129\] + +7. Enhaced the log reading routine to detect that a log file is missing rather than returning that a zero length record was found. \[#20130\] + +8. Added pragma bdbsql_shared_resources to set or report the maximum amount of memory to be used by shared structures in the main environment region and bdbsql_lock_tablesize to set or report the number of buckets in the lock object hash table. These are advanced tuning features for applications with large number of tables or needs to reduce locking on concurrent long running transactions. \[#20156\] + +9. Added set_metadata_dir() and get_metadata_dir() to enable storage of persistent metadata files in a location other than the environment home directory. \[#20174\] + +10. Improved the error handling through the SQL API. Errors can be sent to a file with the use of the BDBSQL_ERROR_FILE pragma. \[#20213\] + +11. Database handles can now be configured to give exclusive access to the database. \[#20331\] + +12. XA transactions will now use transaction snapshots if the XA databases they operate on were configured with DB_MULTIVERSION. \[#20332\] + +13. Added additional stats fields into the C# API \[#20693\] + +14. Added pragma bdbsql_single_process to keep the Berkeley DB environment information on the heap instead of in shared memory. This option cannot be used if the database is accessed from multiple processes. \[#20789\] + +15. Improved the ability of DB-\>compact to move DB_HASH database pages to the begining of the file. \[#20815\] + +### Database Environment Changes + +1. Fixed a bug that could cause a segmentation violation when closing an environment handle which has open database handles on partition databases. \[#20281\] + +2. Fixed a bug that could cause a segmentation violation if a region was extended leaving a very small fragment at the end. \[#20414\] + +3. Changed the behavior of the DB_REGISTER \| DB_RECOVER flag combination, so that recovery is always run if the environment panic bit is set. \[#20767\] + +### Access Method Changes + +1. Fixed a bug were database configuration settings could be lost when the database was opened if the open operation was blocked for any amount of time. \[#20860\] + +2. Fixed a bug that bulk update operations did not work correctly on compressed databases. \[#19017\] + +3. Improved the log flushing performance when ftruncate() is not available on a system. \[#19725\] + +4. When performing partial puts in a heap database, empty pieces will no longer be left in a split record chain. \[#20052\] + +5. Fixed a bug where, on systems without FTRUNCATE, db_verify will return an error for truncated heap databases. \[#20195\] + +6. Fixed a bug where BDB could run out of mutexes when many databases are renamed. \[#20228\] + +7. Fixed a bug where the metadata page in hash databases would not be flushed to disk. \[#20265\] + +8. Fixed a bug that could leave deleted pages from a HEAP database in the buffer cache. \[#20309\] + +9. Fixed a bug where the library would fail to put records with overflow keys into hash duplicate database. \[#20329\] + +10. Fixed a bug in DB-\>compact of btrees that could cause a bad pointer reference. \[#20360\] + +11. Fixed a bug that could cause the last page number stored on the metadata page to be wrong after rolling forward a db-\>compact operation that freed more pages than will fit in a single log record. \[#20646\] + +12. Fixed a bug that could cause DB-\>stat to block on a mutex while holding a lock on the metadata page. \[#20770\] + +13. Fixed a bug that could cause DB-\>compact of a DB_HASH database to fail to mark a page it updated as dirty. \[#20778\] + +14. Fixed a bug where internal HEAP structures were not rebuilt during database handle refresh. \[#20821\] + +15. Fixed a bug with secondary indices, off-page duplicates and DB_READ_COMMITTED which could erroneously release the lock on the page holding a returned record. \[#20853\] + +16. Fixed a bug that could cause a hang or improperly report an empty queue when the queue record numbers wrapped around at 2^32. \[#20956\] + +17. Fixed a bug on Linux or Windows that could generate a checksum error if a database file was being opened while the meta data page happened to be flushed from the cache. \[#20996\] + +### SQL API Changes + +1. Fixed several memory leaks in the Online Backup API. \[#19850\] + +2. Fixed a bug in the SQL API when using large blob items and multiple concurrent connections. \[#19945\] + +3. To avoid a race condition that could cause a snapshot reader to see a wrong version it is now not permitted to open a DB handle specifying DB_MULTIVERSION if that database is currently opened by a handle which did not specify DB_MULTIVERSION. \[#19940\] + +4. Pragma replication=on can now enable replication on an existing database. Turning replication off is now permanent. \[#20180\] + +5. Fixed a bug in the SQL API where it was possible for a schema update to be ignored when accessing a database from multiple processes. \[#20319\] + +6. Fixed a bug where aborting an exclusive transaction followed by an auto-commit read operation causes an assert failure. \[#20567\] + +7. Fixed a bug in the SQL API where using the journal_mode pragma could cause a crash when used as the first operation in a connection on an existing database. \[#20620\] + +8. Turn off the DBSQL encryption option on Windows/WinCE by default to match the behavior on the other platforms. \[#20671\] + +9. Renamed the BDBSQL_OMIT_SHARING preprocessor flag to BDBSQL_SINGLE_PROCESS. \[#20789\] + +10. Fixed a bug dealing with handle lock modes not reflecting the correct state which was causing a deadlock in the SQL API. \[#20862\] + +### Java-specific API changes + +1. Added ReplicationManagerConnectionStatus class and ReplicationManagerSiteInfo.getConnectionStatus(). Deprecated ReplicationManagerSiteInfo.isConnected(). \[#18068\] + +2. Updated EID_MASTER to be "public static final" so that it would be exposed in Java docs. \[#20184\] + +3. Fixed a bug where calls that return Stat objects could cause a segfault. \[#20377\] + +### Replication Changes + +1. Fixed quorum computation when most sites are unelectable. \[#15251\] + +2. Made Replication more resilient to random input on its port. \[#15712\] + +3. Fixed a bug where the datadir structure was not maintained during an internal init. \[#19041\] + +4. Fixed a repmgr memory leak when using DB_PRIVATE. \[#19363\] + +5. Fixed a minor bug to handle ENOMEM when using an in-memory temp database. \[#20197\] + +6. Fixed a bug where multiple long running transactions across checkpoints could cause Log Sequence errors on client systems. \[#20421\] + +7. Fixed a bug where multiple Replication Manager processes would sometimes not all conform to replication-group-aware log archiving. \[#20342\] + +8. Fixed a bug where a Replication Manager master could stop functioning after accepting an obsolete group membership site list from another site. \[#21804\] + +### Locking Subsystem Changes + +1. Fixed a bug that could cause an early lock timeout if a previous error left a lock timeout value set. \[#19973\] + +### Logging Subsystem Changes + +1. Fixed a bug which could cause an incompletely written log record to be recognized as valid, resulting in recovery failing with the message "Illegal record type \ in log". \[#17851\] + +2. Fixed a bug where printlog would fail on in-memory heap databases. \[#20269\] + +### Memory Pool Subsystem Changes + +1. Fixed a bug which overstated the number of clean and dirty pages evicted from the cache. \[#20410\] + +2. Fixed a bug that left a small fragment at the end of a region when extending. \[#20414\] + +3. Fixed a bug where the file bucket was always zero when creating a mpoolfile using the mpool API. \[#20468\] + +4. Fixed a bug with multiversion concurrency control which could cause versions of pages to remain in the cache even though they are no longer needed. \[#20570\] + +5. The memory pool allocator will now start freezing MVCC versions of buffers if it sees more than 1/4 of the available buffers are taken up by versions. \[#20836\] + +### Mutex Subsystem Changes + +1. Fixed a bug in which DB_ENV-\>mutex_set_align() could cause DB_ENV-\>mutex_stat_print(dbenv, DB_STAT_ALL) to display only the first mutex. \[#20522\] + +2. Fixed a bug with DB_ENV-\>mutex_stat_print() in which the information on some mutexes would not be displayed, if any mutex had been freed and not yet reallocated. \[#20533\] + +3. Fix a race condition between DB_ENV-\>failchk() and the allocation of a mutex. \[#21796\] + +### Transaction Subsystem Changes + +1. Fixed a bug where a malloc failure could result in a segfault when doing a put on a database with secondaries. \[#20641\] + +### Utility Changes + +1. Fixed a bug that would cause verify to call the wrong compare function if there are user defined compare functions used and the database has multilevel off page sorted duplicate trees. \[#20284\] + +2. Fixed a bug that could cause recovery to fail if DB-\>compact moved the meta data page of a HASH subdatabase. \[#20708\] + +3. Fixed three problems with db_hotbackup's backup of transaction logs. A hot backup did not use any configured log directory, but would try to open the logs in the environment home. The second fix corrected an error path, in which the memory was freed by the wrong function, possibly causing a guard byte error. The third fix fixed the issue that a wrong message would be displayed when only "-l" was specified. \[#21313\] + +### Configuration, Documentation, Sample Apps, Portability and Build Changes + +1. The DB_CONFIG configuration commands which specify directory pathnames ("set_data_dir", "set_lg_dir", and "set_tmp_dir") now accept names containing whitespace characters. \[#20158\] + +### Known Bugs + +1. If two SQL processes are concurrently altering the schema of the same tables in a database, there is a race condition that can cause the application to hang. \[#20513\] + +2. Replication groups including machines of different endianness do not support the heap access method. \[#21016\] + +3. If a txn that is attempting to remove a region page from a heap database is aborted and another txn is trying to update that same page then it can cause the original txn to abort. This is timing dependant. \[#20939\] + +4. Utilities can operate on a replication client which is being automatically initialized and may therefore be in an inconsistent state. This can cause the utility to fail or to return invalid results. You can use replication statistics to check the site's role (st_status) and master generation (st_gen) before and after a utility runs; if neither has changed, the utility's results are valid. \[#21593\] diff --git a/docs-src/guides/installation/cross_compile_unix.md b/docs-src/guides/installation/cross_compile_unix.md new file mode 100644 index 000000000..c78a9bb2e --- /dev/null +++ b/docs-src/guides/installation/cross_compile_unix.md @@ -0,0 +1,36 @@ +--- +title: "Cross-Compiling on Unix" +api-name: "Cross-Compiling on Unix" +source: docs/installation/cross_compile_unix.html +--- +## Cross-Compiling on Unix + +The purpose of cross-compiling is to build a Berkeley DB library on one platform that is to be used on a different platform. This section demonstrates how to build a library compatible with the ARM platform from a 32-bit x86 Linux platform. You will need to adjust the build tools and options to match your particular platforms. + +1. Download, install, and test a toolchain that can build the desired target binaries. In this example, we use the `arm-linux-gnueabi-gcc` package. + +2. Decide on your configuration options, and set up your environment to use the appropriate compiler and compiler tools. It is often easiest to set this up as a small shell script. For example: + + ``` c + #!/bin/sh -f + env \ + CC=/usr/bin/arm-linux-gnueabi-gcc \ + STRIP=/usr/bin/arm-linux-gnueabi-strip \ + ../dist/configure \ + --build=i686-pc-linux-gnu \ + --host=arm-linux-gnueabi \ + --enable-java \ + --enable-sql \ + --enable-jdbc \ + --enable-smallbuild \ + --enable-shared \ + --enable-stripped_messages \ + --prefix=$HOME/ARM-linux/install \ + $* + ``` + + The `--build` flag indicates the system on which you are compiling and the `--host` flag indicates the target platform. Adjust or omit the `--enable` and `--prefix` flag settings as dictated by your own needs. + +3. Unpack your Berkeley DB distribution and go to the `build_unix` directory. Alternatively, you may create a more appropriately-named directory at the same level and build in it. For example, `build_arm`. + +4. In your build directory, configure by executing your script, then `make` and `make install`. Review any compiler warnings and fix if necessary, repeating the `configure` and `make` steps until you are satisfied. The resulting libraries and executables can then be transferred to your target system. diff --git a/docs-src/guides/installation/debug.md b/docs-src/guides/installation/debug.md new file mode 100644 index 000000000..5e6f1ade7 --- /dev/null +++ b/docs-src/guides/installation/debug.md @@ -0,0 +1,49 @@ +--- +title: "Chapter 3.  Debugging Applications" +api-name: "Chapter 3.  Debugging Applications" +source: docs/installation/debug.html +--- +## Chapter 3.  Debugging Applications + +**Table of Contents** + + [Introduction to debugging](debug.md#debug_intro) + + [Compile-time configuration](debug_compile.md) + + [Run-time error information](debug_runtime.md) + + [Reviewing Berkeley DB log files](debug_printlog.md) + + [Augmenting the Log for Debugging](debug_printlog.md#idp121880) + + [Extracting Committed Transactions and Transaction Status](debug_printlog.md#idp53840) + + [Extracting Transaction Histories](debug_printlog.md#idp41744) + + [Extracting File Histories](debug_printlog.md#idp154152) + + [Extracting Page Histories](debug_printlog.md#idp158032) + + [Other log processing tools](debug_printlog.md#idp124648) + +## Introduction to debugging + +Because Berkeley DB is an embedded library, debugging applications that use Berkeley DB is both harder and easier than debugging a separate server. Debugging can be harder because when a problem arises, it is not always readily apparent whether the problem is in the application, is in the database library, or is a result of an unexpected interaction between the two. Debugging can be easier because it is easier to track down a problem when you can review a stack trace rather than deciphering interprocess communication messages. This chapter is intended to assist you with debugging applications and reporting bugs to us so that we can provide you with the correct answer or fix as quickly as possible. + +When you encounter a problem, there are a few general actions you can take: + +Review the Berkeley DB error output: +If an error output mechanism has been configured in the Berkeley DB environment, additional run-time error messages are made available to the applications. If you are not using an environment, it is well worth modifying your application to create one so that you can get more detailed error messages. See Run-time error information for more information on configuring Berkeley DB to output these error messages. + +Review the options available for the DB_ENV->set_verbose() method: +Look to see if it offers any additional informational and/or debugging messages that might help you understand the problem. + +Add run-time diagnostics: +You can configure and build Berkeley DB to perform run-time diagnostics. (By default, these checks are not done because they can seriously impact performance.) See Compile-time configuration for more information. + +Apply all available patches: +Before reporting a problem in Berkeley DB, please upgrade to the latest Berkeley DB release, if possible, or at least make sure you have applied any updates available for your release from the Berkeley DB web site . + +Run the test suite: +If you see repeated failures or failures of simple test cases, run the Berkeley DB test suite to determine whether the distribution of Berkeley DB you are using was built and configured correctly. diff --git a/docs-src/guides/installation/debug_compile.md b/docs-src/guides/installation/debug_compile.md new file mode 100644 index 000000000..c50b31f66 --- /dev/null +++ b/docs-src/guides/installation/debug_compile.md @@ -0,0 +1,17 @@ +--- +title: "Compile-time configuration" +api-name: "Compile-time configuration" +source: docs/installation/debug_compile.html +--- +## Compile-time configuration + +There are three compile-time configuration options that assist in debugging Berkeley DB and Berkeley DB applications: + + --enable-debug +If you want to build Berkeley DB with **-g** as the C and C++ compiler flag, enter --enable-debug as an argument to configure. This will create Berkeley DB with debugging symbols, as well as load various Berkeley DB routines that can be called directly from a debugger to display database page content, cursor queues, and so forth. (Note that the **-O** optimization flag will still be specified. To compile with only the **-g**, explicitly set the `CFLAGS` environment variable before configuring.) + + --enable-diagnostic +If you want to build Berkeley DB with debugging run-time sanity checks and with DIAGNOSTIC \#defined during compilation, enter --enable-diagnostic as an argument to configure. This will cause a number of special checks to be performed when Berkeley DB is running. This flag should not be defined when configuring to build production binaries because it degrades performance. + + --enable-umrw +When compiling Berkeley DB for use in run-time memory consistency checkers (in particular, programs that look for reads and writes of uninitialized memory), use --enable-umrw as an argument to configure. This guarantees, among other things, that Berkeley DB will completely initialize allocated pages rather than initializing only the minimum necessary amount. diff --git a/docs-src/guides/installation/debug_printlog.md b/docs-src/guides/installation/debug_printlog.md new file mode 100644 index 000000000..1570b42a8 --- /dev/null +++ b/docs-src/guides/installation/debug_printlog.md @@ -0,0 +1,152 @@ +--- +title: "Reviewing Berkeley DB log files" +api-name: "Reviewing Berkeley DB log files" +source: docs/installation/debug_printlog.html +--- +## Reviewing Berkeley DB log files + + [Augmenting the Log for Debugging](debug_printlog.md#idp121880) + + [Extracting Committed Transactions and Transaction Status](debug_printlog.md#idp53840) + + [Extracting Transaction Histories](debug_printlog.md#idp41744) + + [Extracting File Histories](debug_printlog.md#idp154152) + + [Extracting Page Histories](debug_printlog.md#idp158032) + + [Other log processing tools](debug_printlog.md#idp124648) + +If you are running with transactions and logging, the db_printlog utility can be a useful debugging aid. The db_printlog utility will display the contents of your log files in a human readable (and machine-readable) format. + +The db_printlog utility will attempt to display any and all log files present in a designated db_home directory. For each log record, the db_printlog utility will display a line of the form: + +``` c +[22][28]db_big: rec: 43 txnid 80000963 prevlsn [21][10483281] +``` + +The opening numbers in square brackets are the *log sequence number* (*LSN*) of the log record being displayed. The first number indicates the log file in which the record appears, and the second number indicates the offset in that file of the record. + +The first character string identifies the particular log operation being reported. The log records corresponding to particular operations are described following. The rest of the line consists of name/value pairs. + +The rec field indicates the record type (this is used to dispatch records in the log to appropriate recovery functions). + +The txnid field identifies the transaction for which this record was written. A txnid of 0 means that the record was written outside the context of any transaction. You will see these most frequently for checkpoints. + +Finally, the prevlsn contains the LSN of the last record for this transaction. By following prevlsn fields, you can accumulate all the updates for a particular transaction. During normal abort processing, this field is used to quickly access all the records for a particular transaction. + +After the initial line identifying the record type, each field of the log record is displayed, one item per line. There are several fields that appear in many different records and a few fields that appear only in some records. + +The following table presents each currently written log record type with a brief description of the operation it describes. Any of these record types may have the string "\_debug" appended if they were written because DB_TXN_NOT_DURABLE was specified and the system was configured with --enable-diagnostic. + +| Log Record Type | Description | +|----|----| +| bam_adj | Used when we insert/remove an index into/from the page header of a Btree page. | +| bam_cadjust | Keeps track of record counts in a Btree or Recno database. | +| bam_cdel | Used to mark a record on a page as deleted. | +| bam_curadj | Used to adjust a cursor location when a nearby record changes in a Btree database. | +| bam_merge | Used to merge two Btree database pages during compaction. | +| bam_pgno | Used to replace a page number in a Btree record. | +| bam_rcuradj | Used to adjust a cursor location when a nearby record changes in a Recno database. | +| bam_relink | Fix leaf page prev/next chain when a page is removed. | +| bam_repl | Describes a replace operation on a record. | +| bam_root | Describes an assignment of a root page. | +| bam_rsplit | Describes a reverse page split. | +| bam_split | Describes a page split. | +| crdel_inmem_create | Record the creation of an in-memory named database. | +| crdel_inmem_remove | Record the removal of an in-memory named database. | +| crdel_inmem_rename | Record the rename of an in-memory named database. | +| crdel_metasub | Describes the creation of a metadata page for a subdatabase. | +| db_addrem | Add or remove an item from a page of duplicates. | +| db_big | Add an item to an overflow page (*overflow pages* contain items too large to place on the main page) | +| db_cksum | Unable to checksum a page. | +| db_debug | Log debugging message. | +| db_noop | This marks an operation that did nothing but update the LSN on a page. | +| db_ovref | Increment or decrement the reference count for a big item. | +| db_pg_alloc | Indicates we allocated a page to a database. | +| db_pg_free | Indicates we freed a page (freed pages are added to a freelist and reused). | +| db_pg_freedata | Indicates we freed a page that still contained data entries (freed pages are added to a freelist and reused.) | +| db_pg_init | Indicates we reinitialized a page during a truncate. | +| db_pg_sort | Sort the free page list and free pages at the end of the file. | +| dbreg_register | Records an open of a file (mapping the filename to a log-id that is used in subsequent log operations). | +| fop_create | Create a file in the file system. | +| fop_file_remove | Remove a name in the file system. | +| fop_remove | Remove a file in the file system. | +| fop_rename | Rename a file in the file system. | +| fop_write | Write bytes to an object in the file system. | +| ham_chgpg | Used to adjust a cursor location when a Hash page is removed, and its elements are moved to a different Hash page. | +| ham_copypage | Used when we empty a bucket page, but there are overflow pages for the bucket; one needs to be copied back into the actual bucket. | +| ham_curadj | Used to adjust a cursor location when a nearby record changes in a Hash database. | +| ham_groupalloc | Allocate some number of contiguous pages to the Hash database. | +| ham_insdel | Insert/delete an item on a Hash page. | +| ham_metagroup | Update the metadata page to reflect the allocation of a sequence of contiguous pages. | +| ham_newpage | Adds or removes overflow pages from a Hash bucket. | +| ham_replace | Handle updates to records that are on the main page. | +| ham_splitdata | Record the page data for a split. | +| heap_addrem | Add or remove an entry from a Heap database. | +| heap_pg_alloc | Indicates we allocated a page to a Heap database. | +| heap_trunc_meta | Records the truncation of the meta page in a Heap database. | +| heap_trunc_page | Records the truncation of a data page in a Heap database. | +| qam_add | Describes the actual addition of a new record to a Queue. | +| qam_del | Delete a record in a Queue. | +| qam_delext | Delete a record in a Queue with extents. | +| qam_incfirst | Increments the record number that refers to the first record in the database. | +| qam_mvptr | Indicates we changed the reference to either or both of the first and current records in the file. | +| txn_child | Commit a child transaction. | +| txn_ckp | Transaction checkpoint. | +| txn_recycle | Transaction IDs wrapped. | +| txn_regop | Logs a regular (non-child) transaction commit. | +| txn_xa_regop | Logs a prepare message. | + +### Augmenting the Log for Debugging + +When debugging applications, it is sometimes useful to log not only the actual operations that modify pages, but also the underlying Berkeley DB functions being executed. This form of logging can add significant bulk to your log, but can permit debugging application errors that are almost impossible to find any other way. To turn on these log messages, specify the --enable-debug_rop and --enable-debug_wop configuration options when configuring Berkeley DB. See Configuring Berkeley DB for more information. + +### Extracting Committed Transactions and Transaction Status + +Sometimes, it is helpful to use the human-readable log output to determine which transactions committed and aborted. The awk script, commit.awk, (found in the db_printlog directory of the Berkeley DB distribution) allows you to do just that. The following command, where log_output is the output of db_printlog, will display a list of the transaction IDs of all committed transactions found in the log: + +``` c +awk -f commit.awk log_output +``` + +If you need a complete list of both committed and aborted transactions, then the script status.awk will produce it. The syntax is as follows: + +``` c +awk -f status.awk log_output +``` + +### Extracting Transaction Histories + +Another useful debugging aid is to print out the complete history of a transaction. The awk script txn.awk allows you to do that. The following command line, where log_output is the output of the db_printlog utility and txnlist is a comma-separated list of transaction IDs, will display all log records associated with the designated transaction ids: + +``` c +awk -f txn.awk TXN=txnlist log_output +``` + +### Extracting File Histories + +The awk script fileid.awk allows you to extract all log records that refer to a designated file. The syntax for the fileid.awk script is the following, where log_output is the output of db_printlog and fids is a comma-separated list of fileids: + +``` c +awk -f fileid.awk PGNO=fids log_output +``` + +### Extracting Page Histories + +The awk script pgno.awk allows you to extract all log records that refer to designated page numbers. However, because this script will extract records with the designated page numbers for all files, it is most useful in conjunction with the fileid script. The syntax for the pgno.awk script is the following, where log_output is the output of db_printlog and pgnolist is a comma-separated list of page numbers: + +``` c +awk -f pgno.awk PGNO=pgnolist log_output +``` + +### Other log processing tools + +The awk script count.awk prints out the number of log records encountered that belonged to some transaction (that is, the number of log records excluding those for checkpoints and non-transaction-protected operations). + +The script range.awk will extract a subset of a log. This is useful when the output of db_printlog utility is too large to be reasonably manipulated with an editor or other tool. The syntax for range.awk is the following, where **sf** and **so** represent the LSN of the beginning of the sublog you want to extract, and **ef** and **eo** represent the LSN of the end of the sublog you want to extract: + +``` c + awk -f range.awk START_FILE=sf START_OFFSET=so END_FILE=ef \ + END_OFFSET=eo log_output +``` diff --git a/docs-src/guides/installation/debug_runtime.md b/docs-src/guides/installation/debug_runtime.md new file mode 100644 index 000000000..998811b4d --- /dev/null +++ b/docs-src/guides/installation/debug_runtime.md @@ -0,0 +1,14 @@ +--- +title: "Run-time error information" +api-name: "Run-time error information" +source: docs/installation/debug_runtime.html +--- +## Run-time error information + +Normally, when an error occurs in the Berkeley DB library, an integer value (either a Berkeley DB specific value or a system `errno` value) is returned by Berkeley DB. In some cases, however, this value may be insufficient to completely describe the cause of the error, especially during initial application debugging. + +Most Berkeley DB errors will result in additional information being written to a standard file descriptor or output stream. Additionally, Berkeley DB can be configured to pass these verbose error messages to an application function. There are four methods intended to provide applications with additional error information: DB_ENV->set_errcall(), DB_ENV->set_errfile(), DB_ENV->set_errpfx() and DB_ENV->set_verbose(). + +The Berkeley DB error-reporting facilities do not slow performance or significantly increase application size, and may be run during normal operation as well as during debugging. Where possible, we recommend these options always be configured and the output saved in the filesystem. We have found that this often saves time when debugging installation or other system-integration problems. + +In addition, there are three methods to assist applications in displaying their own error messages: db_strerror(), DB_ENV->err(), and `DB_ENV->errx()`. The first is a superset of the ANSI C strerror function, and returns a descriptive string for any error return from the Berkeley DB library. The DB_ENV->err() and `DB_ENV->errx()` methods use the error message configuration options described previously to format and display error messages to appropriate output devices. diff --git a/docs-src/guides/installation/img/arch_bigpic.gif b/docs-src/guides/installation/img/arch_bigpic.gif new file mode 100644 index 000000000..48c52aed5 Binary files /dev/null and b/docs-src/guides/installation/img/arch_bigpic.gif differ diff --git a/docs-src/guides/installation/img/arch_smallpic.gif b/docs-src/guides/installation/img/arch_smallpic.gif new file mode 100644 index 000000000..5eb7ae8da Binary files /dev/null and b/docs-src/guides/installation/img/arch_smallpic.gif differ diff --git a/docs-src/guides/installation/index.md b/docs-src/guides/installation/index.md new file mode 100644 index 000000000..901bf0792 --- /dev/null +++ b/docs-src/guides/installation/index.md @@ -0,0 +1,574 @@ +--- +title: "Berkeley DB Installation and Build Guide" +api-name: "Berkeley DB Installation and Build Guide" +source: docs/installation/index.html +--- +# Berkeley DB Installation and Build Guide + +**Legal Notice** + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction](introduction.md) + + [Installation Overview](introduction.md#install-overview) + + [Supported Platforms](ch01s02.md) + + [2. System Installation Notes](install.md) + + [File utility /etc/magic information](install.md#install_file) + + [Magic information](install.md#magic) + + [Big-endian magic information](install.md#big-endian) + + [Little-endian magic information](install.md#little-endian) + + [Building with multiple versions of Berkeley DB](install_multiple.md) + + [3. Debugging Applications](debug.md) + + [Introduction to debugging](debug.md#debug_intro) + + [Compile-time configuration](debug_compile.md) + + [Run-time error information](debug_runtime.md) + + [Reviewing Berkeley DB log files](debug_printlog.md) + + [Augmenting the Log for Debugging](debug_printlog.md#idp121880) + + [Extracting Committed Transactions and Transaction Status](debug_printlog.md#idp53840) + + [Extracting Transaction Histories](debug_printlog.md#idp41744) + + [Extracting File Histories](debug_printlog.md#idp154152) + + [Extracting Page Histories](debug_printlog.md#idp158032) + + [Other log processing tools](debug_printlog.md#idp124648) + + [4. Building Berkeley DB for Android](build_android_intro.md) + + [Building the Drop-In Replacement for Android](build_android_intro.md#build_android) + + [Migrating from SQLite to Berkeley DB](build_android_intro.md#build_android_migrate) + + [Building the Android JDBC Driver](build_android_jdbc.md) + + [Android Configuration Options](build_android_config.md) + + [5. Building Berkeley DB for Windows](build_win.md) + + [Building Berkeley DB for 32 bit Windows](build_win.md#win_build32) + + [Visual C++ .NET 2010](build_win.md#idp242512) + + [Visual C++ .NET 2008](build_win.md#idp249264) + + [Visual C++ .NET 2005](build_win.md#idp220616) + + [Build results](build_win.md#idp205672) + + [Building Berkeley DB for 64-bit Windows](win_build64.md) + + [x64 build with Visual Studio 2005 or newer](win_build64.md#idp259672) + + [Building Berkeley DB with Cygwin](win_build_cygwin.md) + + [Building the C++ API](win_build_cxx.md) + + [Building the C++ STL API](win_build_stl.md) + + [Building the Java API](build_win_java.md) + + [Building the C# API](build_win_csharp.md) + + [Building the SQL API](build_win_sql.md) + + [Binary Compatibility With SQLite](build_win_sql.md#idp290248) + + [Setting Preprocessor Flags](build_win_sql.md#idp276576) + + [Enabling Extensions](build_win_sql.md#idp288280) + + [Disabling Log Checksums](build_win_sql.md#win-disablechecksums) + + [Building the JDBC Driver](build_win_sql.md#build_jdbc) + + [Using the JDBC Driver](build_win_sql.md#idp266616) + + [Building the ODBC Driver](build_win_sql.md#idp305704) + + [Using the ADO.NET Driver](build_win_sql.md#idp320888) + + [Building the Tcl API](build_win_tcl.md) + + [Distributing DLLs](win_build_dist_dll.md) + + [Additional build options](win_additional_options.md) + + [Building a small memory footprint library](build_win_small.md) + + [Running the test suite under Windows](build_win_test.md) + + [Building the software needed by the tests](build_win_test.md#idp368040) + + [Running the test suite under Windows](build_win_test.md#idp379184) + + [Building the software needed by the SQL tests](build_win_test.md#build_win_test_sql) + + [Windows notes](build_win_notes.md) + + [Windows FAQ](build_win_faq.md) + + [7. Building Berkeley DB for UNIX/POSIX](build_unix.md) + + [Building for UNIX/POSIX](build_unix.md#build_unix_intro) + + [Building the Berkeley DB SQL Interface](build_unix.md#build_unix_sqlinter) + + [Configuring Berkeley DB](build_unix_conf.md) + + [Configuring the SQL Interface](build_unix_sql.md) + + [Changing Compile Options](build_unix_sql.md#config_sql) + + [Enabling Extensions](build_unix_sql.md#idp500824) + + [Building the JDBC Driver](build_unix_sql.md#build_unix_jdbc) + + [Using the JDBC Driver](build_unix_sql.md#idp571856) + + [Building the ODBC Driver](build_unix_sql.md#idp593744) + + [Building the BFILE extension](build_unix_sql.md#bfile) + + [Building a small memory footprint library](build_unix_small.md) + + [Changing compile or load options](build_unix_flags.md) + + [Cross-Compiling on Unix](cross_compile_unix.md) + + [Installing Berkeley DB](build_unix_install.md) + + [Dynamic shared libraries](build_unix_shlib.md) + + [Running the test suite under UNIX](build_unix_test.md) + + [Building SQL Test Suite on Unix](build_unix_test.md#build_unix_test_sql) + + [Architecture independent FAQ](build_unix_notes.md) + + [AIX](build_unix_aix.md) + + [FreeBSD](build_unix_freebsd.md) + + [Apple iOS (iPhone OS)](build_unix_iphone.md) + + [IRIX](build_unix_irix.md) + + [Linux](build_unix_linux.md) + + [Mac OS X](build_unix_macosx.md) + + [QNX](build_unix_qnx.md) + + [SCO](build_unix_sco.md) + + [Solaris](build_unix_solaris.md) + + [SunOS](build_unix_sunos.md) + + [9. Upgrading Berkeley DB 11.2.5.2 applications to Berkeley DB 11.2.5.3](upgrade_53_toc.md) + + [Introduction](upgrade_53_toc.md#upgrade_53_intro) + + [Changes to the build_windows Folder](upgrade_11gr2_53_build_windows.md) + + [Replication Connection Status in the Java API](upgrade_11gr2_53_conn_status.md) + + [New Function](upgrade_11gr2_53_conn_status.md#idp804776) + + [New Class](upgrade_11gr2_53_conn_status.md#idp771568) + + [Deprecated Function](upgrade_11gr2_53_conn_status.md#idp809200) + + [Exclusive Database Handles](upgrade_11gr2_53_excl.md) + + [New Functions](upgrade_11gr2_53_excl.md#idp811424) + + [Configure the Region Size of Heap Databases](upgrade_11gr2_53_heap_regionsize.md) + + [New Functions](upgrade_11gr2_53_heap_regionsize.md#idp775064) + + [New Hotbackup Interface](upgrade_11gr2_53_hotbackup.md) + + [New Functions](upgrade_11gr2_53_hotbackup.md#idp815256) + + [Flags Accepted by DB_ENV-\>backup()](upgrade_11gr2_53_hotbackup.md#idp805032) + + [Flags Accepted by DB_ENV-\>dbbackup()](upgrade_11gr2_53_hotbackup.md#idp822632) + + [Enumerations Accepted by DB_ENV-\>set_backup_config()](upgrade_11gr2_53_hotbackup.md#idp828456) + + [Updated JDBC Version](upgrade_11gr2_53_jdbc.md) + + [Configure Directory to Store Metadata Files](upgrade_11gr2_53_meta_dir.md) + + [New Functions](upgrade_11gr2_53_meta_dir.md#idp837576) + + [Changes in the SQL API Build](upgrade_11gr2_53_sql_build.md) + + [New Berkeley DB SQL API PRAGMAs](upgrade_11gr2_53_sql_pragma.md) + + [New PRAGMAs](upgrade_11gr2_53_sql_pragma.md#idp843792) + + [Replication for Existing Databases in the SQL API](upgrade_11gr2_53_sql_rep.md) + + [PRAGMAs With Permanent Effects](upgrade_11gr2_53_sql_rep.md#idp837896) + + [PRAGMAs That Can Now Operate on Existing Databases](upgrade_11gr2_53_sql_rep.md#idp844568) + + [Berkeley DB X/Open Compliant XA Resource Manager and Transaction Snapshots](upgrade_11gr2_53_xa_mvcc.md) + + [Berkeley DB Library Version 11.2.5.3 Change Log](changelog_5_3.md) + + [Changes between 11.2.5.3.21 and 11.2.5.3.28](changelog_5_3.md#idp839120) + + [Changes between 11.2.5.3.15 and 11.2.5.3.21](changelog_5_3.md#idp845408) + + [Database or Log File On-Disk Format Changes](changelog_5_3.md#idp636088) + + [New Features](changelog_5_3.md#idp856040) + + [Database Environment Changes](changelog_5_3.md#idp853696) + + [Access Method Changes](changelog_5_3.md#idp844240) + + [SQL API Changes](changelog_5_3.md#idp838728) + + [Java-specific API changes](changelog_5_3.md#idp863240) + + [Replication Changes](changelog_5_3.md#idp867984) + + [Locking Subsystem Changes](changelog_5_3.md#idp853912) + + [Logging Subsystem Changes](changelog_5_3.md#idp844888) + + [Memory Pool Subsystem Changes](changelog_5_3.md#idp868368) + + [Mutex Subsystem Changes](changelog_5_3.md#idp883216) + + [Transaction Subsystem Changes](changelog_5_3.md#idp875448) + + [Utility Changes](changelog_5_3.md#idp889064) + + [Configuration, Documentation, Sample Apps, Portability and Build Changes](changelog_5_3.md#idp892136) + + [Known Bugs](changelog_5_3.md#idp892656) + + [10. Upgrading Berkeley DB 11.2.5.1 applications to Berkeley DB 11.2.5.2](upgrade_52_toc.md) + + [Introduction](upgrade_52_toc.md#upgrade_52_intro) + + [SQLite Interface Upgrade](upgrade_11gr2_52_sqlite_ver.md) + + [32bit/64bit Compatibility on Windows](upgrade_11gr2_52_bit_cmp_win.md) + + [Read Only flag for DBT](upgrade_11gr2_52_rep_dbt_readonly.md) + + [New Flag](upgrade_11gr2_52_rep_dbt_readonly.md#idp907000) + + [Dynamic Environment Configuration](upgrade_11gr2_52_dyn_env.md) + + [New Functions](upgrade_11gr2_52_dyn_env.md#idp902144) + + [Deprecated Functions](upgrade_11gr2_52_dyn_env.md#idp912000) + + [Exclusive Transactions in the SQL Layer](upgrade_11gr2_52_excl_txn_sql.md) + + [Group Membership in Repmgr](upgrade_11gr2_52_grp_mbr.md) + + [Upgrading](upgrade_11gr2_52_grp_mbr.md#idp929720) + + [New Functions](upgrade_11gr2_52_grp_mbr.md#idp910056) + + [Modified Functions](upgrade_11gr2_52_grp_mbr.md#idp901088) + + [New Events](upgrade_11gr2_52_grp_mbr.md#idp924520) + + [Removed Functions](upgrade_11gr2_52_grp_mbr.md#idp937928) + + [New Parameters](upgrade_11gr2_52_grp_mbr.md#idp909344) + + [New Structure](upgrade_11gr2_52_grp_mbr.md#idp924776) + + [Heap Access Method](upgrade_11gr2_52_heap.md) + + [New Functions](upgrade_11gr2_52_heap.md#idp936848) + + [Modified Functions](upgrade_11gr2_52_heap.md#idp930424) + + [New Definition](upgrade_11gr2_52_heap.md#idp931776) + + [Enabling Transaction Snapshots in the SQL Layer](upgrade_11gr2_52_mvcc_sql.md) + + [New Pragmas](upgrade_11gr2_52_mvcc_sql.md#idp951464) + + [2SITE_STRICT Enabled by Default in Replication](upgrade_11gr2_52_rep_2site_strict.md) + + [Enabling Replication in the SQL Layer](upgrade_11gr2_52_rep_sql.md) + + [New Pragmas](upgrade_11gr2_52_rep_sql.md#idp962696) + + [Repmgr Message Channels](upgrade_11gr2_52_repmgr_channels.md) + + [New Functions](upgrade_11gr2_52_repmgr_channels.md#idp919280) + + [Sequence Support in the SQL Layer](upgrade_11gr2_52_seq_sql.md) + + [New Functions](upgrade_11gr2_52_seq_sql.md#idp963480) + + [Berkeley DB X/Open Compliant XA Resource Manager](upgrade_11gr2_52_xa.md) + + [Constraints](upgrade_11gr2_52_xa.md#idp973264) + + [New Flag](upgrade_11gr2_52_xa.md#idp978200) + + [Modified Function](upgrade_11gr2_52_xa.md#idp982256) + + [Hot Backup Changes](upgrade_11gr2_52_hot_backup.md) + + [Berkeley DB Library Version 11.2.5.2 Change Log](changelog_5_2.md) + + [Database or Log File On-Disk Format Changes](changelog_5_2.md#idp972456) + + [New Features](changelog_5_2.md#idp978720) + + [Database Environment Changes](changelog_5_2.md#idp984720) + + [Concurrent Data Store Changes](changelog_5_2.md#idp995752) + + [Access Method Changes](changelog_5_2.md#idp989160) + + [SQL API Changes](changelog_5_2.md#idp989544) + + [C API Changes](changelog_5_2.md#idp971912) + + [Tcl-specific API Changes](changelog_5_2.md#idp996528) + + [C#-specific API Changes](changelog_5_2.md#idp972000) + + [Replication Changes](changelog_5_2.md#idp994456) + + [Locking Subsystem Changes](changelog_5_2.md#idp996912) + + [Logging Subsystem Changes](changelog_5_2.md#idp1010640) + + [Memory Pool Subsystem Changes](changelog_5_2.md#idp992728) + + [Mutex Subsystem Changes](changelog_5_2.md#idp1018872) + + [Transaction Subsystem Changes](changelog_5_2.md#idp1011056) + + [Test Suite Changes](changelog_5_2.md#idp1003424) + + [Utility Changes](changelog_5_2.md#idp1029752) + + [Configuration, Documentation, Sample Apps, Portability and Build Changes](changelog_5_2.md#idp1031368) + + [Example Changes](changelog_5_2.md#idp1003200) + + [Miscellaneous Bug Fixes](changelog_5_2.md#idp1034280) + + [Deprecated Features](changelog_5_2.md#idp1035816) + + [Known Bugs](changelog_5_2.md#idp1037736) + + [11. Upgrading Berkeley DB 11.2.5.0 applications to Berkeley DB 11.2.5.1](upgrade_51_toc.md) + + [Introduction](upgrade_51_toc.md#upgrade_51_intro) + + [DPL Applications must be recompiled](upgrade_11gr2_51_dpl_recompile.md) + + [Source Tree Rearranged](upgrade_11gr2_51_src_reorg.md) + + [SQLite Interface Upgrade](upgrade_11gr2_51_sqlite_ver.md) + + [Mod_db4 Support Discontinued](upgrade_11gr2_51_mod_db4_unsupp.md) + + [Berkeley DB Library Version 11.2.5.1 Change Log](changelog_5_1.md) + + [Database or Log File On-Disk Format Changes](changelog_5_1.md#idp1052992) + + [New Features](changelog_5_1.md#idp953176) + + [Database Environment Changes](changelog_5_1.md#idp1045336) + + [Concurrent Data Store Changes](changelog_5_1.md#idp1059760) + + [Access Method Changes](changelog_5_1.md#idp981016) + + [API Changes](changelog_5_1.md#idp1049008) + + [SQL-Specific API Changes](changelog_5_1.md#idp1055592) + + [Tcl-Specific API Changes](changelog_5_1.md#idp1056952) + + [Java-Specific API Changes](changelog_5_1.md#idp1052280) + + [C#-Specific API Changes](changelog_5_1.md#idp987592) + + [Direct Persistence Layer (DPL), Bindings and Collections API](changelog_5_1.md#idp1060648) + + [Replication Changes](changelog_5_1.md#idp1070000) + + [Locking Subsystem Changes](changelog_5_1.md#idp1080936) + + [Logging Subsystem Changes](changelog_5_1.md#idp1092608) + + [Memory Pool Subsystem Changes](changelog_5_1.md#idp1076376) + + [Mutex Subsystem Changes](changelog_5_1.md#idp1080752) + + [Transaction Subsystem Changes](changelog_5_1.md#idp1089584) + + [Test Suite Changes](changelog_5_1.md#idp1067160) + + [Utility Changes](changelog_5_1.md#idp1088000) + + [Configuration, Documentation, Sample Apps, Portability, and Build Changes](changelog_5_1.md#idp1091312) + + [Example Changes](changelog_5_1.md#idp1081576) + + [Miscellaneous Bug Fixes](changelog_5_1.md#idp1102152) + + [Deprecated Features](changelog_5_1.md#idp1100024) + + [Known Bugs](changelog_5_1.md#idp1100672) + + [12. Upgrading Berkeley DB 4.8 applications to Berkeley DB 11.2.5.0](upgrade_11gr2_toc.md) + + [Introduction](upgrade_11gr2_toc.md#upgrade_11gr2_intro) + + [db_sql Renamed to db_sql_codegen](upgrade_11gr2_dbsqlcodegen.md) + + [DB_REP_CONF_NOAUTOINIT Replaced](upgrade_11gr2_autoinit.md) + + [Support for Multiple Client-to-Client Peers](upgrade_11gr2_repmgr.md) + + [Cryptography Support](build_unix_encrypt.md) + + [DB_NOSYNC Flag to Flush Files](build_unix_db_nosync.md) + + [Dropped Support](upgrade_11gr2_remsupp.md) + + [Changing Stack Size](build_unix_stacksize.md) + + [Berkeley DB 11g Release 2 Change Log](changelog_5_0.md) + + [Changes between 11.2.5.0.26 and 11.2.5.0.32](changelog_5_0.md#idp1125968) + + [Changes between 11.2.5.0.21 and 11.2.5.0.26](changelog_5_0.md#idp1126872) + + [Changes between 4.8 and 11.2.5.0.21](changelog_5_0.md#idp1125192) + + [Known Bugs](changelog_5_0.md#idp1131672) + + [13. Upgrading Berkeley DB 4.7 applications to Berkeley DB 4.8](upgrade_4_8_toc.md) + + [Introduction](upgrade_4_8_toc.md#upgrade_4_8_intro) + + [Registering DPL Secondary Keys](upgrade_4_8_dpl.md) + + [Minor Change in Behavior of DB_MPOOLFILE-\>get](upgrade_4_8_mpool.md) + + [Dropped Support for fcntl System Calls](upgrade_4_8_fcntl.md) + + [Upgrade Requirements](upgrade_4_8_disk.md) + + [Berkeley DB 4.8.28 Change Log](changelog_4_8.md) + + [Changes between 4.8.26 and 4.8.28:](changelog_4_8.md#idp1162104) + + [Known bugs in 4.8](changelog_4_8.md#idp1184264) + + [Changes between 4.8.24 and 4.8.26:](changelog_4_8.md#idp1139288) + + [Changes between 4.8.21 and 4.8.24:](changelog_4_8.md#idp1091200) + + [Changes between 4.7 and 4.8.21:](changelog_4_8.md#idp1199520) + + [Database or Log File On-Disk Format Changes:](changelog_4_8.md#idp1200208) + + [New Features:](changelog_4_8.md#idp981712) + + [Database Environment Changes:](changelog_4_8.md#idp1130224) + + [Concurrent Data Store Changes:](changelog_4_8.md#idp1209320) + + [General Access Method Changes:](changelog_4_8.md#idp1209720) + + [Btree Access Method Changes:](changelog_4_8.md#idp1218064) + + [Hash Access Method Changes:](changelog_4_8.md#idp1215560) + + [Queue Access Method Changes:](changelog_4_8.md#idp1226120) + + [Recno Access Method Changes:](changelog_4_8.md#idp1163928) + + [C-specific API Changes:](changelog_4_8.md#idp1138904) + + [C++-specific API Changes:](changelog_4_8.md#idp1218344) + + [Java-specific API Changes:](changelog_4_8.md#idp1238856) + + [Direct Persistence Layer (DPL), Bindings and Collections API:](changelog_4_8.md#idp1232112) + + [Tcl-specific API Changes:](changelog_4_8.md#idp1232384) + + [RPC-specific Client/Server Changes:](changelog_4_8.md#idp1244368) + + [Replication Changes:](changelog_4_8.md#idp1245896) + + [XA Resource Manager Changes:](changelog_4_8.md#idp1242240) + + [Locking Subsystem Changes:](changelog_4_8.md#idp1247728) + + [Logging Subsystem Changes:](changelog_4_8.md#idp1241128) + + [Memory Pool Subsystem Changes:](changelog_4_8.md#idp1258328) + + [Mutex Subsystem Changes:](changelog_4_8.md#idp1258720) + + [Test Suite Changes](changelog_4_8.md#idp1240832) + + [Transaction Subsystem Changes:](changelog_4_8.md#idp1249776) + + [Utility Changes:](changelog_4_8.md#idp1271664) + + [Configuration, Documentation, Sample Application, Portability and Build Changes:](changelog_4_8.md#idp1274104) + + [14. Test Suite](test.md) + + [Running the test suite](test.md#test_run) + + [Running SQL Test Suite on Unix](test.md#idp1298736) + + [Running SQL Test Suite on Windows](test.md#idp1289848) + + [Test suite FAQ](test_faq.md) diff --git a/docs-src/guides/installation/install.md b/docs-src/guides/installation/install.md new file mode 100644 index 000000000..43f7d9ef5 --- /dev/null +++ b/docs-src/guides/installation/install.md @@ -0,0 +1,356 @@ +--- +title: "Chapter 2.  System Installation Notes" +api-name: "Chapter 2.  System Installation Notes" +source: docs/installation/install.html +--- +## Chapter 2.  System Installation Notes + +**Table of Contents** + + [File utility /etc/magic information](install.md#install_file) + + [Magic information](install.md#magic) + + [Big-endian magic information](install.md#big-endian) + + [Little-endian magic information](install.md#little-endian) + + [Building with multiple versions of Berkeley DB](install_multiple.md) + +## File utility /etc/magic information + + [Magic information](install.md#magic) + + [Big-endian magic information](install.md#big-endian) + + [Little-endian magic information](install.md#little-endian) + +The `file`(1) utility is a UNIX utility that examines and classifies files, based on information found in its database of file types, the /etc/magic file. The following information may be added to your system's /etc/magic file to enable `file`(1) to correctly identify Berkeley DB database files. + +The `file`(1) utility `magic`(5) information for the standard System V UNIX implementation of the `file`(1) utility is included in the Berkeley DB distribution for both big-endian (for example, Sparc) and little-endian (for example, x86) architectures. See Big-endian magic information and Little-endian magic information respectively for this information. + +The `file`(1) utility `magic`(5) information for Release 3.X of Ian Darwin's implementation of the file utility (as distributed by FreeBSD and most Linux distributions) is included in the Berkeley DB distribution. This `magic.txt` information is correct for both big-endian and little-endian architectures. See the next section for this information. + +### Magic information + +``` c +# Berkeley DB + +# Ian Darwin's file /etc/magic files: big/little-endian version. + +# Hash 1.85/1.86 databases store metadata in network byte order. +# Btree 1.85/1.86 databases store the metadata in host byte order. +# Hash and Btree 2.X and later databases store the metadata in +# host byte order. + +0 long 0x00061561 Berkeley DB +>8 belong 4321 +>>4 belong >2 1.86 +>>4 belong <3 1.85 +>>4 belong >0 (Hash, version %d, native byte-order) +>8 belong 1234 +>>4 belong >2 1.86 +>>4 belong <3 1.85 +>>4 belong >0 (Hash, version %d, little-endian) + +0 belong 0x00061561 Berkeley DB +>8 belong 4321 +>>4 belong >2 1.86 +>>4 belong <3 1.85 +>>4 belong >0 (Hash, version %d, big-endian) +>8 belong 1234 +>>4 belong >2 1.86 +>>4 belong <3 1.85 +>>4 belong >0 (Hash, version %d, native byte-order) + +0 long 0x00053162 Berkeley DB 1.85/1.86 +>4 long >0 (Btree, version %d, native byte-order) +0 belong 0x00053162 Berkeley DB 1.85/1.86 +>4 belong >0 (Btree, version %d, big-endian) +0 lelong 0x00053162 Berkeley DB 1.85/1.86 +>4 lelong >0 (Btree, version %d, little-endian) + +12 long 0x00061561 Berkeley DB +>16 long >0 (Hash, version %d, native byte-order) +12 belong 0x00061561 Berkeley DB +>16 belong >0 (Hash, version %d, big-endian) +12 lelong 0x00061561 Berkeley DB +>16 lelong >0 (Hash, version %d, little-endian) + +12 long 0x00053162 Berkeley DB +>16 long >0 (Btree, version %d, native byte-order) +12 belong 0x00053162 Berkeley DB +>16 belong >0 (Btree, version %d, big-endian) +12 lelong 0x00053162 Berkeley DB +>16 lelong >0 (Btree, version %d, little-endian) + +12 long 0x00042253 Berkeley DB +>16 long >0 (Queue, version %d, native byte-order) +12 belong 0x00042253 Berkeley DB +>16 belong >0 (Queue, version %d, big-endian) +12 lelong 0x00042253 Berkeley DB +>16 lelong >0 (Queue, version %d, little-endian) + +12 long 0x00040988 Berkeley DB +>16 long >0 (Log, version %d, native byte-order) +12 belong 0x00040988 Berkeley DB +>16 belong >0 (Log, version %d, big-endian) +12 lelong 0x00040988 Berkeley DB +>16 lelong >0 (Log, version %d, little-endian) +``` + +### Big-endian magic information + +``` c +# Berkeley DB + +# System V /etc/magic files: big-endian version. + +# Hash 1.85/1.86 databases store metadata in network byte order. +# Btree 1.85/1.86 databases store the metadata in host byte order. +# Hash and Btree 2.X and later databases store the metadata in +# host byte order. + +0 long 0x00053162 Berkeley DB 1.85/1.86 (Btree, +>4 long 0x00000002 version 2, +>4 long 0x00000003 version 3, +>0 long 0x00053162 native byte-order) + +0 long 0x62310500 Berkeley DB 1.85/1.86 (Btree, +>4 long 0x02000000 version 2, +>4 long 0x03000000 version 3, +>0 long 0x62310500 little-endian) + +12 long 0x00053162 Berkeley DB (Btree, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>12 long 0x00053162 native byte-order) + +12 long 0x62310500 Berkeley DB (Btree, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>12 long 0x62310500 little-endian) + +0 long 0x00061561 Berkeley DB +>4 long >2 1.86 +>4 long <3 1.85 +>0 long 0x00061561 (Hash, +>4 long 2 version 2, +>4 long 3 version 3, +>8 long 0x000004D2 little-endian) +>8 long 0x000010E1 native byte-order) + +12 long 0x00061561 Berkeley DB (Hash, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>12 long 0x00061561 native byte-order) + +12 long 0x61150600 Berkeley DB (Hash, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>12 long 0x61150600 little-endian) + +12 long 0x00042253 Berkeley DB (Queue, +>16 long 0x00000001 version 1, +>16 long 0x00000002 version 2, +>16 long 0x00000003 version 3, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>12 long 0x00042253 native byte-order) + +12 long 0x53220400 Berkeley DB (Queue, +>16 long 0x01000000 version 1, +>16 long 0x02000000 version 2, +>16 long 0x03000000 version 3, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>12 long 0x53220400 little-endian) + +12 long 0x00040988 Berkeley DB (Log, +>16 long 0x00000001 version 1, +>16 long 0x00000002 version 2, +>16 long 0x00000003 version 3, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>16 long 0x0000000a version 10, +>16 long 0x0000000b version 11, +>16 long 0x0000000c version 12, +>16 long 0x0000000d version 13, +>16 long 0x0000000e version 14, +>16 long 0x0000000f version 15, +>12 long 0x00040988 native byte-order) + +12 long 0x88090400 Berkeley DB (Log, +>16 long 0x01000000 version 1, +>16 long 0x02000000 version 2, +>16 long 0x03000000 version 3, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>16 long 0x0a000000 version 10, +>16 long 0x0b000000 version 11, +>16 long 0x0c000000 version 12, +>16 long 0x0d000000 version 13, +>16 long 0x0e000000 version 14, +>16 long 0x0f000000 version 15, +>12 long 0x88090400 little-endian) +``` + +### Little-endian magic information + +``` c +# Berkeley DB + +# System V /etc/magic files: little-endian version. + +# Hash 1.85/1.86 databases store metadata in network byte order. +# Btree 1.85/1.86 databases store the metadata in host byte order. +# Hash and Btree 2.X and later databases store the metadata in +# host byte order. + +0 long 0x00053162 Berkeley DB 1.85/1.86 (Btree, +>4 long 0x00000002 version 2, +>4 long 0x00000003 version 3, +>0 long 0x00053162 native byte-order) + +0 long 0x62310500 Berkeley DB 1.85/1.86 (Btree, +>4 long 0x02000000 version 2, +>4 long 0x03000000 version 3, +>0 long 0x62310500 big-endian) + +12 long 0x00053162 Berkeley DB (Btree, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>12 long 0x00053162 native byte-order) + +12 long 0x62310500 Berkeley DB (Btree, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>12 long 0x62310500 big-endian) + +0 long 0x61150600 Berkeley DB +>4 long >0x02000000 1.86 +>4 long <0x03000000 1.85 +>0 long 0x00061561 (Hash, +>4 long 0x02000000 version 2, +>4 long 0x03000000 version 3, +>8 long 0xD2040000 native byte-order) +>8 long 0xE1100000 big-endian) + +12 long 0x00061561 Berkeley DB (Hash, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>12 long 0x00061561 native byte-order) + +12 long 0x61150600 Berkeley DB (Hash, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>12 long 0x61150600 big-endian) + +12 long 0x00042253 Berkeley DB (Queue, +>16 long 0x00000001 version 1, +>16 long 0x00000002 version 2, +>16 long 0x00000003 version 3, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>12 long 0x00042253 native byte-order) + +12 long 0x53220400 Berkeley DB (Queue, +>16 long 0x01000000 version 1, +>16 long 0x02000000 version 2, +>16 long 0x03000000 version 3, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>12 long 0x53220400 big-endian) + +12 long 0x00040988 Berkeley DB (Log, +>16 long 0x00000001 version 1, +>16 long 0x00000002 version 2, +>16 long 0x00000003 version 3, +>16 long 0x00000004 version 4, +>16 long 0x00000005 version 5, +>16 long 0x00000006 version 6, +>16 long 0x00000007 version 7, +>16 long 0x00000008 version 8, +>16 long 0x00000009 version 9, +>16 long 0x0000000a version 10, +>16 long 0x0000000b version 11, +>16 long 0x0000000c version 12, +>16 long 0x0000000d version 13, +>16 long 0x0000000e version 14, +>16 long 0x0000000f version 15, +>12 long 0x00040988 native byte-order) + +12 long 0x88090400 Berkeley DB (Log, +>16 long 0x01000000 version 1, +>16 long 0x02000000 version 2, +>16 long 0x03000000 version 3, +>16 long 0x04000000 version 4, +>16 long 0x05000000 version 5, +>16 long 0x06000000 version 6, +>16 long 0x07000000 version 7, +>16 long 0x08000000 version 8, +>16 long 0x09000000 version 9, +>16 long 0x0a000000 version 10, +>16 long 0x0b000000 version 11, +>16 long 0x0c000000 version 12, +>16 long 0x0d000000 version 13, +>16 long 0x0e000000 version 14, +>16 long 0x0f000000 version 15, +>12 long 0x88090400 big-endian) +``` diff --git a/docs-src/guides/installation/install_multiple.md b/docs-src/guides/installation/install_multiple.md new file mode 100644 index 000000000..a00444037 --- /dev/null +++ b/docs-src/guides/installation/install_multiple.md @@ -0,0 +1,14 @@ +--- +title: "Building with multiple versions of Berkeley DB" +api-name: "Building with multiple versions of Berkeley DB" +source: docs/installation/install_multiple.html +--- +## Building with multiple versions of Berkeley DB + +In some cases it may be necessary to build applications which include multiple versions of Berkeley DB. Examples include applications which include software from other vendors, or applications running on a system where the system C library itself uses Berkeley DB. In such cases, the two versions of Berkeley DB may be incompatible, that is, they may have different external and internal interfaces, and may even have different underlying database formats. + +To create a Berkeley DB library whose symbols won't collide with other Berkeley DB libraries (or other application or library modules, for that matter), configure Berkeley DB using the --with-uniquename=NAME configuration option, and then build Berkeley DB as usual. (Note that --with-uniquename=NAME only affects the Berkeley DB C language library build; loading multiple versions of the C++ or Java APIs will require additional work.) The modified symbol names are hidden from the application in the Berkeley DB header files, that is, there is no need for the application to be aware that it is using a special library build as long as it includes the appropriate Berkeley DB header file. + +If "NAME" is not specified when configuring with --with-uniquename=NAME, a default value built from the major and minor numbers of the Berkeley DB release will be used. It is rarely necessary to specify NAME; using the major and minor release numbers will ensure that only one copy of the library will be loaded into the application unless two distinct versions really are necessary. + +When distributing any library software that uses Berkeley DB, or any software which will be recompiled by users for their systems, we recommend two things: First, include the Berkeley DB release as part of your release. This will insulate your software from potential Berkeley DB API changes as well as simplifying your coding because you will only have to code to a single version of the Berkeley DB API instead of adapting at compile time to whatever version of Berkeley DB happens to be installed on the target system. Second, use --with-uniquename=NAME when configuring Berkeley DB, because that will insure that you do not unexpectedly collide with other application code or a library already installed on the target system. diff --git a/docs-src/guides/installation/introduction.md b/docs-src/guides/installation/introduction.md new file mode 100644 index 000000000..33d4c4059 --- /dev/null +++ b/docs-src/guides/installation/introduction.md @@ -0,0 +1,30 @@ +--- +title: "Chapter 1. Introduction" +api-name: "Chapter 1. Introduction" +source: docs/installation/introduction.html +--- +## Chapter 1. Introduction + +**Table of Contents** + + [Installation Overview](introduction.md#install-overview) + + [Supported Platforms](ch01s02.md) + +Welcome to the Berkeley DB. This manual describes how to configure, build and install Berkeley DB. Installation of DB for all of the platforms it officially supports is described in this manual. Upgrade instructions and release notes for newer versions of this product are described here. For infomation on upgrading from historical versions, see the Berkeley DB Upgrade Guide. + +Note that some operating systems and distributions might provide DB, either by default or as part of an installation option. If so, those platforms will have installation instructions for DB specific to them. In this situation, you should see the documentation for your operating system or distribution provider for information on how to get DB on your platform. + +## Installation Overview + +Berkeley DB is an open-source product, and as such it is usually offered in source-code format. This means that placing DB on your platform requires you to configure the build scripts, compile it, and then install the product onto your host system. The exception to this are Microsoft Windows platforms for which a binary installer is available. Note that for Windows platforms, you can still compile the product from source if you desire. + +For \*nix systems, including the BSD and Linux systems, the usual `configure`, `make` and `make install` installation process is used to place DB on your platform. + +For information on building and installing Berkeley DB on: + +- Microsoft Windows, see Building Berkeley DB for Windows or Building Berkeley DB for Windows Mobile . + +- Unix/POSIX — including Linux, BSD, Apple iOS (known as iPhone OS previously), and Mac OS X — see Building Berkeley DB for UNIX/POSIX . + +- VxWorks, see Building Berkeley DB for VxWorks . diff --git a/docs-src/guides/installation/moreinfo.md b/docs-src/guides/installation/moreinfo.md new file mode 100644 index 000000000..916b05586 --- /dev/null +++ b/docs-src/guides/installation/moreinfo.md @@ -0,0 +1,38 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/installation/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Getting Started with Transaction Processing for C + +- Berkeley DB Getting Started with Replicated Applications for C + +- Berkeley DB C API Reference Guide + +- Berkeley DB C++ API Reference Guide + +- Berkeley DB STL API Reference Guide + +- Berkeley DB TCL API Reference Guide + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Upgrade Guide + +- Berkeley DB Getting Started with the SQL APIs + +To download the latest documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/installation/preface.md b/docs-src/guides/installation/preface.md new file mode 100644 index 000000000..b4c66b9c1 --- /dev/null +++ b/docs-src/guides/installation/preface.md @@ -0,0 +1,30 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/installation/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +Welcome to Berkeley DB (DB). This document describes how to build, install and upgrade Berkeley DB + +This document reflects Berkeley DB 11*g* Release 2, which provides DB library version 11.2.5.3. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/guides/installation/test.md b/docs-src/guides/installation/test.md new file mode 100644 index 000000000..753f4bd52 --- /dev/null +++ b/docs-src/guides/installation/test.md @@ -0,0 +1,98 @@ +--- +title: "Chapter 14.  Test Suite" +api-name: "Chapter 14.  Test Suite" +source: docs/installation/test.html +--- +## Chapter 14.  Test Suite + +**Table of Contents** + + [Running the test suite](test.md#test_run) + + [Running SQL Test Suite on Unix](test.md#idp1298736) + + [Running SQL Test Suite on Windows](test.md#idp1289848) + + [Test suite FAQ](test_faq.md) + +## Running the test suite + + [Running SQL Test Suite on Unix](test.md#idp1298736) + + [Running SQL Test Suite on Windows](test.md#idp1289848) + +Once you have started tclsh and have loaded the test.tcl source file (see Running the test suite under UNIX and Running the test suite under Windows for more information), you are ready to run the test suite. At the tclsh prompt, to run the standard test suite, enter the following: + +``` c +% run_std +``` + +A more exhaustive version of the test suite runs all the tests several more times, testing encryption, replication, and different page sizes. After you have a clean run for run_std, you may choose to run this lengthier set of tests. At the tclsh prompt, enter: + +``` c +% run_all +``` + +Running the standard tests can take from several hours to a few days to complete, depending on your hardware, and running all the tests will take at least twice as long. For this reason, the output from these commands are redirected to a file in the current directory named `ALL.OUT`. Periodically, a line will be written to the standard output, indicating what test is being run. When the test suite has finished, a final message will be written indicating the test suite has completed successfully or that it has failed. If the run failed, you should review the `ALL.OUT` file to determine which tests failed. Errors will appear in that file as output lines, beginning with the string "FAIL". + +Tests are run in the directory `TESTDIR`, by default. However, the test files are often large, and you should use a filesystem with at least several hundred megabytes of free space. To use a different directory for the test directory, edit the file include.tcl in your build directory, and change the following line to a more appropriate value for your system: + +``` c +set testdir ./TESTDIR +``` + +For example, you might change it to the following: + +``` c +set testdir /var/tmp/db.test +``` + +Alternatively, you can create a symbolic link named TESTDIR in your build directory to an appropriate location for running the tests. Regardless of where you run the tests, the TESTDIR directory should be on a local filesystem. Using a remote filesystem (for example, an NFS mounted filesystem) will almost certainly cause spurious test failures. + +### Running SQL Test Suite on Unix + +Once the test suite is built (see Building SQL Test Suite on Unix for more information), run the entire test suite by executing the following command in the `../build_unix/sql` directory: + +``` c +sh ../../test/sql/bdb-test.sh +``` + +This runs a set of tests and lists the errors each test encountered, if any. A detailed list of the test results is written to `test.log`. + +To run an individual test, such as insert.test, execute the following command in the `../build_unix/sql` directory: + +``` c +./testfixture ../../lang/sql/sqlite/test/insert.test +``` + +### Running SQL Test Suite on Windows + +After the test suite is built (see Building the software needed by the SQL tests for more information) and before running the entire test suite, go to `../sql/adapter/bdb-test.sh` and edit the line: + +``` c +echo $t: `alarm $TIMEOUT ./testfixture.exe +$tpath 2>&1 | tee -a test.log | grep "errors out of" +|| echo "failed"` +``` + +to + +``` c +echo $t: `alarm $TIMEOUT Win32/Debug/testfixture.exe +$tpath 2>&1 | tee -a test.log | grep "errors out of" +|| echo "failed"` +``` + +Running the test suite requires an Unix emulator, such as Cygwin. In a Cygwin window go to the `../build_windows` directory and execute the command: + +``` c +sh ../sql/adapter/bdb-test.sh +``` + +This runs a set of tests and lists errors that each test encountered, if any. A detailed list of the test results is written to `test.log`. + +To run an individual test, such as insert.test, execute the following command in the `../build_windows` directory: + +``` c +Win32/Debug/testfixture.exe ../sql/sqlite/test/insert.test +``` diff --git a/docs-src/guides/installation/test_faq.md b/docs-src/guides/installation/test_faq.md new file mode 100644 index 000000000..456683cd0 --- /dev/null +++ b/docs-src/guides/installation/test_faq.md @@ -0,0 +1,10 @@ +--- +title: "Test suite FAQ" +api-name: "Test suite FAQ" +source: docs/installation/test_faq.html +--- +## Test suite FAQ + +1. **The test suite has been running for over a day. What's wrong?** + + The test suite can take anywhere from some number of hours to several days to run, depending on your hardware configuration. As long as the run is making forward progress and new lines are being written to the `ALL.OUT` files, everything is probably fine. diff --git a/docs-src/guides/installation/upgrade_11gr2_51_dpl_recompile.md b/docs-src/guides/installation/upgrade_11gr2_51_dpl_recompile.md new file mode 100644 index 000000000..7821d5113 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_51_dpl_recompile.md @@ -0,0 +1,8 @@ +--- +title: "DPL Applications must be recompiled" +api-name: "DPL Applications must be recompiled" +source: docs/installation/upgrade_11gr2_51_dpl_recompile.html +--- +## DPL Applications must be recompiled + +Applications that use the Java interface's *Direct Persistence Layer* must be recompiled, due to a change in the return type of the setter methods in StoreConfig and EvolveConfig classes. The setter methods now return `this` instead of `void`. diff --git a/docs-src/guides/installation/upgrade_11gr2_51_mod_db4_unsupp.md b/docs-src/guides/installation/upgrade_11gr2_51_mod_db4_unsupp.md new file mode 100644 index 000000000..fa4fb782d --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_51_mod_db4_unsupp.md @@ -0,0 +1,8 @@ +--- +title: "Mod_db4 Support Discontinued" +api-name: "Mod_db4 Support Discontinued" +source: docs/installation/upgrade_11gr2_51_mod_db4_unsupp.html +--- +## Mod_db4 Support Discontinued + +The mod_db4 apache module is no longer included in the release. diff --git a/docs-src/guides/installation/upgrade_11gr2_51_sqlite_ver.md b/docs-src/guides/installation/upgrade_11gr2_51_sqlite_ver.md new file mode 100644 index 000000000..37f3c4a3d --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_51_sqlite_ver.md @@ -0,0 +1,8 @@ +--- +title: "SQLite Interface Upgrade" +api-name: "SQLite Interface Upgrade" +source: docs/installation/upgrade_11gr2_51_sqlite_ver.html +--- +## SQLite Interface Upgrade + +Berkeley DB's SQL interface includes code from SQLite. The version of SQLite used has been upgraded, so DB SQL is compatible with SQLite version 3.7.0.1. Please see the release notes at http://sqlite.org/changes.html for further information. diff --git a/docs-src/guides/installation/upgrade_11gr2_51_src_reorg.md b/docs-src/guides/installation/upgrade_11gr2_51_src_reorg.md new file mode 100644 index 000000000..899882619 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_51_src_reorg.md @@ -0,0 +1,8 @@ +--- +title: "Source Tree Rearranged" +api-name: "Source Tree Rearranged" +source: docs/installation/upgrade_11gr2_51_src_reorg.html +--- +## Source Tree Rearranged + +The source code hierarchy has been reorganized. Source files that belong to Berkeley DB core are now in a top-level directory named `src`. Files related to language interfaces are in `lang`, and all examples are collected under `examples`. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_bit_cmp_win.md b/docs-src/guides/installation/upgrade_11gr2_52_bit_cmp_win.md new file mode 100644 index 000000000..10195205c --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_bit_cmp_win.md @@ -0,0 +1,8 @@ +--- +title: "32bit/64bit Compatibility on Windows" +api-name: "32bit/64bit Compatibility on Windows" +source: docs/installation/upgrade_11gr2_52_bit_cmp_win.html +--- +## 32bit/64bit Compatibility on Windows + +Berkeley DB can now be compiled on Windows so that 32 bit and 64 bit applications can concurrently access a BDB environment. To enable this feature, build both the 32 bit BDB library and application and the 64 bit library and application with the flag `/D HAVE_MIXED_SIZE_ADDRESSING`. Note that private environments are disabled under the compatibility mode. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_dyn_env.md b/docs-src/guides/installation/upgrade_11gr2_52_dyn_env.md new file mode 100644 index 000000000..3a90bdfc8 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_dyn_env.md @@ -0,0 +1,38 @@ +--- +title: "Dynamic Environment Configuration" +api-name: "Dynamic Environment Configuration" +source: docs/installation/upgrade_11gr2_52_dyn_env.html +--- +## Dynamic Environment Configuration + + [New Functions](upgrade_11gr2_52_dyn_env.md#idp902144) + + [Deprecated Functions](upgrade_11gr2_52_dyn_env.md#idp912000) + +Memory is now allocated incrementally as needed, instead of all at once during environment initialization, for structures that support locks, transactions, threads, and mutexes. With this change new functions have been added that configure how much memory is allocated initially, and how much that memory is allowed to grow. The old memory configuration functions have been deprecated. + +### New Functions + +- DB_ENV->set_memory_init() +- DB_ENV->get_memory_init() +- DB_ENV->set_memory_max() +- DB_ENV->get_memory_max() +- DB_ENV->set_lk_tablesize() +- DB_ENV->get_lk_tablesize() +- DB_ENV->mutex_set_init() +- DB_ENV->mutex_get_init() + +### Deprecated Functions + +- DB_ENV->mutex_set_max() +- DB_ENV->mutex_get_max() +- DB_ENV->set_lk_max_lockers() +- DB_ENV->get_lk_max_lockers() +- DB_ENV->set_lk_max_locks() +- DB_ENV->get_lk_max_locks() +- DB_ENV->set_lk_max_objects() +- DB_ENV->get_lk_max_objects() +- DB_ENV->set_thread_count() +- DB_ENV->get_thread_count() +- DB_ENV->set_tx_max() +- DB_ENV->get_tx_max() diff --git a/docs-src/guides/installation/upgrade_11gr2_52_excl_txn_sql.md b/docs-src/guides/installation/upgrade_11gr2_52_excl_txn_sql.md new file mode 100644 index 000000000..4881d6bdd --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_excl_txn_sql.md @@ -0,0 +1,8 @@ +--- +title: "Exclusive Transactions in the SQL Layer" +api-name: "Exclusive Transactions in the SQL Layer" +source: docs/installation/upgrade_11gr2_52_excl_txn_sql.html +--- +## Exclusive Transactions in the SQL Layer + +Issuing the SQL command `BEGIN TRANSACTION EXCLUSIVE` will now cause any other transactions accessing the database to block, or return a `SQLITE_BUSY` or `SQLITE_LOCK` error, until the exclusive transaction is committed or aborted. Previously, non-exclusive transactions could execute concurrently with an exclusive transaction. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_grp_mbr.md b/docs-src/guides/installation/upgrade_11gr2_52_grp_mbr.md new file mode 100644 index 000000000..0c8ce461d --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_grp_mbr.md @@ -0,0 +1,82 @@ +--- +title: "Group Membership in Repmgr" +api-name: "Group Membership in Repmgr" +source: docs/installation/upgrade_11gr2_52_grp_mbr.html +--- +## Group Membership in Repmgr + + [Upgrading](upgrade_11gr2_52_grp_mbr.md#idp929720) + + [New Functions](upgrade_11gr2_52_grp_mbr.md#idp910056) + + [Modified Functions](upgrade_11gr2_52_grp_mbr.md#idp901088) + + [New Events](upgrade_11gr2_52_grp_mbr.md#idp924520) + + [Removed Functions](upgrade_11gr2_52_grp_mbr.md#idp937928) + + [New Parameters](upgrade_11gr2_52_grp_mbr.md#idp909344) + + [New Structure](upgrade_11gr2_52_grp_mbr.md#idp924776) + +Replication Manager now manages group membership much more closely, making it much easier for applications to add and remove sites from a replication group without risk of transaction loss. In order to accomplish this, the API for configuring group membership has changed significantly. The `repmgr_set_local_site()` and `repmgr_add_remote_site()` methods no longer exist; they are replaced by a new handle type, `DB_SITE`. The `repmgr_get_local_site()` method has been replaced by DB_ENV->repmgr_site(), which now returns a `DB_SITE` handle instead of a raw host/port network address. + +Replication Manager applications may no longer call the DB_ENV->rep_set_nsites() method, because the Replication Manager now tracks the number of sites in the replication group for you. Replication Manager applications may still call DB_ENV->rep_get_nsites(), but only after a successful call to DB_ENV->repmgr_start(). + +For applications using the replication Base API there is no change, except that they may now call DB_ENV->rep_set_nsites() to change the group size even when Master Leases are in use. + +The new Replication Manager group membership functionality is described in the Managing Replication Manager Group Membership chapter in the *Berkeley DB Programmer's Reference Guide*. + +Replication Manager no longer prints an error message on a connection failure. Instead it generates an event with the equivalent information (invoking the application's event-handling call-back function). + +### Upgrading + +An existing application running a previous version of BDB can do a "live upgrade" so that only one site at a time has to be shut down. To do this, restart each site in the group, with the old master being shutdown last. When each site is restarted, use `DB_SITE` to configure the local site with the flag `DB_LEGACY`, and create a `DB_SITE` handle with a full specification of all the remote site addresses for all other sites currently in the group, and configure each handle with the `DB_LEGACY` flag. When the old master is restarted and a new master has been established, the new master is ready to manage membership changes, and new sites can be added as usual. But the application must not try to add new sites, or remove existing sites, during the mixed-version transitional phase. + +To do a non-live upgrade shutdown the entire replication group. Then restart the group with each site configured with the `DB_LEGACY` flag, and in `DB_REP_ELECTION` mode. + +### New Functions + +- DB_ENV->repmgr_site() +- DB_ENV->repmgr_site_by_eid() +- DB_SITE->set_config() +- DB_SITE->get_config() +- DB_SITE->remove() +- DB_SITE->get_eid() +- DB_SITE->get_address() +- DB_SITE->close() + +### Modified Functions + +- DB_ENV->rep_set_nsites() is no longer used by the Replication Manager, but is still used by the Base API. It can now be used to change the number of sites dynamically, even when master leases are in use. + +### New Events + +- `DB_EVENT_REP_SITE_ADDED` +- `DB_EVENT_REP_SITE_REMOVED` +- `DB_EVENT_REP_LOCAL_SITE_REMOVED` +- `DB_EVENT_REP_CONNECT_BROKEN` +- `DB_EVENT_REP_CONNECT_ESTD` +- `DB_EVENT_REP_CONNECT_TRY_FAILED` +- `DB_EVENT_REP_INIT_DONE` + +### Removed Functions + +- `DB_ENV->repmgr_set_local_site()` +- `DB_ENV->repmgr_add_local_site()` +- `DB_ENV->repmgr_add_remote_site()` +- `DB_ENV->repmgr_get_local_site()` + +### New Parameters + +The following new parameters are passed to DB_SITE->set_config(). + +- `DB_BOOTSTRAP_HELPER` +- `DB_GROUP_CREATOR` +- `DB_LEGACY` +- `DB_LOCAL_SITE` +- `DB_REPMGR_PEER` + +### New Structure + +- `DB_REPMGR_CONN_ERR` encapsulates an EID and an integer system error code. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_heap.md b/docs-src/guides/installation/upgrade_11gr2_52_heap.md new file mode 100644 index 000000000..baab55a96 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_heap.md @@ -0,0 +1,28 @@ +--- +title: "Heap Access Method" +api-name: "Heap Access Method" +source: docs/installation/upgrade_11gr2_52_heap.html +--- +## Heap Access Method + + [New Functions](upgrade_11gr2_52_heap.md#idp936848) + + [Modified Functions](upgrade_11gr2_52_heap.md#idp930424) + + [New Definition](upgrade_11gr2_52_heap.md#idp931776) + +Databases can now be configured as heaps by passing the access type `DB_HEAP` to DB->open(). Heap size can be configured with DB->set_heapsize(), and DB->stat() now returns heap statistics in the structure `DB_HEAP_STAT` when applied to a heap database. + +### New Functions + +- DB->set_heapsize() +- DB->get_heapsize() + +### Modified Functions + +- DB->open() now accepts `DB_HEAP` as an access type. +- DB->stat() now returns heap statistics in the structure `DB_HEAP_STAT`. + +### New Definition + +- `DB_HEAP_RID` is the defined heap key value. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_hot_backup.md b/docs-src/guides/installation/upgrade_11gr2_52_hot_backup.md new file mode 100644 index 000000000..44abd565f --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_hot_backup.md @@ -0,0 +1,8 @@ +--- +title: "Hot Backup Changes" +api-name: "Hot Backup Changes" +source: docs/installation/upgrade_11gr2_52_hot_backup.html +--- +## Hot Backup Changes + +Because non-UNIX systems do not support atomic file system reads, the db_hotbackup utility has been modified to read data through the environment. If your application is running on a UNIX based system such as Solaris, HPUX, BSD or Mac OS, you can specify the **-F** flag to read directly from the filesystem. Please refer to Recovery procedures in the *Berkeley DB Programmer's Reference Guide* for more information on safely backing up your databases. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_mvcc_sql.md b/docs-src/guides/installation/upgrade_11gr2_52_mvcc_sql.md new file mode 100644 index 000000000..ce4c0e8ec --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_mvcc_sql.md @@ -0,0 +1,17 @@ +--- +title: "Enabling Transaction Snapshots in the SQL Layer" +api-name: "Enabling Transaction Snapshots in the SQL Layer" +source: docs/installation/upgrade_11gr2_52_mvcc_sql.html +--- +## Enabling Transaction Snapshots in the SQL Layer + + [New Pragmas](upgrade_11gr2_52_mvcc_sql.md#idp951464) + +Read/write concurrency can now be enabled in the SQL API by using `PRAGMA multiversion=on` before accessing any tables in the database. After multiversion has been enabled, it can be temporarily disabled using the `PRAGMA transaction_snapshots=on/off`. + +### New Pragmas + +For more details on pragmas concerning Transaction Snapshots read Using Multiversion Concurrency Control in the *Berkeley DB Getting Started with the SQL APIs* guide. + +- `PRAGMA multiversion=ON|OFF;` +- `PRAGMA snapshot_isolation=ON|OFF` diff --git a/docs-src/guides/installation/upgrade_11gr2_52_rep_2site_strict.md b/docs-src/guides/installation/upgrade_11gr2_52_rep_2site_strict.md new file mode 100644 index 000000000..f54bb8de4 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_rep_2site_strict.md @@ -0,0 +1,8 @@ +--- +title: "2SITE_STRICT Enabled by Default in Replication" +api-name: "2SITE_STRICT Enabled by Default in Replication" +source: docs/installation/upgrade_11gr2_52_rep_2site_strict.html +--- +## 2SITE_STRICT Enabled by Default in Replication + +The 2SITE_STRICT replication configuration parameter is now turned on by default. This configuration parameter is controlled using the DB_REPMGR_CONF_2SITE_STRICT. flag on the DB_ENV->rep_set_config() method. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_rep_dbt_readonly.md b/docs-src/guides/installation/upgrade_11gr2_52_rep_dbt_readonly.md new file mode 100644 index 000000000..234ed1524 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_rep_dbt_readonly.md @@ -0,0 +1,14 @@ +--- +title: "Read Only flag for DBT" +api-name: "Read Only flag for DBT" +source: docs/installation/upgrade_11gr2_52_rep_dbt_readonly.html +--- +## Read Only flag for DBT + + [New Flag](upgrade_11gr2_52_rep_dbt_readonly.md#idp907000) + +A DBT can now be set as read-only, when passed to the DB->get() method, using the flag `DB_DBT_READONLY`. This is useful when using a static string as a key value, because this flag will prevent Berkeley DB from updating the DBT. + +### New Flag + +- `DB_DBT_READONLY` diff --git a/docs-src/guides/installation/upgrade_11gr2_52_rep_sql.md b/docs-src/guides/installation/upgrade_11gr2_52_rep_sql.md new file mode 100644 index 000000000..6e86c1973 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_rep_sql.md @@ -0,0 +1,28 @@ +--- +title: "Enabling Replication in the SQL Layer" +api-name: "Enabling Replication in the SQL Layer" +source: docs/installation/upgrade_11gr2_52_rep_sql.html +--- +## Enabling Replication in the SQL Layer + + [New Pragmas](upgrade_11gr2_52_rep_sql.md#idp962696) + +Replication can now be enabled and configured in the SQL layer using pragmas. The pragmas `replication_local_site`, `replication_initial_master`, and `replication_remote_site` can be used to configure the replication group. Note that when the BDB SQL replicated application is initially started, a specific master site must be explicitly designated. After configuring the replication group, start replication using `PRAGMA replication=ON`. + +To display replication statistics in the dbsql shell, use: + +``` c +dbsql> .stat :rep +``` + +### New Pragmas + +For more details on the replication pragmas see Replication PRAGMAs in the *Berkeley DB Getting Started with the SQL APIs* guide. + +- `PRAGMA replication=ON|OFF` +- `PRAGMA replication_initial_master=ON|OFF` +- `PRAGMA replication_local_site="hostname:port"` +- `PRAGMA replication_remote_site="hostname:port"` +- `PRAGMA replication_remove_site="host:port"` +- `PRAGMA replication_verbose_output=ON|OFF` +- `PRAGMA replication_verbose_file=filename` diff --git a/docs-src/guides/installation/upgrade_11gr2_52_repmgr_channels.md b/docs-src/guides/installation/upgrade_11gr2_52_repmgr_channels.md new file mode 100644 index 000000000..d09ac0c29 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_repmgr_channels.md @@ -0,0 +1,19 @@ +--- +title: "Repmgr Message Channels" +api-name: "Repmgr Message Channels" +source: docs/installation/upgrade_11gr2_52_repmgr_channels.html +--- +## Repmgr Message Channels + + [New Functions](upgrade_11gr2_52_repmgr_channels.md#idp919280) + +Application components running at various sites within a replication group can now use the Replication Manager's existing TCP/IP communications infrastructure to send and process messages among themselves, using the `DB_CHANNEL` handle. DB_ENV->repmgr_channel() is used to create the `DB_CHANNEL` handle. DB_CHANNEL->send_msg() and DB_CHANNEL->send_request() are used to send sychronous and asychronous messages that are handled by the function set by DB_ENV->repmgr_msg_dispatch(). DB_CHANNEL->set_timeout() is used to configure channel time out, and DB_CHANNEL->close() closes the channel and frees resources held by it. + +### New Functions + +- DB_ENV->repmgr_msg_dispatch() +- DB_ENV->repmgr_channel() +- DB_CHANNEL->send_msg() +- DB_CHANNEL->send_request() +- DB_CHANNEL->set_timeout() +- DB_CHANNEL->close() diff --git a/docs-src/guides/installation/upgrade_11gr2_52_seq_sql.md b/docs-src/guides/installation/upgrade_11gr2_52_seq_sql.md new file mode 100644 index 000000000..f2b9d5273 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_seq_sql.md @@ -0,0 +1,41 @@ +--- +title: "Sequence Support in the SQL Layer" +api-name: "Sequence Support in the SQL Layer" +source: docs/installation/upgrade_11gr2_52_seq_sql.html +--- +## Sequence Support in the SQL Layer + + [New Functions](upgrade_11gr2_52_seq_sql.md#idp963480) + +A partial implementation of the sequence API defined in the SQL 2003 specification has been added to the SQL layer. A sequence is created using the syntax: + +``` c +SELECT create_sequence("sequence_name"...) +``` + +The sequence numbers are accessed using + +``` c +SELECT nextval("sequence_name") +``` + +and + +``` c +SELECT currval("sequence_name") +``` + +Finally, a sequence can be dropped using + +``` c +SELECT drop_sequence("sequence_name") +``` + +### New Functions + +The four new functions, which have to be called as part of a `SELECT` statement, are describe in more detail in Using Sequences in the *Berkeley DB Getting Started with the SQL APIs* guide. + +- create_sequence +- seq_nextval +- seq_currval +- seq_drop_sequence. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_sqlite_ver.md b/docs-src/guides/installation/upgrade_11gr2_52_sqlite_ver.md new file mode 100644 index 000000000..cfee0fdad --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_sqlite_ver.md @@ -0,0 +1,8 @@ +--- +title: "SQLite Interface Upgrade" +api-name: "SQLite Interface Upgrade" +source: docs/installation/upgrade_11gr2_52_sqlite_ver.html +--- +## SQLite Interface Upgrade + +Berkeley DB's SQL interface includes code from SQLite. The version of SQLite used has been upgraded, so DB SQL is compatible with SQLite version 3.7.6.2. Please see the release notes at http://sqlite.org/changes.html for further information. diff --git a/docs-src/guides/installation/upgrade_11gr2_52_xa.md b/docs-src/guides/installation/upgrade_11gr2_52_xa.md new file mode 100644 index 000000000..1fe29f2cc --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_52_xa.md @@ -0,0 +1,32 @@ +--- +title: "Berkeley DB X/Open Compliant XA Resource Manager" +api-name: "Berkeley DB X/Open Compliant XA Resource Manager" +source: docs/installation/upgrade_11gr2_52_xa.html +--- +## Berkeley DB X/Open Compliant XA Resource Manager + + [Constraints](upgrade_11gr2_52_xa.md#idp973264) + + [New Flag](upgrade_11gr2_52_xa.md#idp978200) + + [Modified Function](upgrade_11gr2_52_xa.md#idp982256) + +The Berkeley DB X/open compliant XA resource manager has been restored. (It was removed from the product after the 4.7 release.) The new implementation includes support for multi-threaded servers. Consult the documentation of your chosen transaction manager to learn how to implement a multi-threaded server. + +### Constraints + +Applictions that use a BDB XA resource manager must now take into account the following constraints. + +- No in-memory logging. +- No application-level child transactions. +- All database-level operations (open, close, create and the like) must be performed outside of a global transactions (i.e., they can be performed in local BDB transactions, but not while a distributed XA transaction is active). +- Environment configuration must be done using a DB_CONFIG file. +- Cursors must be closed before a service invocation returns. + +### New Flag + +- `DB_XA_CREATE` - This flag is passed to db_create() to create a `DB` handle that supports XA transactions. + +### Modified Function + +- DB->stat() now returns the field `DB_TXN_STAT->DB_TXN_ACTIVE->xa_status`, which contains information on the XA transactions. diff --git a/docs-src/guides/installation/upgrade_11gr2_53_build_windows.md b/docs-src/guides/installation/upgrade_11gr2_53_build_windows.md new file mode 100644 index 000000000..90a0c7050 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_build_windows.md @@ -0,0 +1,8 @@ +--- +title: "Changes to the build_windows Folder" +api-name: "Changes to the build_windows Folder" +source: docs/installation/upgrade_11gr2_53_build_windows.html +--- +## Changes to the build_windows Folder + +Visual Studios projects in the `build_windows` folder have been moved into two new sub folders `build_windows/VS8` and `build_windows/VS10`. Visual Studios 2010 projects have been moved into the folder VS10, and Visual Studios 2008 projects have been moved into the folder VS8. diff --git a/docs-src/guides/installation/upgrade_11gr2_53_conn_status.md b/docs-src/guides/installation/upgrade_11gr2_53_conn_status.md new file mode 100644 index 000000000..839e14663 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_conn_status.md @@ -0,0 +1,26 @@ +--- +title: "Replication Connection Status in the Java API" +api-name: "Replication Connection Status in the Java API" +source: docs/installation/upgrade_11gr2_53_conn_status.html +--- +## Replication Connection Status in the Java API + + [New Function](upgrade_11gr2_53_conn_status.md#idp804776) + + [New Class](upgrade_11gr2_53_conn_status.md#idp771568) + + [Deprecated Function](upgrade_11gr2_53_conn_status.md#idp809200) + +The Java function `ReplicationManagerSiteInfo.isConnected()` is now deprecated. To get the replication connection status, use `ReplicationManagerSiteInfo.getConnectionStatus()`, which returns the new class `ReplicationManagerConnectionStatus`, which has the values `CONNECTED`, `DISCONNECTED`, and `UNKNOWN`. + +### New Function + +- `ReplicationManagerSiteInfo.getConnectionStatus()` + +### New Class + +- `ReplicationManagerConnectionStatus` + +### Deprecated Function + +- `ReplicationManagerSiteInfo.isConnected()` diff --git a/docs-src/guides/installation/upgrade_11gr2_53_excl.md b/docs-src/guides/installation/upgrade_11gr2_53_excl.md new file mode 100644 index 000000000..b6c3f44e7 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_excl.md @@ -0,0 +1,15 @@ +--- +title: "Exclusive Database Handles" +api-name: "Exclusive Database Handles" +source: docs/installation/upgrade_11gr2_53_excl.html +--- +## Exclusive Database Handles + + [New Functions](upgrade_11gr2_53_excl.md#idp811424) + +Database handles can now be configured to allow exclusive access to the database. To enable exclusive access, call DB->set_lk_exclusive() before calling DB->open(). Set nowait_onoff to non-zero to have DB->open() return immediately, with the error `DB_LOCK_NOTGRANTED` if it cannot immediately get exclusive access to the database, and to 0 to have DB->open() block until it can gain exclusive access. + +### New Functions + +- DB->set_lk_exclusive() +- DB->get_lk_exclusive() diff --git a/docs-src/guides/installation/upgrade_11gr2_53_heap_regionsize.md b/docs-src/guides/installation/upgrade_11gr2_53_heap_regionsize.md new file mode 100644 index 000000000..fe82e7b4c --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_heap_regionsize.md @@ -0,0 +1,15 @@ +--- +title: "Configure the Region Size of Heap Databases" +api-name: "Configure the Region Size of Heap Databases" +source: docs/installation/upgrade_11gr2_53_heap_regionsize.html +--- +## Configure the Region Size of Heap Databases + + [New Functions](upgrade_11gr2_53_heap_regionsize.md#idp775064) + +The region size of heap databases is now configurable. Configuring the region size is useful in controlling the growth of a heap database. To set the region size, call DB->set_heap_regionsize() with the number of pages that the region should have, before the database is created. The function is ignored if it is called after the database is created. + +### New Functions + +- DB->set_heap_regionsize() +- DB->get_heap_regionsize() diff --git a/docs-src/guides/installation/upgrade_11gr2_53_hotbackup.md b/docs-src/guides/installation/upgrade_11gr2_53_hotbackup.md new file mode 100644 index 000000000..74e948566 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_hotbackup.md @@ -0,0 +1,46 @@ +--- +title: "New Hotbackup Interface" +api-name: "New Hotbackup Interface" +source: docs/installation/upgrade_11gr2_53_hotbackup.html +--- +## New Hotbackup Interface + + [New Functions](upgrade_11gr2_53_hotbackup.md#idp815256) + + [Flags Accepted by DB_ENV-\>backup()](upgrade_11gr2_53_hotbackup.md#idp805032) + + [Flags Accepted by DB_ENV-\>dbbackup()](upgrade_11gr2_53_hotbackup.md#idp822632) + + [Enumerations Accepted by DB_ENV-\>set_backup_config()](upgrade_11gr2_53_hotbackup.md#idp828456) + +Two new functions have been added to the API that perform hotbackups, DB_ENV->backup() and DB_ENV->dbbackup(). DB_ENV->backup() creates a hotbackup of all databases in the specified environment, and DB_ENV->dbbackup() creates a hotbackup of the specified database. The functions DB_ENV->set_backup_callbacks() and DB_ENV->set_backup_config() can be called to customize the behavior of hotbackup. Note that this interface must be used to create a hotbackup on all non-BSD or Unix based systems. + +### New Functions + +- DB_ENV->backup() +- DB_ENV->dbbackup() +- DB_ENV->set_backup_callbacks() +- DB_ENV->set_backup_config() + +### Flags Accepted by DB_ENV->backup() + +- `DB_BACKUP_CLEAN` +- `DB_BACKUP_FILES` +- `DB_BACKUP_NO_LOGS` +- `DB_BACKUP_SINGLE_DIR` +- `DB_BACKUP_UPDATE` +- `DB_CREATE` +- `DB_EXCL` +- `DB_VERB_BACKUP` + +### Flags Accepted by DB_ENV->dbbackup() + +- `DB_CREATE` +- `DB_EXCL` + +### Enumerations Accepted by DB_ENV->set_backup_config() + +- `DB_BACKUP_WRITE_DIRECT` +- `DB_BACKUP_READ_COUNT` +- `DB_BACKUP_READ_SLEEP` +- `DB_BACKUP_SIZE` diff --git a/docs-src/guides/installation/upgrade_11gr2_53_jdbc.md b/docs-src/guides/installation/upgrade_11gr2_53_jdbc.md new file mode 100644 index 000000000..74a8b3e05 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_jdbc.md @@ -0,0 +1,8 @@ +--- +title: "Updated JDBC Version" +api-name: "Updated JDBC Version" +source: docs/installation/upgrade_11gr2_53_jdbc.html +--- +## Updated JDBC Version + +The JDBC library version included with Berkeley DB has been updated to version 20110827. The new version supports the Embedded Java JSR 169 standard, which is the official specification for JDBC with embedded Java platforms. diff --git a/docs-src/guides/installation/upgrade_11gr2_53_meta_dir.md b/docs-src/guides/installation/upgrade_11gr2_53_meta_dir.md new file mode 100644 index 000000000..0a4567af5 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_meta_dir.md @@ -0,0 +1,17 @@ +--- +title: "Configure Directory to Store Metadata Files" +api-name: "Configure Directory to Store Metadata Files" +source: docs/installation/upgrade_11gr2_53_meta_dir.html +--- +## Configure Directory to Store Metadata Files + + [New Functions](upgrade_11gr2_53_meta_dir.md#idp837576) + +The directory in which persistent metadata files are stored can now be configured. By default persistent metadata files are stored in the environment home directory. The files that will be stored in the metadata directory are \_\_db.rep.system, \_\_db.rep.gen, \_\_db.rep.egen and \_\_db.rep.init. + +To set the metadata file directory, call DB_ENV->set_metadata_dir() with the path to the directory in which to store metadata files. The metadata directory can also be set in the `DB_CONFIG` file using `set_metadata_dir`. + +### New Functions + +- DB_ENV->set_metadata_dir() +- DB_ENV->get_metadata_dir() diff --git a/docs-src/guides/installation/upgrade_11gr2_53_sql_build.md b/docs-src/guides/installation/upgrade_11gr2_53_sql_build.md new file mode 100644 index 000000000..8b239e8d7 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_sql_build.md @@ -0,0 +1,10 @@ +--- +title: "Changes in the SQL API Build" +api-name: "Changes in the SQL API Build" +source: docs/installation/upgrade_11gr2_53_sql_build.html +--- +## Changes in the SQL API Build + +Several changes have been made to the SQL API build. Encryption is now disabled by default on Windows, and the compile time flag BDBSQL_OMIT_SHARING has been changed to BDBSQL_SINGLE_PROCESS. + +Encryption is now disabled by default in the Windows SQL API build. This makes it consistent with builds of the SQL API on other systems. To enable encryption in Visual Studios right click the db_sql project, and select `Properties->Configuration Properties->C/C++->Preprocessor` and add `SQLITE_HAS_CODEC` to `Preprocessor Definitions`. diff --git a/docs-src/guides/installation/upgrade_11gr2_53_sql_pragma.md b/docs-src/guides/installation/upgrade_11gr2_53_sql_pragma.md new file mode 100644 index 000000000..f57a91cfe --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_sql_pragma.md @@ -0,0 +1,15 @@ +--- +title: "New Berkeley DB SQL API PRAGMAs" +api-name: "New Berkeley DB SQL API PRAGMAs" +source: docs/installation/upgrade_11gr2_53_sql_pragma.html +--- +## New Berkeley DB SQL API PRAGMAs + + [New PRAGMAs](upgrade_11gr2_53_sql_pragma.md#idp843792) + +Two new Berkeley DB SQL API specific pragmas have been added, `bdbsql_shared_resources` and `bdbsql_set_lock_tablesize`. `bdbsql_shared_resources` is used to set the maximum amount of memory, in bytes, to be used by shared structures in the main environment region, which is useful in applications with a large number of tables, transactions, or threads. `bdbsql_set_lock_tablesize` is used to set the number of buckets in the lock object hash table in the Berkeley DB environment, which is useful if an application has many concurrent long running transactions. + +### New PRAGMAs + +- `PRAGMA bdbsql_shared_resources[=N]` +- `PRAGMA bdbsql_set_lock_tablesize[=N]` diff --git a/docs-src/guides/installation/upgrade_11gr2_53_sql_rep.md b/docs-src/guides/installation/upgrade_11gr2_53_sql_rep.md new file mode 100644 index 000000000..9be206c96 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_sql_rep.md @@ -0,0 +1,22 @@ +--- +title: "Replication for Existing Databases in the SQL API" +api-name: "Replication for Existing Databases in the SQL API" +source: docs/installation/upgrade_11gr2_53_sql_rep.html +--- +## Replication for Existing Databases in the SQL API + + [PRAGMAs With Permanent Effects](upgrade_11gr2_53_sql_rep.md#idp837896) + + [PRAGMAs That Can Now Operate on Existing Databases](upgrade_11gr2_53_sql_rep.md#idp844568) + +Replication can now be enabled on existing SQL databases, and replication is now disabled permanently instead of temporarily. Replication is enabled on an existing database the same way it is enabled on a new database, with one restriction. The existing database must configure itself as the initial master of a new replication group. To disable replication on a database permanently, use `pragma replication=OFF;`. + +### PRAGMAs With Permanent Effects + +- `pragma replication=OFF;` + +### PRAGMAs That Can Now Operate on Existing Databases + +- `pragma replication_local_site="host:port";` +- `pragma replication_initial_master=ON;` +- `pragma replication=ON;` diff --git a/docs-src/guides/installation/upgrade_11gr2_53_xa_mvcc.md b/docs-src/guides/installation/upgrade_11gr2_53_xa_mvcc.md new file mode 100644 index 000000000..4f51dae12 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_53_xa_mvcc.md @@ -0,0 +1,8 @@ +--- +title: "Berkeley DB X/Open Compliant XA Resource Manager and Transaction Snapshots" +api-name: "Berkeley DB X/Open Compliant XA Resource Manager and Transaction Snapshots" +source: docs/installation/upgrade_11gr2_53_xa_mvcc.html +--- +## Berkeley DB X/Open Compliant XA Resource Manager and Transaction Snapshots + +The transactions managed by the Berkeley DB X/open compliant XA resource manager can now be enabled for transaction snapshots. To enable snapshots open an XA managed database with the flag, `DB_MULTIVERSION`, and all XA managed transactions that operate on that database will use transaction snapshots. diff --git a/docs-src/guides/installation/upgrade_11gr2_autoinit.md b/docs-src/guides/installation/upgrade_11gr2_autoinit.md new file mode 100644 index 000000000..34a368fb8 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_autoinit.md @@ -0,0 +1,8 @@ +--- +title: "DB_REP_CONF_NOAUTOINIT Replaced" +api-name: "DB_REP_CONF_NOAUTOINIT Replaced" +source: docs/installation/upgrade_11gr2_autoinit.html +--- +## DB_REP_CONF_NOAUTOINIT Replaced + +In this release, the `DB_REP_CONF_NOAUTOINIT` flag is replaced by the `DB_REP_CONF_AUTOINIT` flag. This option is ON by default. To turn off automatic internal initialization, call the `DB_ENV->rep_set_config` method with the **which** parameter set to `DB_REP_CONF_AUTOINIT` and the **onoff** parameter set to zero. diff --git a/docs-src/guides/installation/upgrade_11gr2_dbsqlcodegen.md b/docs-src/guides/installation/upgrade_11gr2_dbsqlcodegen.md new file mode 100644 index 000000000..f44bb6a71 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_dbsqlcodegen.md @@ -0,0 +1,8 @@ +--- +title: "db_sql Renamed to db_sql_codegen" +api-name: "db_sql Renamed to db_sql_codegen" +source: docs/installation/upgrade_11gr2_dbsqlcodegen.html +--- +## db_sql Renamed to db_sql_codegen + +The db_sql utility is now called db_sql_codegen. This command line utility is not built by default. To build db_sql_codegen, specify `--enable-sql_codegen` when configuring Berkeley DB. diff --git a/docs-src/guides/installation/upgrade_11gr2_remsupp.md b/docs-src/guides/installation/upgrade_11gr2_remsupp.md new file mode 100644 index 000000000..16fcd7505 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_remsupp.md @@ -0,0 +1,10 @@ +--- +title: "Dropped Support" +api-name: "Dropped Support" +source: docs/installation/upgrade_11gr2_remsupp.html +--- +## Dropped Support + +Berkeley DB no longer supports Visual Studio 6.0. The earliest version supported is Visual Studio 2005. The build files for Windows Visual Studio 6.0 are removed. + +Berkeley DB no longer supports Win9X, Windows Me (Millenium edition), and Windows NT 4.0. The minimum supported windows platform is Windows 2000. diff --git a/docs-src/guides/installation/upgrade_11gr2_repmgr.md b/docs-src/guides/installation/upgrade_11gr2_repmgr.md new file mode 100644 index 000000000..dbf504897 --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_repmgr.md @@ -0,0 +1,14 @@ +--- +title: "Support for Multiple Client-to-Client Peers" +api-name: "Support for Multiple Client-to-Client Peers" +source: docs/installation/upgrade_11gr2_repmgr.html +--- +## Support for Multiple Client-to-Client Peers + +A Berkeley DB Replication Manager application can now designate one or more remote sites (called its "peers") to receive client-to-client requests. + +In previous releases, there could be only one peer at a time. If you called the `DB_ENV->repmgr_add_remote_site` method specifying site "A" as a peer and you made another call specifying site "B" as a peer, site "B" would become the only peer, and site "A" would no longer be a peer. + +Starting with Berkeley DB 11gR2, the same sequence of calls results in both site "A" and site "B" being possible peers. Replication Manager may select any of a site's possible peers to use for client-to-client requests. If the first peer that the Replication Manager selects cannot be used (for example, it is unavailable or it is the current master), Replication Manager attempts to use a different peer if there is more than one peer. + +To get the pre-11gR2 peer behavior in this example, you must make an additional call to the `DB_ENV->repmgr_add_remote_site` method, specifying site "A" and a flag value that excludes the `DB_REPMGR_PEER` bit value to remove site "A" as a possible peer. diff --git a/docs-src/guides/installation/upgrade_11gr2_toc.md b/docs-src/guides/installation/upgrade_11gr2_toc.md new file mode 100644 index 000000000..d71f79d4e --- /dev/null +++ b/docs-src/guides/installation/upgrade_11gr2_toc.md @@ -0,0 +1,38 @@ +--- +title: "Chapter 12.  Upgrading Berkeley DB 4.8 applications to Berkeley DB 11.2.5.0" +api-name: "Chapter 12.  Upgrading Berkeley DB 4.8 applications to Berkeley DB 11.2.5.0" +source: docs/installation/upgrade_11gr2_toc.html +--- +## Chapter 12.  Upgrading Berkeley DB 4.8 applications to Berkeley DB 11.2.5.0 + +**Table of Contents** + + [Introduction](upgrade_11gr2_toc.md#upgrade_11gr2_intro) + + [db_sql Renamed to db_sql_codegen](upgrade_11gr2_dbsqlcodegen.md) + + [DB_REP_CONF_NOAUTOINIT Replaced](upgrade_11gr2_autoinit.md) + + [Support for Multiple Client-to-Client Peers](upgrade_11gr2_repmgr.md) + + [Cryptography Support](build_unix_encrypt.md) + + [DB_NOSYNC Flag to Flush Files](build_unix_db_nosync.md) + + [Dropped Support](upgrade_11gr2_remsupp.md) + + [Changing Stack Size](build_unix_stacksize.md) + + [Berkeley DB 11g Release 2 Change Log](changelog_5_0.md) + + [Changes between 11.2.5.0.26 and 11.2.5.0.32](changelog_5_0.md#idp1125968) + + [Changes between 11.2.5.0.21 and 11.2.5.0.26](changelog_5_0.md#idp1126872) + + [Changes between 4.8 and 11.2.5.0.21](changelog_5_0.md#idp1125192) + + [Known Bugs](changelog_5_0.md#idp1131672) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.8 release interfaces to the Berkeley DB 11*g* Release 2 interfaces. (Library version 11.2.5.0). This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/installation/upgrade_4_8_disk.md b/docs-src/guides/installation/upgrade_4_8_disk.md new file mode 100644 index 000000000..f8c8399c3 --- /dev/null +++ b/docs-src/guides/installation/upgrade_4_8_disk.md @@ -0,0 +1,14 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/installation/upgrade_4_8_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.8 release. + +No database formats changed in the Berkeley DB 4.8 release. + +The Berkeley DB 4.8 release does not support live replication upgrade from the 4.2 or 4.3 releases, only from the 4.4 and later releases. + +For further information on upgrading Berkeley DB installations, see the Berkeley DB Upgrade Guide. diff --git a/docs-src/guides/installation/upgrade_4_8_dpl.md b/docs-src/guides/installation/upgrade_4_8_dpl.md new file mode 100644 index 000000000..abf0a5a5d --- /dev/null +++ b/docs-src/guides/installation/upgrade_4_8_dpl.md @@ -0,0 +1,16 @@ +--- +title: "Registering DPL Secondary Keys" +api-name: "Registering DPL Secondary Keys" +source: docs/installation/upgrade_4_8_dpl.html +--- +## Registering DPL Secondary Keys + +Entity subclasses that define secondary keys must now be registered prior to storing an instance of the class. This can be done in two ways: + +- The `EntityModel.registerClass()` method may be called to register the subclass before opening the entity store. + +- The `EntityStore.getSubclassIndex()` method may be called to implicitly register the subclass after opening the entity store. + +Failure to register the entity subclass will result in an `IllegalArgumentException` the first time an attempt is made to store an instance of the subclass. An exception will not occur if instances of the subclass have previously been stored, which allows existing applications to run unmodified in most cases. + +This behavioral change was made to increase reliability. In several cases, registering an entity subclass has been necessary as a workaround. The requirement to register the subclass will ensure that such errors do not occur in deployed applications. diff --git a/docs-src/guides/installation/upgrade_4_8_fcntl.md b/docs-src/guides/installation/upgrade_4_8_fcntl.md new file mode 100644 index 000000000..2809ba5a1 --- /dev/null +++ b/docs-src/guides/installation/upgrade_4_8_fcntl.md @@ -0,0 +1,8 @@ +--- +title: "Dropped Support for fcntl System Calls" +api-name: "Dropped Support for fcntl System Calls" +source: docs/installation/upgrade_4_8_fcntl.html +--- +## Dropped Support for fcntl System Calls + +Berkeley DB no longer supports mutex implementations based on the `fcntl` system call. If you have been configuring Berkeley DB to use this type of mutex, you need to either switch to a different mutex type or contact the Berkeley DB team for support. diff --git a/docs-src/guides/installation/upgrade_4_8_mpool.md b/docs-src/guides/installation/upgrade_4_8_mpool.md new file mode 100644 index 000000000..4fa121d6b --- /dev/null +++ b/docs-src/guides/installation/upgrade_4_8_mpool.md @@ -0,0 +1,12 @@ +--- +title: "Minor Change in Behavior of DB_MPOOLFILE->get" +api-name: "Minor Change in Behavior of DB_MPOOLFILE->get" +source: docs/installation/upgrade_4_8_mpool.html +--- +## Minor Change in Behavior of DB_MPOOLFILE-\>get + +DB 4.8 introduces some performance enhancements, based on the use of shared/exclusive latches instead of locks in some areas of the internal buffer management code. This change will affect how the `DB_MPOOL` interface handles dirty buffers. + +Because of these changes, `DB_MPOOLFILE->get` will now acquire an exclusive latch on the buffer if the `DB_MPOOL_DIRTY` or `DB_MPOOL_EDIT` flags are specified. This could lead to an application deadlock if the application tries to fetch the buffer again, without an intervening `DB_MPOOLFILE->put` call. + +If your application uses the `DB_MPOOL` interface, and especially the `DB_MPOOL_DIRTY` and `DB_MPOOL_EDIT` flags, you should review your code to ensure that this behavior change does not cause your application to deadlock. diff --git a/docs-src/guides/installation/upgrade_4_8_toc.md b/docs-src/guides/installation/upgrade_4_8_toc.md new file mode 100644 index 000000000..fd9baf34d --- /dev/null +++ b/docs-src/guides/installation/upgrade_4_8_toc.md @@ -0,0 +1,84 @@ +--- +title: "Chapter 13. Upgrading Berkeley DB 4.7 applications to Berkeley DB 4.8" +api-name: "Chapter 13. Upgrading Berkeley DB 4.7 applications to Berkeley DB 4.8" +source: docs/installation/upgrade_4_8_toc.html +--- +## Chapter 13. Upgrading Berkeley DB 4.7 applications to Berkeley DB 4.8 + +**Table of Contents** + + [Introduction](upgrade_4_8_toc.md#upgrade_4_8_intro) + + [Registering DPL Secondary Keys](upgrade_4_8_dpl.md) + + [Minor Change in Behavior of DB_MPOOLFILE-\>get](upgrade_4_8_mpool.md) + + [Dropped Support for fcntl System Calls](upgrade_4_8_fcntl.md) + + [Upgrade Requirements](upgrade_4_8_disk.md) + + [Berkeley DB 4.8.28 Change Log](changelog_4_8.md) + + [Changes between 4.8.26 and 4.8.28:](changelog_4_8.md#idp1162104) + + [Known bugs in 4.8](changelog_4_8.md#idp1184264) + + [Changes between 4.8.24 and 4.8.26:](changelog_4_8.md#idp1139288) + + [Changes between 4.8.21 and 4.8.24:](changelog_4_8.md#idp1091200) + + [Changes between 4.7 and 4.8.21:](changelog_4_8.md#idp1199520) + + [Database or Log File On-Disk Format Changes:](changelog_4_8.md#idp1200208) + + [New Features:](changelog_4_8.md#idp981712) + + [Database Environment Changes:](changelog_4_8.md#idp1130224) + + [Concurrent Data Store Changes:](changelog_4_8.md#idp1209320) + + [General Access Method Changes:](changelog_4_8.md#idp1209720) + + [Btree Access Method Changes:](changelog_4_8.md#idp1218064) + + [Hash Access Method Changes:](changelog_4_8.md#idp1215560) + + [Queue Access Method Changes:](changelog_4_8.md#idp1226120) + + [Recno Access Method Changes:](changelog_4_8.md#idp1163928) + + [C-specific API Changes:](changelog_4_8.md#idp1138904) + + [C++-specific API Changes:](changelog_4_8.md#idp1218344) + + [Java-specific API Changes:](changelog_4_8.md#idp1238856) + + [Direct Persistence Layer (DPL), Bindings and Collections API:](changelog_4_8.md#idp1232112) + + [Tcl-specific API Changes:](changelog_4_8.md#idp1232384) + + [RPC-specific Client/Server Changes:](changelog_4_8.md#idp1244368) + + [Replication Changes:](changelog_4_8.md#idp1245896) + + [XA Resource Manager Changes:](changelog_4_8.md#idp1242240) + + [Locking Subsystem Changes:](changelog_4_8.md#idp1247728) + + [Logging Subsystem Changes:](changelog_4_8.md#idp1241128) + + [Memory Pool Subsystem Changes:](changelog_4_8.md#idp1258328) + + [Mutex Subsystem Changes:](changelog_4_8.md#idp1258720) + + [Test Suite Changes](changelog_4_8.md#idp1240832) + + [Transaction Subsystem Changes:](changelog_4_8.md#idp1249776) + + [Utility Changes:](changelog_4_8.md#idp1271664) + + [Configuration, Documentation, Sample Application, Portability and Build Changes:](changelog_4_8.md#idp1274104) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.7 release interfaces to the Berkeley DB 4.8 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/installation/upgrade_51_toc.md b/docs-src/guides/installation/upgrade_51_toc.md new file mode 100644 index 000000000..0c49655d9 --- /dev/null +++ b/docs-src/guides/installation/upgrade_51_toc.md @@ -0,0 +1,74 @@ +--- +title: "Chapter 11.  Upgrading Berkeley DB 11.2.5.0 applications to Berkeley DB 11.2.5.1" +api-name: "Chapter 11.  Upgrading Berkeley DB 11.2.5.0 applications to Berkeley DB 11.2.5.1" +source: docs/installation/upgrade_51_toc.html +--- +## Chapter 11.  Upgrading Berkeley DB 11.2.5.0 applications to Berkeley DB 11.2.5.1 + +**Table of Contents** + + [Introduction](upgrade_51_toc.md#upgrade_51_intro) + + [DPL Applications must be recompiled](upgrade_11gr2_51_dpl_recompile.md) + + [Source Tree Rearranged](upgrade_11gr2_51_src_reorg.md) + + [SQLite Interface Upgrade](upgrade_11gr2_51_sqlite_ver.md) + + [Mod_db4 Support Discontinued](upgrade_11gr2_51_mod_db4_unsupp.md) + + [Berkeley DB Library Version 11.2.5.1 Change Log](changelog_5_1.md) + + [Database or Log File On-Disk Format Changes](changelog_5_1.md#idp1052992) + + [New Features](changelog_5_1.md#idp953176) + + [Database Environment Changes](changelog_5_1.md#idp1045336) + + [Concurrent Data Store Changes](changelog_5_1.md#idp1059760) + + [Access Method Changes](changelog_5_1.md#idp981016) + + [API Changes](changelog_5_1.md#idp1049008) + + [SQL-Specific API Changes](changelog_5_1.md#idp1055592) + + [Tcl-Specific API Changes](changelog_5_1.md#idp1056952) + + [Java-Specific API Changes](changelog_5_1.md#idp1052280) + + [C#-Specific API Changes](changelog_5_1.md#idp987592) + + [Direct Persistence Layer (DPL), Bindings and Collections API](changelog_5_1.md#idp1060648) + + [Replication Changes](changelog_5_1.md#idp1070000) + + [Locking Subsystem Changes](changelog_5_1.md#idp1080936) + + [Logging Subsystem Changes](changelog_5_1.md#idp1092608) + + [Memory Pool Subsystem Changes](changelog_5_1.md#idp1076376) + + [Mutex Subsystem Changes](changelog_5_1.md#idp1080752) + + [Transaction Subsystem Changes](changelog_5_1.md#idp1089584) + + [Test Suite Changes](changelog_5_1.md#idp1067160) + + [Utility Changes](changelog_5_1.md#idp1088000) + + [Configuration, Documentation, Sample Apps, Portability, and Build Changes](changelog_5_1.md#idp1091312) + + [Example Changes](changelog_5_1.md#idp1081576) + + [Miscellaneous Bug Fixes](changelog_5_1.md#idp1102152) + + [Deprecated Features](changelog_5_1.md#idp1100024) + + [Known Bugs](changelog_5_1.md#idp1100672) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 11.2.5.0 library to the Berkeley DB 11.2.5.1 library (both of which belong to Berkeley DB 11*g* Release 2). This information does not describe how to upgrade Berkeley DB 1.85 release applications. + +For information on the general process of upgrading Berkeley DB installations and upgrade instructions related to historical releases, see the Berkeley DB Upgrade Guide. diff --git a/docs-src/guides/installation/upgrade_52_toc.md b/docs-src/guides/installation/upgrade_52_toc.md new file mode 100644 index 000000000..a405b9dfb --- /dev/null +++ b/docs-src/guides/installation/upgrade_52_toc.md @@ -0,0 +1,130 @@ +--- +title: "Chapter 10.  Upgrading Berkeley DB 11.2.5.1 applications to Berkeley DB 11.2.5.2" +api-name: "Chapter 10.  Upgrading Berkeley DB 11.2.5.1 applications to Berkeley DB 11.2.5.2" +source: docs/installation/upgrade_52_toc.html +--- +## Chapter 10.  Upgrading Berkeley DB 11.2.5.1 applications to Berkeley DB 11.2.5.2 + +**Table of Contents** + + [Introduction](upgrade_52_toc.md#upgrade_52_intro) + + [SQLite Interface Upgrade](upgrade_11gr2_52_sqlite_ver.md) + + [32bit/64bit Compatibility on Windows](upgrade_11gr2_52_bit_cmp_win.md) + + [Read Only flag for DBT](upgrade_11gr2_52_rep_dbt_readonly.md) + + [New Flag](upgrade_11gr2_52_rep_dbt_readonly.md#idp907000) + + [Dynamic Environment Configuration](upgrade_11gr2_52_dyn_env.md) + + [New Functions](upgrade_11gr2_52_dyn_env.md#idp902144) + + [Deprecated Functions](upgrade_11gr2_52_dyn_env.md#idp912000) + + [Exclusive Transactions in the SQL Layer](upgrade_11gr2_52_excl_txn_sql.md) + + [Group Membership in Repmgr](upgrade_11gr2_52_grp_mbr.md) + + [Upgrading](upgrade_11gr2_52_grp_mbr.md#idp929720) + + [New Functions](upgrade_11gr2_52_grp_mbr.md#idp910056) + + [Modified Functions](upgrade_11gr2_52_grp_mbr.md#idp901088) + + [New Events](upgrade_11gr2_52_grp_mbr.md#idp924520) + + [Removed Functions](upgrade_11gr2_52_grp_mbr.md#idp937928) + + [New Parameters](upgrade_11gr2_52_grp_mbr.md#idp909344) + + [New Structure](upgrade_11gr2_52_grp_mbr.md#idp924776) + + [Heap Access Method](upgrade_11gr2_52_heap.md) + + [New Functions](upgrade_11gr2_52_heap.md#idp936848) + + [Modified Functions](upgrade_11gr2_52_heap.md#idp930424) + + [New Definition](upgrade_11gr2_52_heap.md#idp931776) + + [Enabling Transaction Snapshots in the SQL Layer](upgrade_11gr2_52_mvcc_sql.md) + + [New Pragmas](upgrade_11gr2_52_mvcc_sql.md#idp951464) + + [2SITE_STRICT Enabled by Default in Replication](upgrade_11gr2_52_rep_2site_strict.md) + + [Enabling Replication in the SQL Layer](upgrade_11gr2_52_rep_sql.md) + + [New Pragmas](upgrade_11gr2_52_rep_sql.md#idp962696) + + [Repmgr Message Channels](upgrade_11gr2_52_repmgr_channels.md) + + [New Functions](upgrade_11gr2_52_repmgr_channels.md#idp919280) + + [Sequence Support in the SQL Layer](upgrade_11gr2_52_seq_sql.md) + + [New Functions](upgrade_11gr2_52_seq_sql.md#idp963480) + + [Berkeley DB X/Open Compliant XA Resource Manager](upgrade_11gr2_52_xa.md) + + [Constraints](upgrade_11gr2_52_xa.md#idp973264) + + [New Flag](upgrade_11gr2_52_xa.md#idp978200) + + [Modified Function](upgrade_11gr2_52_xa.md#idp982256) + + [Hot Backup Changes](upgrade_11gr2_52_hot_backup.md) + + [Berkeley DB Library Version 11.2.5.2 Change Log](changelog_5_2.md) + + [Database or Log File On-Disk Format Changes](changelog_5_2.md#idp972456) + + [New Features](changelog_5_2.md#idp978720) + + [Database Environment Changes](changelog_5_2.md#idp984720) + + [Concurrent Data Store Changes](changelog_5_2.md#idp995752) + + [Access Method Changes](changelog_5_2.md#idp989160) + + [SQL API Changes](changelog_5_2.md#idp989544) + + [C API Changes](changelog_5_2.md#idp971912) + + [Tcl-specific API Changes](changelog_5_2.md#idp996528) + + [C#-specific API Changes](changelog_5_2.md#idp972000) + + [Replication Changes](changelog_5_2.md#idp994456) + + [Locking Subsystem Changes](changelog_5_2.md#idp996912) + + [Logging Subsystem Changes](changelog_5_2.md#idp1010640) + + [Memory Pool Subsystem Changes](changelog_5_2.md#idp992728) + + [Mutex Subsystem Changes](changelog_5_2.md#idp1018872) + + [Transaction Subsystem Changes](changelog_5_2.md#idp1011056) + + [Test Suite Changes](changelog_5_2.md#idp1003424) + + [Utility Changes](changelog_5_2.md#idp1029752) + + [Configuration, Documentation, Sample Apps, Portability and Build Changes](changelog_5_2.md#idp1031368) + + [Example Changes](changelog_5_2.md#idp1003200) + + [Miscellaneous Bug Fixes](changelog_5_2.md#idp1034280) + + [Deprecated Features](changelog_5_2.md#idp1035816) + + [Known Bugs](changelog_5_2.md#idp1037736) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 11.2.5.1 library to the Berkeley DB 11.2.5.2 library (both of which belong to Berkeley DB 11*g* Release 2). This information does not describe how to upgrade Berkeley DB 1.85 release applications. + +For information on the general process of upgrading Berkeley DB installations and upgrade instructions related to historical releases, see the Berkeley DB Upgrade Guide. diff --git a/docs-src/guides/installation/upgrade_53_toc.md b/docs-src/guides/installation/upgrade_53_toc.md new file mode 100644 index 000000000..5a5ff60b2 --- /dev/null +++ b/docs-src/guides/installation/upgrade_53_toc.md @@ -0,0 +1,100 @@ +--- +title: "Chapter 9.  Upgrading Berkeley DB 11.2.5.2 applications to Berkeley DB 11.2.5.3" +api-name: "Chapter 9.  Upgrading Berkeley DB 11.2.5.2 applications to Berkeley DB 11.2.5.3" +source: docs/installation/upgrade_53_toc.html +--- +## Chapter 9.  Upgrading Berkeley DB 11.2.5.2 applications to Berkeley DB 11.2.5.3 + +**Table of Contents** + + [Introduction](upgrade_53_toc.md#upgrade_53_intro) + + [Changes to the build_windows Folder](upgrade_11gr2_53_build_windows.md) + + [Replication Connection Status in the Java API](upgrade_11gr2_53_conn_status.md) + + [New Function](upgrade_11gr2_53_conn_status.md#idp804776) + + [New Class](upgrade_11gr2_53_conn_status.md#idp771568) + + [Deprecated Function](upgrade_11gr2_53_conn_status.md#idp809200) + + [Exclusive Database Handles](upgrade_11gr2_53_excl.md) + + [New Functions](upgrade_11gr2_53_excl.md#idp811424) + + [Configure the Region Size of Heap Databases](upgrade_11gr2_53_heap_regionsize.md) + + [New Functions](upgrade_11gr2_53_heap_regionsize.md#idp775064) + + [New Hotbackup Interface](upgrade_11gr2_53_hotbackup.md) + + [New Functions](upgrade_11gr2_53_hotbackup.md#idp815256) + + [Flags Accepted by DB_ENV-\>backup()](upgrade_11gr2_53_hotbackup.md#idp805032) + + [Flags Accepted by DB_ENV-\>dbbackup()](upgrade_11gr2_53_hotbackup.md#idp822632) + + [Enumerations Accepted by DB_ENV-\>set_backup_config()](upgrade_11gr2_53_hotbackup.md#idp828456) + + [Updated JDBC Version](upgrade_11gr2_53_jdbc.md) + + [Configure Directory to Store Metadata Files](upgrade_11gr2_53_meta_dir.md) + + [New Functions](upgrade_11gr2_53_meta_dir.md#idp837576) + + [Changes in the SQL API Build](upgrade_11gr2_53_sql_build.md) + + [New Berkeley DB SQL API PRAGMAs](upgrade_11gr2_53_sql_pragma.md) + + [New PRAGMAs](upgrade_11gr2_53_sql_pragma.md#idp843792) + + [Replication for Existing Databases in the SQL API](upgrade_11gr2_53_sql_rep.md) + + [PRAGMAs With Permanent Effects](upgrade_11gr2_53_sql_rep.md#idp837896) + + [PRAGMAs That Can Now Operate on Existing Databases](upgrade_11gr2_53_sql_rep.md#idp844568) + + [Berkeley DB X/Open Compliant XA Resource Manager and Transaction Snapshots](upgrade_11gr2_53_xa_mvcc.md) + + [Berkeley DB Library Version 11.2.5.3 Change Log](changelog_5_3.md) + + [Changes between 11.2.5.3.21 and 11.2.5.3.28](changelog_5_3.md#idp839120) + + [Changes between 11.2.5.3.15 and 11.2.5.3.21](changelog_5_3.md#idp845408) + + [Database or Log File On-Disk Format Changes](changelog_5_3.md#idp636088) + + [New Features](changelog_5_3.md#idp856040) + + [Database Environment Changes](changelog_5_3.md#idp853696) + + [Access Method Changes](changelog_5_3.md#idp844240) + + [SQL API Changes](changelog_5_3.md#idp838728) + + [Java-specific API changes](changelog_5_3.md#idp863240) + + [Replication Changes](changelog_5_3.md#idp867984) + + [Locking Subsystem Changes](changelog_5_3.md#idp853912) + + [Logging Subsystem Changes](changelog_5_3.md#idp844888) + + [Memory Pool Subsystem Changes](changelog_5_3.md#idp868368) + + [Mutex Subsystem Changes](changelog_5_3.md#idp883216) + + [Transaction Subsystem Changes](changelog_5_3.md#idp875448) + + [Utility Changes](changelog_5_3.md#idp889064) + + [Configuration, Documentation, Sample Apps, Portability and Build Changes](changelog_5_3.md#idp892136) + + [Known Bugs](changelog_5_3.md#idp892656) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 11.2.5.2 library to the Berkeley DB 11.2.5.3 library (both of which belong to Berkeley DB 11*g* Release 2). This information does not describe how to upgrade Berkeley DB 1.85 release applications. + +For information on the general process of upgrading Berkeley DB installations and upgrade instructions related to historical releases, see the Berkeley DB Upgrade Guide. diff --git a/docs-src/guides/installation/win_additional_options.md b/docs-src/guides/installation/win_additional_options.md new file mode 100644 index 000000000..6f5777d06 --- /dev/null +++ b/docs-src/guides/installation/win_additional_options.md @@ -0,0 +1,28 @@ +--- +title: "Additional build options" +api-name: "Additional build options" +source: docs/installation/win_additional_options.html +--- +## Additional build options + +There are several build options that you can configure when building Berkeley DB on Windows. To specify these, select `Project Properties`-\>`C/C++`-\>`Command Line` and add the property. + +These are some of the additional properties that you can specify when you are building Berkeley DB on Windows: + +- **/D HAVE_LOCALIZATION** + + Enable localized error message text, if available. This option should not be used when enabling stripped messages. + +- **/D HAVE_MIXED_SIZE_ADDRESSING** + + Allows for the sharing of the BDB database environment between 32-bit and 64-bit applications. Note that if you use this macro to rebuild your Berkeley DB library, then you need to also rebuild both your 32-bit and 64-bit applications using **/D HAVE_MIXED_SIZE_ADDRESSING**. + + Note that use of this macro means that private environments are disabled for the library. + +- **/D HAVE_STRIPPED_MESSAGES** + + Causes all error messages to be stripped of their textual information. This option should not be used when enabling localization support. Use of this property can reduce your library footprint by up to 42KB (for DLLs) or 98KB (for a .lib). + + Note that this option is automatically enabled if you build using the `db_small` project name. For more information on building a small library, see Building a small memory footprint library. + + If you use this build option, you can get an idea of what text should be issued for a given error message by using the Message Reference for Stripped Libraries guide. diff --git a/docs-src/guides/installation/win_build64.md b/docs-src/guides/installation/win_build64.md new file mode 100644 index 000000000..be5012731 --- /dev/null +++ b/docs-src/guides/installation/win_build64.md @@ -0,0 +1,25 @@ +--- +title: "Building Berkeley DB for 64-bit Windows" +api-name: "Building Berkeley DB for 64-bit Windows" +source: docs/installation/win_build64.html +--- +## Building Berkeley DB for 64-bit Windows + + [x64 build with Visual Studio 2005 or newer](win_build64.md#idp259672) + +The following procedure can be used to build natively on a 64-bit system or to cross-compile from a 32-bit system. + +When building 64-bit binaries, the output directory will be one of the following Berkeley DB subdirectories, depending upon the configuration that you chose: + +| | +|------------------------------------| +| `build_windows\x64\Debug` | +| `build_windows\x64\Release` | +| `build_windows\x64\Debug_static` | +| `build_windows\x64\Release_static` | + +### x64 build with Visual Studio 2005 or newer + +1. Follow the build instructions for your version of Visual Studio, as described in Building Berkeley DB for 32 bit Windows. +2. Select *x64* from the *Platform Configuration* dropdown. +3. Right click on *Solution 'Berkeley_DB'* in the Solution Explorer, and select *Build Solution* diff --git a/docs-src/guides/installation/win_build_cxx.md b/docs-src/guides/installation/win_build_cxx.md new file mode 100644 index 000000000..5fa4f61ab --- /dev/null +++ b/docs-src/guides/installation/win_build_cxx.md @@ -0,0 +1,8 @@ +--- +title: "Building the C++ API" +api-name: "Building the C++ API" +source: docs/installation/win_build_cxx.html +--- +## Building the C++ API + +C++ support is built automatically on Windows. diff --git a/docs-src/guides/installation/win_build_cygwin.md b/docs-src/guides/installation/win_build_cygwin.md new file mode 100644 index 000000000..9df308dc6 --- /dev/null +++ b/docs-src/guides/installation/win_build_cygwin.md @@ -0,0 +1,8 @@ +--- +title: "Building Berkeley DB with Cygwin" +api-name: "Building Berkeley DB with Cygwin" +source: docs/installation/win_build_cygwin.html +--- +## Building Berkeley DB with Cygwin + +To build Berkeley DB with Cygwin, follow the instructions in Building for UNIX/POSIX. diff --git a/docs-src/guides/installation/win_build_dist_dll.md b/docs-src/guides/installation/win_build_dist_dll.md new file mode 100644 index 000000000..adb4c36eb --- /dev/null +++ b/docs-src/guides/installation/win_build_dist_dll.md @@ -0,0 +1,21 @@ +--- +title: "Distributing DLLs" +api-name: "Distributing DLLs" +source: docs/installation/win_build_dist_dll.html +--- +## Distributing DLLs + +When distributing applications linked against the DLL (not static) version of the library, the DLL files you need will be found in one of the following Berkeley DB subdirectories, depending upon the configuration that you chose: + +| | +|--------------------------------------| +| `build_windows\Win32\Debug` | +| `build_windows\Win32\Release` | +| `build_windows\Win32\Debug_static` | +| `build_windows\Win32\Release_static` | +| `build_windows\x64\Debug` | +| `build_windows\x64\Release` | +| `build_windows\x64\Debug_static` | +| `build_windows\x64\Release_static` | + +You may also need to redistribute DLL files needed for the compiler's runtime. Generally, these runtime DLL files can be installed in the same directory that will contain your installed Berkeley DB DLLs. This directory may need to be added to your System PATH environment variable. Check your compiler's license and documentation for specifics on redistributing runtime DLLs. diff --git a/docs-src/guides/installation/win_build_stl.md b/docs-src/guides/installation/win_build_stl.md new file mode 100644 index 000000000..6fb1b390a --- /dev/null +++ b/docs-src/guides/installation/win_build_stl.md @@ -0,0 +1,10 @@ +--- +title: "Building the C++ STL API" +api-name: "Building the C++ STL API" +source: docs/installation/win_build_stl.html +--- +## Building the C++ STL API + +In the project list of the `Berkeley_DB.sln ` solution, build the "db_stl" project and "db_stl_static" project to build STL API as a dynamic or static library respectively. And in your application, you should link this library file as well as the Berkeley DB library file to your application. The STL API library file is by default always put at the same location as the Berkeley DB library file. + +And you need to include the STL API header files in your application code. If you are using the Berkeley DB source tree, the header files are in \/stl directory; If you are using the pre-built installed version, these header files are in \< Berkeley DB Installed Directory\>/include, as well as the db.h and db_cxx.h header files. diff --git a/docs-src/guides/porting/_meta.toml b/docs-src/guides/porting/_meta.toml new file mode 100644 index 000000000..f4f6c4a7e --- /dev/null +++ b/docs-src/guides/porting/_meta.toml @@ -0,0 +1,23 @@ +# Nav/index metadata for the porting guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Porting Berkeley DB" +landing = "index.md" +order = [ + "preface", + "audience", + "moreinfo", + "introduction", + "portprocess", + "newbinary", + "modscope", + "buildtarget", + "layout", + "testport", + "modifytest", + "testrun", + "testreview", + "sourceintegrate", + "certport", +] diff --git a/docs-src/guides/porting/audience.md b/docs-src/guides/porting/audience.md new file mode 100644 index 000000000..49ae865fc --- /dev/null +++ b/docs-src/guides/porting/audience.md @@ -0,0 +1,12 @@ +--- +title: "Audience" +api-name: "Audience" +source: docs/porting/audience.html +--- +## Audience + +This guide is intended for programmers porting Berkeley DB to a new platform. It assumes that these programmers possess: + +- Familiarity with standard ANSI C and POSIX C 1003.1 and 1003.2 library and system calls. + +- Working knowledge of the target platform as well as the development tools (for example, compilers, linkers, and debuggers) available on that platform. diff --git a/docs-src/guides/porting/buildtarget.md b/docs-src/guides/porting/buildtarget.md new file mode 100644 index 000000000..2b4a16e94 --- /dev/null +++ b/docs-src/guides/porting/buildtarget.md @@ -0,0 +1,8 @@ +--- +title: "Building on the Target Platform" +api-name: "Building on the Target Platform" +source: docs/porting/buildtarget.html +--- +## Building on the Target Platform + +Once you have an idea of the scope of the modifications, use the files generated on the UNIX system to help you create any compiler or linker tool chain files that you need to build on the target platform. At this point, start building the Berkeley DB on the target platform making the changes to the code that you identified earlier in the process. Once you have identified the modifications that you need to make, change the code accordingly. diff --git a/docs-src/guides/porting/certport.md b/docs-src/guides/porting/certport.md new file mode 100644 index 000000000..6ee12b2b8 --- /dev/null +++ b/docs-src/guides/porting/certport.md @@ -0,0 +1,12 @@ +--- +title: "Certifying a Port of Berkeley DB" +api-name: "Certifying a Port of Berkeley DB" +source: docs/porting/certport.html +--- +## Certifying a Port of Berkeley DB + +When the target platform supports using Tcl, the port is considered certified after a successful standard run (`run_std`) of the Test Suite . + +When the target platform does not support using Tcl, a port is considered certified if you see successful message reports for each of the tests in `test_micro` and `test_mutex`. + +Additionally, the configuration and compilation of Berkeley DB must be complete without errors or warninigs of any kind. You must provide specific information about the hardware, software, and compiler used during the porting process, especially versions of all thrid party tools used. If you have other diagnostic tools available, such as memory allocation checking tools, please conduct tests using them as well and provide Oracle engineering with the results. Finally, do not constrain your testing to one configuration. There are many different ways to configure Berkeley DB (that is many options to the "configure" script), please configure, built, and test Berkeley DB on the target platform with as many combinations of these flags as possible to ensure that nothing is missed. diff --git a/docs-src/guides/porting/index.md b/docs-src/guides/porting/index.md new file mode 100644 index 000000000..c36221624 --- /dev/null +++ b/docs-src/guides/porting/index.md @@ -0,0 +1,76 @@ +--- +title: "Porting Berkeley DB" +api-name: "Porting Berkeley DB" +source: docs/porting/index.html +--- +# Porting Berkeley DB + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [Audience](audience.md) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction to Porting Berkeley DB](introduction.md) + + [Types of Berkeley DB ports](introduction.md#porttypes) + + [When Oracle Has Agreed to Support Berkeley DB on the New Platform](introduction.md#idp139688) + + [When Oracle has Not Agreed to Support Berkeley DB on the New Platform](introduction.md#idp79768) + + [Berkeley DB Porting Process](portprocess.md) + + [2. Creating a New Berkeley DB Binary](newbinary.md) + + [Creating a Base Build of Berkeley DB](newbinary.md#basebuild) + + [Determining the Scope of the Modifications](modscope.md) + + [Do Changes Need to be Made to the Operating System Functionality?](modscope.md#osfunc) + + [Are Some Standard Functions Missing on the Target Platform?](modscope.md#standardfunc) + + [How Will the Port Handle Shared Memory?](modscope.md#sharedmem) + + [What Type of Mutexes Will the Port Use?](modscope.md#typemutex) + + [Do Any Other Changes Need to be Made?](modscope.md#otherchanges) + + [Building on the Target Platform](buildtarget.md) + + [Source Code Layout](layout.md) + + [3. Testing and Certifying the Port](testport.md) + + [Types of Tests for Berkeley DB](testport.md#testtypes) + + [Modifying the Tests](modifytest.md) + + [Running the Tests](testrun.md) + + [Reviewing the Results of the Tests](testreview.md) + + [Integrating Changes into the Berkeley DB Source Code](sourceintegrate.md) + + [Certifying a Port of Berkeley DB](certport.md) diff --git a/docs-src/guides/porting/introduction.md b/docs-src/guides/porting/introduction.md new file mode 100644 index 000000000..b6e689324 --- /dev/null +++ b/docs-src/guides/porting/introduction.md @@ -0,0 +1,56 @@ +--- +title: "Chapter 1. Introduction to Porting Berkeley DB" +api-name: "Chapter 1. Introduction to Porting Berkeley DB" +source: docs/porting/introduction.html +--- +## Chapter 1. Introduction to Porting Berkeley DB + +**Table of Contents** + + [Types of Berkeley DB ports](introduction.md#porttypes) + + [When Oracle Has Agreed to Support Berkeley DB on the New Platform](introduction.md#idp139688) + + [When Oracle has Not Agreed to Support Berkeley DB on the New Platform](introduction.md#idp79768) + + [Berkeley DB Porting Process](portprocess.md) + +Berkeley DB is an open source database product that supports a variety of platforms. When there is a need to run Berkeley DB on a platform that is currently not supported, DB is distributed in source code form that you can use as base source to port Berkeley DB to that platform. + +Berkeley DB is designed to be as portable as possible, and has been ported to a wide variety of systems, from Wind River's Tornado system, to VMS, to Windows/NT and Windows/95, and most existing UNIX platforms. It runs on 16-bit, 32-bit, and 64-bit machines, little or big-endian. The difficulty of a port depends on how much of the ANSI C and POSIX 1003.1 standards the new architecture offers. + +Before you begin actually porting Berkeley DB, you need an understanding of the: + +- Types of Berkeley DB ports + +- Berkeley DB Porting Process + +## Types of Berkeley DB ports + + [When Oracle Has Agreed to Support Berkeley DB on the New Platform](introduction.md#idp139688) + + [When Oracle has Not Agreed to Support Berkeley DB on the New Platform](introduction.md#idp79768) + +There are several types of Berkeley DB ports: + +- Ports developed and supported by Oracle + +- Ports developed by a customer or a partner, but which Oracle has agreed to support. + +- Ports developed, maintained, and supported by a customer or partner. + +For a port developed by a customer or a partner, the general steps for porting Berkeley DB to a new platform are the same whether or not Oracle has agreed to support Berkeley DB on the new platform. For example, after you complete the port you send it to Berkeley DB as described in Integrating Changes into the Berkeley DB Source Code. However, there are some differences. + +### When Oracle Has Agreed to Support Berkeley DB on the New Platform + +When porting Berkeley DB to a platform that Oracle has agreed to support, you need to have Berkeley DB engineering review your port at various points. These review points are discussed more fully in Integrating Changes into the Berkeley DB Source Code, Modifying the Tests, and Reviewing the Results of the Tests. + +It is up to you to submit the results of the tests (test_micro, test_mutex, and, if possible, the entire tcl test suit) for review by Oracle Berkelely DB engineering in order for Oracle to consider providing support for Berkeley DB on a new platform. + +You must also assign copyrights for your changes to any part of Berkeley DB to "Oracle Corporation" and attest to the fact that you are not infringing on any software patents for the changes to be included in the general Berekely DB distribution. + +Once the port is certified, Oracle provides support for Berkeley DB on the new platform in the same manner that it does for Berkeley DB running on other established platforms. + +### When Oracle has Not Agreed to Support Berkeley DB on the New Platform + +When Oracle has *not* agreed to support Berkeley DB on the new platform, the customer or partner assume the responsibility of front-line support. When it is determined that there is a problem in the code that was not modified by the customer or partner, then Berkeley DB engineering provides support to the customer or vendor who implemented the port, However, in these cases, Oracle needs access to the platform and hardware for diagnosing, debugging, and testing. diff --git a/docs-src/guides/porting/layout.md b/docs-src/guides/porting/layout.md new file mode 100644 index 000000000..da28aed68 --- /dev/null +++ b/docs-src/guides/porting/layout.md @@ -0,0 +1,24 @@ +--- +title: "Source Code Layout" +api-name: "Source Code Layout" +source: docs/porting/layout.html +--- +## Source Code Layout + +The following table describes the directories in the Berkeley DB distribution. + +| Directory | Description | +|----|----| +| LICENSE | Berkeley DB License | +| build_android | Android build directory | +| build_unix | UNIX build directory | +| build_vxworks | VxWorks build directory | +| build_wince | Windows CE build directory | +| build_windows | Windows build directory | +| dist | Scripts used to auto-generate code for Berkeley DB distribution and administration | +| docs | Documentation | +| examples | Example programs for various language APIs | +| lang | Implementation of various APIs that work with the Berkeley DB library | +| src | Implementation of the Berkeley DB library | +| test | Test suites | +| util | Implementation of utilities that can be used with Berkeley DB | diff --git a/docs-src/guides/porting/modifytest.md b/docs-src/guides/porting/modifytest.md new file mode 100644 index 000000000..d59ec3e87 --- /dev/null +++ b/docs-src/guides/porting/modifytest.md @@ -0,0 +1,10 @@ +--- +title: "Modifying the Tests" +api-name: "Modifying the Tests" +source: docs/porting/modifytest.html +--- +## Modifying the Tests + +There should be no need to make modifications to the tests. However, in a few situations, the test hardware may have some constraints (for example, small amount of memory) which may cause certain tests (which expect more memory) to fail. In these rare situations, you may need to modify the tests to work within the constraints of the platform. + +When Oracle has agreed to support Berkeley DB on the new platform, submit any proposed test changes for review and approval by Oracle Engineering before running the tests. diff --git a/docs-src/guides/porting/modscope.md b/docs-src/guides/porting/modscope.md new file mode 100644 index 000000000..f270a9212 --- /dev/null +++ b/docs-src/guides/porting/modscope.md @@ -0,0 +1,201 @@ +--- +title: "Determining the Scope of the Modifications" +api-name: "Determining the Scope of the Modifications" +source: docs/porting/modscope.html +--- +## Determining the Scope of the Modifications + + [Do Changes Need to be Made to the Operating System Functionality?](modscope.md#osfunc) + + [Are Some Standard Functions Missing on the Target Platform?](modscope.md#standardfunc) + + [How Will the Port Handle Shared Memory?](modscope.md#sharedmem) + + [What Type of Mutexes Will the Port Use?](modscope.md#typemutex) + + [Do Any Other Changes Need to be Made?](modscope.md#otherchanges) + +Once you have a good build of Berkeley DB on a UNIX or UNIX-like system, look over the code to determine what type of code changes you need to make so that you can successfully build Berkeley DB on your target system. This process involves determining: + +- Do Changes Need to be Made to the Operating System Functionality? + +- Are Some Standard Functions Missing on the Target Platform? + +- How Will the Port Handle Shared Memory? + +- What Type of Mutexes Will the Port Use? + +- Do Any Other Changes Need to be Made? + +### Do Changes Need to be Made to the Operating System Functionality? + +Berkeley DB uses about forty operating system primitives. The Berkeley DB distribution contains files which are wrappers around these operating system primitives that act as an abstraction layer to separate the main Berkeley DB code from operating system and architecture-specific components. You *must* port these files (or versions of these files) whenever you port Berkeley DB to a new platform. + +Within a Berkeley DB distribution, typically, there is only a single version of these files for all platforms that Berkeley DB supports. Those versions of the files live in the `os` directory of the distribution and follow the ANSI C and POSIX 1003.1 standards. Within each file, there is usually one, but sometimes several functions (for example, the `os_alloc.c` file contains functions such as `malloc`, `realloc`, `strdup`, `free`). The following table describes the files in the os directory of the Berkeley DB distribution along with the POSIX functions that must be ported. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

POSIX Functions

Internal Function Name

Source File

abort() is required if diagnostic build is used or if snprintf is not provided by the platform

__os_abort()

os_abort.c

freeaddrinfo()

+

getaddrinfo(), htonl(), htons(), inet_addr(), and gethostbyname() are required for Replication Manager

__os_getaddrinfo(), __os_freeaddrinfo()

os_addrinfo.c

malloc(), realloc(), strdup(), free(), memcpy(), memset(), strlen()

__os_umalloc(), __os_urealloc(), __os_ufree(), __os_strdup(), __os_calloc(), __os_malloc(), __os_realloc(), __os_free(), __os_guard(), __ua_memcpy()

os_alloc.c

clock_gettime(), time(), gettimeofday()

__os_gettime()

os_clock.c

sysconf()

__os_cpu_count()

os_cpu.c

ctime(), ctime_r()

__os_ctime()

os_ctime.c

opendir(), closdir(), readdir(), stat()

__os_dirlist(), __os_dirfree()

os_dir.c

strncpy()

__os_get_errno_ret_zero(), __os_get_errno(), __os_get_syserr(), __os_set_errno(), __os_strerror(), __os_posix_err()

os_errno.c

fcntl() is required for DB_REGISTER

__os_fdlock()

os_flock.c

fsync(), fdatasync()

__vx_fsync(), __os_fsync()

os_fsync.c

getenv() and strcpy() are required when environment variables are used to configure the database

__os_getenv()

os_getenv.c

close(), open()

__os_openhandle(), __os_closehandle()

os_handle.c

getpid()

+

pthread_self() is required for replication and failchk functionality

__os_id()

os_pid.c

shmget(), shmdt(), shmctl(), and shmat() are required when envrionment uses share memory for regions

+

munmap() is required when envrionment uses memory mapped files for regions or read-only databases

+

munlock() is required when environment is configured with DB_LOCKDOWN

__os_attach(), __os_detach(), __os_mapfile(), __os_unmapfile(), __os_map(), __shm_mode(), __no_system_mem()

os_map.c

mkdir(), chmod()

__os_mkdir()

os_mkdir.c

fchmod()

+

directio() is required when explicitly enabling DIRECTIO_ON

__os_open()

os_open.c

rename()

__os_rename()

os_rename.c

getuid() is required when environment variables are used to configure the database

__os_isroot()

os_root.c

read(), write(), pread(), pwrite()

__os_io(), __os_read(), __os_write(), __os_physwrite()

os_rw.c

lseek()

__os_seek()

os_seek.c

stat(), fstat()

__os_exists(), __os_ioinfo()

os_stat.c

ftruncate() is required when using truncate

__os_truncate()

os_truncate.c

unlink()

__os_unlink()

os_unlink.c

yield(), sched_yield()

__os_yield(), __os_sleep()

os.yield.c

+ +When the operating system primitives on the target platform are identical or close to the POSIX semantics that Berkeley DB requires, then no code changes or minimal code changes to the files in the `os` directory are required. If the operating system primitives are quite different, then some code changes may be required to bridge the gap between the requirements of Berkeley DB and what the operating system provides. + +Where different code is required, you write an entirely different version of the file and place it in an `os`\_*xxx* directory where *xxx* represents a platform name. There are `os`\_*xxx* subdirectories in the Berkeley DB distribution for several established non-POSIX platforms. For example, there is a `os_vxworks` directory that contains VxWorks versions of some of the files in the os directory, and Windows versions of some files are in the `os_windows` directory. If your target platform needs a different version of a file, you will need to write that file and place it in a new `os`\_*xxx* directory that you create for your target platform. + +### Are Some Standard Functions Missing on the Target Platform? + +In some cases, the target platform may not provide the few POSIX functions required by Berkeley DB or the functions provided by the target platform may not operate in a standard compliant way. Berkeley DB provides replacement functions in the `clib` directory of the Berkeley DB distribution. + +You need to determine how your target platfrom handles these functions: + +- When the target platform does *not* have a POSIX function required by Berkeley DB, no action is required on your part. When Berekely DB cannot find one of these functions on the target platform, it automatically uses the replacement functions supplied in the `clib` directory of the Berkeley DB distribution. For example, if the target platform does not have the `atoi` or `strtol` functions, Berkeley DB uses `clib/atoi.c` and `clib/strtol.c`. + +- When the target platform has a function required by Berekely DB, but that function operates in a non-standard compliant way, you can code to the replacement functions supplied in the `clib` directory. + +### How Will the Port Handle Shared Memory? + +In order to write multiprocess database applications (not multithreaded, but threads of control running in different address spaces), Berkeley DB must be able to name pieces of shared memory and access them from multiple processes. + +On UNIX/POSIX systems, Berkeley DB uses `mmap` and `shmget` for that purpose, but any interface that provides access to named shared memory is sufficient. If you have a simple, flat address space, you should be able to use the code in `os_vxworks/os_map.c` as a starting point for the port. + +If you are not intending to write multiprocess database applications, then this won't be necessary, as Berkeley DB can simply allocate memory from the heap if all threads of control will live in a single address space. + +### What Type of Mutexes Will the Port Use? + +Berkeley DB requires some form of self-blocking mutual exclusion mutex. Blocking mutexes are preferred as they tend to be less CPU-expensive and less likely to cause thrashing. If blocking mutexes are not available, however, test-and-set will work as well. The code for mutexes is in two places in the system: the include file `dbinc/mutex_int.h`, and the distribution directory `mutex`. + +### Do Any Other Changes Need to be Made? + +In most cases, you do not need to make any changes to the Berkeley DB source code that is not in the abstraction layer (that is, in the `os` directory) as that code is designed to be platform-independent code. However, in some situations, the compiler for the target platform is non-standard and may raise errors when compiling some aspects of the Berkeley DB code (for example, additional casting may be required, or a certain type may cause a problem). In these cases, you will need to modify the generic Berkeley DB code in order to have error-free compilation. diff --git a/docs-src/guides/porting/moreinfo.md b/docs-src/guides/porting/moreinfo.md new file mode 100644 index 000000000..ecb3b4c26 --- /dev/null +++ b/docs-src/guides/porting/moreinfo.md @@ -0,0 +1,34 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/porting/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Getting Started with Berkeley DB for C + +- Getting Started with Transaction Processing for C + +- Berkeley DB Getting Started with Replicated Applications for C + +- Berkeley DB Installation and Build Guide + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Getting Started with the SQL APIs + +- Berkeley DB C API Reference Guide + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/porting/newbinary.md b/docs-src/guides/porting/newbinary.md new file mode 100644 index 000000000..a1d717eab --- /dev/null +++ b/docs-src/guides/porting/newbinary.md @@ -0,0 +1,46 @@ +--- +title: "Chapter 2. Creating a New Berkeley DB Binary" +api-name: "Chapter 2. Creating a New Berkeley DB Binary" +source: docs/porting/newbinary.html +--- +## Chapter 2. Creating a New Berkeley DB Binary + +**Table of Contents** + + [Creating a Base Build of Berkeley DB](newbinary.md#basebuild) + + [Determining the Scope of the Modifications](modscope.md) + + [Do Changes Need to be Made to the Operating System Functionality?](modscope.md#osfunc) + + [Are Some Standard Functions Missing on the Target Platform?](modscope.md#standardfunc) + + [How Will the Port Handle Shared Memory?](modscope.md#sharedmem) + + [What Type of Mutexes Will the Port Use?](modscope.md#typemutex) + + [Do Any Other Changes Need to be Made?](modscope.md#otherchanges) + + [Building on the Target Platform](buildtarget.md) + + [Source Code Layout](layout.md) + +Creating a new Berkeley DB executable on the target platform, involves: + +1. Creating a Base Build of Berkeley DB + +2. Determining the Scope of the Modifications + +3. Building on the Target Platform + +## Creating a Base Build of Berkeley DB + +The simplest way to begin a port is to attempt to configure and build Berkeley DB on a UNIX or UNIX-like system. This gives you a list of the files that you needed to build Berkeley DB as well as the configuration files you can use as a starting point for building on your target port. + +To create a base build of Berkeley DB, following the instructions in the *Berkeley DB Programmer's Reference Guide*: + +1. Download a Berkeley DB distribution from http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +2. Build Berkeley DB. + +Berkeley DB uses the GNU autoconf tools for configuration on almost all of the platforms it supports. Specifically, the include file `db_config.h` configures the Berkeley DB build. The simplest way to begin a port is to configure and build Berkeley DB on a UNIX or UNIX-like system, and then take the `Makefile` and `db_config.h` file created by that configuration, and modify it by hand to reflect the needs of the new architecture. Unless you are already familiar with the GNU autoconf toolset, we do not recommend you take the time to integrate your changes back into the Berkeley DB autoconfiguration framework. Instead, send us context diffs of your changes and any new source files you created, and we can integrate the changes into our source tree. diff --git a/docs-src/guides/porting/portprocess.md b/docs-src/guides/porting/portprocess.md new file mode 100644 index 000000000..1359bc477 --- /dev/null +++ b/docs-src/guides/porting/portprocess.md @@ -0,0 +1,12 @@ +--- +title: "Berkeley DB Porting Process" +api-name: "Berkeley DB Porting Process" +source: docs/porting/portprocess.html +--- +## Berkeley DB Porting Process + +As with any porting project, porting Berkeley DB to a new platform consists of the following process: + +1. Determine the modifications that you need to make in the base code and create a working executable of Berkeley DB on the target platform as described in Creating a New Berkeley DB Binary. + +2. Perform final test and quality assurance functions on the target platform as described in Testing and Certifying the Port. diff --git a/docs-src/guides/porting/preface.md b/docs-src/guides/porting/preface.md new file mode 100644 index 000000000..1fb3cb0e4 --- /dev/null +++ b/docs-src/guides/porting/preface.md @@ -0,0 +1,48 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/porting/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [Audience](audience.md) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +The Berkeley DB family of open source, embeddable databases provides developers with fast, reliable persistence with zero administration. Often deployed as "edge" databases, the Berkeley DB family provides very high performance, reliability, scalability, and availability for application use cases that do not require SQL. + +As an open source database, Berkeley DB works on many different platforms, from Wind River's Tornado system, to VMS, to Windows NT and Windows 95, and most existing UNIX platforms. It runs on 32 and 64-bit machines, little or big-endian. + +*Berkeley DB Porting Guide* provides the information you need to port Berkeley DB 11*g* Release 2 (library version 11.2.5.3) to additional platforms. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Structure names are represented in `monospaced font`, as are `method names`. For example: "`DB->open()` is a method on a `DB` handle." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/guides/porting/sourceintegrate.md b/docs-src/guides/porting/sourceintegrate.md new file mode 100644 index 000000000..1ea2cf224 --- /dev/null +++ b/docs-src/guides/porting/sourceintegrate.md @@ -0,0 +1,16 @@ +--- +title: "Integrating Changes into the Berkeley DB Source Code" +api-name: "Integrating Changes into the Berkeley DB Source Code" +source: docs/porting/sourceintegrate.html +--- +## Integrating Changes into the Berkeley DB Source Code + +Once you have have completed your port, send all code changes that you made as "diff" files to Berkeley DB development for review. + +Exactly how you integrate changes into the Berkeley DB source code varies depending on whether or not Oracle has agreed to support Berkeley DB on the new platform: + +At this point, when Oracle has agreed to support Berkeley DB on the new platform: + +1. Berkeley DB development reviews the changes, makes the changes (with modifications if necessary) in the source code, creates another snapshot which includes the new platform-specific changes, and sends you the new snapshot. + +2. On receipt of the new snapshot, recompile this new snapshot. If the new snapshot compiles correctly without any changes, test the port again. diff --git a/docs-src/guides/porting/testport.md b/docs-src/guides/porting/testport.md new file mode 100644 index 000000000..8d06299bc --- /dev/null +++ b/docs-src/guides/porting/testport.md @@ -0,0 +1,52 @@ +--- +title: "Chapter 3. Testing and Certifying the Port" +api-name: "Chapter 3. Testing and Certifying the Port" +source: docs/porting/testport.html +--- +## Chapter 3. Testing and Certifying the Port + +**Table of Contents** + + [Types of Tests for Berkeley DB](testport.md#testtypes) + + [Modifying the Tests](modifytest.md) + + [Running the Tests](testrun.md) + + [Reviewing the Results of the Tests](testreview.md) + + [Integrating Changes into the Berkeley DB Source Code](sourceintegrate.md) + + [Certifying a Port of Berkeley DB](certport.md) + +There are several different types of tests available for validating your port of Berkeley DB as discussed in Types of Tests for Berkeley DB. Testing your port involves: + +- Modifying the Tests + +- Running the Tests + +- Reviewing the Results of the Tests + +- Integrating Changes into the Berkeley DB Source Code + +- Certifying a Port of Berkeley DB + +## Types of Tests for Berkeley DB + +There are two types of tests available for testing your port of Berkeley DB: + +- The C Tests for Berkeley DB + + There are two types of C tests for Berkeley DB. Each of these is in its own directory: + + - `test_mutex` contains files that test the use of mutexes in Berkeley DB. + + - `test_micro` contains the C tests that exercise the most common code paths, but it is not intended to be an exhaustive Test Suite. Additionally, it tests the different versions of Berkeley DB (including the new port) against each other. The `test_micro` tests can either be run in a shell or as simple C tests. + +- The Berkeley DB Test Suite + + The `test` directory contains the Berkeley DB Test Suite that tests all of the code in Berkeley DB. Using the Test Suite involves using Tool Command Language (Tcl) version 8.5 or later. Running the standard version of the Test Suite executes tests the major functionality of Berkeley DB. A more exhaustive version of the Test Suite runs all the tests several more times, testing encryption, replication, and different page sizes. + +### Note + +Contact the Oracle Berkelely DB engineering team for a platform compatibility test suite. diff --git a/docs-src/guides/porting/testreview.md b/docs-src/guides/porting/testreview.md new file mode 100644 index 000000000..8f6af8d5d --- /dev/null +++ b/docs-src/guides/porting/testreview.md @@ -0,0 +1,10 @@ +--- +title: "Reviewing the Results of the Tests" +api-name: "Reviewing the Results of the Tests" +source: docs/porting/testreview.html +--- +## Reviewing the Results of the Tests + +It is up to you to submit the results of the tests (`test_micro`, `test_mutex`, and, if possible, the entire tcl test suit) for review by Oracle Berkelely DB engineering in order for Oracle to consider providing support for Berkeley DB on a new platform. + +When Oracle has *not* agreed to support Berkeley DB on the new platform, you are responsible for ensuring that the tests run successfully. diff --git a/docs-src/guides/porting/testrun.md b/docs-src/guides/porting/testrun.md new file mode 100644 index 000000000..0eb4d9f47 --- /dev/null +++ b/docs-src/guides/porting/testrun.md @@ -0,0 +1,20 @@ +--- +title: "Running the Tests" +api-name: "Running the Tests" +source: docs/porting/testrun.html +--- +## Running the Tests + +You test your new port of Berkeley DB by running the tests in the following order: + +1. Run the C tests in the following order: + + 1. Tests for mutexes located in the `test_mutex` directory. To run the tests, follow the instructions in the `test_mutex/readme` file. + + 2. Tests for the common code paths located in the `test_micro` directory. To run the tests in a shell script, follow the instructions in the `test_micro/readme` file. To run the tests as simple C tests, follow the instructions in the `test_micro/readme_embedded` file. + +2. If the target platform supports the use of Tcl (version 8.5 or later), run the Test Suite. How you run the Test Suite varies depending on the target platform: + + - If the target platform supports a UNIX-like version of Tcl, then set up Tcl and build the Test Suite as described in "Running the Test Suite under UNIX" in *Berkeley DB Installation and Build Guide* at http://download.oracle.com/docs/cd/E17076_02/html/installation/build_unix_test.html and, then, run the test suite. + + - If the target platform supports a Windows-like version of Tcl, then setup Tcl, and build and run the Test Suite as described in "Running the Test Suite under Windows" in *Berkeley DB Programmer's Reference Guide* at http://download.oracle.com/docs/cd/E17076_02/html/installation/build_win_test.html diff --git a/docs-src/guides/programmer_reference/_meta.toml b/docs-src/guides/programmer_reference/_meta.toml new file mode 100644 index 000000000..622f16342 --- /dev/null +++ b/docs-src/guides/programmer_reference/_meta.toml @@ -0,0 +1,208 @@ +# Nav/index metadata for the programmer_reference guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Berkeley DB Programmer's Reference Guide" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "intro", + "intro_terrain", + "intro_dbis", + "intro_dbisnot", + "intro_need", + "intro_what", + "intro_distrib", + "intro_where", + "intro_products", + "am_conf", + "am_conf_select", + "am_conf_logrec", + "general_am_conf", + "bt_conf", + "hash_conf", + "heap_conf", + "rq_conf", + "am", + "am_opensub", + "am_partition", + "am_get", + "am_put", + "am_delete", + "am_stat", + "am_truncate", + "am_upgrade", + "am_verify", + "am_sync", + "am_close", + "am_second", + "am_foreign", + "am_cursor", + "am_misc", + "am_misc_bulk", + "am_misc_partial", + "am_misc_struct", + "am_misc_perm", + "am_misc_error", + "am_misc_stability", + "am_misc_dbsizes", + "am_misc_diskspace", + "am_misc_db_sql", + "am_misc_tune", + "am_misc_faq", + "java", + "java_compat", + "java_program", + "java_faq", + "csharp", + "stl", + "stl_usecase", + "stl_examples", + "stl_db_usage", + "stl_db_advanced_usage", + "stl_txn_usage", + "stl_mt_usage", + "stl_primitive_rw", + "stl_complex_rw", + "stl_persistence", + "stl_container_specific", + "stl_efficienct_use", + "stl_memory_mgmt", + "stl_misc", + "stl_known_issues", + "arch", + "arch_progmodel", + "arch_apis", + "arch_script", + "arch_utilities", + "env", + "env_create", + "env_size", + "env_open", + "env_error", + "env_db_config", + "env_naming", + "env_region", + "env_security", + "env_encrypt", + "env_remote", + "env_faq", + "cam", + "cam_fail", + "cam_app", + "transapp", + "transapp_why", + "transapp_term", + "transapp_fail", + "transapp_app", + "transapp_env_open", + "transapp_data_open", + "transapp_put", + "transapp_atomicity", + "transapp_inc", + "transapp_read", + "transapp_cursor", + "transapp_nested", + "transapp_admin", + "transapp_deadlock", + "transapp_checkpoint", + "transapp_archival", + "transapp_logfile", + "transapp_recovery", + "transapp_hotfail", + "transapp_journal", + "transapp_filesys", + "transapp_reclimit", + "transapp_tune", + "transapp_throughput", + "transapp_faq", + "rep", + "rep_id", + "rep_pri", + "rep_app", + "rep_mgr_meth", + "rep_base_meth", + "rep_comm", + "rep_newsite", + "group_membership", + "rep_filename", + "rep_mgrmulti", + "rep_replicate", + "rep_mgr_ack", + "rep_elect", + "rep_mastersync", + "rep_init", + "rep_bulk", + "rep_trans", + "rep_lease", + "rep_ryw", + "rep_clock_skew", + "repmgr_channels", + "rep_twosite", + "rep_partition", + "rep_faq", + "rep_ex", + "rep_ex_comm", + "rep_ex_rq", + "rep_ex_chan", + "xa", + "ch13s02", + "xa_build", + "xa_xa_intro", + "xa_xa_config", + "xa_xa_restrict", + "xa_faq", + "apprec", + "apprec_def", + "apprec_auto", + "apprec_config", + "program", + "program_errorret", + "program_environ", + "program_mt", + "program_scope", + "program_namespace", + "program_ram", + "program_cache", + "program_copy", + "program_compatible", + "program_runtime", + "program_perfmon", + "program_faq", + "lock", + "lock_config", + "lock_max", + "lock_stdmode", + "lock_dead", + "lock_timeout", + "lock_deaddbg", + "lock_page", + "lock_notxn", + "lock_twopl", + "lock_cam_conv", + "lock_am_conv", + "lock_nondb", + "log", + "log_config", + "log_limits", + "mp", + "mp_config", + "mp_warm", + "txn", + "txn_config", + "txn_limits", + "sequence", + "tcl", + "tcl_using", + "tcl_program", + "tcl_error", + "tcl_faq", + "ext", + "ext_perl", + "ext_php", + "dumpload", + "dumpload_format", + "dumpload_text", + "refs", +] diff --git a/docs-src/guides/programmer_reference/am.md b/docs-src/guides/programmer_reference/am.md new file mode 100644 index 000000000..8d6346b8a --- /dev/null +++ b/docs-src/guides/programmer_reference/am.md @@ -0,0 +1,110 @@ +--- +title: "Chapter 3.  Access Method Operations" +api-name: "Chapter 3.  Access Method Operations" +source: docs/programmer_reference/am.html +--- +## Chapter 3.  Access Method Operations + +**Table of Contents** + + [Database open](am.md#am_open) + + [Opening multiple databases in a single file](am_opensub.md) + + [Configuring databases sharing a file](am_opensub.md#idp50943544) + + [Caching databases sharing a file](am_opensub.md#idp50944288) + + [Locking in databases based on sharing a file](am_opensub.md#idp50944984) + + [Partitioning databases](am_partition.md) + + [Specifying partition keys](am_partition.md#am_partition_keys) + + [Partitioning callback](am_partition.md#am_partition_function) + + [Placing partition files](am_partition.md#partition_file_placement) + + [Retrieving records](am_get.md) + + [Storing records](am_put.md) + + [Deleting records](am_delete.md) + + [Database statistics](am_stat.md) + + [Database truncation](am_truncate.md) + + [Database upgrade](am_upgrade.md) + + [Database verification and salvage](am_verify.md) + + [Flushing the database cache](am_sync.md) + + [Database close](am_close.md) + + [Secondary indexes](am_second.md) + + [Error Handling With Secondary Indexes](am_second.md#idp51040080) + + [Foreign key indexes](am_foreign.md) + + [Cursor operations](am_cursor.md) + + [Retrieving records with a cursor](am_cursor.md#am_curget) + + [Storing records with a cursor](am_cursor.md#am_curput) + + [Deleting records with a cursor](am_cursor.md#am_curdel) + + [Duplicating a cursor](am_cursor.md#am_curdup) + + [Equality Join](am_cursor.md#am_join) + + [Data item count](am_cursor.md#am_count) + + [Cursor close](am_cursor.md#am_curclose) + +Once a database handle has been created using db_create(), there are several standard access method operations. Each of these operations is performed using a method referred to by the returned handle. Generally, the database will be opened using DB->open(). If the database is from an old release of Berkeley DB, it may need to be upgraded to the current release before it is opened using DB->upgrade(). + +Once a database has been opened, records may be retrieved (DB->get()), stored (DB->put()), and deleted (DB->del()). + +Additional operations supported by the database handle include statistics (DB->stat()), truncation (DB->truncate()), version upgrade (DB->upgrade()), verification and salvage (DB->verify()), flushing to a backing file (DB->sync()), and association of secondary indices (DB->associate()). Database handles are eventually closed using DB->close(). + +For more information on the access method operations supported by the database handle, see the Database and Related Methods section in the *Berkeley DB C API Reference Guide.* + +## Database open + +The DB->open() method opens a database, and takes five arguments: + +file +The name of the file to be opened. + +database +An optional database name. + +type +The type of database to open. This value will be one of the five access methods Berkeley DB supports: DB_BTREE, DB_HASH, DB_HEAP, DB_QUEUE or DB_RECNO, or the special value DB_UNKNOWN, which allows you to open an existing file without knowing its type. + +mode +The permissions to give to any created file. + +There are a few flags that you can set to customize open: + + DB_CREATE +Create the underlying database and any necessary physical files. + + DB_NOMMAP +Do not map this database into process memory. + + DB_RDONLY +Treat the data base as read-only. + + DB_THREAD +The returned handle is free-threaded, that is, it can be used simultaneously by multiple threads within the process. + + DB_TRUNCATE +Physically truncate the underlying database file, discarding all databases it contained. Underlying filesystem primitives are used to implement this flag. For this reason it is only applicable to the physical file and cannot be used to discard individual databases from within physical files. + + DB_UPGRADE +Upgrade the database format as necessary. diff --git a/docs-src/guides/programmer_reference/am_close.md b/docs-src/guides/programmer_reference/am_close.md new file mode 100644 index 000000000..828d6d793 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_close.md @@ -0,0 +1,21 @@ +--- +title: "Database close" +api-name: "Database close" +source: docs/programmer_reference/am_close.html +--- +## Database close + +The DB->close() database handle closes the DB handle. By default, DB->close() also flushes all modified records from the database cache to disk. + +There is one flag that you can set to customize DB->close(): + + DB_NOSYNC +Do not flush cached information to disk. + +**It is important to understand that flushing cached information to disk only minimizes the window of opportunity for corrupted data, it does not eliminate the possibility.** + +While unlikely, it is possible for database corruption to happen if a system or application crash occurs while writing data to the database. To ensure that database corruption never occurs, applications must either: + +- Use transactions and logging with automatic recovery. +- Use logging and application-specific recovery. +- Edit a copy of the database, and, once all applications using the database have successfully called DB->close(), use system operations (for example, the POSIX rename system call) to atomically replace the original database with the updated copy. diff --git a/docs-src/guides/programmer_reference/am_conf.md b/docs-src/guides/programmer_reference/am_conf.md new file mode 100644 index 000000000..db38be6d6 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_conf.md @@ -0,0 +1,110 @@ +--- +title: "Chapter 2.  Access Method Configuration" +api-name: "Chapter 2.  Access Method Configuration" +source: docs/programmer_reference/am_conf.html +--- +## Chapter 2.  Access Method Configuration + +**Table of Contents** + + [What are the available access methods?](am_conf.md#am_conf_intro) + + [Btree](am_conf.md#idp50599376) + + [Hash](am_conf.md#idp50705400) + + [Heap](am_conf.md#idp50708952) + + [Queue](am_conf.md#idm1385000) + + [Recno](am_conf.md#idp50715336) + + [Selecting an access method](am_conf_select.md) + + [Btree or Heap?](am_conf_select.md#idp50702528) + + [Hash or Btree?](am_conf_select.md#idp50755552) + + [Queue or Recno?](am_conf_select.md#idp50569200) + + [Logical record numbers](am_conf_logrec.md) + + [General access method configuration](general_am_conf.md) + + [Selecting a page size](general_am_conf.md#am_conf_pagesize) + + [Selecting a cache size](general_am_conf.md#am_conf_cachesize) + + [Selecting a byte order](general_am_conf.md#am_conf_byteorder) + + [Duplicate data items](general_am_conf.md#am_conf_dup) + + [Non-local memory allocation](general_am_conf.md#am_conf_malloc) + + [Btree access method specific configuration](bt_conf.md) + + [Btree comparison](bt_conf.md#am_conf_bt_compare) + + [Btree prefix comparison](bt_conf.md#am_conf_bt_prefix) + + [Minimum keys per page](bt_conf.md#am_conf_bt_minkey) + + [Retrieving Btree records by logical record number](bt_conf.md#am_conf_bt_recnum) + + [Compression](bt_conf.md#am_conf_bt_compress) + + [Hash access method specific configuration](hash_conf.md) + + [Page fill factor](hash_conf.md#am_conf_h_ffactor) + + [Specifying a database hash](hash_conf.md#am_conf_h_hash) + + [Hash table size](hash_conf.md#am_conf_h_nelem) + + [Heap access method specific configuration](heap_conf.md) + + [Queue and Recno access method specific configuration](rq_conf.md) + + [Managing record-based databases](rq_conf.md#am_conf_recno) + + [Selecting a Queue extent size](rq_conf.md#am_conf_extentsize) + + [Flat-text backing files](rq_conf.md#am_conf_re_source) + + [Logically renumbering records](rq_conf.md#am_conf_renumber) + +## What are the available access methods? + + [Btree](am_conf.md#idp50599376) + + [Hash](am_conf.md#idp50705400) + + [Heap](am_conf.md#idp50708952) + + [Queue](am_conf.md#idm1385000) + + [Recno](am_conf.md#idp50715336) + +Berkeley DB currently offers five access methods: Btree, Hash, Heap, Queue and Recno. + +### Btree + +The Btree access method is an implementation of a sorted, balanced tree structure. Searches, insertions, and deletions in the tree all take *O(height)* time, where *height* is the number of levels in the Btree from the root to the leaf pages. The upper bound on the height is *log base_b N*, where *base_b* is the smallest number of keys on a page, and *N* is the total number of keys stored. + +Inserting unordered data into a Btree can result in pages that are only half-full. DB makes ordered (or inverse ordered) insertion the best case, resulting in nearly full-page space utilization. + +### Hash + +The Hash access method data structure is an implementation of Extended Linear Hashing, as described in "Linear Hashing: A New Tool for File and Table Addressing", Witold Litwin, *Proceedings of the 6th International Conference on Very Large Databases (VLDB)*, 1980. + +### Heap + +The Heap access method stores records in a heap file. Records are referenced solely by the page and offset at which they are written. Because records are written in a heap file, compaction is not necessary when deleting records, which allows for more efficient use of space than if Btree is in use. The Heap access method is intended for platforms with constrained disk space, especially if those systems are performing a great many record creation and deletions. + +### Queue + +The Queue access method stores fixed-length records with logical record numbers as keys. It is designed for fast inserts at the tail and has a special cursor consume operation that deletes and returns a record from the head of the queue. The Queue access method uses record level locking. + +### Recno + +The Recno access method stores both fixed and variable-length records with logical record numbers as keys, optionally backed by a flat text (byte stream) file. diff --git a/docs-src/guides/programmer_reference/am_conf_logrec.md b/docs-src/guides/programmer_reference/am_conf_logrec.md new file mode 100644 index 000000000..e7aa40413 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_conf_logrec.md @@ -0,0 +1,82 @@ +--- +title: "Logical record numbers" +api-name: "Logical record numbers" +source: docs/programmer_reference/am_conf_logrec.html +--- +## Logical record numbers + +The Berkeley DB Btree, Queue and Recno access methods can operate on logical record numbers. Record numbers are 1-based, not 0-based, that is, the first record in a database is record number 1. + +In all cases for the Queue and Recno access methods, and when calling the Btree access method using the DB->get() and DBC->get() methods with the DB_SET_RECNO flag specified, the **data** field of the key DBT must be a pointer to a memory location of type **db_recno_t**, as typedef'd in the standard Berkeley DB include file. The **size** field of the key DBT should be the size of that type (for example, "sizeof(db_recno_t)" in the C programming language). The **db_recno_t** type is a 32-bit unsigned type, which limits the number of logical records in a Queue or Recno database, and the maximum logical record which may be directly retrieved from a Btree database, to 4,294,967,295. + +Record numbers in Recno databases can be configured to run in either mutable or fixed mode: mutable, where logical record numbers change as records are deleted or inserted, and fixed, where record numbers never change regardless of the database operation. Record numbers in Queue databases are always fixed, and never change regardless of the database operation. Record numbers in Btree databases are always mutable, and as records are deleted or inserted, the logical record number for other records in the database can change. See Logically renumbering records for more information. + +When appending new data items into Queue databases, record numbers wrap around. When the tail of the queue reaches the maximum record number, the next record appended will be given record number 1. If the head of the queue ever catches up to the tail of the queue, Berkeley DB will return the system error EFBIG. Record numbers do not wrap around when appending new data items into Recno databases. + +Configuring Btree databases to support record numbers can severely limit the throughput of applications with multiple concurrent threads writing the database, because locations used to store record counts often become hot spots that many different threads all need to update. In the case of a Btree supporting duplicate data items, the logical record number refers to a key and all of its data items, as duplicate data items are not individually numbered. + +The following is an example function that reads records from standard input and stores them into a Recno database. The function then uses a cursor to step through the database and display the stored records. + +``` c +int +recno_build(DB *dbp) +{ + DBC *dbcp; + DBT key, data; + db_recno_t recno; + u_int32_t len; + int ret; + char buf[1024]; + + /* Insert records into the database. */ + memset(&key, 0, sizeof(DBT)); + memset(&data, 0, sizeof(DBT)); + for (recno = 1;; ++recno) { + printf("record #%lu> ", (u_long)recno); + fflush(stdout); + if (fgets(buf, sizeof(buf), stdin) == NULL) + break; + if ((len = strlen(buf)) <= 1) + continue; + + key.data = &recno; + key.size = sizeof(recno); + data.data = buf; + data.size = len - 1; + + switch (ret = dbp->put(dbp, NULL, &key, &data, 0)) { + case 0: + break; + default: + dbp->err(dbp, ret, "DB->put"); + break; + } + } + printf("\n"); + + /* Acquire a cursor for the database. */ + if ((ret = dbp->cursor(dbp, NULL, &dbcp, 0)) != 0) { + dbp->err(dbp, ret, "DB->cursor"); + return (1); + } + + /* Re-initialize the key/data pair. */ + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + + /* Walk through the database and print out the key/data pairs. */ + while ((ret = dbcp->get(dbcp, &key, &data, DB_NEXT)) == 0) + printf("%lu : %.*s\n", + *(u_long *)key.data, (int)data.size, + (char *)data.data); + if (ret != DB_NOTFOUND) + dbp->err(dbp, ret, "DBcursor->get"); + + /* Close the cursor. */ + if ((ret = dbcp->close(dbcp)) != 0) { + dbp->err(dbp, ret, "DBcursor->close"); + return (1); + } + return (0); +} +``` diff --git a/docs-src/guides/programmer_reference/am_conf_select.md b/docs-src/guides/programmer_reference/am_conf_select.md new file mode 100644 index 000000000..1b1cfa5d2 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_conf_select.md @@ -0,0 +1,112 @@ +--- +title: "Selecting an access method" +api-name: "Selecting an access method" +source: docs/programmer_reference/am_conf_select.html +--- +## Selecting an access method + + [Btree or Heap?](am_conf_select.md#idp50702528) + + [Hash or Btree?](am_conf_select.md#idp50755552) + + [Queue or Recno?](am_conf_select.md#idp50569200) + +The Berkeley DB access method implementation unavoidably interacts with each application's data set, locking requirements and data access patterns. For this reason, one access method may result in dramatically better performance for an application than another one. Applications whose data could be stored using more than one access method may want to benchmark their performance using the different candidates. + +One of the strengths of Berkeley DB is that it provides multiple access methods with nearly identical interfaces to the different access methods. This means that it is simple to modify an application to use a different access method. Applications can easily benchmark the different Berkeley DB access methods against each other for their particular data set and access pattern. + +Most applications choose between using the Btree or Heap access methods, between Btree or Hash, or between Queue and Recno, because each of these pairs offer similar functionality. + +### Btree or Heap? + +Most applications use Btree because it performs well for most general-purpose database workloads. But there are circumstances where Heap is the better choice. This section describes the differences between the two access methods so that you can better understand when Heap might be the superior choice for your application. + +Before continuing, it is helpful to have a high level understanding of the operating differences between Btree and Heap. + +#### Disk Space Usage + +The Heap access method was developed for use in systems with constrained disk space (such as an embedded system). Because of the way it reuses page space, for some workloads it can be much better than Btree on disk space usage because it will not grow the on-disk database file as fast as Btree. Of course, this assumes that your application is characterized by a roughly equal number of record creations and deletions. + +Also, Heap can actively control the space used by the database with the use of the DB->set_heapsize() method. When the limit specified by that method is reached, no additional pages will be allocated and existing pages will be aggressively searched for free space. Also records in the heap can be split to fill space on two or more pages. + +#### Record Access + +Btree and Heap are fundamentally different because of the way that you access records in them. In Btree, you access a record by using the record's key. This lookup occurs fairly quickly because Btree places records in the database according to a pre-defined sorting order. Your application is responsible for constructing the key, which means that it is relatively easy for your application to know what key is in use by any given record. + +Conversely, Heap accesses records based on their offset location within the database. You retrieve a record in a Heap database using the record's Record ID (RID), which is created when the record is added to the database. The RID is created for you; you cannot specify this yourself. Because the RID is created for you, your application does not have control over the key value. For this reason, retrieval operations for a Heap database are usually performed using secondary databases. You can then use this secondary index to retrieve records stored in your Heap database. + +Note that an application's data access requirements grow complex, Btree databases also frequently require secondary databases. So at a certain level of complexity you will be using secondary databases regardless of the access method that you choose. + +Secondary databases are described in Secondary indexes. + +#### Record Creation/Deletion + +When Btree creates a new record, it places the record on the database page that is appropriate for the sorting order in use by the database. If Btree can not find a page to put the new record on, it locates a page that is in the proper location for the new record, splits it so that the existing records are divided between the two pages, and then adds the new record to the appropriate page. + +On deletion, Btree simply removes the deleted record from whatever page it is stored on. This leaves some amount of unused space ("holes") on the page. Only new records that sort to this page can fill that space. However, once a page is completely empty, it can be reused to hold records with a different sort value. + +In order to reclaim unused disk space, you must run the DB->compact() method, which attempts to fill holes in existing pages by moving records from other pages. If it is successful in moving enough records, it might be left with entire pages that have no data on them. In this event, the unused pages might be removed from the database (depending on the flags that you provide to DB->compact()), which causes the database file to be reduced in size. + +Both tree searching and page compaction are relatively expensive operations. Heap avoids these operations, and so is able to perform better under some circumstances. + +Heap does not care about record order. When a record is created in a Heap database, it is placed on the first page that has space to store the record. No sorting is involved, and so the overhead from record sorting is removed. + +On deletion, both Btree and Heap free space within a page when a record is deleted. However, unlike Btree, Heap has no compaction operation, nor does it have to wait for a record with the proper sort order to fill a hole on a page. Instead, Heap simply reuses empty page space whenever any record is added that will fit into the space. + +#### Cursor Operations + +When considering Heap, be aware that this access method does not support the full range of cursor operations that Btree does. + +- On sequential cursor scans of the database, the retrieval order of the records is not predictable for Heap because the records are not sorted. Btree, of course, sorts its records so the retrieval order is predictable. + +- When using a Heap database, you cannot create new records using a cursor. Also, this means that Heap does not support the DBC->put() `DB_AFTER` and `DB_BEFORE` flags. You can, however, update existing records using a cursor. + +- For concurrent applications, iterating through the records in a Heap database is not recommended due to performance considerations. This is because there is a good chance that there are a lot of empty pages in the database if you have a concurrent application. + + For a Heap database, entire regions are locked when a lock is acquired for a database page. If there is then contention for that region, and a new database page needs to be added, then Berkeley DB simply creates a whole new region. The locked region is then padded with empty pages in order to reach the new region. + + The result is that if the last used page in a region is 10, and a new region is created at page 100, then there are empty pages from 11 to 99. If you are iterating with a cursor, then all those empty pages must be examined by the cursor before it can reach the data at page 100. + +#### Which Access Method Should You Use? + +Ultimately, you can only determine which access method is superior for your application through performance testing using both access methods. To be effective, this performance testing must use a production-equivalent workload. + +That said, there are a few times when you absolutely must use Btree: + +- If you want to use bulk put and get operations. + +- If having your database clustered on sort order is important to you. + +- If you want to be able to create records using cursors. + +- If you have multiple threads/processes simultaneously creating new records, and you want to be able to efficiently iterate over those records using a cursor. + +But beyond those limitations, there are some application characteristics that should cause you to suspect that Heap will work better for your application than Btree. They are: + +- Your application will run in an environment with constrained resources and you want to set a hard limit on the size of the database file. + +- You want to limit the disk space growth of your database file, and your application performs a roughly equivalent number of record creations and deletions. + +- Inserts into a Btree require sorting the new record onto its proper page. This operation can require multiple page reads. A Heap database can simply reuse whatever empty page space it can find in the cache. Insert-intensive applications will typically find that Heap is much more efficient than Btree, especially as the size of the database increases. + +### Hash or Btree? + +The Hash and Btree access methods should be used when logical record numbers are not the primary key used for data access. (If logical record numbers are a secondary key used for data access, the Btree access method is a possible choice, as it supports simultaneous access by a key and a record number.) + +Keys in Btrees are stored in sorted order and the relationship between them is defined by that sort order. For this reason, the Btree access method should be used when there is any locality of reference among keys. Locality of reference means that accessing one particular key in the Btree implies that the application is more likely to access keys near to the key being accessed, where "near" is defined by the sort order. For example, if keys are timestamps, and it is likely that a request for an 8AM timestamp will be followed by a request for a 9AM timestamp, the Btree access method is generally the right choice. Or, for example, if the keys are names, and the application will want to review all entries with the same last name, the Btree access method is again a good choice. + +There is little difference in performance between the Hash and Btree access methods on small data sets, where all, or most of, the data set fits into the cache. However, when a data set is large enough that significant numbers of data pages no longer fit into the cache, then the Btree locality of reference described previously becomes important for performance reasons. For example, there is no locality of reference for the Hash access method, and so key "AAAAA" is as likely to be stored on the same database page with key "ZZZZZ" as with key "AAAAB". In the Btree access method, because items are sorted, key "AAAAA" is far more likely to be near key "AAAAB" than key "ZZZZZ". So, if the application exhibits locality of reference in its data requests, then the Btree page read into the cache to satisfy a request for key "AAAAA" is much more likely to be useful to satisfy subsequent requests from the application than the Hash page read into the cache to satisfy the same request. This means that for applications with locality of reference, the cache is generally much more effective for the Btree access method than the Hash access method, and the Btree access method will make many fewer I/O calls. + +However, when a data set becomes even larger, the Hash access method can outperform the Btree access method. The reason for this is that Btrees contain more metadata pages than Hash databases. The data set can grow so large that metadata pages begin to dominate the cache for the Btree access method. If this happens, the Btree can be forced to do an I/O for each data request because the probability that any particular data page is already in the cache becomes quite small. Because the Hash access method has fewer metadata pages, its cache stays "hotter" longer in the presence of large data sets. In addition, once the data set is so large that both the Btree and Hash access methods are almost certainly doing an I/O for each random data request, the fact that Hash does not have to walk several internal pages as part of a key search becomes a performance advantage for the Hash access method as well. + +Application data access patterns strongly affect all of these behaviors, for example, accessing the data by walking a cursor through the database will greatly mitigate the large data set behavior describe above because each I/O into the cache will satisfy a fairly large number of subsequent data requests. + +In the absence of information on application data and data access patterns, for small data sets either the Btree or Hash access methods will suffice. For data sets larger than the cache, we normally recommend using the Btree access method. If you have truly large data, then the Hash access method may be a better choice. The db_stat utility is a useful tool for monitoring how well your cache is performing. + +### Queue or Recno? + +The Queue or Recno access methods should be used when logical record numbers are the primary key used for data access. The advantage of the Queue access method is that it performs record level locking and for this reason supports significantly higher levels of concurrency than the Recno access method. The advantage of the Recno access method is that it supports a number of additional features beyond those supported by the Queue access method, such as variable-length records and support for backing flat-text files. + +Logical record numbers can be mutable or fixed: mutable, where logical record numbers can change as records are deleted or inserted, and fixed, where record numbers never change regardless of the database operation. It is possible to store and retrieve records based on logical record numbers in the Btree access method. However, those record numbers are always mutable, and as records are deleted or inserted, the logical record number for other records in the database will change. The Queue access method always runs in fixed mode, and logical record numbers never change regardless of the database operation. The Recno access method can be configured to run in either mutable or fixed mode. + +In addition, the Recno access method provides support for databases whose permanent storage is a flat text file and the database is used as a fast, temporary storage area while the data is being read or modified. diff --git a/docs-src/guides/programmer_reference/am_cursor.md b/docs-src/guides/programmer_reference/am_cursor.md new file mode 100644 index 000000000..6697ca0af --- /dev/null +++ b/docs-src/guides/programmer_reference/am_cursor.md @@ -0,0 +1,388 @@ +--- +title: "Cursor operations" +api-name: "Cursor operations" +source: docs/programmer_reference/am_cursor.html +--- +## Cursor operations + + [Retrieving records with a cursor](am_cursor.md#am_curget) + + [Storing records with a cursor](am_cursor.md#am_curput) + + [Deleting records with a cursor](am_cursor.md#am_curdel) + + [Duplicating a cursor](am_cursor.md#am_curdup) + + [Equality Join](am_cursor.md#am_join) + + [Data item count](am_cursor.md#am_count) + + [Cursor close](am_cursor.md#am_curclose) + +A database cursor refers to a single key/data pair in the database. It supports traversal of the database and is the only way to access individual duplicate data items. Cursors are used for operating on collections of records, for iterating over a database, and for saving handles to individual records, so that they can be modified after they have been read. + +The DB->cursor() method opens a cursor into a database. Upon return the cursor is uninitialized, cursor positioning occurs as part of the first cursor operation. + +Once a database cursor has been opened, records may be retrieved (DBC->get()), stored (DBC->put()), and deleted (DBC->del()). + +Additional operations supported by the cursor handle include duplication (DBC->dup()), equality join (DB->join()), and a count of duplicate data items (DBC->count()). Cursors are eventually closed using DBC->close(). + +For more information on the operations supported by the cursor handle, see the Database Cursors and Related Methods section in the *Berkeley DB C API Reference Guide.* + +### Retrieving records with a cursor + +The DBC->get() method retrieves records from the database using a cursor. The DBC->get() method takes a flag which controls how the cursor is positioned within the database and returns the key/data item associated with that positioning. Similar to DB->get(), DBC->get() may also take a supplied key and retrieve the data associated with that key from the database. There are several flags that you can set to customize retrieval. + +#### Cursor position flags + +DB_FIRST, DB_LAST +Return the first (last) record in the database. + +DB_NEXT, DB_PREV +Return the next (previous) record in the database. + + DB_NEXT_DUP +Return the next record in the database, if it is a duplicate data item for the current key. For Heap databases, this flag always results in the cursor returning the `DB_NOTFOUND` error. + +DB_NEXT_NODUP, DB_PREV_NODUP +Return the next (previous) record in the database that is not a duplicate data item for the current key. + + DB_CURRENT +Return the record from the database to which the cursor currently refers. + +#### Retrieving specific key/data pairs + + DB_SET +Return the record from the database that matches the supplied key. In the case of duplicates the first duplicate is returned and the cursor is positioned at the beginning of the duplicate list. The user can then traverse the duplicate entries for the key. + + DB_SET_RANGE +Return the smallest record in the database greater than or equal to the supplied key. This functionality permits partial key matches and range searches in the Btree access method. + + DB_GET_BOTH +Return the record from the database that matches both the supplied key and data items. This is particularly useful when there are large numbers of duplicate records for a key, as it allows the cursor to easily be positioned at the correct place for traversal of some part of a large set of duplicate records. + + DB_GET_BOTH_RANGE +If used on a database configured for sorted duplicates, this returns the smallest record in the database greater than or equal to the supplied key and data items. If used on a database that is *not* configured for sorted duplicates, this flag behaves identically to `DB_GET_BOTH`. + +#### Retrieving based on record numbers + + DB_SET_RECNO +If the underlying database is a Btree, and was configured so that it is possible to search it by logical record number, retrieve a specific record based on a record number argument. + + DB_GET_RECNO +If the underlying database is a Btree, and was configured so that it is possible to search it by logical record number, return the record number for the record to which the cursor refers. + +#### Special-purpose flags + + DB_CONSUME +Read-and-delete: the first record (the head) of the queue is returned and deleted. The underlying database must be a Queue. + + DB_RMW +Read-modify-write: acquire write locks instead of read locks during retrieval. This can enhance performance in threaded applications by reducing the chance of deadlock. + +In all cases, the cursor is repositioned by a DBC->get() operation to point to the newly-returned key/data pair in the database. + +The following is a code example showing a cursor walking through a database and displaying the records it contains to the standard output: + +``` c +int +display(char *database) + +{ + DB *dbp; + DBC *dbcp; + DBT key, data; + int close_db, close_dbc, ret; + + close_db = close_dbc = 0; + + /* Open the database. */ + if ((ret = db_create(&dbp, NULL, 0)) != 0) { + fprintf(stderr, + "%s: db_create: %s\n", progname, db_strerror(ret)); + return (1); + } + close_db = 1; + + /* Turn on additional error output. */ + dbp->set_errfile(dbp, stderr); + dbp->set_errpfx(dbp, progname); + + /* Open the database. */ + if ((ret = dbp->open(dbp, NULL, database, NULL, + DB_UNKNOWN, DB_RDONLY, 0)) != 0) { + dbp->err(dbp, ret, "%s: DB->open", database); + goto err; + } + + /* Acquire a cursor for the database. */ + if ((ret = dbp->cursor(dbp, NULL, &dbcp, 0)) != 0) { + dbp->err(dbp, ret, "DB->cursor"); + goto err; + } + close_dbc = 1; + + /* Initialize the key/data return pair. */ + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + + /* Walk through the database and print out the key/data pairs. */ + while ((ret = dbcp->get(dbcp, &key, &data, DB_NEXT)) == 0) + printf("%.*s : %.*s\n", + (int)key.size, (char *)key.data, + (int)data.size, (char *)data.data); + if (ret != DB_NOTFOUND) { + dbp->err(dbp, ret, "DBcursor->get"); + goto err; + } + +err: if (close_dbc && (ret = dbcp->close(dbcp)) != 0) + dbp->err(dbp, ret, "DBcursor->close"); + if (close_db && (ret = dbp->close(dbp, 0)) != 0) + fprintf(stderr, + "%s: DB->close: %s\n", progname, db_strerror(ret)); + return (0); +} +``` + +### Storing records with a cursor + +The DBC->put() method stores records into the database using a cursor. In general, DBC->put() takes a key and inserts the associated data into the database, at a location controlled by a specified flag. + +There are several flags that you can set to customize storage: + + DB_AFTER +Create a new record, immediately after the record to which the cursor refers. + + DB_BEFORE +Create a new record, immediately before the record to which the cursor refers. + + DB_CURRENT +Replace the data part of the record to which the cursor refers. + + DB_KEYFIRST +Create a new record as the first of the duplicate records for the supplied key. + + DB_KEYLAST +Create a new record, as the last of the duplicate records for the supplied key. + +In all cases, the cursor is repositioned by a DBC->put() operation to point to the newly inserted key/data pair in the database. + +The following is a code example showing a cursor storing two data items in a database that supports duplicate data items: + +``` c +int +store(DB *dbp) + +{ + DBC *dbcp; + DBT key, data; + int ret; + + /* + * The DB handle for a Btree database supporting duplicate data + * items is the argument; acquire a cursor for the database. + */ + if ((ret = dbp->cursor(dbp, NULL, &dbcp, 0)) != 0) { + dbp->err(dbp, ret, "DB->cursor"); + goto err; + } + + /* Initialize the key. */ + memset(&key, 0, sizeof(key)); + key.data = "new key"; + key.size = strlen(key.data) + 1; + + /* Initialize the data to be the first of two duplicate records. */ + memset(&data, 0, sizeof(data)); + data.data = "new key's data: entry #1"; + data.size = strlen(data.data) + 1; + + /* Store the first of the two duplicate records. */ + if ((ret = dbcp->put(dbcp, &key, &data, DB_KEYFIRST)) != 0) + dbp->err(dbp, ret, "DB->cursor"); + + /* Initialize the data to be the second of two duplicate records. */ + data.data = "new key's data: entry #2"; + data.size = strlen(data.data) + 1; + + /* + * Store the second of the two duplicate records. No duplicate + * record sort function has been specified, so we explicitly + * store the record as the last of the duplicate set. + */ + if ((ret = dbcp->put(dbcp, &key, &data, DB_KEYLAST)) != 0) + dbp->err(dbp, ret, "DB->cursor"); + +err: if ((ret = dbcp->close(dbcp)) != 0) + dbp->err(dbp, ret, "DBcursor->close"); + + return (0); +} +``` + +### Note + +If you are using the Heap access method and you are creating a new record in the database, then the key that you provide to the DBC->put() method should be empty. The DBC->put() method will return the record's ID (RID) in the key. The RID is automatically created for you when Heap database records are created. + +### Deleting records with a cursor + +The DBC->del() method deletes records from the database using a cursor. The DBC->del() method deletes the record to which the cursor currently refers. In all cases, the cursor position is unchanged after a delete. + +### Duplicating a cursor + +Once a cursor has been initialized (for example, by a call to DBC->get()), it can be thought of as identifying a particular location in a database. The DBC->dup() method permits an application to create a new cursor that has the same locking and transactional information as the cursor from which it is copied, and which optionally refers to the same position in the database. + +In order to maintain a cursor position when an application is using locking, locks are maintained on behalf of the cursor until the cursor is closed. In cases when an application is using locking without transactions, cursor duplication is often required to avoid self-deadlocks. For further details, refer to Berkeley DB Transactional Data Store locking conventions. + +### Equality Join + +Berkeley DB supports "equality" (also known as "natural"), joins on secondary indices. An equality join is a method of retrieving data from a primary database using criteria stored in a set of secondary indices. It requires the data be organized as a primary database which contains the primary key and primary data field, and a set of secondary indices. Each of the secondary indices is indexed by a different secondary key, and, for each key in a secondary index, there is a set of duplicate data items that match the primary keys in the primary database. + +For example, let's assume the need for an application that will return the names of stores in which one can buy fruit of a given color. We would first construct a primary database that lists types of fruit as the key item, and the store where you can buy them as the data item: + +| Primary key: | Primary data: | +|:-------------|:------------------| +| apple | Convenience Store | +| blueberry | Farmer's Market | +| peach | Shopway | +| pear | Farmer's Market | +| raspberry | Shopway | +| strawberry | Farmer's Market | + +We would then create a secondary index with the key **color**, and, as the data items, the names of fruits of different colors. + +| Secondary key: | Secondary data: | +|:---------------|:----------------| +| blue | blueberry | +| red | apple | +| red | raspberry | +| red | strawberry | +| yellow | peach | +| yellow | pear | + +This secondary index would allow an application to look up a color, and then use the data items to look up the stores where the colored fruit could be purchased. For example, by first looking up **blue**, the data item **blueberry** could be used as the lookup key in the primary database, returning **Farmer's Market**. + +Your data must be organized in the following manner in order to use the DB->join() method: + +1. The actual data should be stored in the database represented by the DB object used to invoke this method. Generally, this DB object is called the *primary*. + +2. Secondary indices should be stored in separate databases, whose keys are the values of the secondary indices and whose data items are the primary keys corresponding to the records having the designated secondary key value. It is acceptable (and expected) that there may be duplicate entries in the secondary indices. + + These duplicate entries should be sorted for performance reasons, although it is not required. For more information see the DB_DUPSORT flag to the DB->set_flags() method. + +What the DB->join() method does is review a list of secondary keys, and, when it finds a data item that appears as a data item for all of the secondary keys, it uses that data item as a lookup into the primary database, and returns the associated data item. + +If there were another secondary index that had as its key the **cost** of the fruit, a similar lookup could be done on stores where inexpensive fruit could be purchased: + +| Secondary key: | Secondary data: | +|:---------------|:----------------| +| expensive | blueberry | +| expensive | peach | +| expensive | pear | +| expensive | strawberry | +| inexpensive | apple | +| inexpensive | pear | +| inexpensive | raspberry | + +The DB->join() method provides equality join functionality. While not strictly cursor functionality, in that it is not a method off a cursor handle, it is more closely related to the cursor operations than to the standard DB operations. + +It is also possible to do lookups based on multiple criteria in a single operation. For example, it is possible to look up fruits that are both red and expensive in a single operation. If the same fruit appeared as a data item in both the color and expense indices, then that fruit name would be used as the key for retrieval from the primary index, and would then return the store where expensive, red fruit could be purchased. + +#### Example + +Consider the following three databases: + +personnel +- key = SSN +- data = record containing name, address, phone number, job title + +lastname +- key = lastname +- data = SSN + +jobs +- key = job title +- data = SSN + +Consider the following query: + +``` c +Return the personnel records of all people named smith with the job +title manager. +``` + +This query finds are all the records in the primary database (personnel) for whom the criteria **lastname=smith and job title=manager** is true. + +Assume that all databases have been properly opened and have the handles: pers_db, name_db, job_db. We also assume that we have an active transaction to which the handle txn refers. + +``` c +DBC *name_curs, *job_curs, *join_curs; +DBC *carray[3]; +DBT key, data; +int ret, tret; + +name_curs = NULL; +job_curs = NULL; +memset(&key, 0, sizeof(key)); +memset(&data, 0, sizeof(data)); + +if ((ret = + name_db->cursor(name_db, txn, &name_curs, 0)) != 0) + goto err; +key.data = "smith"; +key.size = sizeof("smith"); +if ((ret = + name_curs->get(name_curs, &key, &data, DB_SET)) != 0) + goto err; + +if ((ret = job_db->cursor(job_db, txn, &job_curs, 0)) != 0) + goto err; +key.data = "manager"; +key.size = sizeof("manager"); +if ((ret = + job_curs->get(job_curs, &key, &data, DB_SET)) != 0) + goto err; + +carray[0] = name_curs; +carray[1] = job_curs; +carray[2] = NULL; + +if ((ret = + pers_db->join(pers_db, carray, &join_curs, 0)) != 0) + goto err; +while ((ret = + join_curs->get(join_curs, &key, &data, 0)) == 0) { + /* Process record returned in key/data. */ +} + +/* + * If we exited the loop because we ran out of records, + * then it has completed successfully. + */ +if (ret == DB_NOTFOUND) + ret = 0; + +err: +if (join_curs != NULL && + (tret = join_curs->close(join_curs)) != 0 && ret == 0) + ret = tret; +if (name_curs != NULL && + (tret = name_curs->close(name_curs)) != 0 && ret == 0) + ret = tret; +if (job_curs != NULL && + (tret = job_curs->close(job_curs)) != 0 && ret == 0) + ret = tret; + +return (ret); +``` + +The name cursor is positioned at the beginning of the duplicate list for **smith** and the job cursor is placed at the beginning of the duplicate list for **manager**. The join cursor is returned from the join method. This code then loops over the join cursor getting the personnel records of each one until there are no more. + +### Data item count + +Once a cursor has been initialized to refer to a particular key in the database, it can be used to determine the number of data items that are stored for any particular key. The DBC->count() method returns this number of data items. The returned value is always one, unless the database supports duplicate data items, in which case it may be any number of items. + +### Cursor close + +The DBC->close() method closes the DBC cursor, after which the cursor may no longer be used. Although cursors are implicitly closed when the database they point to are closed, it is good programming practice to explicitly close cursors. In addition, in transactional systems, cursors may not exist outside of a transaction and so must be explicitly closed. diff --git a/docs-src/guides/programmer_reference/am_delete.md b/docs-src/guides/programmer_reference/am_delete.md new file mode 100644 index 000000000..e685523c6 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_delete.md @@ -0,0 +1,10 @@ +--- +title: "Deleting records" +api-name: "Deleting records" +source: docs/programmer_reference/am_delete.html +--- +## Deleting records + +The DB->del() method deletes records from the database. In general, DB->del() takes a key and deletes the data item associated with it from the database. + +If the database has been configured to support duplicate records, the DB->del() method will remove all of the duplicate records. To remove individual duplicate records, you must use a Berkeley DB cursor interface. diff --git a/docs-src/guides/programmer_reference/am_foreign.md b/docs-src/guides/programmer_reference/am_foreign.md new file mode 100644 index 000000000..e8cc2bcbe --- /dev/null +++ b/docs-src/guides/programmer_reference/am_foreign.md @@ -0,0 +1,135 @@ +--- +title: "Foreign key indexes" +api-name: "Foreign key indexes" +source: docs/programmer_reference/am_foreign.html +--- +## Foreign key indexes + +Foreign keys are used to ensure a level of consistency between two different databases in terms of the keys that the databases use. In a foreign key relationship, one database is the *constrained* database. This database is actually a secondary database which is associated with a primary database. The other database in this relationship is the *foreign key* database. Once this relationship has been established between a constrained database and a foreign key database, then: + +1. Key/data items cannot be added to the constrained database unless that same key already exists in the foreign key database. + +2. A key/data pair cannot be deleted from the foreign key database unless some action is also taken to keep the constrained database consistent with the foreign key database. + +Because the constrained database is a secondary database, by ensuring it's consistency with a foreign key database you are actually ensuring that a primary database (the one to which the secondary database is associated) is consistent with the foreign key database. + +Deletions of keys in the foreign key database affect the constrained database in one of three ways, as specified by the application: + +- `Abort` + + The deletion of a record from the foreign database will not proceed if that key exists in the constrained primary database. Transactions must be used to prevent the aborted delete from corrupting either of the databases. + +- `Cascade` + + The deletion of a record from the foreign database will also cause any records in the constrained primary database that use that key to also be automatically deleted. + +- `Nullify` + + The deletion of a record from the foreign database will cause a user specified callback function to be called, in order to alter or nullify any records using that key in the constrained primary database. + +Note that it is possible to delete a key from the constrained database, but not from the foreign key database. For this reason, if you want the keys used in both databases to be 100% accurate, then you will have to write code to ensure that when a key is removed from the constrained database, it is also removed from the foreign key database. + +As an example of how foreign key indexes might be used, consider a database of customer information and a database of order information. A typical customer database would use a customer ID as the key and those keys would also appear in the order database. To ensure an order is not booked for a non-existent customer, the customer database can be associated with the order database as a foreign index. + +In order to do this, you create a secondary index of the order database, which uses customer IDs as the key for its key/data pairs. This secondary index is, then, the constrained database. But because the secondary index is constrained, so too is the order database because the contents of the secondary index are programmatically tied to the contents of the order database. + +The customer database, then, is the foreign key database. It is associated to the order database's secondary index using the DB->associate_foreign() method. In this way, an order cannot be added to the order database unless the customer ID already exists in the customer database. + +Note that this relationship can also be configured to delete any outstanding orders for a customer when that customer is deleted from the customer database. + +In SQL, this would be done by executing something like the following: + +``` c +CREATE TABLE customers(cust_id CHAR(4) NOT NULL, + lastname CHAR(15), firstname CHAR(15), PRIMARY KEY(cust_id)); +CREATE TABLE orders(order_id CHAR(4) NOT NULL, order_num int NOT NULL, + cust_id CHAR(4), PRIMARY KEY (order_id), + FOREIGN KEY (cust_id) REFERENCES customers(cust_id) + ON DELETE CASCADE); +``` + +In Berkeley DB, this would work as follows: + +``` c +struct customer { + char cust_id[4]; + char last_name[15]; + char first_name[15]; +}; +struct order { + char order_id[4]; + int order_number; + char cust_id[4]; +}; + +.... + +void +foreign() +{ + DB *dbp, *sdbp, *fdbp; + int ret; + + /* Open/create order database */ + if ((ret = db_create(&dbp, dbenv, 0)) != 0) + handle_error(ret); + if ((ret = dbp->open(dbp, NULL, + "orders.db", NULL, DB_BTREE, DB_CREATE, 0600)) != 0) + handle_error(ret); + + /* + * Open/create secondary index on customer id. Note that it + * supports duplicates because a customer may have multiple + * orders. + */ + if ((ret = db_create(&sdbp, dbenv, 0)) != 0) + handle_error(ret); + if ((ret = sdbp->set_flags(sdbp, DB_DUP | DB_DUPSORT)) != 0) + handle_error(ret); + if ((ret = sdbp->open(sdbp, NULL, "orders_cust_ids.db", + NULL, DB_BTREE, DB_CREATE, 0600)) != 0) + handle_error(ret); + + /* Associate the secondary with the primary. */ + if ((ret = dbp->associate(dbp, NULL, sdbp, getcustid, 0)) != 0) + handle_error(ret); + + /* Open/create customer database */ + if ((ret = db_create(&fdbp, dbenv, 0)) != 0) + handle_error(ret); + if ((ret = fdbp->open(fdbp, NULL, + "customers.db", NULL, DB_BTREE, DB_CREATE, 0600)) != 0) + handle_error(ret); + + /* Associate the foreign with the secondary. */ + if ((ret = fdbp->associate_foreign( + fdbp, sdbp, NULL, DB_FOREIGN_CASCADE)) != 0) + handle_error(ret); + +} + +/* +* getcustid -- extracts a secondary key (the customer id) from a primary +* key/data pair +*/ +int +getcustid(secondary, pkey, pdata, skey) + DB *secondary; + const DBT *pkey, *pdata; + DBT *skey; +{ + /* + * Since the secondary key is a simple structure member of the + * record, we don't have to do anything fancy to return it. If + * we have composite keys that need to be constructed from the + * record, rather than simply pointing into it, then the user's + * function might need to allocate space and copy data. In + * this case, the DB_DBT_APPMALLOC flag should be set in the + * secondary key DBT. + */ + memset(skey, 0, sizeof(DBT)); + skey->data = ((struct order *)pdata->data)->cust_id; + skey->size = 4; + return (0); +} +``` diff --git a/docs-src/guides/programmer_reference/am_get.md b/docs-src/guides/programmer_reference/am_get.md new file mode 100644 index 000000000..e327dedeb --- /dev/null +++ b/docs-src/guides/programmer_reference/am_get.md @@ -0,0 +1,21 @@ +--- +title: "Retrieving records" +api-name: "Retrieving records" +source: docs/programmer_reference/am_get.html +--- +## Retrieving records + +The DB->get() method retrieves records from the database. In general, DB->get() takes a key and returns the associated data from the database. + +There are a few flags that you can set to customize retrieval: + + DB_GET_BOTH +Search for a matching key and data item, that is, only return success if both the key and the data items match those stored in the database. + + DB_RMW +Read-modify-write: acquire write locks instead of read locks during retrieval. This can enhance performance in threaded applications by reducing the chance of deadlock. + + DB_SET_RECNO +If the underlying database is a Btree, and was configured so that it is possible to search it by logical record number, retrieve a specific record. + +If the database has been configured to support duplicate records, DB->get() will always return the first data item in the duplicate set. diff --git a/docs-src/guides/programmer_reference/am_misc.md b/docs-src/guides/programmer_reference/am_misc.md new file mode 100644 index 000000000..43a124619 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc.md @@ -0,0 +1,46 @@ +--- +title: "Chapter 4.  Access Method Wrapup" +api-name: "Chapter 4.  Access Method Wrapup" +source: docs/programmer_reference/am_misc.html +--- +## Chapter 4.  Access Method Wrapup + +**Table of Contents** + + [Data alignment](am_misc.md#am_misc_align) + + [Retrieving and updating records in bulk](am_misc_bulk.md) + + [Bulk retrieval](am_misc_bulk.md#am_misc_bulk_get) + + [Bulk updates](am_misc_bulk.md#am_misc_bulk_put) + + [Bulk deletes](am_misc_bulk.md#am_misc_bulk_del) + + [Partial record storage and retrieval](am_misc_partial.md) + + [Storing C/C++ structures/objects](am_misc_struct.md) + + [Retrieved key/data permanence for C/C++](am_misc_perm.md) + + [Error support](am_misc_error.md) + + [Cursor stability](am_misc_stability.md) + + [Database limits](am_misc_dbsizes.md) + + [Disk space requirements](am_misc_diskspace.md) + + [Btree](am_misc_diskspace.md#idp51253016) + + [Hash](am_misc_diskspace.md#idp51253080) + + [Specifying a Berkeley DB schema using SQL DDL](am_misc_db_sql.md) + + [Access method tuning](am_misc_tune.md) + + [Access method FAQ](am_misc_faq.md) + +## Data alignment + +The Berkeley DB access methods provide no guarantees about byte alignment for returned key/data pairs, or callback functions which take DBT references as arguments, and applications are responsible for arranging any necessary alignment. The DB_DBT_MALLOC, DB_DBT_REALLOC, and DB_DBT_USERMEM flags may be used to store returned items in memory of arbitrary alignment. diff --git a/docs-src/guides/programmer_reference/am_misc_bulk.md b/docs-src/guides/programmer_reference/am_misc_bulk.md new file mode 100644 index 000000000..396c569e4 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_bulk.md @@ -0,0 +1,112 @@ +--- +title: "Retrieving and updating records in bulk" +api-name: "Retrieving and updating records in bulk" +source: docs/programmer_reference/am_misc_bulk.html +--- +## Retrieving and updating records in bulk + + [Bulk retrieval](am_misc_bulk.md#am_misc_bulk_get) + + [Bulk updates](am_misc_bulk.md#am_misc_bulk_put) + + [Bulk deletes](am_misc_bulk.md#am_misc_bulk_del) + +When retrieving or modifying large numbers of records, the number of method calls can often dominate performance. Berkeley DB offers bulk get, put and delete interfaces which can significantly increase performance for some applications. + +### Bulk retrieval + +To retrieve records in bulk, an application buffer must be specified to the DB->get() or DBC->get() methods. This is done in the C API by setting the **data** and **ulen** fields of the **data** DBT to reference an application buffer, and the **flags** field of that structure to DB_DBT_USERMEM. In the Berkeley DB C++ and Java APIs, the actions are similar, although there are API-specific methods to set the DBT values. Then, the DB_MULTIPLE or DB_MULTIPLE_KEY flags are specified to the DB->get() or DBC->get() methods, which cause multiple records to be returned in the specified buffer. + +The difference between DB_MULTIPLE and DB_MULTIPLE_KEY is as follows: DB_MULTIPLE returns multiple data items for a single key. For example, the DB_MULTIPLE flag would be used to retrieve all of the duplicate data items for a single key in a single call. The DB_MULTIPLE_KEY flag is used to retrieve multiple key/data pairs, where each returned key may or may not have duplicate data items. + +Once the DB->get() or DBC->get() method has returned, the application will walk through the buffer handling the returned records. This is implemented for the C and C++ APIs using four macros: DB_MULTIPLE_INIT, DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE_RECNO_NEXT. For the Java API, this is implemented as three iterator classes: MultipleDataEntry, MultipleKeyDataEntry, and MultipleRecnoDataEntry. + +The DB_MULTIPLE_INIT macro is always called first. It initializes a local application variable and the **data** DBT for stepping through the set of returned records. Then, the application calls one of the remaining three macros: DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE_RECNO_NEXT. + +If the DB_MULTIPLE flag was specified to the DB->get() or DBC->get() method, the application will always call the DB_MULTIPLE_NEXT macro. If the DB_MULTIPLE_KEY flag was specified to the DB->get() or DBC->get() method, and the underlying database is a Btree or Hash database, the application will always call the DB_MULTIPLE_KEY_NEXT macro. If the DB_MULTIPLE_KEY flag was specified to the DB->get() or DBC->get() method, and the underlying database is a Queue or Recno database, the application will always call the DB_MULTIPLE_RECNO_NEXT macro. The DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE_RECNO_NEXT macros are called repeatedly, until the end of the returned records is reached. The end of the returned records is detected by the application's local pointer variable being set to NULL. + +Note that if you want to use a cursor for bulk retrieval of records in a Btree database, you should open the cursor using the `DB_CURSOR_BULK` flag. This optimizes the cursor for bulk retrieval. + +The following is an example of a routine that displays the contents of a Btree database using the bulk return interfaces. + +``` c +int +rec_display(DB *dbp) +{ + DBC *dbcp; + DBT key, data; + size_t retklen, retdlen; + void *retkey, *retdata; + int ret, t_ret; + void *p; + + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + + /* Review the database in 5MB chunks. */ +#define BUFFER_LENGTH (5 * 1024 * 1024) + if ((data.data = malloc(BUFFER_LENGTH)) == NULL) + return (errno); + data.ulen = BUFFER_LENGTH; + data.flags = DB_DBT_USERMEM; + + /* Acquire a cursor for the database. */ + if ((ret = dbp->cursor(dbp, NULL, &dbcp, DB_CURSOR_BULK)) + != 0) { + dbp->err(dbp, ret, "DB->cursor"); + free(data.data); + return (ret); + } + + for (;;) { + /* + * Acquire the next set of key/data pairs. This code + * does not handle single key/data pairs that won't fit + * in a BUFFER_LENGTH size buffer, instead returning + * DB_BUFFER_SMALL to our caller. + */ + if ((ret = dbcp->get(dbcp, + &key, &data, DB_MULTIPLE_KEY | DB_NEXT)) != 0) { + if (ret != DB_NOTFOUND) + dbp->err(dbp, ret, "DBcursor->get"); + break; + } + + for (DB_MULTIPLE_INIT(p, &data);;) { + DB_MULTIPLE_KEY_NEXT(p, + &data, retkey, retklen, retdata, retdlen); + if (p == NULL) + break; + printf("key: %.*s, data: %.*s\n", + (int)retklen, (char *)retkey, (int)retdlen, + (char *)retdata); + } + } + + if ((t_ret = dbcp->close(dbcp)) != 0) { + dbp->err(dbp, ret, "DBcursor->close"); + if (ret == 0) + ret = t_ret; + } + + free(data.data); + + return (ret); +} +``` + +### Bulk updates + +To put records in bulk with the btree or hash access methods, construct bulk buffers in the **key** and **data** DBT using DB_MULTIPLE_WRITE_INIT and DB_MULTIPLE_WRITE_NEXT. To put records in bulk with the recno or queue access methods, construct bulk buffers in the **data** DBT as before, but construct the **key** DBT using DB_MULTIPLE_RECNO_WRITE_INIT and DB_MULTIPLE_RECNO_WRITE_NEXT with a data size of zero;. In both cases, set the DB_MULTIPLE flag to DB->put(). + +Alternatively, for btree and hash access methods, construct a single bulk buffer in the **key** DBT using DB_MULTIPLE_WRITE_INIT and DB_MULTIPLE_KEY_WRITE_NEXT. For recno and queue access methods, construct a bulk buffer in the **key** DBT using DB_MULTIPLE_RECNO_WRITE_INIT and DB_MULTIPLE_RECNO_WRITE_NEXT. In both cases, set the DB_MULTIPLE_KEY flag to DB->put(). + +A successful bulk operation is logically equivalent to a loop through each key/data pair, performing a DB->put() for each one. + +### Bulk deletes + +To delete all records with a specified set of keys with the btree or hash access methods, construct a bulk buffer in the **key** DBT using DB_MULTIPLE_WRITE_INIT and DB_MULTIPLE_WRITE_NEXT. To delete a set of records with the recno or queue access methods, construct the **key** DBT using DB_MULTIPLE_RECNO_WRITE_INIT and DB_MULTIPLE_RECNO_WRITE_NEXT with a data size of zero. In both cases, set the DB_MULTIPLE flag to DB->del(). This is equivalent to calling DB->del() for each key in the bulk buffer. In particular, if the database supports duplicates, all records with the matching key are deleted. + +Alternatively, to delete a specific set of key/data pairs, which may be items within a set of duplicates, there are also two cases depending on whether the access method uses record numbers for keys. For btree and hash access methods, construct a single bulk buffer in the **key** DBT using DB_MULTIPLE_WRITE_INIT and DB_MULTIPLE_KEY_WRITE_NEXT. For recno and queue access methods, construct a bulk buffer in the **key** DBT using DB_MULTIPLE_RECNO_WRITE_INIT and DB_MULTIPLE_RECNO_WRITE_NEXT. In both cases, set the DB_MULTIPLE_KEY flag to DB->del(). + +A successful bulk operation is logically equivalent to a loop through each key/data pair, performing a DB->del() for each one. diff --git a/docs-src/guides/programmer_reference/am_misc_db_sql.md b/docs-src/guides/programmer_reference/am_misc_db_sql.md new file mode 100644 index 000000000..4cded94d1 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_db_sql.md @@ -0,0 +1,18 @@ +--- +title: "Specifying a Berkeley DB schema using SQL DDL" +api-name: "Specifying a Berkeley DB schema using SQL DDL" +source: docs/programmer_reference/am_misc_db_sql.html +--- +## Specifying a Berkeley DB schema using SQL DDL + +When starting a new Berkeley DB project, much of the code that you must write is dedicated to defining the BDB environment: what databases it contains, the types of the databases, and so forth. Also, since records in BDB are just byte arrays, you must write code that assembles and interprets these byte arrays. + +Much of this code can be written automatically (in C) by the db_sql_codegen utility. To use it, you first specify the schema of your Berkeley DB environment in SQL Data Definition Language (DDL). Then you invoke the db_sql_codegen command, giving the DDL as input. **db_sql_codegen** reads the DDL, and writes C code that implements a storage-layer API suggested by the DDL. + +The generated API includes a general-purpose initialization function, which sets up the environment and the databases (creating them if they don't already exist). It also includes C structure declarations for each record type, and numerous specialized functions for storing and retrieving those records. + +**db_sql_codegen** can also produce a simple test program that exercises the generated API. This program is useful as an example of how to use the API. It contains calls to all of the interface functions, along with commentary explaining what the code is doing. + +Once the storage layer API is produced, your application may use it as is, or you may customize it as much as you like by editing the generated source code. Be warned, however: **db_sql_codegen** is a one-way process; there is no way to automatically incorporate customizations into newly generated code, if you decide to run **db_sql_codegen** again. + +To learn more about **db_sql_codegen**, please consult the db_sql_codegen utility manual page in the Berkeley DB C API Reference Guide. diff --git a/docs-src/guides/programmer_reference/am_misc_dbsizes.md b/docs-src/guides/programmer_reference/am_misc_dbsizes.md new file mode 100644 index 000000000..b581f5bf2 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_dbsizes.md @@ -0,0 +1,14 @@ +--- +title: "Database limits" +api-name: "Database limits" +source: docs/programmer_reference/am_misc_dbsizes.html +--- +## Database limits + +The largest database file that Berkeley DB can handle depends on the page size selected by the application. Berkeley DB stores database file page numbers as unsigned 32-bit numbers and database file page sizes as unsigned 16-bit numbers. Using the maximum database page size of 65536, this results in a maximum database file size of 248 (256 terabytes). The minimum database page size is 512 bytes, which results in a minimum maximum database size of 241 (2 terabytes). + +The largest database file Berkeley DB can support is potentially further limited if the host system does not have filesystem support for files larger than 232, including the ability to seek to absolute offsets within those files. + +The largest key or data item that Berkeley DB can support is 232, or more likely limited by available memory. Specifically, while key and data byte strings may be of essentially unlimited length, any one of them must fit into available memory so that it can be returned to the application. As some of the Berkeley DB interfaces return both key and data items to the application, those interfaces will require that any key/data pair fit simultaneously into memory. Further, as the access methods may need to compare key and data items with other key and data items, it may be a requirement that any two key or two data items fit into available memory. Finally, when writing applications supporting transactions, it may be necessary to have an additional copy of any data item in memory for logging purposes. + +The maximum Btree depth is 255. diff --git a/docs-src/guides/programmer_reference/am_misc_diskspace.md b/docs-src/guides/programmer_reference/am_misc_diskspace.md new file mode 100644 index 000000000..a2aba8bd7 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_diskspace.md @@ -0,0 +1,130 @@ +--- +title: "Disk space requirements" +api-name: "Disk space requirements" +source: docs/programmer_reference/am_misc_diskspace.html +--- +## Disk space requirements + + [Btree](am_misc_diskspace.md#idp51253016) + + [Hash](am_misc_diskspace.md#idp51253080) + +It is possible to estimate the total database size based on the size of the data. The following calculations are an estimate of how many bytes you will need to hold a set of data and then how many pages it will take to actually store it on disk. + +Space freed by deleting key/data pairs from a Btree or Hash database is never returned to the filesystem, although it is reused where possible. This means that the Btree and Hash databases are grow-only. If enough keys are deleted from a database that shrinking the underlying file is desirable, you should use the DB->compact() method to reclaim disk space. Alternatively, you can create a new database and copy the records from the old one into it. + +These are rough estimates at best. For example, they do not take into account overflow records, filesystem metadata information, large sets of duplicate data items (where the key is only stored once), or real-life situations where the sizes of key and data items are wildly variable, and the page-fill factor changes over time. + +### Btree + +The formulas for the Btree access method are as follows: + +``` c +useful-bytes-per-page = (page-size - page-overhead) * page-fill-factor + +bytes-of-data = n-records * + (bytes-per-entry + page-overhead-for-two-entries) + +n-pages-of-data = bytes-of-data / useful-bytes-per-page + +total-bytes-on-disk = n-pages-of-data * page-size +``` + +The **useful-bytes-per-page** is a measure of the bytes on each page that will actually hold the application data. It is computed as the total number of bytes on the page that are available to hold application data, corrected by the percentage of the page that is likely to contain data. The reason for this correction is that the percentage of a page that contains application data can vary from close to 50% after a page split to almost 100% if the entries in the database were inserted in sorted order. Obviously, the **page-fill-factor** can drastically alter the amount of disk space required to hold any particular data set. The page-fill factor of any existing database can be displayed using the db_stat utility. + +The page-overhead for Btree databases is 26 bytes. As an example, using an 8K page size, with an 85% page-fill factor, there are 6941 bytes of useful space on each page: + +``` c +6941 = (8192 - 26) * .85 +``` + +The total **bytes-of-data** is an easy calculation: It is the number of key or data items plus the overhead required to store each item on a page. The overhead to store a key or data item on a Btree page is 5 bytes. So, it would take 1560000000 bytes, or roughly 1.34GB of total data to store 60,000,000 key/data pairs, assuming each key or data item was 8 bytes long: + +``` c +1560000000 = 60000000 * ((8 + 5) * 2) +``` + +The total pages of data, **n-pages-of-data**, is the **bytes-of-data** divided by the **useful-bytes-per-page**. In the example, there are 224751 pages of data. + +``` c +224751 = 1560000000 / 6941 +``` + +The total bytes of disk space for the database is **n-pages-of-data** multiplied by the **page-size**. In the example, the result is 1841160192 bytes, or roughly 1.71GB. + +``` c +1841160192 = 224751 * 8192 +``` + +### Hash + +The formulas for the Hash access method are as follows: + +``` c +useful-bytes-per-page = (page-size - page-overhead) + +bytes-of-data = n-records * + (bytes-per-entry + page-overhead-for-two-entries) + +n-pages-of-data = bytes-of-data / useful-bytes-per-page + +total-bytes-on-disk = n-pages-of-data * page-size +``` + +The **useful-bytes-per-page** is a measure of the bytes on each page that will actually hold the application data. It is computed as the total number of bytes on the page that are available to hold application data. If the application has explicitly set a page-fill factor, pages will not necessarily be kept full. For databases with a preset fill factor, see the calculation below. The page-overhead for Hash databases is 26 bytes and the page-overhead-for-two-entries is 6 bytes. + +As an example, using an 8K page size, there are 8166 bytes of useful space on each page: + +``` c +8166 = (8192 - 26) +``` + +The total **bytes-of-data** is an easy calculation: it is the number of key/data pairs plus the overhead required to store each pair on a page. In this case that's 6 bytes per pair. So, assuming 60,000,000 key/data pairs, each of which is 8 bytes long, there are 1320000000 bytes, or roughly 1.23GB of total data: + +``` c +1320000000 = 60000000 * (16 + 6) +``` + +The total pages of data, **n-pages-of-data**, is the **bytes-of-data** divided by the **useful-bytes-per-page**. In this example, there are 161646 pages of data. + +``` c +161646 = 1320000000 / 8166 +``` + +The total bytes of disk space for the database is **n-pages-of-data** multiplied by the **page-size**. In the example, the result is 1324204032 bytes, or roughly 1.23GB. + +``` c +1324204032 = 161646 * 8192 +``` + +Now, let's assume that the application specified a fill factor explicitly. The fill factor indicates the target number of items to place on a single page (a fill factor might reduce the utilization of each page, but it can be useful in avoiding splits and preventing buckets from becoming too large). Using our estimates above, each item is 22 bytes (16 + 6), and there are 8166 useful bytes on a page (8192 - 26). That means that, on average, you can fit 371 pairs per page. + +``` c +371 = 8166 / 22 +``` + +However, let's assume that the application designer knows that although most items are 8 bytes, they can sometimes be as large as 10, and it's very important to avoid overflowing buckets and splitting. Then, the application might specify a fill factor of 314. + +``` c +314 = 8166 / 26 +``` + +With a fill factor of 314, then the formula for computing database size is + +``` c +n-pages-of-data = npairs / pairs-per-page +``` + +or 191082. + +``` c +191082 = 60000000 / 314 +``` + +At 191082 pages, the total database size would be 1565343744, or 1.46GB. + +``` c +1565343744 = 191082 * 8192 +``` + +There are a few additional caveats with respect to Hash databases. This discussion assumes that the hash function does a good job of evenly distributing keys among hash buckets. If the function does not do this, you may find your table growing significantly larger than you expected. Secondly, in order to provide support for Hash databases coexisting with other databases in a single file, pages within a Hash database are allocated in power-of-two chunks. That means that a Hash database with 65 buckets will take up as much space as a Hash database with 128 buckets; each time the Hash database grows beyond its current power-of-two number of buckets, it allocates space for the next power-of-two buckets. This space may be sparsely allocated in the file system, but the files will appear to be their full size. Finally, because of this need for contiguous allocation, overflow pages and duplicate pages can be allocated only at specific points in the file, and this too can lead to sparse hash tables. diff --git a/docs-src/guides/programmer_reference/am_misc_error.md b/docs-src/guides/programmer_reference/am_misc_error.md new file mode 100644 index 000000000..4a0281ee9 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_error.md @@ -0,0 +1,51 @@ +--- +title: "Error support" +api-name: "Error support" +source: docs/programmer_reference/am_misc_error.html +--- +## Error support + +Berkeley DB offers programmatic support for displaying error return values. + +The db_strerror() function returns a pointer to the error message corresponding to any Berkeley DB error return, similar to the ANSI C strerror function, but is able to handle both system error returns and Berkeley DB specific return values. + +For example: + +``` c +int ret; +... +if ((ret = dbp->put(dbp, NULL, &key, &data, 0)) != 0) { + fprintf(stderr, "put failed: %s\n", db_strerror(ret)); + return (1); +} +``` + +There are also two additional error methods, DB->err() and `DB->errx()`. These methods work like the ANSI C X3.159-1989 (ANSI C) printf function, taking a printf-style format string and argument list, and writing a message constructed from the format string and arguments. + +The DB->err() method appends the standard error string to the constructed message; the `DB->errx()` method does not. These methods provide simpler ways of displaying Berkeley DB error messages. For example, if your application tracks session IDs in a variable called session_id, it can include that information in its error messages: + +Error messages can additionally be configured to always include a prefix (for example, the program name) using the DB->set_errpfx() method. + +``` c +#define DATABASE "access.db" + +int ret; + +(void)dbp->set_errpfx(dbp, program_name); + +if ((ret = dbp->open(dbp, + NULL, DATABASE, NULL, DB_BTREE, DB_CREATE, 0664)) != 0) { + dbp->err(dbp, ret, "%s", DATABASE); + dbp->errx(dbp, + "contact your system administrator: session ID was %d", + session_id); + return (1); +} +``` + +For example, if the program were called my_app and the open call returned an EACCESS system error, the error messages shown would appear as follows: + +``` c +my_app: access.db: Permission denied. +my_app: contact your system administrator: session ID was 14 +``` diff --git a/docs-src/guides/programmer_reference/am_misc_faq.md b/docs-src/guides/programmer_reference/am_misc_faq.md new file mode 100644 index 000000000..2a13cb049 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_faq.md @@ -0,0 +1,75 @@ +--- +title: "Access method FAQ" +api-name: "Access method FAQ" +source: docs/programmer_reference/am_misc_faq.html +--- +## Access method FAQ + +1. **Is a Berkeley DB database the same as a "table"?** + + Yes; "tables" are databases, "rows" are key/data pairs, and "columns" are application-encapsulated fields within a data item (to which Berkeley DB does not directly provide access). + +2. **I'm getting an error return in my application, but I can't figure out what the library is complaining about.** + + See DB_ENV->set_errcall(), DB_ENV->set_errfile() and DB->set_errfile() for ways to get additional information about error returns from Berkeley DB. + +3. **Are Berkeley DB databases portable between architectures with different integer sizes and different byte orders ?** + + Yes. Specifically, databases can be moved between 32- and 64-bit machines, as well as between little- and big-endian machines. See Selecting a byte order for more information. + +4. **I'm seeing database corruption when creating multiple databases in a single physical file.** + + This problem is usually the result of DB handles not sharing an underlying database environment. See Opening multiple databases in a single file for more information. + +5. **I'm using integers as keys for a Btree database, and even though the key/data pairs are entered in sorted order, the page-fill factor is low.** + + This is usually the result of using integer keys on little-endian architectures such as the x86. Berkeley DB sorts keys as byte strings, and little-endian integers don't sort well when viewed as byte strings. For example, take the numbers 254 through 257. Their byte patterns on a little-endian system are: + + ``` c + 254 fe 0 0 0 + 255 ff 0 0 0 + 256 0 1 0 0 + 257 1 1 0 0 + ``` + + If you treat them as strings, then they sort badly: + + ``` c + 256 + 257 + 254 + 255 + ``` + + On a big-endian system, their byte patterns are: + + ``` c + 254 0 0 0 fe + 255 0 0 0 ff + 256 0 0 1 0 + 257 0 0 1 1 + ``` + + and so, if you treat them as strings they sort nicely. Which means, if you use steadily increasing integers as keys on a big-endian system Berkeley DB behaves well and you get compact trees, but on a little-endian system Berkeley DB produces much less compact trees. To avoid this problem, you may want to convert the keys to flat text or big-endian representations, or provide your own Btree comparison + +6. **Is there any way to avoid double buffering in the Berkeley DB system?** + + While you cannot avoid double buffering entirely, there are a few things you can do to address this issue: + + First, the Berkeley DB cache size can be explicitly set. Rather than allocate additional space in the Berkeley DB cache to cover unexpectedly heavy load or large table sizes, double buffering may suggest you size the cache to function well under normal conditions, and then depend on the file buffer cache to cover abnormal conditions. Obviously, this is a trade-off, as Berkeley DB may not then perform as well as usual under abnormal conditions. + + Second, depending on the underlying operating system you're using, you may be able to alter the amount of physical memory devoted to the system's file buffer cache. Altering this type of resource configuration may require appropriate privileges, or even operating system reboots and/or rebuilds, on some systems. + + Third, changing the size of the Berkeley DB environment regions can change the amount of space the operating system makes available for the file buffer cache, and it's often worth considering exactly how the operating system is dividing up its available memory. Further, moving the Berkeley DB database environment regions from filesystem backed memory into system memory (or heap memory), can often make additional system memory available for the file buffer cache, especially on systems without a unified buffer cache and VM system. + + Finally, for operating systems that allow buffering to be turned off, specifying the DB_DIRECT_DB and DB_LOG_DIRECT flags will attempt to do so. + +7. **I'm seeing database corruption when I run out of disk space.** + + Berkeley DB can continue to run when when out-of-disk-space errors occur, but it requires the application to be transaction protected. Applications which do not enclose update operations in transactions cannot recover from out-of-disk-space errors, and the result of running out of disk space may be database corruption. + +8. **How can I associate application information with a DB or DB_ENV handle?** + + In the C API, the DB and DB_ENV structures each contain an "app_private" field intended to be used to reference application-specific information. See the db_create() and db_env_create() documentation for more information. + + In the C++ or Java APIs, the easiest way to associate application-specific data with a handle is to subclass the Db or DbEnv, for example subclassing Db to get MyDb. Objects of type MyDb will still have the Berkeley DB API methods available on them, and you can put any extra data or methods you want into the MyDb class. If you are using "callback" APIs that take Db or DbEnv arguments (for example, DB->set_bt_compare()) these will always be called with the Db or DbEnv objects you create. So if you always use MyDb objects, you will be able to take the first argument to the callback function and cast it to a MyDb (in C++, cast it to (MyDb\*)). That will allow you to access your data members or methods. diff --git a/docs-src/guides/programmer_reference/am_misc_partial.md b/docs-src/guides/programmer_reference/am_misc_partial.md new file mode 100644 index 000000000..ad0e39f35 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_partial.md @@ -0,0 +1,118 @@ +--- +title: "Partial record storage and retrieval" +api-name: "Partial record storage and retrieval" +source: docs/programmer_reference/am_misc_partial.html +--- +## Partial record storage and retrieval + +It is possible to both store and retrieve parts of data items in all Berkeley DB access methods. This is done by setting the DB_DBT_PARTIAL flag DBT structure passed to the Berkeley DB method. + +The DB_DBT_PARTIAL flag is based on the values of two fields of the DBT structure: **dlen** and **doff**. The value of **dlen** is the number of bytes of the record in which the application is interested. The value of **doff** is the offset from the beginning of the data item where those bytes start. + +For example, if the data item were **ABCDEFGHIJKL**, a **doff** value of 3 would indicate that the bytes of interest started at **D**, and a **dlen** value of 4 would indicate that the bytes of interest were **DEFG**. + +When retrieving a data item from a database, the **dlen** bytes starting **doff** bytes from the beginning of the record are returned, as if they comprised the entire record. If any or all of the specified bytes do not exist in the record, the retrieval is still successful and any existing bytes are returned. + +When storing a data item into the database, the **dlen** bytes starting **doff** bytes from the beginning of the specified key's data record are replaced by the data specified by the **data** and **size** fields. If **dlen** is smaller than **size**, the record will grow, and if **dlen** is larger than **size**, the record will shrink. If the specified bytes do not exist, the record will be extended using nul bytes as necessary, and the store call will still succeed. + +The following are various examples of the put case for the DB_DBT_PARTIAL flag. In all examples, the initial data item is 20 bytes in length: + +**ABCDEFGHIJ0123456789** + +1. ``` c + size = 20 + doff = 0 + dlen = 20 + data = abcdefghijabcdefghij + + Result: The 20 bytes at offset 0 are replaced by the 20 bytes of + data; that is, the entire record is replaced. + + ABCDEFGHIJ0123456789 -> abcdefghijabcdefghij + ``` + +2. ``` c + size = 10 + doff = 20 + dlen = 0 + data = abcdefghij + + Result: The 0 bytes at offset 20 are replaced by the 10 bytes of + data; that is, the record is extended by 10 bytes. + + ABCDEFGHIJ0123456789 -> ABCDEFGHIJ0123456789abcdefghij + ``` + +3. ``` c + size = 10 + doff = 10 + dlen = 5 + data = abcdefghij + + Result: The 5 bytes at offset 10 are replaced by the 10 bytes of + data. + + ABCDEFGHIJ0123456789 -> ABCDEFGHIJabcdefghij56789 + ``` + +4. ``` c + size = 10 + doff = 10 + dlen = 0 + data = abcdefghij + + Result: The 0 bytes at offset 10 are replaced by the 10 bytes of + data; that is, 10 bytes are inserted into the record. + + ABCDEFGHIJ0123456789 -> ABCDEFGHIJabcdefghij0123456789 + ``` + +5. ``` c + size = 10 + doff = 2 + dlen = 15 + data = abcdefghij + + Result: The 15 bytes at offset 2 are replaced by the 10 bytes of + data. + + ABCDEFGHIJ0123456789 -> ABabcdefghij789 + ``` + +6. ``` c + size = 10 + doff = 0 + dlen = 0 + data = abcdefghij + + Result: The 0 bytes at offset 0 are replaced by the 10 bytes of + data; that is, the 10 bytes are inserted at the beginning of the + record. + + ABCDEFGHIJ0123456789 -> abcdefghijABCDEFGHIJ0123456789 + ``` + +7. ``` c + size = 0 + doff = 0 + dlen = 10 + data = "" + + Result: The 10 bytes at offset 0 are replaced by the 0 bytes of + data; that is, the first 10 bytes of the record are discarded. + + ABCDEFGHIJ0123456789 -> 0123456789 + ``` + +8. ``` c + size = 10 + doff = 25 + dlen = 0 + data = abcdefghij + + Result: The 0 bytes at offset 25 are replaced by the 10 bytes of + data; that is, 10 bytes are inserted into the record past the end + of the current data (\0 represents a nul byte). + + ABCDEFGHIJ0123456789 -> ABCDEFGHIJ0123456789\0\0\0\0\0abcdefghij + ``` diff --git a/docs-src/guides/programmer_reference/am_misc_perm.md b/docs-src/guides/programmer_reference/am_misc_perm.md new file mode 100644 index 000000000..a032ca1cd --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_perm.md @@ -0,0 +1,10 @@ +--- +title: "Retrieved key/data permanence for C/C++" +api-name: "Retrieved key/data permanence for C/C++" +source: docs/programmer_reference/am_misc_perm.html +--- +## Retrieved key/data permanence for C/C++ + +When using the non-cursor Berkeley DB calls to retrieve key/data items under the C/C++ APIs (for example, DB->get()), the memory to which the pointer stored into the DBT refers is only valid until the next call to Berkeley DB using the DB handle. (This includes **any** use of the returned DB handle, including by another thread of control within the process. For this reason, when multiple threads are using the returned DB handle concurrently, one of the DB_DBT_MALLOC, DB_DBT_REALLOC or DB_DBT_USERMEM flags must be specified with any non-cursor DBT used for key or data retrieval.) + +When using the cursor Berkeley DB calls to retrieve key/data items under the C/C++ APIs (for example, DBC->get()), the memory to which the pointer stored into the DBT refers is only valid until the next call to Berkeley DB using the DBC returned by DB->cursor(). diff --git a/docs-src/guides/programmer_reference/am_misc_stability.md b/docs-src/guides/programmer_reference/am_misc_stability.md new file mode 100644 index 000000000..3f457553a --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_stability.md @@ -0,0 +1,14 @@ +--- +title: "Cursor stability" +api-name: "Cursor stability" +source: docs/programmer_reference/am_misc_stability.html +--- +## Cursor stability + +In the absence of locking, no guarantees are made about the stability of cursors in different threads of control. However, the Btree, Queue and Recno access methods guarantee that cursor operations, interspersed with any other operation in the same thread of control will always return keys in order and will return each non-deleted key/data pair exactly once. Because the Hash access method uses a dynamic hashing algorithm, it cannot guarantee any form of stability in the presence of inserts and deletes unless transactional locking is performed. + +If locking was specified when the Berkeley DB environment was opened, but transactions are not in effect, the access methods provide repeatable reads with respect to the cursor. That is, a DB_CURRENT call on the cursor is guaranteed to return the same record as was returned on the last call to the cursor. + +In the presence of transactions, the Btree, Hash and Recno access methods provide degree 3 isolation (serializable transactions). The Queue access method provides degree 3 isolation with the exception that it permits phantom records to appear between calls. That is, deleted records are not locked, therefore another transaction may replace a deleted record between two calls to retrieve it. The record would not appear in the first call but would be seen by the second call. For readers not enclosed in transactions, all access method calls provide degree 2 isolation, that is, reads are not repeatable. A transaction may be declared to run with degree 2 isolation by specifying the DB_READ_COMMITTED flag. Finally, Berkeley DB provides degree 1 isolation when the DB_READ_UNCOMMITTED flag is specified; that is, reads may see data modified in transactions which have not yet committed. + +For all access methods, a cursor scan of the database performed within the context of a transaction is guaranteed to return each key/data pair once and only once, except in the following case. If, while performing a cursor scan using the Hash access method, the transaction performing the scan inserts a new pair into the database, it is possible that duplicate key/data pairs will be returned. diff --git a/docs-src/guides/programmer_reference/am_misc_struct.md b/docs-src/guides/programmer_reference/am_misc_struct.md new file mode 100644 index 000000000..4e19bf086 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_struct.md @@ -0,0 +1,86 @@ +--- +title: "Storing C/C++ structures/objects" +api-name: "Storing C/C++ structures/objects" +source: docs/programmer_reference/am_misc_struct.html +--- +## Storing C/C++ structures/objects + +Berkeley DB can store any kind of data, that is, it is entirely 8-bit clean. How you use this depends, to some extent, on the application language you are using. In the C/C++ languages, there are a couple of different ways to store structures and objects. + +First, you can do some form of run-length encoding and copy your structure into another piece of memory before storing it: + +``` c +struct { + char *data1; + u_int32_t data2; + ... +} info; +size_t len; +u_int8_t *p, data_buffer[1024]; + +p = &data_buffer[0]; +len = strlen(info.data1); +memcpy(p, &len, sizeof(len)); +p += sizeof(len); +memcpy(p, info.data1, len); +p += len; +memcpy(p, &info.data2, sizeof(info.data2)); +p += sizeof(info.data2); +... +``` + +and so on, until all the fields of the structure have been loaded into the byte array. If you want more examples, see the Berkeley DB logging routines (for example, btree/btree_auto.c:\_\_bam_split_log()). This technique is generally known as "marshalling". If you use this technique, you must then un-marshall the data when you read it back: + +``` c +struct { + char *data1; + u_int32_t data2; + ... +} info; +size_t len; +u_int8_t *p, data_buffer[1024]; +... +p = &data_buffer[0]; +memcpy(&len, p, sizeof(len)); +p += sizeof(len); +info.data1 = malloc(len); +memcpy(info.data1, p, len); +p += len; +memcpy(&info.data2, p, sizeof(info.data2)); +p += sizeof(info.data2); +... +``` + +and so on. + +The second way to solve this problem only works if you have just one variable length field in the structure. In that case, you can declare the structure as follows: + +``` c +struct { + int a, b, c; + u_int8_t buf[1]; +} info; +``` + +Then, let's say you have a string you want to store in this structure. When you allocate the structure, you allocate it as: + +``` c +malloc(sizeof(struct info) + strlen(string)); +``` + +Since the allocated memory is contiguous, you can the initialize the structure as: + +``` c +info.a = 1; +info.b = 2; +info.c = 3; +memcpy(&info.buf[0], string, strlen(string) + 1); +``` + +and give it to Berkeley DB to store, with a length of: + +``` c +sizeof(struct info) + strlen(string); +``` + +In this case, the structure can be copied out of the database and used without any additional work. diff --git a/docs-src/guides/programmer_reference/am_misc_tune.md b/docs-src/guides/programmer_reference/am_misc_tune.md new file mode 100644 index 000000000..4ea5bda39 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_misc_tune.md @@ -0,0 +1,34 @@ +--- +title: "Access method tuning" +api-name: "Access method tuning" +source: docs/programmer_reference/am_misc_tune.html +--- +## Access method tuning + +There are a few different issues to consider when tuning the performance of Berkeley DB access method applications. + +access method +An application's choice of a database access method can significantly affect performance. Applications using fixed-length records and integer keys are likely to get better performance from the Queue access method. Applications using variable-length records are likely to get better performance from the Btree access method, as it tends to be faster for most applications than either the Hash or Recno access methods. Because the access method APIs are largely identical between the Berkeley DB access methods, it is easy for applications to benchmark the different access methods against each other. See Selecting an access method for more information. + +cache size +The Berkeley DB database cache defaults to a fairly small size, and most applications concerned with performance will want to set it explicitly. Using a too-small cache will result in horrible performance. The first step in tuning the cache size is to use the db_stat utility (or the statistics returned by the DB->stat() function) to measure the effectiveness of the cache. The goal is to maximize the cache's hit rate. Typically, increasing the size of the cache until the hit rate reaches 100% or levels off will yield the best performance. However, if your working set is sufficiently large, you will be limited by the system's available physical memory. Depending on the virtual memory and file system buffering policies of your system, and the requirements of other applications, the maximum cache size will be some amount smaller than the size of physical memory. If you find that the db_stat utility shows that increasing the cache size improves your hit rate, but performance is not improving (or is getting worse), then it's likely you've hit other system limitations. At this point, you should review the system's swapping/paging activity and limit the size of the cache to the maximum size possible without triggering paging activity. Finally, always remember to make your measurements under conditions as close as possible to the conditions your deployed application will run under, and to test your final choices under worst-case conditions. + +shared memory +By default, Berkeley DB creates its database environment shared regions in filesystem backed memory. Some systems do not distinguish between regular filesystem pages and memory-mapped pages backed by the filesystem, when selecting dirty pages to be flushed back to disk. For this reason, dirtying pages in the Berkeley DB cache may cause intense filesystem activity, typically when the filesystem sync thread or process is run. In some cases, this can dramatically affect application throughput. The workaround to this problem is to create the shared regions in system shared memory (DB_SYSTEM_MEM) or application private memory (DB_PRIVATE), or, in cases where this behavior is configurable, to turn off the operating system's flushing of memory-mapped pages. + +large key/data items +Storing large key/data items in a database can alter the performance characteristics of Btree, Hash and Recno databases. The first parameter to consider is the database page size. When a key/data item is too large to be placed on a database page, it is stored on "overflow" pages that are maintained outside of the normal database structure (typically, items that are larger than one-quarter of the page size are deemed to be too large). Accessing these overflow pages requires at least one additional page reference over a normal access, so it is usually better to increase the page size than to create a database with a large number of overflow pages. Use the db_stat utility (or the statistics returned by the DB->stat() method) to review the number of overflow pages in the database. + +The second issue is using large key/data items instead of duplicate data items. While this can offer performance gains to some applications (because it is possible to retrieve several data items in a single get call), once the key/data items are large enough to be pushed off-page, they will slow the application down. Using duplicate data items is usually the better choice in the long run. + +A common question when tuning Berkeley DB applications is scalability. For example, people will ask why, when adding additional threads or processes to an application, the overall database throughput decreases, even when all of the operations are read-only queries. + +First, while read-only operations are logically concurrent, they still have to acquire mutexes on internal Berkeley DB data structures. For example, when searching a linked list and looking for a database page, the linked list has to be locked against other threads of control attempting to add or remove pages from the linked list. The more threads of control you add, the more contention there will be for those shared data structure resources. + +Second, once contention starts happening, applications will also start to see threads of control convoy behind locks (especially on architectures supporting only test-and-set spin mutexes, rather than blocking mutexes). On test-and-set architectures, threads of control waiting for locks must attempt to acquire the mutex, sleep, check the mutex again, and so on. Each failed check of the mutex and subsequent sleep wastes CPU and decreases the overall throughput of the system. + +Third, every time a thread acquires a shared mutex, it has to shoot down other references to that memory in every other CPU on the system. Many modern snoopy cache architectures have slow shoot down characteristics. + +Fourth, schedulers don't care what application-specific mutexes a thread of control might hold when de-scheduling a thread. If a thread of control is descheduled while holding a shared data structure mutex, other threads of control will be blocked until the scheduler decides to run the blocking thread of control again. The more threads of control that are running, the smaller their quanta of CPU time, and the more likely they will be descheduled while holding a Berkeley DB mutex. + +The results of adding new threads of control to an application, on the application's throughput, is application and hardware specific and almost entirely dependent on the application's data access pattern and hardware. In general, using operating systems that support blocking mutexes will often make a tremendous difference, and limiting threads of control to to some small multiple of the number of CPUs is usually the right choice to make. diff --git a/docs-src/guides/programmer_reference/am_opensub.md b/docs-src/guides/programmer_reference/am_opensub.md new file mode 100644 index 000000000..6feda8b1b --- /dev/null +++ b/docs-src/guides/programmer_reference/am_opensub.md @@ -0,0 +1,38 @@ +--- +title: "Opening multiple databases in a single file" +api-name: "Opening multiple databases in a single file" +source: docs/programmer_reference/am_opensub.html +--- +## Opening multiple databases in a single file + + [Configuring databases sharing a file](am_opensub.md#idp50943544) + + [Caching databases sharing a file](am_opensub.md#idp50944288) + + [Locking in databases based on sharing a file](am_opensub.md#idp50944984) + +Applications may create multiple databases within a single physical file. This is useful when the databases are both numerous and reasonably small, in order to avoid creating a large number of underlying files, or when it is desirable to include secondary index databases in the same file as the primary index database. Putting multiple databases in a single physical file is an administrative convenience and unlikely to affect database performance. + +To open or create a file that will include more than a single database, specify a database name when calling the DB->open() method. + +Physical files do not need to be comprised of a single type of database, and databases in a file may be of any mixture of types, except for Queue and Heap databases. Queue and Heap databases must be created one per file and cannot share a file with any other database type. There is no limit on the number of databases that may be created in a single file other than the standard Berkeley DB file size and disk space limitations. + +It is an error to attempt to open a second database in a file that was not initially created using a database name, that is, the file must initially be specified as capable of containing multiple databases for a second database to be created in it. + +It is not an error to open a file that contains multiple databases without specifying a database name, however the database type should be specified as DB_UNKNOWN and the database must be opened read-only. The handle that is returned from such a call is a handle on a database whose key values are the names of the databases stored in the database file and whose data values are opaque objects. No keys or data values may be modified or stored using this database handle. + +### Configuring databases sharing a file + +There are four pieces of configuration information which must be specified consistently for all databases in a file, rather than differing on a per-database basis. They are: byte order, checksum and encryption behavior, and page size. When creating additional databases in a file, any of these configuration values specified must be consistent with the existing databases in the file or an error will be returned. + +### Caching databases sharing a file + +When storing multiple databases in a single physical file rather than in separate files, if any of the databases in a file is opened for update, all of the databases in the file must share a memory pool. In other words, they must be opened in the same database environment. This is so per-physical-file information common between the two databases is updated correctly. + +### Locking in databases based on sharing a file + +If databases are in separate files (and access to each separate database is single-threaded), there is no reason to perform any locking of any kind, and the two databases may be read and written simultaneously. Further, there would be no requirement to create a shared database environment in which to open those two databases. + +However, since multiple databases in a file exist in a single physical file, opening two databases in the same file simultaneously requires locking be enabled, unless all of the databases are read-only. As the locks for the two databases can only conflict during page allocation, this additional locking is unlikely to affect performance. The exception is when Berkeley DB Concurrent Data Store is configured; a single lock is used for all databases in the file when Berkeley DB Concurrent Data Store is configured, and a write to one database will block all accesses to all databases. + +In summary, programmers writing applications that open multiple databases in a single file will almost certainly need to create a shared database environment in the application as well. For more information on database environments, see Database environment introduction diff --git a/docs-src/guides/programmer_reference/am_partition.md b/docs-src/guides/programmer_reference/am_partition.md new file mode 100644 index 000000000..96151210f --- /dev/null +++ b/docs-src/guides/programmer_reference/am_partition.md @@ -0,0 +1,207 @@ +--- +title: "Partitioning databases" +api-name: "Partitioning databases" +source: docs/programmer_reference/am_partition.html +--- +## Partitioning databases + + [Specifying partition keys](am_partition.md#am_partition_keys) + + [Partitioning callback](am_partition.md#am_partition_function) + + [Placing partition files](am_partition.md#partition_file_placement) + +You can improve concurrency on your database reads and writes by splitting access to a single database into multiple databases. This helps to avoid contention for internal database pages, as well as allowing you to spread your databases across multiple disks, which can help to improve disk I/O. + +### Note + +Database partitions are not supported by the C# and Java APIs at this time. + +While you can manually do this by creating and using more than one database for your data, DB is capable of partitioning your database for you. When you use DB's built-in database partitioning feature, your access to your data is performed in exactly the same way as if you were only using one database; all the work of knowing which database to use to access a particular record is handled for you under the hood. + +Only the BTree and Hash access methods are supported for partitioned databases. + +You indicate that you want your database to be partitioned by calling DB->set_partition() before opening your database the first time. You can indicate the directory in which each partition is contained using the DB->set_partition_dirs() method. + +Once you have partitioned a database, you cannot change your partitioning scheme. + +There are two ways to indicate what key/data pairs should go on which partition. The first is by specifying an array of DBTs that indicate the minimum key value for a given partition. The second is by providing a callback that returns the number of the partition on which a specified key is placed. + +### Specifying partition keys + +For simple cases, you can partition your database by providing an array of DBTs, each element of which provides the minimum key value to be placed on a partition. There must be one fewer elements in this array than you have partitions. The first element of the array indicates the minimum key value for the second partition in your database. Key values that are less than the first key value provided in this array are placed on the first partition (partition 0). + +### Note + +You can use partition keys only if you are using the Btree access method. + +For example, suppose you had a database of fruit, and you want three partitions for your database. Then you need a DBT array of size two. The first element in this array indicates the minimum keys that should be placed on partition 1. The second element in this array indicates the minimum key value placed on partition 2. Keys that compare less than the first DBT in the array are placed on partition 0. + +All comparisons are performed according to the lexicographic comparison used by your platform. + +For example, suppose you want all fruits whose names begin with: + +- 'a' - 'f' to go on partition 0 + +- 'g' - 'p' to go on partition 1 + +- 'q' - 'z' to go on partition 2. + +Then you would accomplish this with the following code fragment: + +### Note + +The DB->set_partition() partition callback parameter must be `NULL` if you are using an array of DBTs to partition your database. + +``` c +DB *dbp = NULL; + DB_ENV *envp = NULL; + DBT partKeys[2]; + u_int32_t db_flags; + const char *file_name = "mydb.db"; + int ret; + +... + + /* Skipping environment open to shorten this example */ + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + fprintf(stderr, "%s\n", db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* Setup the partition keys */ + memset(&partKeys[0], 0, sizeof(DBT)); + partKeys[0].data = "g"; + partKeys[0].size = sizeof("g") - 1; + + memset(&partKeys[1], 0, sizeof(DBT)); + partKeys[1].data = "q"; + partKeys[1].size = sizeof("q") - 1; + + dbp->set_partition(dbp, 3, partKeys, NULL); + + /* Now open the database */ + db_flags = DB_CREATE; /* Allow database creation */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed", + file_name); + return (EXIT_FAILURE); + } +``` + +### Partitioning callback + +In some cases, a simple lexicographical comparison of key data will not sufficiently support a partitioning scheme. For those situations, you should write a partitioning function. This function accepts a pointer to the DB and the DBT, and it returns the number of the partition on which the key belongs. + +Note that DB actually places the key on the partition calculated by: + +``` c +returned_partition modulo number_of_partitions +``` + +Also, remember that if you use a partitioning function when you create your database, then you must use the same partitioning function every time you open that database in the future. + +The following code fragment illustrates a partition callback: + +``` c +u_int32_t db_partition_fn(DB *db, DBT *key) { + char *key_data; + u_int32_t ret_number; + /* Obtain your key data, unpacking it as necessary + * Here, we do the very simple thing just for illustrative purposes. + */ + + key_data = (char *)key->data; + + /* Here you would perform whatever comparison you require to determine + * what partition the key belongs on. If you return either 0 or the + * number of partitions in the database, the key is placed in the first + * database partition. Else, it is placed on: + * + * returned_number mod number_of_partitions + */ + + ret_number = 0; + + return ret_number; +} +``` + +You then cause your partition callback to be used by providing it to the DB->set_partition() method, as illustrated by the following code fragment. + +### Note + +The DB->set_partition() DBT array parameter must be `NULL` if you are using a partition call back to partition your database. + +``` c +DB *dbp = NULL; + DB_ENV *envp = NULL; + u_int32_t db_flags; + const char *file_name = "mydb.db"; + int ret; + +... + + /* Skipping environment open to shorten this example */ + + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + fprintf(stderr, "%s\n", db_strerror(ret)); + return (EXIT_FAILURE); + } + + dbp->set_partition(dbp, 3, NULL, db_partition_fn); + + /* Now open the database */ + db_flags = DB_CREATE; /* Allow database creation */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed", + file_name); + return (EXIT_FAILURE); + } +``` + +### Placing partition files + +When you partition a database, a database file is created on disk in the same way as if you were not partitioning the database. That is, this file uses the name you provide to the DB->open() `file` parameter. + +However, DB then also creates a series of database files on disk, one for each partition that you want to use. These partition files share the same name as the database file name, but are also number sequentially. So if you create a database named `mydb.db`, and you create 3 partitions for it, then you will see the following database files on disk: + +``` c + mydb.db + __dbp.mydb.db.000 + __dbp.mydb.db.001 + __dbp.mydb.db.002 +``` + +All of the database's contents go into the numbered database files. You can cause these files to be placed in different directories (and, hence, different disk partitions or even disks) by using the DB->set_partition_dirs() method. + +DB->set_partition_dirs() takes a NULL-terminated array of strings, each one of which should represent an existing filesystem directory. + +If you are using an environment, the directories specified using DB->set_partition_dirs() must also be included in the environment list specified by DB_ENV->add_data_dir(). + +If you are not using an environment, then the the directories specified to DB->set_partition_dirs() can be either complete paths to currently existing directories, or paths relative to the application's current working directory. + +Ideally, you will provide DB->set_partition_dirs() with an array that is the same size as the number of partitions you are creating for your database. Partition files are then placed according to the order that directories are contained in the array; partition 0 is placed in directory_array\[0\], partition 1 in directory_array\[1\], and so forth. However, if you provide an array of directories that is smaller than the number of database partitions, then the directories are used on a round-robin fashion. + +You must call DB->set_partition_dirs() before you create your database, and before you open your database each time thereafter. The array provided to DB->set_partition_dirs() must not change after the database has been created. diff --git a/docs-src/guides/programmer_reference/am_put.md b/docs-src/guides/programmer_reference/am_put.md new file mode 100644 index 000000000..0ae0e32a2 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_put.md @@ -0,0 +1,22 @@ +--- +title: "Storing records" +api-name: "Storing records" +source: docs/programmer_reference/am_put.html +--- +## Storing records + +The DB->put() method stores records into the database. In general, DB->put() takes a key and stores the associated data into the database. + +There are a few flags that you can set to customize storage: + + DB_APPEND +Simply append the data to the end of the database, treating the database much like a simple log. This flag is only valid for the Heap, Queue and Recno access methods. This flag is required if you are creating a new record in a Heap database. + + DB_NOOVERWRITE +Only store the data item if the key does not already appear in the database. + +If the database has been configured to support duplicate records, the DB->put() method will add the new data value at the end of the duplicate set. If the database supports sorted duplicates, the new data value is inserted at the correct sorted location. + +### Note + +If you are using the Heap access method and you are creating a new record in the database, then the key that you provide to the DB->put() method should be empty. The DB->put() method will return the record's ID (RID) in the key. The RID is automatically created for you when Heap database records are created. diff --git a/docs-src/guides/programmer_reference/am_second.md b/docs-src/guides/programmer_reference/am_second.md new file mode 100644 index 000000000..fc4ce435d --- /dev/null +++ b/docs-src/guides/programmer_reference/am_second.md @@ -0,0 +1,209 @@ +--- +title: "Secondary indexes" +api-name: "Secondary indexes" +source: docs/programmer_reference/am_second.html +--- +## Secondary indexes + + [Error Handling With Secondary Indexes](am_second.md#idp51040080) + +A secondary index, put simply, is a way to efficiently access records in a database (the primary) by means of some piece of information other than the usual (primary) key. In Berkeley DB, this index is simply another database whose keys are these pieces of information (the secondary keys), and whose data are the primary keys. Secondary indexes can be created manually by the application; there is no disadvantage, other than complexity, to doing so. However, when the secondary key can be mechanically derived from the primary key and datum that it points to, as is frequently the case, Berkeley DB can automatically and transparently manage secondary indexes. + +As an example of how secondary indexes might be used, consider a database containing a list of students at a college, each of whom has a unique student ID number. A typical database would use the student ID number as the key; however, one might also reasonably want to be able to look up students by last name. To do this, one would construct a secondary index in which the secondary key was this last name. + +In SQL, this would be done by executing something like the following: + +``` c +CREATE TABLE students(student_id CHAR(4) NOT NULL, + lastname CHAR(15), firstname CHAR(15), PRIMARY KEY(student_id)); +CREATE INDEX lname ON students(lastname); +``` + +In Berkeley DB, this would work as follows (a Java API example is also available): + +``` c +struct student_record { + char student_id[4]; + char last_name[15]; + char first_name[15]; +}; + +.... + +void +second() +{ + DB *dbp, *sdbp; + int ret; + + /* Open/create primary */ + if ((ret = db_create(&dbp, dbenv, 0)) != 0) + handle_error(ret); + if ((ret = dbp->open(dbp, NULL, + "students.db", NULL, DB_BTREE, DB_CREATE, 0600)) != 0) + handle_error(ret); + + /* + * Open/create secondary. Note that it supports duplicate data + * items, since last names might not be unique. + */ + if ((ret = db_create(&sdbp, dbenv, 0)) != 0) + handle_error(ret); + if ((ret = sdbp->set_flags(sdbp, DB_DUP | DB_DUPSORT)) != 0) + handle_error(ret); + if ((ret = sdbp->open(sdbp, NULL, + "lastname.db", NULL, DB_BTREE, DB_CREATE, 0600)) != 0) + handle_error(ret); + + /* Associate the secondary with the primary. */ + if ((ret = dbp->associate(dbp, NULL, sdbp, getname, 0)) != 0) + handle_error(ret); +} + +/* + * getname -- extracts a secondary key (the last name) from a primary + * key/data pair + */ +int +getname(DB *secondary, const DBT *pkey, const DBT *pdata, DBT *skey) + +{ + /* + * Since the secondary key is a simple structure member of the + * record, we don't have to do anything fancy to return it. If + * we have composite keys that need to be constructed from the + * record, rather than simply pointing into it, then the user's + * function might need to allocate space and copy data. In + * this case, the DB_DBT_APPMALLOC flag should be set in the + * secondary key DBT. + */ + memset(skey, 0, sizeof(DBT)); + skey->data = ((struct student_record *)pdata->data)->last_name; + skey->size = sizeof(((struct student_record *)pdata->data)->last_name); + return (0); +} +``` + +From the application's perspective, putting things into the database works exactly as it does without a secondary index; one can simply insert records into the primary database. In SQL one would do the following: + +``` c +INSERT INTO student + VALUES ("WC42", "Churchill ", "Winston "); +``` + +and in Berkeley DB, one does: + +``` c +struct student_record s; +DBT data, key; + +memset(&key, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); +memset(&s, 0, sizeof(struct student_record)); +key.data = "WC42"; +key.size = 4; +memcpy(&s.student_id, "WC42", sizeof(s.student_id)); +memcpy(&s.last_name, "Churchill ", sizeof(s.last_name)); +memcpy(&s.first_name, "Winston ", sizeof(s.first_name)); +data.data = &s; +data.size = sizeof(s); +if ((ret = dbp->put(dbp, txn, &key, &data, 0)) != 0) + handle_error(ret); + +``` + +Internally, a record with secondary key "Churchill" is inserted into the secondary database (in addition to the insertion of "WC42" into the primary, of course). + +Deletes are similar. The SQL clause: + +``` c +DELETE FROM student WHERE (student_id = "WC42"); +``` + +looks like: + +``` c +DBT key; + +memset(&key, 0, sizeof(DBT)); +key.data = "WC42"; +key.size = 4; +if ((ret = dbp->del(dbp, txn, &key, 0)) != 0) + handle_error(ret); +``` + +Deletes can also be performed on the secondary index directly; a delete done this way will delete the "real" record in the primary as well. If the secondary supports duplicates and there are duplicate occurrences of the secondary key, then all records with that secondary key are removed from both the secondary index and the primary database. In SQL: + +``` c +DELETE FROM lname WHERE (lastname = "Churchill "); +``` + +In Berkeley DB: + +``` c +DBT skey; + +memset(&skey, 0, sizeof(DBT)); +skey.data = "Churchill "; +skey.size = 15; +if ((ret = sdbp->del(sdbp, txn, &skey, 0)) != 0) + handle_error(ret); +``` + +Gets on a secondary automatically return the primary datum. If DB->pget() or DBC->pget() is used in lieu of DB->get() or DBC->get(), the primary key is returned as well. Thus, the equivalent of: + +``` c +SELECT * from lname WHERE (lastname = "Churchill "); +``` + +would be: + +``` c +DBT data, pkey, skey; + +memset(&skey, 0, sizeof(DBT)); +memset(&pkey, 0, sizeof(DBT)); +memset(&data, 0, sizeof(DBT)); +skey.data = "Churchill "; +skey.size = 15; +if ((ret = sdbp->pget(sdbp, txn, &skey, &pkey, &data, 0)) != 0) + handle_error(ret); +/* + * Now pkey contains "WC42" and data contains Winston's record. + */ +``` + +To create a secondary index to a Berkeley DB database, open the database that is to become a secondary index normally, then pass it as the "secondary" argument to the DB->associate() method for some primary database. + +After a DB->associate() call is made, the secondary indexes become alternate interfaces to the primary database. All updates to the primary will be automatically reflected in each secondary index that has been associated with it. All get operations using the DB->get() or DBC->get() methods on the secondary index return the primary datum associated with the specified (or otherwise current, in the case of cursor operations) secondary key. The DB->pget() and DBC->pget() methods also become usable; these behave just like DB->get() and DBC->get(), but return the primary key in addition to the primary datum, for those applications that need it as well. + +Cursor get operations on a secondary index perform as expected; although the data returned will by default be those of the primary database, a position in the secondary index is maintained normally, and records will appear in the order determined by the secondary key and the comparison function or other structure of the secondary database. + +Delete operations on a secondary index delete the item from the primary database and all relevant secondaries, including the current one. + +Put operations of any kind are forbidden on secondary indexes, as there is no way to specify a primary key for a newly put item. Instead, the application should use the DB->put() or DBC->put() methods on the primary database. + +Any number of secondary indexes may be associated with a given primary database, up to limitations on available memory and the number of open file descriptors. + +Note that although Berkeley DB guarantees that updates made using any DB handle with an associated secondary will be reflected in the that secondary, associating each primary handle with all the appropriate secondaries is the responsibility of the application and is not enforced by Berkeley DB. It is generally unsafe, but not forbidden by Berkeley DB, to modify a database that has secondary indexes without having those indexes open and associated. Similarly, it is generally unsafe, but not forbidden, to modify a secondary index directly. Applications that violate these rules face the possibility of outdated or incorrect results if the secondary indexes are later used. + +If a secondary index becomes outdated for any reason, it should be discarded using the DB->remove() method and a new one created using the DB->associate() method. If a secondary index is no longer needed, all of its handles should be closed using the DB->close() method, and then the database should be removed using a new database handle and the DB->remove() method. + +Closing a primary database handle automatically dis-associates all secondary database handles associated with it. + +### Error Handling With Secondary Indexes + +An error return during a secondary update in CDS or DS (which requires an abort in TDS) may leave a secondary index inconsistent in CDS or DS. There are a few non-error returns: + +- 0 +- DB_BUFFER_SMALL +- DB_NOTFOUND +- DB_KEYEMPTY +- DB_KEYEXIST + +In the case of any other error return during a secondary update in CDS or DS, delete the secondary indices, recreate them and set the `DB_CREATE flag` to the `DB->associate` method. Some examples of error returns that need to be handled this way are: + +- ENOMEM - indicating there is insufficient memory to return the requested item +- EINVAL - indicating that an invalid flag value or parameter is specified + +Note that `DB_RUNRECOVERY` and `DB_PAGE_NOTFOUND` are fatal errors which should never occur during normal use of CDS or DS. If those errors are returned by Berkeley DB when running without transactions, check the database integrity with the `DB->verify` method before rebuilding the secondary indices. diff --git a/docs-src/guides/programmer_reference/am_stat.md b/docs-src/guides/programmer_reference/am_stat.md new file mode 100644 index 000000000..e1cda909d --- /dev/null +++ b/docs-src/guides/programmer_reference/am_stat.md @@ -0,0 +1,13 @@ +--- +title: "Database statistics" +api-name: "Database statistics" +source: docs/programmer_reference/am_stat.html +--- +## Database statistics + +The DB->stat() method returns a set of statistics about the underlying database, for example, the number of key/data pairs in the database, how the database was originally configured, and so on. + +There is a flag you can set to avoid time-consuming operations: + + DB_FAST_STAT +Return only information that can be acquired without traversing the entire database. diff --git a/docs-src/guides/programmer_reference/am_sync.md b/docs-src/guides/programmer_reference/am_sync.md new file mode 100644 index 000000000..971584ce3 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_sync.md @@ -0,0 +1,16 @@ +--- +title: "Flushing the database cache" +api-name: "Flushing the database cache" +source: docs/programmer_reference/am_sync.html +--- +## Flushing the database cache + +The DB->sync() method flushes all modified records from the database cache to disk. + +**It is important to understand that flushing cached information to disk only minimizes the window of opportunity for corrupted data, it does not eliminate the possibility.** + +While unlikely, it is possible for database corruption to happen if a system or application crash occurs while writing data to the database. To ensure that database corruption never occurs, applications must either: + +- Use transactions and logging with automatic recovery. +- Use logging and application-specific recovery. +- Edit a copy of the database, and, once all applications using the database have successfully called DB->close(), use system operations (for example, the POSIX rename system call) to atomically replace the original database with the updated copy. diff --git a/docs-src/guides/programmer_reference/am_truncate.md b/docs-src/guides/programmer_reference/am_truncate.md new file mode 100644 index 000000000..ea4613804 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_truncate.md @@ -0,0 +1,8 @@ +--- +title: "Database truncation" +api-name: "Database truncation" +source: docs/programmer_reference/am_truncate.html +--- +## Database truncation + +The DB->truncate() method empties a database of all records. diff --git a/docs-src/guides/programmer_reference/am_upgrade.md b/docs-src/guides/programmer_reference/am_upgrade.md new file mode 100644 index 000000000..97a9c8c5c --- /dev/null +++ b/docs-src/guides/programmer_reference/am_upgrade.md @@ -0,0 +1,12 @@ +--- +title: "Database upgrade" +api-name: "Database upgrade" +source: docs/programmer_reference/am_upgrade.html +--- +## Database upgrade + +When upgrading to a new release of Berkeley DB, it may be necessary to upgrade the on-disk format of already-created database files. **Berkeley DB database upgrades are done in place, and so are potentially destructive.** This means that if the system crashes during the upgrade procedure, or if the upgrade procedure runs out of disk space, the databases may be left in an inconsistent and unrecoverable state. To guard against failure, the procedures outlined in the Upgrading Berkeley DB installations chapter of the Berkeley DB Installation and Build Guide should be carefully followed. If you are not performing catastrophic archival as part of your application upgrade process, you should at least copy your database to archival media, verify that your archival media is error-free and readable, and that copies of your backups are stored offsite! + +The actual database upgrade is done using the DB->upgrade() method, or by dumping the database using the old version of the Berkeley DB software and reloading it using the current version. + +After an upgrade, Berkeley DB applications must be recompiled to use the new Berkeley DB library before they can access an upgraded database. **There is no guarantee that applications compiled against previous releases of Berkeley DB will work correctly with an upgraded database format. Nor is there any guarantee that applications compiled against newer releases of Berkeley DB will work correctly with the previous database format.** We do guarantee that any archived database may be upgraded using a current Berkeley DB software release and the DB->upgrade() method, and there is no need to step-wise upgrade the database using intermediate releases of Berkeley DB. Sites should consider archiving appropriate copies of their application or application sources if they may need to access archived databases without first upgrading them. diff --git a/docs-src/guides/programmer_reference/am_verify.md b/docs-src/guides/programmer_reference/am_verify.md new file mode 100644 index 000000000..1afcd8645 --- /dev/null +++ b/docs-src/guides/programmer_reference/am_verify.md @@ -0,0 +1,11 @@ +--- +title: "Database verification and salvage" +api-name: "Database verification and salvage" +source: docs/programmer_reference/am_verify.html +--- +## Database verification and salvage + +The DB->verify() method verifies that a file, and any databases it may contain, are uncorrupted. In addition, the method may optionally be called with a file stream argument to which all key/data pairs found in the database are output. There are two modes for finding key/data pairs to be output: + +1. If the DB_SALVAGE flag is specified, the key/data pairs in the database are output. When run in this mode, the database is assumed to be largely uncorrupted. For example, the DB->verify() method will search for pages that are no longer linked into the database, and will output key/data pairs from such pages. However, key/data items that have been marked as deleted in the database will not be output, as the page structures are generally trusted in this mode. +2. If both the DB_SALVAGE and DB_AGGRESSIVE flags are specified, all possible key/data pairs are output. When run in this mode, the database is assumed to be seriously corrupted. For example, key/data pairs that have been deleted will re-appear in the output. In addition, because pages may have been subsequently reused and modified during normal database operations after the key/data pairs were deleted, it is not uncommon for apparently corrupted key/data pairs to be output in this mode, even when there is no corruption in the underlying database. The output will almost always have to be edited by hand or other means before the data is ready for reload into another database. We recommend that DB_SALVAGE be tried first, and DB_AGGRESSIVE only tried if the output from that first attempt is obviously missing data items or the data is sufficiently valuable that human review of the output is preferable to any kind of data loss. diff --git a/docs-src/guides/programmer_reference/apprec.md b/docs-src/guides/programmer_reference/apprec.md new file mode 100644 index 000000000..ebc620a4d --- /dev/null +++ b/docs-src/guides/programmer_reference/apprec.md @@ -0,0 +1,36 @@ +--- +title: "Chapter 14.  Application Specific Logging and Recovery" +api-name: "Chapter 14.  Application Specific Logging and Recovery" +source: docs/programmer_reference/apprec.html +--- +## Chapter 14.  Application Specific Logging and Recovery + +**Table of Contents** + + [Introduction to application specific logging and recovery](apprec.md#apprec_intro) + + [Defining application-specific log records](apprec_def.md) + + [Automatically generated functions](apprec_auto.md) + + [Application configuration](apprec_config.md) + +## Introduction to application specific logging and recovery + +It is possible to use the Locking, Logging and Transaction subsystems of Berkeley DB to provide transaction semantics on objects other than those described by the Berkeley DB access methods. In these cases, the application will need application-specific logging and recovery functions. + +For example, consider an application that provides transaction semantics on data stored in plain text files accessed using the POSIX read and write system calls. The read and write operations for which transaction protection is desired will be bracketed by calls to the standard Berkeley DB transactional interfaces, DB_ENV->txn_begin() and DB_TXN->commit(), and the transaction's locker ID will be used to acquire relevant read and write locks. + +Before data is accessed, the application must make a call to the lock manager, DB_ENV->lock_get(), for a lock of the appropriate type (for example, read) on the object being locked. The object might be a page in the file, a byte, a range of bytes, or some key. It is up to the application to ensure that appropriate locks are acquired. Before a write is performed, the application should acquire a write lock on the object by making an appropriate call to the lock manager, DB_ENV->lock_get(). Then, the application should make a call to the log manager, via the automatically-generated log-writing function described as follows. This record should contain enough information to redo the operation in case of failure after commit and to undo the operation in case of abort. + +When designing applications that will use the log subsystem, it is important to remember that the application is responsible for providing any necessary structure to the log record. For example, the application must understand what part of the log record is an operation code, what part identifies the file being modified, what part is redo information, and what part is undo information. + +After the log message is written, the application may issue the write system call. After all requests are issued, the application may call DB_TXN->commit(). When DB_TXN->commit() returns, the caller is guaranteed that all necessary log writes have been written to disk. + +At any time before issuing a DB_TXN->commit(), the application may call DB_TXN->abort(), which will result in restoration of the database to a consistent pretransaction state. (The application may specify its own recovery function for this purpose using the DB_ENV->set_app_dispatch() method. The recovery function must be able to either reapply or undo the update depending on the context, for each different type of log record. The recovery functions must not use Berkeley DB methods to access data in the environment as there is no way to coordinate these accesses with either the aborting transaction or the updates done by recovery or replication.) + +If the application crashes, the recovery process uses the log to restore the database to a consistent state. + +Berkeley DB includes tools to assist in the development of application-specific logging and recovery. Specifically, given a description of information to be logged in a family of log records, these tools will automatically create log-writing functions (functions that marshall their arguments into a single log record), log-reading functions (functions that read a log record and unmarshall it into a structure containing fields that map into the arguments written to the log), log-printing functions (functions that print the contents of a log record for debugging), and templates for recovery functions (functions that review log records during transaction abort or recovery). The tools and generated code are C-language and POSIX-system based, but the generated code should be usable on any system, not just POSIX systems. + +A sample application that does application-specific recovery is included in the Berkeley DB distribution, in the directory `examples_c/ex_apprec`. diff --git a/docs-src/guides/programmer_reference/apprec_auto.md b/docs-src/guides/programmer_reference/apprec_auto.md new file mode 100644 index 000000000..0275a4524 --- /dev/null +++ b/docs-src/guides/programmer_reference/apprec_auto.md @@ -0,0 +1,134 @@ +--- +title: "Automatically generated functions" +api-name: "Automatically generated functions" +source: docs/programmer_reference/apprec_auto.html +--- +## Automatically generated functions + +The XXX.src file is processed using the gen_rec.awk script included in the dist directory of the Berkeley DB distribution. This is an awk script that is executed from with the following command line: + +``` c +awk -f gen_rec.awk \ + -v source_file=C_FILE \ + -v header_file=H_FILE \ + -v print_file=P_FILE \ + -v template_file=TMP_FILE < XXX.src +``` + +where *C_FILE* is the name of the file into which to place the automatically generated C code, *H_FILE* is the name of the file into which to place the automatically generated data structures and declarations, *P_FILE* is the name of the file into which to place the automatically generated C code that prints log records, and *TMP_FILE* is the name of the file into which to place a template for the recovery routines. + +Because the gen_rec.awk script uses sources files located relative to the Berkeley DB dist directory, it must be run from the dist directory. For example, in building the Berkeley DB logging and recovery routines for ex_apprec, the following script is used to rebuild the automatically generated files: + +``` c +E=../examples_c/ex_apprec + +cd ../../dist +awk -f gen_rec.awk \ + -v source_file=$E/ex_apprec_auto.c \ + -v header_file=$E/ex_apprec_auto.h \ + -v print_file=$E/ex_apprec_autop.c \ + -v template_file=$E/ex_apprec_template < $E/ex_apprec.src +``` + +For each log record description found in the XXX.src file, the following structure declarations and \#defines will be created in the file *header_file*: + +``` c +#define DB_PREFIX_RECORD_TYPE /* Integer ID number */ + +typedef struct _PREFIX_RECORD_TYPE_args { + /* + * These three fields are generated for every record. + */ + u_int32_t type; /* Record type used for dispatch. */ + + /* + * Transaction handle that identifies the transaction on whose + * behalf the record is being logged. + */ + DB_TXN *txnid; + + /* + * The log sequence number returned by the previous call to log_put + * for this transaction. + */ + DB_LSN *prev_lsn; + + /* + * The rest of the structure contains one field for each of + * the entries in the record statement. + */ +}; +``` + +Thus, the auto-generated ex_apprec_mkdir_args structure looks as follows: + +``` c +typedef struct _ex_apprec_mkdir_args { + u_int32_t type; + DB_TXN *txnid; + DB_LSN prev_lsn; + DBT dirname; +} ex_apprec_mkdir_args; +``` + +The template_file will contain a template for a recovery function. The recovery function is called on each record read from the log during system recovery, transaction abort, or the application of log records on a replication client, and is expected to redo or undo the operations described by that record. The details of the recovery function will be specific to the record being logged and need to be written manually, but the template provides a good starting point. (See ex_apprec_template and ex_apprec_rec.c for an example of both the template produced and the resulting recovery function.) + +The template file should be copied to a source file in the application (but not the automatically generated source_file, as that will get overwritten each time gen_rec.awk is run) and fully developed there. The recovery function takes the following parameters: + +> dbenv +> The environment in which recovery is running. +> +> rec +> The record being recovered. +> +> lsn +> The log sequence number of the record being recovered. The prev_lsn field, automatically included in every auto-generated log record, should be returned through this argument. The prev_lsn field is used to chain log records together to allow transaction aborts; because the recovery function is the only place that a log record gets parsed, the responsibility for returning this value lies with the recovery function writer. +> +> op +> A parameter of type db_recops, which indicates what operation is being run (DB_TXN_ABORT, DB_TXN_APPLY, DB_TXN_BACKWARD_ROLL, DB_TXN_FORWARD_ROLL or DB_TXN_PRINT). + +In addition to the header_file and template_file, a source_file is created, containing a log, read, recovery, and print function for each record type. + +The log function marshalls the parameters into a buffer, and calls DB_ENV->log_put() on that buffer returning 0 on success and non-zero on failure. The log function takes the following parameters: + +> dbenv +> The environment in which recovery is running. +> +> txnid +> The transaction identifier for the transaction handle returned by DB_ENV->txn_begin(). +> +> lsnp +> A pointer to storage for a log sequence number into which the log sequence number of the new log record will be returned. +> +> syncflag +> A flag indicating whether the record must be written synchronously. Valid values are 0 and DB_FLUSH. +> +> args +> The remaining parameters to the log message are the fields described in the XXX.src file, in order. + +The read function takes a buffer and unmarshalls its contents into a structure of the appropriate type. It returns 0 on success and non-zero on error. After the fields of the structure have been used, the pointer returned from the read function should be freed. The read function takes the following parameters: + +> dbenv +> The environment in which recovery is running. +> +> recbuf +> A buffer. +> +> argp +> A pointer to a structure of the appropriate type. + +The print function displays the contents of the record. The print function takes the same parameters as the recovery function described previously. Although some of the parameters are unused by the print function, taking the same parameters allows a single dispatch loop to dispatch to a variety of functions. The print function takes the following parameters: + +> dbenv +> The environment in which recovery is running. +> +> rec +> The record being recovered. +> +> lsn +> The log sequence number of the record being recovered. +> +> op +> Unused. + +Finally, the source file will contain a function (named XXX_init_print, where XXX is replaced by the prefix) which should be added to the initialization part of the standalone db_printlog utility code so that utility can be used to display application-specific log records. diff --git a/docs-src/guides/programmer_reference/apprec_config.md b/docs-src/guides/programmer_reference/apprec_config.md new file mode 100644 index 000000000..654e644e5 --- /dev/null +++ b/docs-src/guides/programmer_reference/apprec_config.md @@ -0,0 +1,63 @@ +--- +title: "Application configuration" +api-name: "Application configuration" +source: docs/programmer_reference/apprec_config.html +--- +## Application configuration + +The application should include a dispatch function that dispatches to appropriate printing and/or recovery functions based on the log record type and the operation code. The dispatch function should take the same arguments as the recovery function, and should call the appropriate recovery and/or printing functions based on the log record type and the operation code. For example, the ex_apprec dispatch function is as follows: + +``` c +int +apprec_dispatch(dbenv, dbt, lsn, op) + DB_ENV *dbenv; + DBT *dbt; + DB_LSN *lsn; + db_recops op; +{ + u_int32_t rectype; + /* Pull the record type out of the log record. */ + memcpy(&rectype, dbt->data, sizeof(rectype)); + switch (rectype) { + case DB_ex_apprec_mkdir: + return (ex_apprec_mkdir_recover(dbenv, dbt, lsn, op)); + default: + /* + * We've hit an unexpected, allegedly user-defined record + * type. + */ + dbenv->errx(dbenv, "Unexpected log record type encountered"); + return (EINVAL); + } +} +``` + +Applications use this dispatch function and the automatically generated functions as follows: + +1. When the application starts, call the DB_ENV->set_app_dispatch() with your dispatch function. +2. Issue a DB_ENV->txn_begin() call before any operations you want to be transaction-protected. +3. Before accessing any data, issue the appropriate lock call to lock the data (either for reading or writing). +4. Before modifying any data that is transaction-protected, issue a call to the appropriate log function. +5. Call DB_TXN->commit() to cancel all of the modifications. + +The recovery functions are called in the three following cases: + +1. During recovery after application or system failure, with op set to DB_TXN_FORWARD_ROLL or DB_TXN_BACKWARD_ROLL. +2. During transaction abort, with op set to DB_TXN_ABORT. +3. On a replicated client to apply updates from the master, with op set to DB_TXN_APPLY. + +For each log record type you declare, you must write the appropriate function to undo and redo the modifications. The shell of these functions will be generated for you automatically, but you must fill in the details. + +Your code must be able to detect whether the described modifications have been applied to the data. The function will be called with the "op" parameter set to DB_TXN_ABORT when a transaction that wrote the log record aborts, with DB_TXN_FORWARD_ROLL and DB_TXN_BACKWARD_ROLL during recovery, and with DB_TXN_APPLY on a replicated client. + +The actions for DB_TXN_ABORT and DB_TXN_BACKWARD_ROLL should generally be the same, and the actions for DB_TXN_FORWARD_ROLL and DB_TXN_APPLY should generally be the same. However, if the application is using Berkeley DB replication and another thread of control may be performing read operations while log records are applied on a replication client, the recovery function should perform appropriate locking during DB_TXN_APPLY operations. In this case, the recovery function may encounter deadlocks when issuing locking calls. The application should run with the deadlock detector, and the recovery function should simply return DB_LOCK_DEADLOCK if a deadlock is detected and a locking operation fails with that error. + +The DB_TXN_PRINT operation should print the log record, typically using the auto-generated print function; it is not used in the Berkeley DB library, but may be useful for debugging, as in the db_printlog utility. Applications may safely ignore this operation code, they may handle printing from the recovery function, or they may dispatch directly to the auto-generated print function. + +One common way to determine whether operations need to be undone or redone is the use of log sequence numbers (LSNs). For example, each access method database page contains the LSN of the most recent log record that describes a modification to the page. When the access method changes a page, it writes a log record describing the change and including the LSN that was on the page before the change. This LSN is referred to as the previous LSN. The recovery functions read the page described by a log record, and compare the LSN on the page to the LSN they were passed. + +If the page LSN is less than the passed LSN and the operation is an undo, no action is necessary (because the modifications have not been written to the page). If the page LSN is the same as the previous LSN and the operation is a redo, the actions described are reapplied to the page. If the page LSN is equal to the passed LSN and the operation is an undo, the actions are removed from the page; if the page LSN is greater than the passed LSN and the operation is a redo, no further action is necessary. If the action is a redo and the LSN on the page is less than the previous LSN in the log record, it is an error because it could happen only if some previous log record was not processed. + +Examples of other recovery functions can be found in the Berkeley DB library recovery functions (found in files named XXX_rec.c) and in the application-specific recovery example (specifically, ex_apprec_rec.c). + +Finally, applications need to ensure that any data modifications they have made, that were part of a committed transaction, must be written to stable storage before calling the DB_ENV->txn_checkpoint() method. This is to allow the periodic removal of database environment log files. diff --git a/docs-src/guides/programmer_reference/apprec_def.md b/docs-src/guides/programmer_reference/apprec_def.md new file mode 100644 index 000000000..e33c85916 --- /dev/null +++ b/docs-src/guides/programmer_reference/apprec_def.md @@ -0,0 +1,84 @@ +--- +title: "Defining application-specific log records" +api-name: "Defining application-specific log records" +source: docs/programmer_reference/apprec_def.html +--- +## Defining application-specific log records + +By convention, log records are described in files named `XXX.src`, where "XXX" is typically a descriptive name for a subsystem or other logical group of logging functions. These files contain interface definition language descriptions for each type of log record that is used by the subsystem. + +All blank lines and lines beginning with a hash ("#") character in the XXX.src files are ignored. + +The first non-comment line in the file should begin with the keyword PREFIX, followed by a string that will be prepended to every generated function name. Frequently, the PREFIX is either identical or similar to the name of the `XXX.src` file. For example, the Berkeley DB application-specific recovery example uses the file `ex_apprec.src`, which begins with the following PREFIX line: + +``` c +PREFIX ex_apprec +``` + +Following the PREFIX line are the include files required by the automatically generated functions. The include files should be listed in order, prefixed by the keyword INCLUDE. For example, the Berkeley DB application-specific recovery example lists the following include files: + +``` c +INCLUDE #include "ex_apprec.h" +``` + +The rest of the XXX.src file consists of log record descriptions. Each log record description begins with one of the following lines: + +``` c +BEGIN RECORD_NAME DB_VERSION_NUMBER RECORD_NUMBER +``` + +``` c +BEGIN_COMPAT RECORD_NAME DB_VERSION_NUMBER RECORD_NUMBER +``` + +and ends with the line: + +``` c +END +``` + +The *BEGIN* line should be used for most record types. + +The *BEGIN_COMPAT* is used for log record compatibility to facilitate online upgrades of replication groups. Records created with this keyword will produce reading and printing routines, but no logging routines. The recovery routines are retrieved from older releases, so no recovery templates will be generated for these records. + +The *DB_VERSION_NUMBER* variable should be replaced with the current major and minor version of Berkeley DB, with all punctuation removed. For example, Berkeley DB version 4.2 should be 42, version 4.5 should be 45. + +The *RECORD_NAME* variable should be replaced with a record name for this log record. The *RECORD_NUMBER* variable should be replaced with a record number. + +The combination of PREFIX name and *RECORD_NAME*, and the *RECORD_NUMBER* must be unique for the application, that is, values for application-specific and Berkeley DB log records may not overlap. Further, because record numbers are stored in log files, which are usually portable across application and Berkeley DB releases, any change to the record numbers or log record format or should be handled as described in the section on log format changes in the Upgrading Berkeley DB installations chapter of the Berkeley DB Installation and Build Guide. The record number space below 10,000 is reserved for Berkeley DB itself; applications should choose record number values equal to or greater than 10,000. + +Between the BEGIN and END keywords there should be one optional *DUPLICATE* line and one line for each data item logged as part of this log record. + +The *DUPLICATE* line is of the form: + +``` c +DUPLICATE RECORD_NAME DB_VERSION_NUMBER RECORD_NUMBER +``` + +The *DUPLICATE* specifier should be used when creating a record that requires its own record number but can use the argument structure, reading and printing routines from another record. In this case, we will create a new log record type, but use the enclosing log record type for the argument structure and the log reading and printing routines. + +The format of lines for each data item logged is as follows: + +``` c +ARG | DBT | POINTER variable_name variable_type printf_format +``` + +The keyword ARG indicates that the argument is a simple parameter of the type specified. For example, a file ID might be logged as: + +``` c +ARG fileID int d +``` + +The keyword DBT indicates that the argument is a Berkeley DB DBT structure, containing a length and pointer to a byte string. The keyword POINTER indicates that the argument is a pointer to the data type specified (of course the data type, not the pointer, is what is logged). + +The *variable_name* is the field name within the structure that will be used to refer to this item. The *variable_type* is the C-language type of the variable, and the printf format is the C-language format string, without the leading percent ("%") character, that should be used to display the contents of the field (for example, "s" for string, "d" for signed integral type, "u" for unsigned integral type, "ld" for signed long integral type, "lu" for long unsigned integral type, and so on). + +For example, ex_apprec.src defines a single log record type, used to log a directory name that has been stored in a DBT: + +``` c +BEGIN mkdir 10000 +DBT dirname DBT s +END +``` + +As the name suggests, this example of an application-defined log record will be used to log the creation of a directory. There are many more examples of XXX.src files in the Berkeley DB distribution. For example, the file btree/btree.src contains the definitions for the log records supported by the Berkeley DB Btree access method. diff --git a/docs-src/guides/programmer_reference/arch.md b/docs-src/guides/programmer_reference/arch.md new file mode 100644 index 000000000..2fcdbf60f --- /dev/null +++ b/docs-src/guides/programmer_reference/arch.md @@ -0,0 +1,120 @@ +--- +title: "Chapter 8.  Berkeley DB Architecture" +api-name: "Chapter 8.  Berkeley DB Architecture" +source: docs/programmer_reference/arch.html +--- +## Chapter 8.  Berkeley DB Architecture + +**Table of Contents** + + [The big picture](arch.md#arch_bigpic) + + [Programming model](arch_progmodel.md) + + [Programmatic APIs](arch_apis.md) + + [C](arch_apis.md#idp51640232) + + [C++](arch_apis.md#idp51656168) + + [STL](arch_apis.md#idp51646944) + + [Java](arch_apis.md#idp51647760) + + [Dbm/Ndbm, Hsearch](arch_apis.md#idp51664896) + + [Scripting languages](arch_script.md) + + [Perl](arch_script.md#idp51640920) + + [PHP](arch_script.md#idp51639128) + + [Tcl](arch_script.md#idp51657264) + + [Supporting utilities](arch_utilities.md) + +## The big picture + +The previous chapters in this Reference Guide have described applications that use the Berkeley DB access methods for fast data storage and retrieval. The applications described in the following chapters are similar in nature to the access method applications, but they are also threaded and/or recoverable in the face of application or system failure. + +Application code that uses only the Berkeley DB access methods might appear as follows: + +``` c +switch (ret = dbp->/put(dbp, NULL, &key, &data, 0)) { +case 0: + printf("db: %s: key stored.\n", (char *)key.data); + break; +default: + dbp->/err(dbp, ret, "dbp->/put"); + exit (1); +} +``` + +The underlying Berkeley DB architecture that supports this is + +![](arch_smallpic.gif) + +As you can see from this diagram, the application makes calls into the access methods, and the access methods use the underlying shared memory buffer cache to hold recently used file pages in main memory. + +When applications require recoverability, their calls to the Access Methods must be wrapped in calls to the transaction subsystem. The application must inform Berkeley DB where to begin and end transactions, and must be prepared for the possibility that an operation may fail at any particular time, causing the transaction to abort. + +An example of transaction-protected code might appear as follows: + +``` c +for (fail = 0;;) { + /* Begin the transaction. */ + if ((ret = dbenv->/txn_begin(dbenv, NULL, &tid, 0)) != 0) { + dbenv->/err(dbenv, ret, "dbenv->/txn_begin"); + exit (1); + } + + /* Store the key. */ + switch (ret = dbp->/put(dbp, tid, &key, &data, 0)) { + case 0: + /* Success: commit the change. */ + printf("db: %s: key stored.\n", (char *)key.data); + if ((ret = tid->/commit(tid, 0)) != 0) { + dbenv->/err(dbenv, ret, "DB_TXN->/commit"); + exit (1); + } + return (0); + case DB_LOCK_DEADLOCK: + default: + /* Failure: retry the operation. */ + if ((t_ret = tid->/abort(tid)) != 0) { + dbenv->/err(dbenv, t_ret, "DB_TXN->/abort"); + exit (1); + } + if (fail++ == MAXIMUM_RETRY) + return (ret); + continue; + } +} +``` + +In this example, the same operation is being done as before; however, it is wrapped in transaction calls. The transaction is started with DB_ENV->txn_begin() and finished with DB_TXN->commit(). If the operation fails due to a deadlock, the transaction is aborted using DB_TXN->abort(), after which the operation may be retried. + +There are actually five major subsystems in Berkeley DB, as follows: + +Access Methods +The access methods subsystem provides general-purpose support for creating and accessing database files formatted as Btrees, Hashed files, and Fixed- and Variable-length records. These modules are useful in the absence of transactions for applications that need fast formatted file support. See DB->open() and DB->cursor() for more information. These functions were already discussed in detail in the previous chapters. + +Memory Pool +The Memory Pool subsystem is the general-purpose shared memory buffer pool used by Berkeley DB. This is the shared memory cache that allows multiple processes and threads within processes to share access to databases. This module is useful outside of the Berkeley DB package for processes that require portable, page-oriented, cached, shared file access. + +Transaction +The Transaction subsystem allows a group of database changes to be treated as an atomic unit so that either all of the changes are done, or none of the changes are done. The transaction subsystem implements the Berkeley DB transaction model. This module is useful outside of the Berkeley DB package for processes that want to transaction-protect their own data modifications. + +Locking +The Locking subsystem is the general-purpose lock manager used by Berkeley DB. This module is useful outside of the Berkeley DB package for processes that require a portable, fast, configurable lock manager. + +Logging +The Logging subsystem is the write-ahead logging used to support the Berkeley DB transaction model. It is largely specific to the Berkeley DB package, and unlikely to be useful elsewhere except as a supporting module for the Berkeley DB transaction subsystem. + +Here is a more complete picture of the Berkeley DB library: + +![](arch_bigpic.gif) + +In this model, the application makes calls to the access methods and to the Transaction subsystem. The access methods and Transaction subsystems in turn make calls into the Memory Pool, Locking and Logging subsystems on behalf of the application. + +The underlying subsystems can be used independently by applications. For example, the Memory Pool subsystem can be used apart from the rest of Berkeley DB by applications simply wanting a shared memory buffer pool, or the Locking subsystem may be called directly by applications that are doing their own locking outside of Berkeley DB. However, this usage is not common, and most applications will either use only the access methods subsystem, or the access methods subsystem wrapped in calls to the Berkeley DB transaction interfaces. diff --git a/docs-src/guides/programmer_reference/arch_apis.md b/docs-src/guides/programmer_reference/arch_apis.md new file mode 100644 index 000000000..4ddd53d73 --- /dev/null +++ b/docs-src/guides/programmer_reference/arch_apis.md @@ -0,0 +1,80 @@ +--- +title: "Programmatic APIs" +api-name: "Programmatic APIs" +source: docs/programmer_reference/arch_apis.html +--- +## Programmatic APIs + + [C](arch_apis.md#idp51640232) + + [C++](arch_apis.md#idp51656168) + + [STL](arch_apis.md#idp51646944) + + [Java](arch_apis.md#idp51647760) + + [Dbm/Ndbm, Hsearch](arch_apis.md#idp51664896) + +The Berkeley DB subsystems can be accessed through interfaces from multiple languages. Applications can use Berkeley DB via C, C++ or Java, as well as a variety of scripting languages such as Perl, Python, Ruby or Tcl. Environments can be shared among applications written by using any of these interfaces. For example, you might have a local server written in C or C++, a script for an administrator written in Perl or Tcl, and a Web-based user interface written in Java -- all sharing a single database environment. + +### C + +The Berkeley DB library is written entirely in ANSI C. C applications use a single include file: + +``` c +#include +``` + +### C++ + +The C++ classes provide a thin wrapper around the C API, with the major advantages being improved encapsulation and an optional exception mechanism for errors. C++ applications use a single include file: + +``` c +#include +``` + +The classes and methods are named in a fashion that directly corresponds to structures and functions in the C interface. Likewise, arguments to methods appear in the same order as the C interface, except to remove the explicit **this** pointer. The \#defines used for flags are identical between the C and C++ interfaces. + +As a rule, each C++ object has exactly one structure from the underlying C API associated with it. The C structure is allocated with each constructor call and deallocated with each destructor call. Thus, the rules the user needs to follow in allocating and deallocating structures are the same between the C and C++ interfaces. + +To ensure portability to many platforms, both new and old, Berkeley DB makes as few assumptions as possible about the C++ compiler and library. For example, it does not expect STL, templates, or namespaces to be available. The newest C++ feature used is exceptions, which are used liberally to transmit error information. Even the use of exceptions can be disabled at runtime. + +### STL + +dbstl is an C++ STL style API for Berkeley DB, based on the C++ API above. With it, you can store data/objects of any type into or retrieve them from Berkeley DB databases as if you are using C++ STL containers. The full functionality of Berkeley DB can still be utilized via dbstl with little performance overhead, e.g. you can use all transaction and/or replication functionality of Berkeley DB. + +dbstl container/iterator class templates reside in header files dbstl_vector.h, dbstl_map.h and dbstl_set.h. Among them, dbstl_vector.h contains dbstl::db_vector and its iterators; dbstl_map.h contains dbstl::db_map, dbstl::db_multimap and their iterators; dbstl_set.h contains dbstl::db_set and dbstl::db_multiset and their iterators. You should include needed header file(s) to use the container/iterator. Note that we don't use the file name with no extention --- To use dbstl::db_vector, you should do this: + +``` c +#include "dbstl_vector.h" +``` + +rather than this: + +``` c +#include "dbstl_vector" +``` + +And these header files reside in "stl" directory inside Berkeley DB source root directory. If you have installed Berkeley DB, they are also available in the "include" directory in the directory where Berkeley DB is installed. + +Apart from the above three header files, you may also need to include db_exception.h and db_utility.h files. The db_exception.h file contains all exception classes of dbstl, which integrate seamlessly with Berkeley DB C++ API exceptions and C++ standard exception classes in std namespace. And the db_utility.h file contains the DbstlElemTraits which helps you to store complex objects. These five header files are all that you need to include in order to make use of dbstl. + +All symbols of dbstl, including classes, class templates, global functions, etc, reside in the namespace "dbstl", so in order to use them, you may also want to do this: + +``` c +using namespace dbstl; +``` + +The dbstl library is always at the same place where Berkeley DB library is located, you will need to build it and link with it to use dbstl. + +While making use of dbstl, you will probably want to create environment or databases directly, or set/get configurations to Berkeley DB environment or databases, etc. You are allowed to do so via Berkeley DB C/C++ API. + +### Java + +The Java classes provide a layer around the C API that is almost identical to the C++ layer. The classes and methods are, for the most part identical to the C++ layer. Berkeley DB constants and \#defines are represented as "static final int" values. Error conditions are communicated as Java exceptions. + +As in C++, each Java object has exactly one structure from the underlying C API associated with it. The Java structure is allocated with each constructor or open call, but is deallocated only by the Java garbage collector. Because the timing of garbage collection is not predictable, applications should take care to do a close when finished with any object that has a close method. + +### Dbm/Ndbm, Hsearch + +Berkeley DB supports the standard UNIX dbm and hsearch interfaces. After including a new header file and recompiling, programs will run orders of magnitude faster, and underlying databases can grow as large as necessary. Also, historic dbm applications can fail once some number of entries are inserted into the database, in which the number depends on the effectiveness of the internal hashing function on the particular data set. This is not a problem with Berkeley DB. diff --git a/docs-src/guides/programmer_reference/arch_progmodel.md b/docs-src/guides/programmer_reference/arch_progmodel.md new file mode 100644 index 000000000..aec3117b0 --- /dev/null +++ b/docs-src/guides/programmer_reference/arch_progmodel.md @@ -0,0 +1,8 @@ +--- +title: "Programming model" +api-name: "Programming model" +source: docs/programmer_reference/arch_progmodel.html +--- +## Programming model + +Berkeley DB is a database library, in which the library is linked into the address space of the application using it. One or more applications link the Berkeley DB library directly into their address spaces. There may be many threads of control in this model because Berkeley DB supports locking for both multiple processes and for multiple threads within a process. This model provides significantly faster access to the database functionality, but implies trust among all threads of control sharing the database environment because they will have the ability to read, write and potentially corrupt each other's data. diff --git a/docs-src/guides/programmer_reference/arch_script.md b/docs-src/guides/programmer_reference/arch_script.md new file mode 100644 index 000000000..0782b68b8 --- /dev/null +++ b/docs-src/guides/programmer_reference/arch_script.md @@ -0,0 +1,24 @@ +--- +title: "Scripting languages" +api-name: "Scripting languages" +source: docs/programmer_reference/arch_script.html +--- +## Scripting languages + + [Perl](arch_script.md#idp51640920) + + [PHP](arch_script.md#idp51639128) + + [Tcl](arch_script.md#idp51657264) + +### Perl + +Two Perl wrappers are distributed with the Berkeley DB release. The Perl interface to Berkeley DB version 1.85 is called DB_File. The Perl interface to Berkeley DB version 2 and later is called BerkeleyDB. See Using Berkeley DB with Perl for more information. + +### PHP + +A PHP wrapper is distributed with the Berkeley DB release. See Using Berkeley DB with PHP for more information. + +### Tcl + +A Tcl wrapper is distributed with the Berkeley DB release. See Loading Berkeley DB with Tcl for more information. diff --git a/docs-src/guides/programmer_reference/arch_utilities.md b/docs-src/guides/programmer_reference/arch_utilities.md new file mode 100644 index 000000000..32f72203d --- /dev/null +++ b/docs-src/guides/programmer_reference/arch_utilities.md @@ -0,0 +1,49 @@ +--- +title: "Supporting utilities" +api-name: "Supporting utilities" +source: docs/programmer_reference/arch_utilities.html +--- +## Supporting utilities + +The following are the standalone utilities that provide supporting functionality for the Berkeley DB environment: + +db_archive utility +The db_archive utility supports database backup and archival, and log file administration. It facilitates log reclamation and the creation of database snapshots. Generally, some form of log archival must be done if a database environment has been configured for logging or transactions. + +db_checkpoint utility +The db_checkpoint utility runs as a daemon process, monitoring the database log and periodically issuing checkpoints. It facilitates log reclamation and the creation of database snapshots. Generally, some form of database checkpointing must be done if a database environment has been configured for transactions. + +db_deadlock utility +The db_deadlock utility runs as a daemon process, periodically traversing the database lock structures and aborting transactions when it detects a deadlock. Generally, some form of deadlock detection must be done if a database environment has been configured for locking. + +db_dump utility +The db_dump utility writes a copy of the database to a flat-text file in a portable format. + +db_hotbackup utility +The db_hotbackup utility creates "hot backup" or "hot failover" snapshots of Berkeley DB database environments. + +db_load utility +The db_load utility reads the flat-text file produced by the db_load utility and loads it into a database file. + +db_printlog utility +The db_printlog utility displays the contents of Berkeley DB log files in a human-readable and parsable format. + +db_recover utility +The db_recover utility runs after an unexpected Berkeley DB or system failure to restore the database to a consistent state. Generally, some form of database recovery must be done if databases are being modified. + +db_sql_codegen +The db_sql_codegen utility translates a schema description written in a SQL Data Definition Language dialect into C code that implements the schema using Berkeley DB. + +db_stat utility +The db_stat utility displays statistics for databases and database environments. + +db_tuner utility +The db_tuner utility suggests a page size for btree databases that optimizes cache efficiency and storage space requirements. + +db_upgrade utility +The db_upgrade utility provides a command-line interface for upgrading underlying database formats. + +db_verify utility +The db_verify utility provides a command-line interface for verifying the database format. + +All of the functionality implemented for these utilities is also available as part of the standard Berkeley DB API. This means that threaded applications can easily create a thread that calls the same Berkeley DB functions as do the utilities. This often simplifies an application environment by removing the necessity for multiple processes to negotiate database and database environment creation and shut down. diff --git a/docs-src/guides/programmer_reference/bt_conf.md b/docs-src/guides/programmer_reference/bt_conf.md new file mode 100644 index 000000000..aee362a6d --- /dev/null +++ b/docs-src/guides/programmer_reference/bt_conf.md @@ -0,0 +1,410 @@ +--- +title: "Btree access method specific configuration" +api-name: "Btree access method specific configuration" +source: docs/programmer_reference/bt_conf.html +--- +## Btree access method specific configuration + + [Btree comparison](bt_conf.md#am_conf_bt_compare) + + [Btree prefix comparison](bt_conf.md#am_conf_bt_prefix) + + [Minimum keys per page](bt_conf.md#am_conf_bt_minkey) + + [Retrieving Btree records by logical record number](bt_conf.md#am_conf_bt_recnum) + + [Compression](bt_conf.md#am_conf_bt_compress) + +There are a series of configuration tasks which you can perform when using the Btree access method. They are described in the following sections. + +### Btree comparison + +The Btree data structure is a sorted, balanced tree structure storing associated key/data pairs. By default, the sort order is lexicographical, with shorter keys collating before longer keys. The user can specify the sort order for the Btree by using the DB->set_bt_compare() method. + +Sort routines are passed pointers to keys as arguments. The keys are represented as DBT structures. The routine must return an integer less than, equal to, or greater than zero if the first argument is considered to be respectively less than, equal to, or greater than the second argument. The only fields that the routines may examine in the DBT structures are **data** and **size** fields. + +An example routine that might be used to sort integer keys in the database is as follows: + +``` c +int +compare_int(DB *dbp, const DBT *a, const DBT *b) + +{ + int ai, bi; + /* + * Returns: + * < 0 if a < b + * = 0 if a = b + * > 0 if a > b + */ + memcpy(&ai, a->data, sizeof(int)); + memcpy(&bi, b->data, sizeof(int)); + return (ai - bi); +} +``` + +Note that the data must first be copied into memory that is appropriately aligned, as Berkeley DB does not guarantee any kind of alignment of the underlying data, including for comparison routines. When writing comparison routines, remember that databases created on machines of different architectures may have different integer byte orders, for which your code may need to compensate. + +An example routine that might be used to sort keys based on the first five bytes of the key (ignoring any subsequent bytes) is as follows: + +``` c +int +compare_dbt(DB *dbp, const DBT *a, const DBT *b) + +{ + int len; + u_char *p1, *p2; + + /* + * Returns: + * < 0 if a < b + * = 0 if a = b + * > 0 if a > b + */ + for (p1 = a->data, p2 = b->data, len = 5; len--; ++p1, ++p2) + if (*p1 != *p2) + return ((long)*p1 - (long)*p2); + return (0); +} +``` + +All comparison functions must cause the keys in the database to be well-ordered. The most important implication of being well-ordered is that the key relations must be transitive, that is, if key A is less than key B, and key B is less than key C, then the comparison routine must also return that key A is less than key C. + +It is reasonable for a comparison function to not examine an entire key in some applications, which implies partial keys may be specified to the Berkeley DB interfaces. When partial keys are specified to Berkeley DB, interfaces which retrieve data items based on a user-specified key (for example, DB->get() and DBC->get() with the DB_SET flag), will modify the user-specified key by returning the actual key stored in the database. + +### Btree prefix comparison + +The Berkeley DB Btree implementation maximizes the number of keys that can be stored on an internal page by storing only as many bytes of each key as are necessary to distinguish it from adjacent keys. The prefix comparison routine is what determines this minimum number of bytes (that is, the length of the unique prefix), that must be stored. A prefix comparison function for the Btree can be specified by calling DB->set_bt_prefix(). + +The prefix comparison routine must be compatible with the overall comparison function of the Btree, since what distinguishes any two keys depends entirely on the function used to compare them. This means that if a prefix comparison routine is specified by the application, a compatible overall comparison routine must also have been specified. + +Prefix comparison routines are passed pointers to keys as arguments. The keys are represented as DBT structures. The only fields the routines may examine in the DBT structures are **data** and **size** fields. + +The prefix comparison function must return the number of bytes necessary to distinguish the two keys. If the keys are identical (equal and equal in length), the length should be returned. If the keys are equal up to the smaller of the two lengths, then the length of the smaller key plus 1 should be returned. + +An example prefix comparison routine follows: + +``` c +size_t +compare_prefix(DB *dbp, const DBT *a, const DBT *b) + +{ + size_t cnt, len; + u_int8_t *p1, *p2; + + cnt = 1; + len = a->size > b->size ? b->size : a->size; + for (p1 = + a->data, p2 = b->data; len--; ++p1, ++p2, ++cnt) + if (*p1 != *p2) + return (cnt); + /* + * They match up to the smaller of the two sizes. + * Collate the longer after the shorter. + */ + if (a->size < b->size) + return (a->size + 1); + if (b->size < a->size) + return (b->size + 1); + return (b->size); +} +``` + +The usefulness of this functionality is data-dependent, but in some data sets can produce significantly reduced tree sizes and faster search times. + +### Minimum keys per page + +The number of keys stored on each page affects the size of a Btree and how it is maintained. Therefore, it also affects the retrieval and search performance of the tree. For each Btree, Berkeley DB computes a maximum key and data size. This size is a function of the page size and the fact that at least two key/data pairs must fit on any Btree page. Whenever key or data items exceed the calculated size, they are stored on overflow pages instead of in the standard Btree leaf pages. + +Applications may use the DB->set_bt_minkey() method to change the minimum number of keys that must fit on a Btree page from two to another value. Altering this value in turn alters the on-page maximum size, and can be used to force key and data items which would normally be stored in the Btree leaf pages onto overflow pages. + +Some data sets can benefit from this tuning. For example, consider an application using large page sizes, with a data set almost entirely consisting of small key and data items, but with a few large items. By setting the minimum number of keys that must fit on a page, the application can force the outsized items to be stored on overflow pages. That in turn can potentially keep the tree more compact, that is, with fewer internal levels to traverse during searches. + +The following calculation is similar to the one performed by the Btree implementation. (The **minimum_keys** value is multiplied by 2 because each key/data pair requires 2 slots on a Btree page.) + +``` c +maximum_size = page_size / (minimum_keys * 2) +``` + +Using this calculation, if the page size is 8KB and the default **minimum_keys** value of 2 is used, then any key or data items larger than 2KB will be forced to an overflow page. If an application were to specify a **minimum_key** value of 100, then any key or data items larger than roughly 40 bytes would be forced to overflow pages. + +It is important to remember that accesses to overflow pages do not perform as well as accesses to the standard Btree leaf pages, and so setting the value incorrectly can result in overusing overflow pages and decreasing the application's overall performance. + +### Retrieving Btree records by logical record number + +The Btree access method optionally supports retrieval by logical record numbers. To configure a Btree to support record numbers, call the DB->set_flags() method with the DB_RECNUM flag. + +Configuring a Btree for record numbers should not be done lightly. While often useful, it may significantly slow down the speed at which items can be stored into the database, and can severely impact application throughput. Generally it should be avoided in trees with a need for high write concurrency. + +To retrieve by record number, use the DB_SET_RECNO flag to the DB->get() and DBC->get() methods. The following is an example of a routine that displays the data item for a Btree database created with the DB_RECNUM option. + +``` c +int +rec_display(DB *dbp, db_recno_t recno) + +{ + DBT key, data; + int ret; + + memset(&key, 0, sizeof(key)); + key.data = &recno; + key.size = sizeof(recno); + memset(&data, 0, sizeof(data)); + + if ((ret = dbp->get(dbp, NULL, &key, &data, DB_SET_RECNO)) != 0) + return (ret); + printf("data for %lu: %.*s\n", + (u_long)recno, (int)data.size, (char *)data.data); + return (0); +} +``` + +To determine a key's record number, use the DB_GET_RECNO flag to the DBC->get() method. The following is an example of a routine that displays the record number associated with a specific key. + +``` c +int +recno_display(DB *dbp, char *keyvalue) + +{ + DBC *dbcp; + DBT key, data; + db_recno_t recno; + int ret, t_ret; + + /* Acquire a cursor for the database. */ + if ((ret = dbp->cursor(dbp, NULL, &dbcp, 0)) != 0) { + dbp->err(dbp, ret, "DB->cursor"); + goto err; + } + + /* Position the cursor. */ + memset(&key, 0, sizeof(key)); + key.data = keyvalue; + key.size = strlen(keyvalue); + memset(&data, 0, sizeof(data)); + if ((ret = dbcp->get(dbcp, &key, &data, DB_SET)) != 0) { + dbp->err(dbp, ret, "DBC->get(DB_SET): %s", keyvalue); + goto err; + } + + /* + * Request the record number, and store it into appropriately + * sized and aligned local memory. + */ + memset(&data, 0, sizeof(data)); + data.data = &recno; + data.ulen = sizeof(recno); + data.flags = DB_DBT_USERMEM; + if ((ret = dbcp->get(dbcp, &key, &data, DB_GET_RECNO)) != 0) { + dbp->err(dbp, ret, "DBC->get(DB_GET_RECNO)"); + goto err; + } + + printf("key for requested key was %lu\n", (u_long)recno); + +err: /* Close the cursor. */ + if ((t_ret = dbcp->close(dbcp)) != 0) { + if (ret == 0) + ret = t_ret; + dbp->err(dbp, ret, "DBC->close"); + } + return (ret); +} +``` + +### Compression + +The Btree access method supports the automatic compression of key/data pairs upon their insertion into the database. The key/data pairs are decompressed before they are returned to the application, making an application's interaction with a compressed database identical to that for a non-compressed database. To configure Berkeley DB for compression, call the DB->set_bt_compress() method and specify custom compression and decompression functions. If DB->set_bt_compress() is called with NULL compression and decompression functions, Berkeley DB will use its default compression functions. + +### Note + +Compression only works with the Btree access method, and then only so long as your database is not configured for unsorted duplicates. + +### Note + +The default compression function is not guaranteed to reduce the size of the on-disk database in every case. It has been tested and shown to work well with English-language text. Of course, in order to determine if the default compression algorithm is beneficial for your application, it is important to test both the final size and the performance using a representative set of data and access patterns. + +The default compression function performs prefix compression on each key added to the database. This means that, for a key *n* bytes in length, the first *i* bytes that match the first *i* bytes of the previous key exactly are omitted and only the final *n-i* bytes are stored in the database. If the bytes of key being stored match the bytes of the previous key exactly, then the same prefix compression algorithm is applied to the data value being stored. To use Berkeley DB's default compression behavior, both the default compression and decompression functions must be used. + +For example, to configure your database for default compression: + +``` c + DB *dbp = NULL; + DB_ENV *envp = NULL; + u_int32_t db_flags; + const char *file_name = "mydb.db"; + int ret; + +... + + /* Skipping environment open to shorten this example */ + /* Initialize the DB handle */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + fprintf(stderr, "%s\n", db_strerror(ret)); + return (EXIT_FAILURE); + } + + /* Turn on default data compression */ + dbp->set_bt_compress(dbp, NULL, NULL); + + /* Now open the database */ + db_flags = DB_CREATE; /* Allow database creation */ + + ret = dbp->open(dbp, /* Pointer to the database */ + NULL, /* Txn pointer */ + file_name, /* File name */ + NULL, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database '%s' open failed", + file_name); + return (EXIT_FAILURE); + } +``` + +#### Custom compression + +An application wishing to perform its own compression may supply a compression and decompression function which will be called instead of Berkeley DB's default functions. The compression function is passed five DBT structures: + +- The key and data immediately preceeding the key/data pair that is being stored. + +- The key and data being stored in the tree. + +- The buffer where the compressed data should be written. + +The total size of the buffer used to store the compressed data is identified in the DBT's `ulen` field. If the compressed data cannot fit in the buffer, the compression function should store the amount of space needed in DBT's `size` field and then return `DB_BUFFER_SMALL`. Berkeley DB will subsequently re-call the compression function with the required amount of space allocated in the compression data buffer. + +Multiple compressed key/data pairs will likely be written to the same buffer and the compression function should take steps to ensure it does not overwrite data. + +For example, the following code fragments illustrate the use of a custom compression routine. This code is actually a much simplified example of the default compression provided by Berkeley DB. It does simple prefix compression on the key part of the data. + +``` c + int compress(DB *dbp, const DBT *prevKey, const DBT *prevData, + const DBT *key, const DBT *data, DBT *dest) +{ + u_int8_t *dest_data_ptr; + const u_int8_t *key_data, *prevKey_data; + size_t len, prefix, suffix; + + key_data = (const u_int8_t*)key->data; + prevKey_data = (const u_int8_t*)prevKey->data; + len = key->size > prevKey->size ? prevKey->size : key->size; + for (; len-- && *key_data == *prevKey_data; ++key_data, + ++prevKey_data) + continue; + + prefix = (size_t)(key_data - (u_int8_t*)key->data); + suffix = key->size - prefix; + + /* Check that we have enough space in dest */ + dest->size = (u_int32_t)(__db_compress_count_int(prefix) + + __db_compress_count_int(suffix) + + __db_compress_count_int(data->size) + suffix + data->size); + if (dest->size > dest->ulen) + return (DB_BUFFER_SMALL); + + /* prefix length */ + dest_data_ptr = (u_int8_t*)dest->data; + dest_data_ptr += __db_compress_int(dest_data_ptr, prefix); + + /* suffix length */ + dest_data_ptr += __db_compress_int(dest_data_ptr, suffix); + + /* data length */ + dest_data_ptr += __db_compress_int(dest_data_ptr, data->size); + + /* suffix */ + memcpy(dest_data_ptr, key_data, suffix); + dest_data_ptr += suffix; + + /* data */ + memcpy(dest_data_ptr, data->data, data->size); + + return (0); +} +``` + +The corresponding decompression function is likewise passed five DBT structures: + +- The key and data DBTs immediately preceding the decompressed key and data. + +- The compressed data from the database. + +- One to store the decompressed key and another one for the decompressed data. + +Because the compression of `record X` relies upon `record X-1`, the decompression function can be called repeatedly to linearally decompress a set of records stored in the compressed buffer. + +The total size of the buffer available to store the decompressed data is identified in the destination DBT's `ulen` field. If the decompressed data cannot fit in the buffer, the decompression function should store the amount of space needed in the destination DBT's `size` field and then return `DB_BUFFER_SMALL`. Berkeley DB will subsequently re-call the decompression function with the required amount of space allocated in the decompression data buffer. + +For example, the decompression routine that corresponds to the example compression routine provided above is: + +``` c +int decompress(DB *dbp, const DBT *prevKey, const DBT *prevData, + DBT *compressed, DBT *destKey, DBT *destData) +{ + u_int8_t *comp_data, *dest_data; + u_int32_t prefix, suffix, size; + + /* Unmarshal prefix, suffix and data length */ + comp_data = (u_int8_t*)compressed->data; + size = __db_decompress_count_int(comp_data); + if (size > compressed->size) + return (EINVAL); + comp_data += __db_decompress_int32(comp_data, &prefix); + + size += __db_decompress_count_int(comp_data); + if (size > compressed->size) + return (EINVAL); + comp_data += __db_decompress_int32(comp_data, &suffix); + + size += __db_decompress_count_int(comp_data); + if (size > compressed->size) + return (EINVAL); + comp_data += __db_decompress_int32(comp_data, &destData->size); + + /* Check destination lengths */ + destKey->size = prefix + suffix; + if (destKey->size > destKey->ulen || + destData->size > destData->ulen) + return (DB_BUFFER_SMALL); + + /* Write the prefix */ + if (prefix > prevKey->size) + return (EINVAL); + dest_data = (u_int8_t*)destKey->data; + memcpy(dest_data, prevKey->data, prefix); + dest_data += prefix; + + /* Write the suffix */ + size += suffix; + if (size > compressed->size) + return (EINVAL); + memcpy(dest_data, comp_data, suffix); + comp_data += suffix; + + /* Write the data */ + size += destData->size; + if (size > compressed->size) + return (EINVAL); + memcpy(destData->data, comp_data, destData->size); + comp_data += destData->size; + + /* Return bytes read */ + compressed->size = + (u_int32_t)(comp_data - (u_int8_t*)compressed->data); + return (0); +} +``` + +#### Programmer Notes + +As you use compression with your databases, be aware of the following: + +- Compression works by placing key/data pairs from a single database page into a single block of compressed data. This is true whether you use DB's default compression, or you write your own compression. Because all of key/data data is placed in a single block of memory, you cannot decompress data unless you have decompressed everything that came before it in the block. That is, you cannot decompress item *n* in the data block, unless you also decompress items *0* through *n-1*. + +- If you increase the minimum number of key/data pairs placed on a Btree leaf page (using DB->set_bt_minkey()), you will decrease your seek times on a compressed database. However, this will also decrease the effectiveness of the compression. + +- Compressed databases are fastest if bulk load is used to add data to them. See Retrieving and updating records in bulk for information on using bulk load. diff --git a/docs-src/guides/programmer_reference/cam.md b/docs-src/guides/programmer_reference/cam.md new file mode 100644 index 000000000..76d9bd5ae --- /dev/null +++ b/docs-src/guides/programmer_reference/cam.md @@ -0,0 +1,44 @@ +--- +title: "Chapter 10.  Berkeley DB Concurrent Data Store Applications" +api-name: "Chapter 10.  Berkeley DB Concurrent Data Store Applications" +source: docs/programmer_reference/cam.html +--- +## Chapter 10.  Berkeley DB Concurrent Data Store Applications + +**Table of Contents** + + [Concurrent Data Store introduction](cam.md#cam_intro) + + [Handling failure in Data Store and Concurrent Data Store applications](cam_fail.md) + + [Architecting Data Store and Concurrent Data Store applications](cam_app.md) + +## Concurrent Data Store introduction + +It is often desirable to have concurrent read-write access to a database when there is no need for full recoverability or transaction semantics. For this class of applications, Berkeley DB provides interfaces supporting deadlock-free, multiple-reader/single writer access to the database. This means that at any instant in time, there may be either multiple readers accessing data or a single writer modifying data. The application is entirely unaware of which is happening, and Berkeley DB implements the necessary locking and blocking to ensure this behavior. + +To create Berkeley DB Concurrent Data Store applications, you must first initialize an environment by calling DB_ENV->open(). You must specify the DB_INIT_CDB and DB_INIT_MPOOL flags to that method. It is an error to specify any of the other DB_ENV->open() subsystem or recovery configuration flags, for example, DB_INIT_LOCK, DB_INIT_TXN or DB_RECOVER All databases must, of course, be created in this environment by using the db_create() function or Db constructor, and specifying the environment as an argument. + +Berkeley DB performs appropriate locking so that safe enforcement of the deadlock-free, multiple-reader/single-writer semantic is transparent to the application. However, a basic understanding of Berkeley DB Concurrent Data Store locking behavior is helpful when writing Berkeley DB Concurrent Data Store applications. + +Berkeley DB Concurrent Data Store avoids deadlocks without the need for a deadlock detector by performing all locking on an entire database at once (or on an entire environment in the case of the DB_CDB_ALLDB flag), and by ensuring that at any given time only one thread of control is allowed to simultaneously hold a read (shared) lock and attempt to acquire a write (exclusive) lock. + +All open Berkeley DB cursors hold a read lock, which serves as a guarantee that the database will not change beneath them; likewise, all non-cursor DB->get() operations temporarily acquire and release a read lock that is held during the actual traversal of the database. Because read locks will not conflict with each other, any number of cursors in any number of threads of control may be open simultaneously, and any number of DB->get() operations may be concurrently in progress. + +To enforce the rule that only one thread of control at a time can attempt to upgrade a read lock to a write lock, however, Berkeley DB must forbid multiple cursors from attempting to write concurrently. This is done using the DB_WRITECURSOR flag to the DB->cursor() method. This is the only difference between access method calls in Berkeley DB Concurrent Data Store and in the other Berkeley DB products. The DB_WRITECURSOR flag causes the newly created cursor to be a "write" cursor; that is, a cursor capable of performing writes as well as reads. Only cursors thus created are permitted to perform write operations (either deletes or puts), and only one such cursor can exist at any given time. + +Any attempt to create a second write cursor or to perform a non-cursor write operation while a write cursor is open will block until that write cursor is closed. Read cursors may open and perform reads without blocking while a write cursor is extant. However, any attempts to actually perform a write, either using the write cursor or directly using the DB->put() or DB->del() methods, will block until all read cursors are closed. This is how the multiple-reader/single-writer semantic is enforced, and prevents reads from seeing an inconsistent database state that may be an intermediate stage of a write operation. + +By default, Berkeley DB Concurrent Data Store does locking on a per-database basis. For this reason, using cursors to access multiple databases in different orders in different threads or processes, or leaving cursors open on one database while accessing another database, can cause an application to hang. If this behavior is a requirement for the application, Berkeley DB should be configured to do locking on an environment-wide basis. See the DB_CDB_ALLDB flag of the DB_ENV->set_flags() method for more information. + +With these behaviors, Berkeley DB can guarantee deadlock-free concurrent database access, so that multiple threads of control are free to perform reads and writes without needing to handle synchronization themselves or having to run a deadlock detector. Berkeley DB has no direct knowledge of which cursors belong to which threads, so some care must be taken to ensure that applications do not inadvertently block themselves, causing the application to hang and be unable to proceed. + +As a consequence of the Berkeley DB Concurrent Data Store locking model, the following sequences of operations will cause a thread to block itself indefinitely: + +1. Keeping a cursor open while issuing a DB->put() or DB->del() access method call. +2. Attempting to open a write cursor while another cursor is already being held open by the same thread of control. Note that it is correct operation for one thread of control to attempt to open a write cursor or to perform a non-cursor write (DB->put() or DB->del()) while a write cursor is already active in another thread. It is only a problem if these things are done within a single thread of control -- in which case that thread will block and never be able to release the lock that is blocking it. +3. Not testing Berkeley DB error return codes (if any cursor operation returns an unexpected error, that cursor must still be closed). + +If the application needs to open multiple cursors in a single thread to perform an operation, it can indicate to Berkeley DB that the cursor locks should not block each other by creating a Berkeley DB Concurrent Data Store **group**, using DB_ENV->cdsgroup_begin(). This creates a locker ID that is shared by all cursors opened in the group. + +Berkeley DB Concurrent Data Store groups use a TXN handle to indicate the shared locker ID to Berkeley DB calls, and call DB_TXN->commit() to end the group. This is a convenient way to pass the locked ID to the calls where it is needed, but should not be confused with the real transactional semantics provided by Berkeley DB Transactional Data Store. In particular, Berkeley DB Concurrent Data Store groups do not provide any abort or recovery facilities, and have no impact on durability of operations. diff --git a/docs-src/guides/programmer_reference/cam_app.md b/docs-src/guides/programmer_reference/cam_app.md new file mode 100644 index 000000000..4fe6f7c5d --- /dev/null +++ b/docs-src/guides/programmer_reference/cam_app.md @@ -0,0 +1,52 @@ +--- +title: "Architecting Data Store and Concurrent Data Store applications" +api-name: "Architecting Data Store and Concurrent Data Store applications" +source: docs/programmer_reference/cam_app.html +--- +## Architecting Data Store and Concurrent Data Store applications + +When building Data Store and Concurrent Data Store applications, the architecture decisions involve application startup (cleaning up any existing databases, the removal of any existing database environment and creation of a new environment), and handling system or application failure. "Cleaning up" databases involves removal and re-creation of the database, restoration from an archival copy and/or verification and optional salvage, as described in Handling failure in Data Store and Concurrent Data Store applications. + +Data Store or Concurrent Data Store applications without database environments are single process, by definition. These applications should start up, re-create, restore, or verify and optionally salvage their databases and run until eventual exit or application or system failure. After system or application failure, that process can simply repeat this procedure. This document will not discuss the case of these applications further. + +Otherwise, the first question of Data Store and Concurrent Data Store architecture is the cleaning up existing databases and the removal of existing database environments, and the subsequent creation of a new environment. For obvious reasons, the application must serialize the re-creation, restoration, or verification and optional salvage of its databases. Further, environment removal and creation must be single-threaded, that is, one thread of control (where a thread of control is either a true thread or a process) must remove and re-create the environment before any other thread of control can use the new environment. It may simplify matters that Berkeley DB serializes creation of the environment, so multiple threads of control attempting to create a environment will serialize behind a single creating thread. + +Removing a database environment will first mark the environment as "failed", causing any threads of control still running in the environment to fail and return to the application. This feature allows applications to remove environments without concern for threads of control that might still be running in the removed environment. + +One consideration in removing a database environment which may be in use by another thread, is the type of mutex being used by the Berkeley DB library. In the case of database environment failure when using test-and-set mutexes, threads of control waiting on a mutex when the environment is marked "failed" will quickly notice the failure and will return an error from the Berkeley DB API. In the case of environment failure when using blocking mutexes, where the underlying system mutex implementation does not unblock mutex waiters after the thread of control holding the mutex dies, threads waiting on a mutex when an environment is recovered might hang forever. Applications blocked on events (for example, an application blocked on a network socket or a GUI event) may also fail to notice environment recovery within a reasonable amount of time. Systems with such mutex implementations are rare, but do exist; applications on such systems should use an application architecture where the thread recovering the database environment can explicitly terminate any process using the failed environment, or configure Berkeley DB for test-and-set mutexes, or incorporate some form of long-running timer or watchdog process to wake or kill blocked processes should they block for too long. + +Regardless, it makes little sense for multiple threads of control to simultaneously attempt to remove and re-create a environment, since the last one to run will remove all environments created by the threads of control that ran before it. However, for some few applications, it may make sense for applications to have a single thread of control that checks the existing databases and removes the environment, after which the application launches a number of processes, any of which are able to create the environment. + +With respect to cleaning up existing databases, the database environment must be removed before the databases are cleaned up. Removing the environment causes any Berkeley DB library calls made by threads of control running in the failed environment to return failure to the application. Removing the database environment first ensures the threads of control in the old environment do not race with the threads of control cleaning up the databases, possibly overwriting them after the cleanup has finished. Where the application architecture and system permit, many applications kill all threads of control running in the failed database environment before removing the failed database environment, on general principles as well as to minimize overall system resource usage. It does not matter if the new environment is created before or after the databases are cleaned up. + +After having dealt with database and database environment recovery after failure, the next issue to manage is application failure. As described in Handling failure in Data Store and Concurrent Data Store applications, when a thread of control in a Data Store or Concurrent Data Store application fails, it may exit holding data structure mutexes or logical database locks. These mutexes and locks must be released to avoid the remaining threads of control hanging behind the failed thread of control's mutexes or locks. + +There are three common ways to architect Berkeley DB Data Store and Concurrent Data Store applications. The one chosen is usually based on whether or not the application is comprised of a single process or group of processes descended from a single process (for example, a server started when the system first boots), or if the application is comprised of unrelated processes (for example, processes started by web connections or users logging into the system). + +1. The first way to architect Data Store and Concurrent Data Store applications is as a single process (the process may or may not be multithreaded.) + + When this process starts, it removes any existing database environment and creates a new environment. It then cleans up the databases and opens those databases in the environment. The application can subsequently create new threads of control as it chooses. Those threads of control can either share already open Berkeley DB DB_ENV and DB handles, or create their own. In this architecture, databases are rarely opened or closed when more than a single thread of control is running; that is, they are opened when only a single thread is running, and closed after all threads but one have exited. The last thread of control to exit closes the databases and the database environment. + + This architecture is simplest to implement because thread serialization is easy and failure detection does not require monitoring multiple processes. + + If the application's thread model allows the process to continue after thread failure, the DB_ENV->failchk() method can be used to determine if the database environment is usable after the failure. If the application does not call DB_ENV->failchk(), or DB_ENV->failchk() returns DB_RUNRECOVERY, the application must behave as if there has been a system failure, removing the environment and creating a new environment, and cleaning up any databases it wants to continue to use. Once these actions have been taken, other threads of control can continue (as long as all existing Berkeley DB handles are first discarded), or restarted. + +2. The second way to architect Data Store and Concurrent Data Store applications is as a group of related processes (the processes may or may not be multithreaded). + + This architecture requires the order in which threads of control are created be controlled to serialize database environment removal and creation, and database cleanup. + + In addition, this architecture requires that threads of control be monitored. If any thread of control exits with open Berkeley DB handles, the application may call the DB_ENV->failchk() method to determine if the database environment is usable after the exit. If the application does not call DB_ENV->failchk(), or DB_ENV->failchk() returns DB_RUNRECOVERY, the application must behave as if there has been a system failure, removing the environment and creating a new environment, and cleaning up any databases it wants to continue to use. Once these actions have been taken, other threads of control can continue (as long as all existing Berkeley DB handles are first discarded), or restarted. + + The easiest way to structure groups of related processes is to first create a single "watcher" process (often a script) that starts when the system first boots, removes and creates the database environment, cleans up the databases and then creates the processes or threads that will actually perform work. The initial thread has no further responsibilities other than to wait on the threads of control it has started, to ensure none of them unexpectedly exit. If a thread of control exits, the watcher process optionally calls the DB_ENV->failchk() method. If the application does not call DB_ENV->failchk(), or if DB_ENV->failchk() returns DB_RUNRECOVERY, the environment can no longer be used, the watcher kills all of the threads of control using the failed environment, cleans up, and starts new threads of control to perform work. + +3. The third way to architect Data Store and Concurrent Data Store applications is as a group of unrelated processes (the processes may or may not be multithreaded). This is the most difficult architecture to implement because of the level of difficulty in some systems of finding and monitoring unrelated processes. + + One solution is to log a thread of control ID when a new Berkeley DB handle is opened. For example, an initial "watcher" process could open/create the database environment, clean up the databases and then create a sentinel file. Any "worker" process wanting to use the environment would check for the sentinel file. If the sentinel file does not exist, the worker would fail or wait for the sentinel file to be created. Once the sentinel file exists, the worker would register its process ID with the watcher (via shared memory, IPC or some other registry mechanism), and then the worker would open its DB_ENV handles and proceed. When the worker finishes using the environment, it would unregister its process ID with the watcher. The watcher periodically checks to ensure that no worker has failed while using the environment. If a worker fails while using the environment, the watcher removes the sentinel file, kills all of the workers currently using the environment, cleans up the environment and databases, and finally creates a new sentinel file. + + The weakness of this approach is that, on some systems, it is difficult to determine if an unrelated process is still running. For example, POSIX systems generally disallow sending signals to unrelated processes. The trick to monitoring unrelated processes is to find a system resource held by the process that will be modified if the process dies. On POSIX systems, flock- or fcntl-style locking will work, as will LockFile on Windows systems. Other systems may have to use other process-related information such as file reference counts or modification times. In the worst case, threads of control can be required to periodically re-register with the watcher process: if the watcher has not heard from a thread of control in a specified period of time, the watcher will take action, cleaning up the environment. + + If it is not practical to monitor the processes sharing a database environment, it may be possible to monitor the environment to detect if a thread of control has failed holding open Berkeley DB handles. This would be done by having a "watcher" process periodically call the DB_ENV->failchk() method. If DB_ENV->failchk() returns DB_RUNRECOVERY, the watcher would then take action, cleaning up the environment. + + The weakness of this approach is that all threads of control using the environment must specify an "ID" function and an "is-alive" function using the DB_ENV->set_thread_id() method. (In other words, the Berkeley DB library must be able to assign a unique ID to each thread of control, and additionally determine if the thread of control is still running. It can be difficult to portably provide that information in applications using a variety of different programming languages and running on a variety of different platforms.) + +Obviously, when implementing a process to monitor other threads of control, it is important the watcher process' code be as simple and well-tested as possible, because the application may hang if it fails. diff --git a/docs-src/guides/programmer_reference/cam_fail.md b/docs-src/guides/programmer_reference/cam_fail.md new file mode 100644 index 000000000..a8c4c9b71 --- /dev/null +++ b/docs-src/guides/programmer_reference/cam_fail.md @@ -0,0 +1,26 @@ +--- +title: "Handling failure in Data Store and Concurrent Data Store applications" +api-name: "Handling failure in Data Store and Concurrent Data Store applications" +source: docs/programmer_reference/cam_fail.html +--- +## Handling failure in Data Store and Concurrent Data Store applications + +When building Data Store and Concurrent Data Store applications, there are design issues to consider whenever a thread of control with open Berkeley DB handles fails for any reason (where a thread of control may be either a true thread or a process). + +The simplest case is handling system failure for any Data Store or Concurrent Data Store application. In the case of system failure, it doesn't matter if the application has opened a database environment or is just using standalone databases: if the system fails, after the application has modified a database and has not subsequently flushed the database to stable storage (by calling either the DB->close(), DB->sync() or DB_ENV->memp_sync() methods), the database may be left in a corrupted state. In this case, before accessing the database again, the database should either be: + +- removed and re-created, +- removed and restored from the last known good backup, or +- verified using the DB->verify() method or db_verify utility. If the database does not verify cleanly, the contents may be salvaged using the **-R** and **-r** options of the db_dump utility. + +Applications where the potential for data loss is unacceptable should consider the Berkeley DB Transactional Data Store product, which offers standard transactional durability guarantees, including recoverability after failure. + +Additionally, system failure requires that any persistent database environment (that is, any database environment not created using the DB_PRIVATE flag), be removed. Database environments may be removed using the DB_ENV->remove() method. If the persistent database environment was backed by the filesystem (that is, the environment was not created using the DB_SYSTEM_MEM flag), the database environment may also be safely removed by deleting the environment's files with standard system utilities. + +The second case is application failure for a Data Store application, with or without a database environment, or application failure for a Concurrent Data Store application without a database environment: as in the case of system failure, if any thread of control fails, after the application has modified a database and has not subsequently flushed the database to stable storage, the database may be left in a corrupted state. In this case, the database should be handled as described previously in the system failure case. + +The third case is application failure for a Concurrent Data Store application with a database environment. There are resources maintained in database environments that may be left locked if a thread of control exits without first closing all open Berkeley DB handles. Concurrent Data Store applications with database environments have an additional option for handling the unexpected exit of a thread of control, the DB_ENV->failchk() method. + +The DB_ENV->failchk() will return DB_RUNRECOVERY if the database environment is unusable as a result of the thread of control failure. (If a data structure mutex or a database write lock is left held by thread of control failure, the application should not continue to use the database environment, as subsequent use of the environment is likely to result in threads of control convoying behind the held locks.) The DB_ENV->failchk() call will release any database read locks that have been left held by the exit of a thread of control. In this case, the application can continue to use the database environment. + +A Concurrent Data Store application recovering from a thread of control failure should call DB_ENV->failchk(), and, if it returns success, the application can continue. If DB_ENV->failchk() returns DB_RUNRECOVERY, the application should proceed as described for the case of system failure. diff --git a/docs-src/guides/programmer_reference/ch13s02.md b/docs-src/guides/programmer_reference/ch13s02.md new file mode 100644 index 000000000..aeb3379d1 --- /dev/null +++ b/docs-src/guides/programmer_reference/ch13s02.md @@ -0,0 +1,12 @@ +--- +title: "Berkeley DB XA Implementation" +api-name: "Berkeley DB XA Implementation" +source: docs/programmer_reference/ch13s02.html +--- +## Berkeley DB XA Implementation + +Berkeley DB provides support for distributed transactions using the two-phase commit protocol via its DB_TXN->prepare() interfaces. The DB_TXN->prepare() method performs the first phase of a two-phase commit, flushing the log to disk, and associating a global transaction ID with the underlying Berkeley DB transaction. This global transaction ID is used by the global transaction manager to identify the Berkeley DB transaction, and will be returned by the DB_ENV->txn_recover() method when it is called during recovery. + +However, Berkeley DB does not perform distributed deadlock detection. Instead, when being used as an XA resource manager, Berkeley DB acquires all locks in a non-blocking mode. This results in pre-emptive abort of transactions that have the potential to deadlock. While this can lead to more transactions being aborted than is strictly necessary, it avoids system-wide hanging due to distributed deadlocks. + +When using distributed transactions, there is no way to perform hot backups of multiple environments and guarantee that the backups are globally transaction-consistent across these multiple environments. If backups are desired, then all write transactions should be suspended; that is, active write transactions must be allowed to complete and no new write transactions should be begun. Once there are no active write transactions, the logs may be copied for backup purposes and the backup will be consistent across the multiple environments. diff --git a/docs-src/guides/programmer_reference/csharp.md b/docs-src/guides/programmer_reference/csharp.md new file mode 100644 index 000000000..495d81d9f --- /dev/null +++ b/docs-src/guides/programmer_reference/csharp.md @@ -0,0 +1,34 @@ +--- +title: "Chapter 6. C# API" +api-name: "Chapter 6. C# API" +source: docs/programmer_reference/csharp.html +--- +## Chapter 6. C# API + +**Table of Contents** + + [Compatibility](csharp.md#csharp_compat) + +You can use Berkeley DB in your application through the C# API. To understand the application concepts relating to Berkeley DB, see the first few chapters of this manual. For a general discussion on how to build Berkeley DB applications, see the Berkeley DB Getting Started Guides of C or C++. You can also review the example code of C and C++ from the examples_c and examples_cxx directories. For a description of all the classes, functions, and enumerations of Berkeley DB C# API, see the Berkeley DB C# API Reference Guide. + +A separate Visual Studio solution is provided to build the Berkeley DB C# classes, the examples, and the native support library. See Building the C# API in the Berkeley DB Installation and Build Guide for more information. + +The C# API requires .NET framework version 2.0 or above, and expects that it has already been installed on your system. For the sake of discussion, we assume that the Berkeley DB source is in a directory called db-*VERSION*; for example, you downloaded a Berkeley DB archive, and you did not change the top-level directory name. The files related to C# are in four subdirectories of db-*VERSION*: csharp (the C# source files), libdb_csharp (the C++ files that provide the "glue" between C# and Berkeley DB,) examples_csharp (containing all example code) and test\scr037 (containing NUnit tests for the API). + +Building the C# API produces a managed assembly `libdb_dotnet`*`VERSION`*`.dll`, containing the API, and two native libraries: `libdb_csharp`*`VERSION`*`.dll` and `libdb`*`VERSION`*`.dll`. (For all three files, *VERSION* is \[MAJOR\]\[MINOR\], i.e. for version 4.8 the managed assembly is `libdb_dotnet48.dll`.) Following the existing convention, native libraries are placed in either `db-`*`VERSION`*`\build_windows\Win32`or `db-`*`VERSION`*`\build_windows\x64`, depending upon the platform being targeted. In all cases, the managed assembly will be placed in `db-`*`VERSION`*`\build_windows\AnyCPU`. + +Because the C# API uses P/Invoke, for your application to use Berkeley DB successfully, the .NET framework needs to be able to locate the native libaries. This means the native libraries need to either be copied to your application's directory, the Windows or System directory, or the location of the libraries needs to be added to the `PATH` environment variable. See the MSDN documentation of the DllImport attribute and Dynamic-Link Library Search Order for further information. + +If you get the following exception when you run, the .NET platform probably is unable to locate the native libraries: + +``` c +System.TypeInitializationException +``` + +To ensure that everything is running correctly, you may want to try a simple test from the example programs in the `db-`*`VERSION`*`\examples_csharp` directory. + +For example, the ex_access sample program will prompt for text input lines, which are then stored in a Btree database named `access.db`. It is designed to be run from either the `db-`*`VERSION`*`\build_windows\Debug` or `db-`*`VERSION`*`\build_windows\Release` directory. Try giving it a few lines of input text and then a blank line. Before it exits, you should see a list of the lines you entered display with data items. This is a simple check to make sure the fundamental configuration is working correctly. + +## Compatibility + +The Berkeley DB C# API has been tested with the Microsoft .NET Framework versions 2.0, 3.0, 3.5, and 4.0. diff --git a/docs-src/guides/programmer_reference/dumpload.md b/docs-src/guides/programmer_reference/dumpload.md new file mode 100644 index 000000000..6075667cb --- /dev/null +++ b/docs-src/guides/programmer_reference/dumpload.md @@ -0,0 +1,28 @@ +--- +title: "Chapter 23.  Dumping and Reloading Databases" +api-name: "Chapter 23.  Dumping and Reloading Databases" +source: docs/programmer_reference/dumpload.html +--- +## Chapter 23.  Dumping and Reloading Databases + +**Table of Contents** + + [The db_dump and db_load utilities](dumpload.md#dumpload_utility) + + [Dump output formats](dumpload_format.md) + + [Loading text into databases](dumpload_text.md) + +## The db_dump and db_load utilities + +There are three utilities used for dumping and loading Berkeley DB databases: the db_dump utility, the db_dump185 utility and the db_load utility. + +The db_dump utility and the db_dump185 utility dump Berkeley DB databases into a flat-text representation of the data that can be read by db_load utility. The only difference between them is that the db_dump utility reads Berkeley DB version 2 and greater database formats, whereas the db_dump185 utility reads Berkeley DB version 1.85 and 1.86 database formats. + +The db_load utility reads either the output format used by the dump utilities or (optionally) a flat-text representation created using other tools, and stores it into a Berkeley DB database. + +Dumping and reloading Hash databases that use user-defined hash functions will result in new databases that use the default hash function. Although using the default hash function may not be optimal for the new database, it will continue to work correctly. + +Dumping and reloading Btree databases that use user-defined prefix or comparison functions will result in new databases that use the default prefix and comparison functions. In this case, it is quite likely that applications will be unable to retrieve records, and it is possible that the load process itself will fail. + +The only available workaround for either Hash or Btree databases is to modify the sources for the db_load utility to load the database using the correct hash, prefix, and comparison functions. diff --git a/docs-src/guides/programmer_reference/dumpload_format.md b/docs-src/guides/programmer_reference/dumpload_format.md new file mode 100644 index 000000000..3e6f4dd1c --- /dev/null +++ b/docs-src/guides/programmer_reference/dumpload_format.md @@ -0,0 +1,30 @@ +--- +title: "Dump output formats" +api-name: "Dump output formats" +source: docs/programmer_reference/dumpload_format.html +--- +## Dump output formats + +There are two output formats used by the db_dump utility and db_dump185 utility. + +In both output formats, the first few lines of the output contain header information describing the underlying access method, filesystem page size, and other bookkeeping information. + +The header information starts with a single line, VERSION=N, where N is the version number of the dump output format. + +The header information is then output in name=value pairs, where name may be any of the keywords listed in the db_load utility manual page, and value will be its value. Although this header information can be manually edited before the database is reloaded, there is rarely any reason to do so because all of this information can also be specified or overridden by command-line arguments to the db_load utility. + +The header information ends with single line HEADER=END. + +Following the header information are the key/data pairs from the database. If the database being dumped is a Btree or Hash database, or if the **-k** option was specified, the output will be paired lines of text where the first line of the pair is the key item, and the second line of the pair is its corresponding data item. If the database being dumped is a Queue or Recno database, and the **-k** option was not specified, the output will be lines of text where each line is the next data item for the database. Each of these lines is preceded by a single space. + +If the **-p** option was specified to the db_dump utility or db_dump185 utility, the key/data lines will consist of single characters representing any characters from the database that are *printing characters* and backslash (**\\**) escaped characters for any that were not. Backslash characters appearing in the output mean one of two things: if the backslash character precedes another backslash character, it means that a literal backslash character occurred in the key or data item. If the backslash character precedes any other character, the next two characters must be interpreted as hexadecimal specification of a single character; for example, **\0a** is a newline character in the ASCII character set. + +Although some care should be exercised, it is perfectly reasonable to use standard text editors and tools to edit databases dumped using the **-p** option before reloading them using the db_load utility. + +Note that the definition of a printing character may vary from system to system, so database representations created using the **-p** option may be less portable than those created without it. + +If the **-p** option in not specified to db_dump utility or db_dump185 utility, each output line will consist of paired hexadecimal values; for example, the line **726f6f74** is the string **root** in the ASCII character set. + +In all output formats, the key and data items are ended by a single line DATA=END. + +Where multiple databases have been dumped from a file, the overall output will repeat; that is, a new set of headers and a new set of data items. diff --git a/docs-src/guides/programmer_reference/dumpload_text.md b/docs-src/guides/programmer_reference/dumpload_text.md new file mode 100644 index 000000000..3d9db1d12 --- /dev/null +++ b/docs-src/guides/programmer_reference/dumpload_text.md @@ -0,0 +1,15 @@ +--- +title: "Loading text into databases" +api-name: "Loading text into databases" +source: docs/programmer_reference/dumpload_text.html +--- +## Loading text into databases + +The db_load utility can be used to load text into databases. The **-T** option permits nondatabase applications to create flat-text files that are then loaded into databases for fast, highly-concurrent access. For example, the following command loads the standard UNIX `/etc/passwd` file into a database, with the login name as the key item and the entire password entry as the data item: + +``` c +awk -F: '{print $1; print $0}' < /etc/passwd |\ + sed 's/\\/\\\\/g' | db_load -T -t hash passwd.db +``` + +Note that backslash characters naturally occurring in the text are escaped to avoid interpretation as escape characters by the db_load utility. diff --git a/docs-src/guides/programmer_reference/embedded.md b/docs-src/guides/programmer_reference/embedded.md new file mode 100644 index 000000000..c6c177607 --- /dev/null +++ b/docs-src/guides/programmer_reference/embedded.md @@ -0,0 +1,284 @@ +--- +title: "Challenges in Embedded Database System Administration" +api-name: "Challenges in Embedded Database System Administration" +source: docs/programmer_reference/embedded.html +--- +# Challenges in Embedded Database System Administration + +### Margo Seltzer, Harvard University + +### Michael Olson, Sleepycat Software, Inc. + +*{margo,mao}@sleepycat.com* + +Database configuration and maintenance have historically been complex tasks, often requiring expert knowledge of database design and application behavior. In an embedded environment, it is not feasible to require such expertise and ongoing database maintenance. This paper discusses the database administration challenges posed by embedded systems and describes how the Berkeley DB architecture addresses these challenges. + +## 1. Introduction + +Embedded systems provide a combination of opportunities and challenges in application and system configuration and management. As an embedded system is most often dedicated to a single application or small set of tasks, the operating conditions of the system are typically better understood than those of general purpose computing environments. Similarly, as embedded systems are dedicated to a small set of tasks, one would expect that the software to manage them should be small and simple. On the other hand, once an embedded system is deployed, it must continue to function without interruption and without administrator intervention. + +Database administration consists of two components, initial configuration and ongoing maintenance. Initial configuration consists of database design, manifestation, and tuning. The instantiation of the design includes decomposing the design into tables, relations, or objects and designating proper indices and their implementations (e.g., Btrees, hash tables, etc.). Tuning a design requires selecting a location for the log and data files, selecting appropriate database page sizes, specifying the size of in-memory caches, and specifying the limits of multi-threading and concurrency. As embedded systems define a specific environment and set of tasks, requiring expertise during the initial system configuration process is acceptable, and we focus our efforts on the ongoing maintenance of the system. In this way, our emphasis differs from other projects such as Microsoft's AutoAdmin project [\[3\]](#Chaud982), and the "no-knobs" administration that is identified as an area of important future research by the Asilomar authors[\[1\]](#Bern98). + +In this paper, we focus on what the authors of the Asilomar report call "gizmo" databases [\[1\]](#Bern98), databases that reside in devices such as smart cards, toasters, or telephones. The key characteristics of such databases are that their functionality is completely transparent to users, no one ever performs explicit database operations or database maintenance, the database may crash at any time and must recover instantly, the device may undergo a hard reset at any time, requiring that the database return to its initial state, and the semantic integrity of the database must be maintained at all times. In Section 2, we provide more detail on the sorts of tasks typically performed by database administrators (DBAs) that must be automated in an embedded system. + +The rest of this paper is structured as follows. In Section 2, we outline the requirements for embedded database support. In Section 3, we discuss how Berkeley DB is conducive to the hands-off management required in embedded systems. In Section 4, we discuss novel features that enhance Berkeley DB's suitability for the embedded applications. In Section 5, we discuss issues of footprint size. In Section 6 we discuss related work, and we conclude in Section 7. + +## 2. Embedded Database Requirements + +Historically, much of the commercial database industry has been driven by the requirements of high performance online transaction processing (OLTP), complex query processing, and the industry standard benchmarks that have emerged (e.g., TPC-C [\[9\]](#TPCC), TPC-D [\[10\]](#TPCD)) to allow for system comparisons. As embedded systems typically perform fairly simple queries, such metrics are not nearly as relevant for embedded database systems as are ease of maintenance, robustness, and small footprint. Of these three requirements, robustness and ease of maintenance are the key issues. Users must trust the data stored in their devices and must not need to manually perform anything resembling system administration in order to get their unit to work properly. Fortunately, ease of use and robustness are important side effects of simplicity and good design. These, in turn, lead to a small size, providing the third requirement of an embedded system. + +### 2.1 The User Perspective + +In the embedded database arena, it is the ongoing maintenance tasks that must be automated, not necessarily the initial system configuration. There are five tasks that are traditionally performed by DBAs, but must be performed automatically in embedded database systems. These tasks are log archival and reclamation, backup, data compaction/reorganization, automatic and rapid recovery, and reinitialization from scratch. + +Log archival and backup are tightly coupled. Database backups are part of any large database installation, and log archival is analogous to incremental backup. It is not clear what the implications of backup and archival are in an embedded system. Consumers do not back up their VCRs or refrigerators, yet they do (or should) back up their personal computers or personal digital assistants. For the remainder of this paper, we assume that backups, in some form, are required for gizmo databases (imagine having to reprogram, manually, the television viewing access pattern learned by some set-top television systems today). Furthermore, we require that those backups are nearly instantaneous or completely transparent, as users should not be aware that their gizmos are being backed up and should not have to explicitly initiate such backups. + +Data compaction or reorganization has traditionally required periodic dumping and restoration of database tables and the recreation of indices. In an embedded system, such reorganization must happen automatically. + +Recovery issues are similar in embedded and traditional environments with a few exceptions. While a few seconds or even a minute recovery is acceptable for a large server installation, no one is willing to wait for their telephone or television to reboot. As with archival, recovery must be nearly instantaneous in an embedded product. Secondly, it is often the case that a system will be completely reinitialized, rather than simply rebooted. In this case, the embedded database must be restored to its initial state, freeing all its resources. This is not typically a requirement of large server systems. + +### 2.2 The Developer Perspective + +In addition to the maintenance-free operation required of the embedded systems, there are a number of requirements that fall out of the constrained resources typically found in the "gizmos" using gizmo databases. These requirements are: small footprint, short code-path, programmatic interface for tight application coupling and to avoid the overhead (in both time and size) of interfaces such as SQL and ODBC, application configurability and flexibility, support for complete memory-resident operation (e.g., these systems must run on gizmos without file systems), and support for multi-threading. + +A small footprint and short code-path are self-explanatory, however what is not as obvious is that the programmatic interface requirement is the logical result of them. Traditional interfaces such as ODBC and SQL add significant size overhead and frequently add multiple context/thread switches per operation, not to mention several IPC calls. An embedded product is less likely to require the complex query processing that SQL enables. Instead, in the embedded space, the ability for an application to configure the database for the specific tasks in question is more important than a general query interface. + +As some systems do not provide storage other than RAM and ROM, it is essential that an embedded database work seemlessly in memory-only environments. Similarly, many of today's embedded operating systems provide a single address space architecture, so a simple, multi-threaded capability is essential for application requiring any concurrency. + +In general, embedded applications run on gizmos whose native operating system support varies tremendously. For example, the embedded OS may or may not support user-level processing or multi-threading. Even if it does, a particular embedded application may or may not need it. Not all applications need more than one thread of control. An embedded database must provide mechanisms to developers without deciding policy. For example, the threading model in an application is a matter of policy, and depends not on the database software, but on the hardware, operating system, and the application's feature set. Therefore, the data manager must provide for the use of multi-threading, but not require it. + +## 3. Berkeley DB: A Database for Embedded Systems + +Berkeley DB is the result of implementing database functionality using the UNIX tool-based philosophy. The current Berkeley DB package, as distributed by Sleepycat Software, is a descendant of the hash and btree access methods distributed with 4.4BSD and its descendents. The original package (referred to as DB-1.85), while intended as a public domain replacement for dbm and its followers (e.g., ndbm, gdbm, etc), rapidly became widely used as an efficient, easy-to-use data store. It was incorporated into a number of Open Source packages including Perl, Sendmail, Kerberos, and the GNU C-library. + +Versions 2.X and higher are distributed by Sleepycat Software and add functionality for concurrency, logging, transactions, and recovery. Each piece of additional functionality is implemented as an independent module, which means that the subsystems can be used outside the context of Berkeley DB. For example, the locking subsystem can easily be used to implement locking for a non-DB application and the shared memory buffer pool can be used for any application caching data in main memory. This subsystem design allows a designer to pick and choose the functionality necessary for the application, minimizing memory footprint and maximizing performance. This addresses the small footprint and short code-path criteria mentioned in the previous section. + +As Berkeley DB grew out of a replacement for dbm, its primary implementation language has always been C and its interface has been programmatic. The C interface is the native interface, unlike many database systems where the programmatic API is simply a layer on top of an already-costly query interface (e.g. embedded SQL). Berkeley DB's heritage is also apparent in its data model; it has none. The database stores unstructured key/data pairs, specified as variable length byte strings. This leaves schema design and representation issues the responsibility of the application, which is ideal for an embedded environment. Applications retain full control over specification of their data types, representation, index values, and index relationships. In other words, Berkeley DB provides a robust, high-performance, keyed storage system, not a particular database management system. We have designed for simplicity and performance, trading off complex, general purpose support that is better encapsulated in applications. + +Another element of Berkeley DB's programmatic interface is its customizability; applications can specify Btree comparison and prefix compression functions, hash functions, error routines, and recovery models. This means that embedded applications can tailor the underlying database to best suit their data demands. Similarly, the utilities traditionally bundled with a database manager (e.g., recovery, dump/restore, archive) are implemented as tiny wrapper programs around library routines. This means that it is not necessary to run separate applications for the utilities. Instead, independent threads can act as utility daemons, or regular query threads can perform utility functions. Many of the current products built on Berkeley DB are bundled as a single large server with independent threads that perform functions such as checkpoint, deadlock detection, and performance monitoring. + +As mentioned earlier, living in an embedded environment requires flexible management of storage. Berkeley DB does not require any preallocation of disk space for log or data files. While many commercial database systems take complete control of a raw device, Berkeley DB uses a normal file system, and can therefore, safely and easily share a data space with other programs. All databases and log files are native files of the host environment, so whatever utilities are provided by the environment can be used to manage database files as well. + +Berkeley DB provides three different memory models for its management of shared information. Applications can use the IEEE Std 1003.1b-1993 (POSIX) `mmap` interface to share data, they can use system shared memory, as frequently provided by the shmget family of interfaces, or they can use per-process heap memory (e.g., malloc). Applications that require no permanent storage and do not provide shared memory facilities can still use Berkeley DB by requesting strictly private memory and specifying that all databases be memory-resident. This provides pure-memory operation. + +Lastly, Berkeley DB is designed for rapid startup -- recovery can happen automatically as part of system initialization. This means that Berkeley DB works correctly in environments where gizmos are suddenly shut down and restarted. + +## 4. Extensions for Embedded Environments + +While the Berkeley DB library has been designed for use in embedded systems, all the features described above are useful in more conventional systems as well. In this section, we discuss a number of features and "automatic knobs" that are specifically geared toward the more constrained environments found in gizmo databases. + +### 4.1 Automatic compression + +Following the programmatic interface design philosophy, we support application-specific (or default) compression routines. These can be geared toward the particular data types present in the application's dataset, thus providing better compression than a general purpose routine. Note that the application could instead specify an encryption function and create encrypted databases instead of compressed ones. Alternately, the application might specify a function that performs both compression and encryption. + +As applications are also permitted to specify comparison and hash functions, the application can chose to organize its data based either on uncompressed and clear-text data or compressed and encrypted data. If the application indicates that data should be compared in its processed form (i.e., compressed and encrypted), then the compression and encryption are performed on individual data items and the in-memory representation retains these characteristics. However, if the application indicates that data should be compared in its original form, then entire pages are transformed upon being read into or written out of the main memory buffer cache. These two alternatives provide the flexibility to trade space and security for performance. + +### 4.2 In-memory logging & transactions + +One of the four key properties of transaction systems is durability. This means that transaction systems are designed for permanent storage (most commonly disk). However, as mentioned above, embedded systems do not necessarily contain any such storage. Nevertheless, transactions can be useful in this environment to preserve the semantic integrity of the underlying storage. Berkeley DB optionally provides logging functionality and transaction support regardless of whether the database and logs are on disk or in memory. + +### 4.3 Remote Logs + +While we do not expect users to backup their television sets and toasters, it is conceivable that a set-top box provided by a cable carrier should, in fact, be backed up by that cable carrier. The ability to store logs remotely can provide "information appliance" functionality, and can also be used in conjunction with local logs to enhance reliability. Furthermore, remote logs provide for catastrophic recovery, e.g., loss of the gizmo, destruction of the gizmo, etc. + +### 4.4 Application References to Database Buffers + +Typically, when data is returned to the user, it must be copied from the data manager's buffer cache (or data page) into the application's memory. However, in an embedded environment, the robustness of the total software package is of paramount importance, not the isolation between the application and the data manager. As a result, it is possible for the data manager to avoid copies by giving applications direct references to data items in a shared memory cache. This is a significant performance optimization that can be allowed when the application and data manager are tightly integrated. + +### 4.5 Recoverable database creation/deletion + +In a conventional database management system, the creation of database tables (relations) and indices are heavyweight operations that are not recoverable. This is not acceptable in a complex embedded environment where instantaneous recovery and robust operation in the face of all types of database operations is essential. While Berkeley DB files can be removed using normal file system utilities, we provide transaction protected utilities that allow us to recover both database creation and deletion. + +### 4.6 Adaptive concurrency control + +The Berkeley DB package uses page-level locking by default. This trades off fine grain concurrency control for simplicity during recovery. (Finer grain concurrency control can be obtained by reducing the page size in the database.) However, when multiple threads/processes perform page-locking in the presence of writing operations, there is the potential for deadlock. As some environments do not need or desire the overhead of logging and transactions, it is important to provide the ability for concurrent access without the potential for deadlock. + +Berkeley DB provides an option to perform coarser grain, deadlock-free locking. Rather than locking on pages, locking is performed at the interface to the database. Multiple readers or a single writer are allowed to be active in the database at any instant in time, with conflicting requests queued automatically. The presence of cursors, through which applications can both read and write data, complicates this design. If a cursor is currently being used for reading, but will later be used to write, the system will be deadlock prone if no special precautions are taken. To handle this situation, we require that, when a cursor is created, the application specify any future intention to write. If there is an intention to write, the cursor is granted an intention-to-write lock which does not conflict with readers, but does conflict with other intention-to-write locks and write locks. The end result is that the application is limited to a single potentially writing cursor accessing the database at any point in time. + +Under periods of low contention (but potentially high throughput), the normal page-level locking provides the best overall throughput. However, as contention rises, so does the potential for deadlock. As some cross-over point, switching to the less concurrent, but deadlock-free locking protocol will result in higher throughput as operations must never be retried. Given the operating conditions of an embedded database manager, it is useful to make this change automatically as the system itself detects high contention. + +### 4.7 Adaptive synchronization + +In addition to the logical locks that protect the integrity of the database pages, Berkeley DB must synchronize access to shared memory data structures, such as the lock table, in-memory buffer pool, and in-memory log buffer. Each independent module uses a single mutex to protect its shared data structures, under the assumption that operations that require the mutex are very short and the potential for conflict is low. Unfortunately, in highly concurrent environments with multiple processors present, this assumption is not always true. When this assumption becomes invalid (that is, we observe significant contention for the subsystem mutexes), we can switch over to a finer-grained concurrency model for the mutexes. Once again, there is a performance trade-off. Fine-grain mutexes impose a penalty of approximately 25% (due to the increased number of mutexes required for each operation), but allow for higher throughput. Using fine-grain mutexes under low contention would cause a decrease in performance, so it is important to monitor the system carefully, so that the change can be executed only when it will increase system throughput without jeopardizing latency. + +## 5. Footprint of an Embedded System + +While traditional systems compete on price-performance, the embedded players will compete on price, features, and footprint. The earlier sections have focused on features; in this section we focus on footprint. + +Oracle reports that Oracle Lite 3.0 requires 350 KB to 750 KB of memory and approximately 2.5 MB of hard disk space [\[7\]](#Oracle). This includes drivers for interfaces such as ODBC and JDBC. In contrast, Berkeley DB ranges in size from 75 KB to under 200 KB, foregoing heavyweight interfaces such as ODBC and JDBC and providing a variety of deployed sizes that can be used depending on application needs. At the low end, applications requiring a simple single-user access method can choose from either extended linear hashing, B+ trees, or record-number based retrieval and pay only the 75 KB space requirement. Applications requiring all three access methods will observe the 110 KB footprint. At the high end, a fully recoverable, high-performance system occupies less than a quarter megabyte of memory. This is a system you can easily incorporate in your toaster oven. Table 1 shows the per-module break down of the entire Berkeley DB library. Note that this does not include memory used to cache database pages. + +Object sizes in bytes + +Subsystem + +Text + +Data + +Bss + +Btree-specific routines + +28812 + +0 + +0 + +Recno-specific routines + +7211 + +0 + +0 + +Hash-specific routines + +23742 + +0 + +0 + +Memory Pool + +14535 + +0 + +0 + +Access method common code + +23252 + +0 + +0 + +OS compatibility library + +4980 + +52 + +0 + +Support utilities + +6165 + +0 + +0 + +All modules for Btree access method only + +77744 + +52 + +0 + +All modules for Recno access method only + +84955 + +52 + +0 + +All modules for Hash access method only + +72674 + +52 + +0 + +All Access Methods + +108697 + +52 + +0 + + + +Locking + +12533 + +0 + +0 + +Recovery + +26948 + +8 + +4 + +Logging + +37367 + +0 + +0 + +Full Package + +185545 + +60 + +4 + + + +## 6. Related Work + +Every three to five years, leading researchers in the database community convene to identify future directions in database research. They produce a report of this meeting, named for the year and location of the meeting. The most recent of these reports, the 1998 Asilomar report, identifies the embedded database market as one of the high growth areas in database research [\[1\]](#Bern98). Not surprisingly, market analysts identify the embedded database market as a high-growth area in the commercial sector as well [\[5\]](#Host98). + +The Asilomar report identifies a new class of database applications, which they term "gizmo" databases, small databases embedded in tiny mobile appliances, e.g., smart-cards, telephones, personal digital assistants. Such databases must be self-managing, secure and reliable. Thus, the idea is that gizmo databases require plug and play data management with no database administrator (DBA), no human settable parameters, and the ability to adapt to changing conditions. More specifically, the Asilomar authors claim that the goal is self-tuning, including defining the physical DB design, the logical DB design, and automatic reports and utilities [\[1\]](#Bern98) To date, few researchers have accepted this challenge, and there is a dearth of research literature on the subject. + +Our approach to embedded database administration is fundamentally different than that described by the Asilomar authors. We adopt their terminology, but view the challenge in supporting gizmo databases to be that of self-sustenance *after* initial deployment. Therefore, we find it, not only acceptable, but desirable to assume that application developers control initial database design and configuration. To the best of our knowledge, none of the published work in this area addresses this approach. + +As the research community has not provided guidance in this arena, most work in embedded database administration has fallen to the commercial vendors. These vendors fall into two camps, companies selling databases specifically designed for embedding or programmatic access and the major database vendors (e.g., Oracle, Informix, Sybase). + +The embedded vendors all acknowledge the need for automatic administration, but fail to identify precisely how their products actually accomplish this. A notable exception is Interbase whose white paper comparison with Sybase and Microsoft's SQL servers explicitly address features of maintenance ease. Interbase claims that as they use no log files, there is no need for log reclamation, checkpoint tuning, or other tasks associated with log management. However, Interbase uses Transaction Information Pages, and it is unclear how these are reused or reclaimed [\[6\]](#Interbase). Additionally, with a log-free system, they must use a FORCE policy (write all pages to disk at commit), as defined by Haerder and Reuter [\[4\]](#Haerder). This has serious performance consequences for disk-based systems. The approach described in this paper does use logs and therefore requires log reclamation, but provides hooks so the application may reclaim logs safely and programmatically. While Berkeley DB does require checkpoints, the goal of tuning the checkpoint interval is to bound recovery time. Since the checkpoint interval in Berkeley DB can be expressed by the amount of log data written, it requires no tuning. The application designer sets a target recovery time, and selects the amount of log data that can be read in that interval and specifies the checkpoint interval appropriately. Even as load changes, the time to recover does not. + +The backup approaches taken by Interbase and Berkeley DB are similar in that they both allow online backup, but rather different in their affect on transactions running during backup. As Interbase performs backups as transactions [\[6\]](#Interbase), concurrent queries can suffer potentially long delays. Berkeley DB uses native operating system system utilities and recovery for backups, so there is no interference with concurrent activity, other than potential contention on disk arms. + +There are a number of database vendors selling in the embedded market (e.g., Raima, Centura, Pervasive, Faircom), but none highlight the special requirements of embedded database applications. On the other end of the spectrum, the major vendors, Oracle, Sybase, Microsoft, are all becoming convinced of the importance of the embedded market. As mentioned earlier, Oracle has announced its Oracle Lite server for embedded use. Sybase has announced its UltraLite platform for "application-optimized, high-performance, SQL database engine for professional application developers building solutions for mobile and embedded platforms." [\[8\]](#Sybase). We believe that SQL is incompatible with the gizmo database environment or truly embedded systems for which Berkeley DB is most suitable. Microsoft research is taking a different approach, developing technology to assist in automating initial database design and index specification [\[2\]](#Chaud98)[\[3\]](#Chaud982). As mentioned earlier, we believe that such configuration is, not only acceptable in the embedded market, but desirable so that applications can tune their database management for the target environment. + +## 7. Conclusions + +The coming wave of embedded systems poses a new set of challenges for data management. The traditional server-based, big footprint systems designed for high performance on big iron are not the right approach in this environment. Instead, application developers need small, fast, versatile systems that can be tailored to a specific environment. In this paper, we have identified several of the key issues in providing these systems and shown how Berkeley DB provides many of the characteristics necessary for such applications. + +## 8. References + +\[1\] Bernstein, P., Brodie, M., Ceri, S., DeWitt, D., Franklin, M., Garcia-Molina, H., Gray, J., Held, J., Hellerstein, J., Jagadish, H., Lesk, M., Maier, D., Naughton, J., Pirahesh, H., Stonebraker, M., Ullman, J., "The Asilomar Report on Database Research," SIGMOD Record 27(4): 74-80, 1998. + +\[2\] Chaudhuri, S., Narasayya, V., "AutoAdmin 'What-If' Index Analysis Utility," *Proceedings of the ACM SIGMOD Conference*, Seattle, 1998. + +\[3\] Chaudhuri, S., Narasayya, V., "An Efficient, Cost-Driver Index Selection Tool for Microsoft SQL Server," *Proceedings of the 23rd VLDB Conference*, Athens, Greece, 1997. + +\[4\] Haerder, T., Reuter, A., "Principles of Transaction-Oriented Database Recovery," *Computing Surveys 15*,4 (1983), 237-318. + +\[5\] Hostetler, M., "Cover Is Off A New Type of Database," Embedded DB News, http://www.theadvisors.com/embeddeddbnews.htm, 5/6/98. + +\[6\] Interbase, "A Comparison of Borland InterBase 4.0 Sybase SQL Server and Microsoft SQL Server," http://web.interbase.com/products/doc_info_f.html. + +\[7\] Oracle, "Oracle Delivers New Server, Application Suite to Power the Web for Mission-Critical Business," http://www.oracle.com.sg/partners/news/newserver.htm, May 1998. + +\[8\] Sybase, Sybase UltraLite, http://www.sybase.com/products/ultralite/beta. + +\[9\] Transaction Processing Council, "TPC-C Benchmark Specification, Version 3.4," San Jose, CA, August 1998. + +\[10\] Transaction Processing Council, "TPC-D Benchmark Specification, Version 2.1," San Jose, CA, April 1999. diff --git a/docs-src/guides/programmer_reference/env.md b/docs-src/guides/programmer_reference/env.md new file mode 100644 index 000000000..67e011af0 --- /dev/null +++ b/docs-src/guides/programmer_reference/env.md @@ -0,0 +1,50 @@ +--- +title: "Chapter 9.  The Berkeley DB Environment" +api-name: "Chapter 9.  The Berkeley DB Environment" +source: docs/programmer_reference/env.html +--- +## Chapter 9.  The Berkeley DB Environment + +**Table of Contents** + + [Database environment introduction](env.md#env_intro) + + [Creating a database environment](env_create.md) + + [Sizing a database environment](env_size.md) + + [Opening databases within the environment](env_open.md) + + [Error support](env_error.md) + + [DB_CONFIG configuration file](env_db_config.md) + + [File naming](env_naming.md) + + [Specifying file naming to Berkeley DB](env_naming.md#idp51749352) + + [Filename resolution in Berkeley DB](env_naming.md#idp51763728) + + [Examples](env_naming.md#idp51756464) + + [Shared memory regions](env_region.md) + + [Security](env_security.md) + + [Encryption](env_encrypt.md) + + [Remote filesystems](env_remote.md) + + [Environment FAQ](env_faq.md) + +## Database environment introduction + +A Berkeley DB environment is an encapsulation of one or more databases, log files and region files. Region files are the shared memory areas that contain information about the database environment such as memory pool cache pages. Only databases are byte-order independent and only database files can be moved between machines of different byte orders. Log files can be moved between machines of the same byte order. Region files are usually unique to a specific machine and potentially to a specific operating system release. + +The simplest way to administer a Berkeley DB application environment is to create a single **home** directory that stores the files for the applications that will share the environment. The environment home directory must be created before any Berkeley DB applications are run. Berkeley DB itself never creates the environment home directory. The environment can then be identified by the name of that directory. + +An environment may be shared by any number of processes, as well as by any number of threads within those processes. It is possible for an environment to include resources from other directories on the system, and applications often choose to distribute resources to other directories or disks for performance or other reasons. However, by default, the databases, shared regions (the locking, logging, memory pool, and transaction shared memory areas) and log files will be stored in a single directory hierarchy. + +It is important to realize that all applications sharing a database environment implicitly trust each other. They have access to each other's data as it resides in the shared regions, and they will share resources such as buffer space and locks. At the same time, any applications using the same databases **must** share an environment if consistency is to be maintained between them. + +For more information on the operations supported by the database environment handle, see the Database Environments and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/env_create.md b/docs-src/guides/programmer_reference/env_create.md new file mode 100644 index 000000000..c3da2ff68 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_create.md @@ -0,0 +1,79 @@ +--- +title: "Creating a database environment" +api-name: "Creating a database environment" +source: docs/programmer_reference/env_create.html +--- +## Creating a database environment + +The Berkeley DB environment is created and described by the db_env_create() and DB_ENV->open() interfaces. In situations where customization is desired, such as storing log files on a separate disk drive or selection of a particular cache size, applications must describe the customization by either creating an environment configuration file in the environment home directory or by arguments passed to other DB_ENV handle methods. + +Once an environment has been created, database files specified using relative pathnames will be named relative to the home directory. Using pathnames relative to the home directory allows the entire environment to be easily moved, simplifying restoration and recovery of a database in a different directory or on a different system. + +Applications first obtain an environment handle using the db_env_create() method, then call the DB_ENV->open() method which creates or joins the database environment. There are a number of options you can set to customize DB_ENV->open() for your environment. These options fall into four broad categories: + +Subsystem Initialization: +These flags indicate which Berkeley DB subsystems will be initialized for the environment, and what operations will happen automatically when databases are accessed within the environment. The flags include DB_INIT_CDB, DB_INIT_LOCK, DB_INIT_LOG, DB_INIT_MPOOL, and DB_INIT_TXN. The DB_INIT_CDB flag does initialization for Berkeley DB Concurrent Data Store applications. (See Concurrent Data Store introduction for more information.) The rest of the flags initialize a single subsystem; that is, when DB_INIT_LOCK is specified, applications reading and writing databases opened in this environment will be using locking to ensure that they do not overwrite each other's changes. + +Recovery options: +These flags, which include DB_RECOVER and DB_RECOVER_FATAL, indicate what recovery is to be performed on the environment before it is opened for normal use. + +Naming options: +These flags, which include DB_USE_ENVIRON and DB_USE_ENVIRON_ROOT, modify how file naming happens in the environment. + +Miscellaneous: +Finally, there are a number of miscellaneous flags, for example, DB_CREATE which causes underlying files to be created as necessary. See the DB_ENV->open() manual pages for further information. + +Most applications either specify only the DB_INIT_MPOOL flag or they specify all four subsystem initialization flags (DB_INIT_MPOOL, DB_INIT_LOCK, DB_INIT_LOG, and DB_INIT_TXN). The former configuration is for applications that simply want to use the basic Access Method interfaces with a shared underlying buffer pool, but don't care about recoverability after application or system failure. The latter is for applications that need recoverability. There are situations in which other combinations of the initialization flags make sense, but they are rare. + +The DB_RECOVER flag is specified by applications that want to perform any necessary database recovery when they start running. That is, if there was a system or application failure the last time they ran, they want the databases to be made consistent before they start running again. It is not an error to specify this flag when no recovery needs to be done. + +The DB_RECOVER_FATAL flag is more special-purpose. It performs catastrophic database recovery, and normally requires that some initial arrangements be made; that is, archived log files be brought back into the filesystem. Applications should not normally specify this flag. Instead, under these rare conditions, the db_recover utility should be used. + +The following is a simple example of a function that opens a database environment for a transactional program. + +``` c +DB_ENV * +db_setup(char *home, char *data_dir, FILE *errfp, char *progname) + { + DB_ENV *dbenv; + int ret; + + /* + * Create an environment and initialize it for additional error + * reporting. + */ + if ((ret = db_env_create(&dbenv, 0)) != 0) { + fprintf(errfp, "%s: %s\n", progname, db_strerror(ret)); + return (NULL); + } + dbenv->set_errfile(dbenv, errfp); + dbenv->set_errpfx(dbenv, progname); + + /* + * Specify the shared memory buffer pool cachesize: 5MB. + * Databases are in a subdirectory of the environment home. + */ + if ((ret = dbenv->set_cachesize(dbenv, 0, + 5 * 1024 * 1024, 0)) != 0) { + dbenv->err(dbenv, ret, "set_cachesize"); + goto err; + } + if ((ret = dbenv->set_data_dir(dbenv, data_dir)) != 0) { + dbenv->err(dbenv, ret, "set_data_dir: %s", data_dir); + goto err; + } + + /* Open the environment with full transactional support. */ + if ((ret = dbenv->open(dbenv, home, DB_CREATE | DB_INIT_LOG | + DB_INIT_LOCK | DB_INIT_MPOOL | + DB_INIT_TXN, 0)) != 0) { + dbenv->err(dbenv, ret, "environment open: %s", home); + goto err; + } + + return (dbenv); + +err: (void)dbenv->close(dbenv, 0); + return (NULL); +} +``` diff --git a/docs-src/guides/programmer_reference/env_db_config.md b/docs-src/guides/programmer_reference/env_db_config.md new file mode 100644 index 000000000..c69b40e73 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_db_config.md @@ -0,0 +1,12 @@ +--- +title: "DB_CONFIG configuration file" +api-name: "DB_CONFIG configuration file" +source: docs/programmer_reference/env_db_config.html +--- +## DB_CONFIG configuration file + +Almost all of the configuration information that can be specified to DB_ENV class methods can also be specified using a configuration file. If a file named DB_CONFIG exists in the database home directory, it will be read for lines of the format **NAME VALUE**. + +One or more whitespace characters are used to delimit the two parts of the line, and trailing whitespace characters are discarded. All empty lines or lines whose first character is a whitespace or hash (**\#**) character will be ignored. Each line must specify both the NAME and the VALUE of the pair. The specific NAME VALUE pairs are documented in the manual for the corresponding methods (for example, the DB_ENV->set_data_dir() documentation includes NAME VALUE pair information Berkeley DB administrators can use to configure locations for database files). + +The DB_CONFIG configuration file is intended to allow database environment administrators to customize environments independent of applications using the environment. For example, a database administrator can move the database log and data files to a different location without application recompilation. In addition, because the DB_CONFIG file is read when the database environment is opened, it can be used to overrule application configuration done before that time. For example a database administrator could override the compiled-in application cache size to a size more appropriate for a specific machine. diff --git a/docs-src/guides/programmer_reference/env_encrypt.md b/docs-src/guides/programmer_reference/env_encrypt.md new file mode 100644 index 000000000..2956a4643 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_encrypt.md @@ -0,0 +1,49 @@ +--- +title: "Encryption" +api-name: "Encryption" +source: docs/programmer_reference/env_encrypt.html +--- +## Encryption + +Berkeley DB optionally supports encryption using the Rijndael/AES (also known as the Advanced Encryption Standard and Federal Information Processing Standard (FIPS) 197) algorithm for encryption or decryption. The algorithm is configured to use a 128-bit key. Berkeley DB uses a 16-byte initialization vector generated using the Mersenne Twister. All encrypted information is additionally checksummed using the SHA1 Secure Hash Algorithm, using a 160-bit message digest. + +The encryption support provided with Berkeley DB is intended to protect applications from an attacker obtaining physical access to the media on which a Berkeley DB database is stored, or an attacker compromising a system on which Berkeley DB is running but who is unable to read system or process memory on that system. **The encryption support provided with Berkeley DB will not protect applications from attackers able to read system memory on the system where Berkeley DB is running.** + +To encrypt a database, you must configure the database for encryption prior to creating it. If you are using a database environment, you must also configure the environment for encryption. In order to create an encrypted database within an environment, you: + +1. Configure the environment for encryption using the DB_ENV->set_encrypt() method. + +2. Open the database environment. + +3. Specify the DB_ENCRYPT flag to the database handle. + +4. Open the database. + +Once you have done that, all of the databases that you create in the environment are encrypted/decrypted by the password you specify using the DB_ENV->set_encrypt() method. + +For databases not created in an environment: + +1. Specify the DB_ENCRYPT flag to the database handle. + +2. Call the DB->set_encrypt() method. + +3. Open the database. + +Note that databases cannot be converted to an encrypted format after they have been created without dumping and re-creating them. Finally, encrypted databases cannot be read on systems with a different endianness than the system that created the encrypted database. + +Each encrypted database environment (including all its encrypted databases) is encrypted using a single password and a single algorithm. Applications wanting to provide a finer granularity of database access must either use multiple database environments or implement additional access controls outside of Berkeley DB. + +The only encrypted parts of a database environment are its databases and its log files. Specifically, the Shared memory regions supporting the database environment are not encrypted. For this reason, it may be possible for an attacker to read some or all of an encrypted database by reading the on-disk files that back these shared memory regions. To prevent such attacks, applications may want to use in-memory filesystem support (on systems that support it), or the DB_PRIVATE or DB_SYSTEM_MEM flags to the DB_ENV->open() method, to place the shared memory regions in memory that is never written to a disk. As some systems page system memory to a backing disk, it is important to consider the specific operating system running on the machine as well. Finally, when backing database environment shared regions with the filesystem, Berkeley DB can be configured to overwrite the shared regions before removing them by specifying the DB_OVERWRITE flag. This option is only effective in the presence of fixed-block filesystems, journaling or logging filesystems will require operating system support and probably modification of the Berkeley DB sources. + +While all user data is encrypted, parts of the databases and log files in an encrypted environment are maintained in an unencrypted state. Specifically, log record headers are not encrypted, only the actual log records. Additionally, database internal page header fields are not encrypted. These page header fields includes information such as the page's DB_LSN number and position in the database's sort order. + +Log records distributed by a replication master to replicated clients are transmitted to the clients in unencrypted form. If encryption is desired in a replicated application, the use of a secure transport is strongly suggested. + +We gratefully acknowledge: + +- Vincent Rijmen, Antoon Bosselaers and Paulo Barreto for writing the Rijndael/AES code used in Berkeley DB. +- Steve Reid and James H. Brown for writing the SHA1 checksum code used in Berkeley DB. +- Makoto Matsumoto and Takuji Nishimura for writing the Mersenne Twister code used in Berkeley DB. +- Adam Stubblefield for integrating the Rijndael/AES, SHA1 checksum and Mersenne Twister code into Berkeley DB. + +Berkeley DB 11g Release 2 supports encryption using Intel's Performance Primitive (IPP) on Linux. This works only on Intel processors. To use Berkeley DB with IPP encryption, you must have IPP installed along with the cryptography extension. The IPP performance is higher in most cases compared to the current AES implementation. See --with-cryptography for more information. See the Intel Documenation for more information on IPP. diff --git a/docs-src/guides/programmer_reference/env_error.md b/docs-src/guides/programmer_reference/env_error.md new file mode 100644 index 000000000..d36502fd5 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_error.md @@ -0,0 +1,48 @@ +--- +title: "Error support" +api-name: "Error support" +source: docs/programmer_reference/env_error.html +--- +## Error support + +Berkeley DB offers programmatic support for displaying error return values. The db_strerror() function returns a pointer to the error message corresponding to any Berkeley DB error return. This is similar to the ANSI C strerror interface, but can handle both system error returns and Berkeley DB-specific return values. + +For example: + +``` c +int ret; +if ((ret = dbenv->set_cachesize(dbenv, 0, 32 * 1024, 1)) != 0) { + fprintf(stderr, "set_cachesize failed: %s\n", db_strerror(ret)); + return (1); +} +``` + +There are also two additional error methods: DB_ENV->err() and `DB_ENV->errx()`. These methods work like the ANSI C printf function, taking a printf-style format string and argument list, and writing a message constructed from the format string and arguments. + +The DB_ENV->err() function appends the standard error string to the constructed message; the `DB_ENV->errx()` function does not. + +Error messages can be configured always to include a prefix (for example, the program name) using the DB_ENV->set_errpfx() method. + +These functions provide simpler ways of displaying Berkeley DB error messages: + +``` c +int ret; +... +dbenv->set_errpfx(dbenv, program_name); +if ((ret = dbenv->open(dbenv, home, + DB_CREATE | DB_INIT_LOG | DB_INIT_TXN | DB_USE_ENVIRON, 0)) + != 0) { + dbenv->err(dbenv, ret, "open: %s", home); + dbenv->errx(dbenv, + "contact your system administrator: session ID was %d", + session_id); + return (1); +} +``` + +For example, if the program was called "my_app", and it tried to open an environment home directory in "/tmp/home" and the open call returned a permission error, the error messages shown would look like this: + +``` c +my_app: open: /tmp/home: Permission denied. +my_app: contact your system administrator: session ID was 2 +``` diff --git a/docs-src/guides/programmer_reference/env_faq.md b/docs-src/guides/programmer_reference/env_faq.md new file mode 100644 index 000000000..d3b45da0a --- /dev/null +++ b/docs-src/guides/programmer_reference/env_faq.md @@ -0,0 +1,16 @@ +--- +title: "Environment FAQ" +api-name: "Environment FAQ" +source: docs/programmer_reference/env_faq.html +--- +## Environment FAQ + +1. **I'm using multiple processes to access an Berkeley DB database environment; is there any way to ensure that two processes don't run transactional recovery at the same time, or that all processes have exited the database environment so that recovery can be run?** + + See Handling failure in Transactional Data Store applications and Architecting Transactional Data Store applications for a full discussion of this topic. + +2. **How can I associate application information with a DB or DB_ENV handle?** + + In the C API, the DB and DB_ENV structures each contain an "app_private" field intended to be used to reference application-specific information. See the db_create() and db_env_create() documentation for more information. + + In the C++ or Java APIs, the easiest way to associate application-specific data with a handle is to subclass the Db or DbEnv, for example subclassing Db to get MyDb. Objects of type MyDb will still have the Berkeley DB API methods available on them, and you can put any extra data or methods you want into the MyDb class. If you are using "callback" APIs that take Db or DbEnv arguments (for example, Db::set_bt_compare()) these will always be called with the Db or DbEnv objects you create. So if you always use MyDb objects, you will be able to take the first argument to the callback function and cast it to a MyDb (in C++, cast it to (MyDb\*)). That will allow you to access your data members or methods. diff --git a/docs-src/guides/programmer_reference/env_naming.md b/docs-src/guides/programmer_reference/env_naming.md new file mode 100644 index 000000000..73cf9a353 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_naming.md @@ -0,0 +1,108 @@ +--- +title: "File naming" +api-name: "File naming" +source: docs/programmer_reference/env_naming.html +--- +## File naming + + [Specifying file naming to Berkeley DB](env_naming.md#idp51749352) + + [Filename resolution in Berkeley DB](env_naming.md#idp51763728) + + [Examples](env_naming.md#idp51756464) + +One of the most important tasks of the database environment is to structure file naming within Berkeley DB. Cooperating applications (or multiple invocations of the same application) must agree on the location of the database environment, log files and other files used by the Berkeley DB subsystems, and, of course, the database files. Although it is possible to specify full pathnames to all Berkeley DB methods, this is cumbersome and requires applications be recompiled when database files are moved. + +Applications are normally expected to specify a single directory home for the database environment. This can be done easily in the call to DB_ENV->open() by specifying a value for the **db_home** argument. There are more complex configurations in which it may be desirable to override **db_home** or provide supplementary path information. + +### Specifying file naming to Berkeley DB + +The following list describes the possible ways in which file naming information may be specified to the Berkeley DB library. The specific circumstances and order in which these ways are applied are described in a subsequent paragraph. + +db_home +If the **db_home** argument to DB_ENV->open() is non-NULL, its value may be used as the database home, and files named relative to its path. + +DB_HOME +If the DB_HOME environment variable is set when DB_ENV->open() is called, its value may be used as the database home, and files named relative to its path. + +The DB_HOME environment variable is intended to permit users and system administrators to override application and installation defaults. For example: + +``` c +env DB_HOME=/database/my_home application +``` + +Application writers are encouraged to support the **-h** option found in the supporting Berkeley DB utilities to let users specify a database home. + +DB_ENV methods +There are four DB_ENV methods that affect file naming: + +- The DB_ENV->add_data_dir() method specifies a directory to search for database files. + +- The DB_ENV->set_lg_dir() method specifies a directory in which to create logging files. + +- The DB_ENV->set_tmp_dir() method specifies a directory in which to create backing temporary files. + +- The DB_ENV->set_metadata_dir() method specifies the directory in which to create persistent metadata files used by the environment. + +These methods are intended to permit applications to customize a file locations for an environment. For example, an application writer can place data files and log files in different directories or instantiate a new log directory each time the application runs. + + DB_CONFIG +The same information specified to the DB_ENV methods may also be specified using the DB_CONFIG configuration file. + +### Filename resolution in Berkeley DB + +The following list describes the specific circumstances and order in which the different ways of specifying file naming information are applied. Berkeley DB filename processing proceeds sequentially through the following steps: + +absolute pathnames +If the filename specified to a Berkeley DB function is an *absolute pathname*, that filename is used without modification by Berkeley DB. + +On UNIX systems, an absolute pathname is defined as any pathname that begins with a leading slash (**/**). + +On Windows systems, an absolute pathname is any pathname that begins with a leading slash or leading backslash (**\\**); or any pathname beginning with a single alphabetic character, a colon and a leading slash or backslash (for example, `C:/tmp`). + +DB_ENV methods, DB_CONFIG +If a relevant configuration string (for example, set_data_dir), is specified either by calling a DB_ENV method or as a line in the DB_CONFIG configuration file, the value is prepended to the filename. If the resulting filename is an absolute pathname, the filename is used without further modification by Berkeley DB. + +db_home +If the application specified a non-NULL **db_home** argument to DB_ENV->open(), its value is prepended to the filename. If the resulting filename is an absolute pathname, the filename is used without further modification by Berkeley DB. + +DB_HOME +If the **db_home** argument is NULL, the DB_HOME environment variable was set, and the application has set the appropriate DB_USE_ENVIRON or DB_USE_ENVIRON_ROOT flags, its value is prepended to the filename. If the resulting filename is an absolute pathname, the filename is used without further modification by Berkeley DB. + +default +Finally, all filenames are interpreted relative to the current working directory of the process. + +The common model for a Berkeley DB environment is one in which only the DB_HOME environment variable, or the **db_home** argument is specified. In this case, all data filenames are relative to that directory, and all files created by the Berkeley DB subsystems will be created in that directory. + +The more complex model for a transaction environment might be one in which a database home is specified, using either the DB_HOME environment variable or the **db_home** argument to DB_ENV->open(); and then the data directory and logging directory are set to the relative pathnames of directories underneath the environment home. + +### Examples + +Store all files in the directory `/a/database`: + +``` c +dbenv->open(dbenv, "/a/database", flags, mode); +``` + +Create temporary backing files in `/b/temporary`, and all other files in `/a/database`: + +``` c +dbenv->set_tmp_dir(dbenv, "/b/temporary"); +dbenv->open(dbenv, "/a/database", flags, mode); +``` + +Store data files in `/a/database/datadir`, log files in `/a/database/logdir`, and all other files in the directory `/a/database`: + +``` c +dbenv->set_lg_dir(dbenv, "logdir"); +dbenv->set_data_dir(dbenv, "datadir"); +dbenv->open(dbenv, "/a/database", flags, mode); +``` + +Store data files in `/a/database/data1` and `/b/data2`, and all other files in the directory `/a/database`. Any data files that are created will be created in `/b/data2`, because it is the first data file directory specified: + +``` c +dbenv->set_data_dir(dbenv, "/b/data2"); +dbenv->set_data_dir(dbenv, "data1"); +dbenv->open(dbenv, "/a/database", flags, mode); +``` diff --git a/docs-src/guides/programmer_reference/env_open.md b/docs-src/guides/programmer_reference/env_open.md new file mode 100644 index 000000000..d2ac14bd5 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_open.md @@ -0,0 +1,71 @@ +--- +title: "Opening databases within the environment" +api-name: "Opening databases within the environment" +source: docs/programmer_reference/env_open.html +--- +## Opening databases within the environment + +Once the environment has been created, database handles may be created and then opened within the environment. This is done by calling the db_create() function and specifying the appropriate environment as an argument. + +File naming, database operations, and error handling will all be done as specified for the environment. For example, if the DB_INIT_LOCK or DB_INIT_CDB flags were specified when the environment was created or joined, database operations will automatically perform all necessary locking operations for the application. + +The following is a simple example of opening two databases within a database environment: + +``` c +DB_ENV *dbenv; + DB *dbp1, *dbp2; + int ret; + + dbenv = NULL; + dbp1 = dbp2 = NULL; + /* + * Create an environment and initialize it for additional error + * reporting. + */ + if ((ret = db_env_create(&dbenv, 0)) != 0) { + fprintf(errfp, "%s: %s\n", progname, db_strerror(ret)); + return (ret); + } + + dbenv->set_errfile(dbenv, errfp); + dbenv->set_errpfx(dbenv, progname); + + /* Open an environment with just a memory pool. */ + if ((ret = + dbenv->open(dbenv, home, DB_CREATE | DB_INIT_MPOOL, 0)) != 0) { + dbenv->err(dbenv, ret, "environment open: %s", home); + goto err; + } + + /* Open database #1. */ + if ((ret = db_create(&dbp1, dbenv, 0)) != 0) { + dbenv->err(dbenv, ret, "database create"); + goto err; + } + if ((ret = dbp1->open(dbp1, + NULL, DATABASE1, NULL, DB_BTREE, DB_CREATE, 0664)) != 0) { + dbenv->err(dbenv, ret, "DB->open: %s", DATABASE1); + goto err; + } + + /* Open database #2. */ + if ((ret = db_create(&dbp2, dbenv, 0)) != 0) { + dbenv->err(dbenv, ret, "database create"); + goto err; + } + if ((ret = dbp2->open(dbp2, + NULL, DATABASE2, NULL, DB_HASH, DB_CREATE, 0664)) != 0) { + dbenv->err(dbenv, ret, "DB->open: %s", DATABASE2); + goto err; + } + + return (0); + +err: if (dbp2 != NULL) + (void)dbp2->close(dbp2, 0); + if (dbp1 != NULL) + (void)dbp1->close(dbp1, 0); + (void)dbenv->close(dbenv, 0); + return (1); +} +``` diff --git a/docs-src/guides/programmer_reference/env_region.md b/docs-src/guides/programmer_reference/env_region.md new file mode 100644 index 000000000..704a8aa8e --- /dev/null +++ b/docs-src/guides/programmer_reference/env_region.md @@ -0,0 +1,26 @@ +--- +title: "Shared memory regions" +api-name: "Shared memory regions" +source: docs/programmer_reference/env_region.html +--- +## Shared memory regions + +Each of the Berkeley DB subsystems within an environment is described by one or more regions, or chunks of memory. The regions contain all of the per-process and per-thread shared information (including mutexes), that comprise a Berkeley DB environment. These regions are created in one of three types of memory, depending on the flags specified to the DB_ENV->open() method: + +1. If the DB_PRIVATE flag is specified to the DB_ENV->open() method, regions are created in per-process heap memory; that is, memory returned by `malloc`(3). + + If this flag is specified, then you cannot open more than a single handle for the environment. For example, if both a server application and Berkeley DB utilities (for example, the db_archive utility, the db_checkpoint utility or the db_stat utility) are expected to access the environment, the DB_PRIVATE flag should not be specified because the second attempt to open the environment will fail. + +2. If the DB_SYSTEM_MEM flag is specified to DB_ENV->open(), shared regions are created in system memory rather than files. This is an alternative mechanism for sharing the Berkeley DB environment among multiple processes and multiple threads within processes. + + The system memory used by Berkeley DB is potentially useful past the lifetime of any particular process. Therefore, additional cleanup may be necessary after an application fails because there may be no way for Berkeley DB to ensure that system resources backing the shared memory regions are returned to the system. + + The system memory that is used is architecture-dependent. For example, on systems supporting X/Open-style shared memory interfaces, such as UNIX systems, the `shmget`(2) and related System V IPC interfaces are used. Additionally, VxWorks systems use system memory. In these cases, an initial segment ID must be specified by the application to ensure that applications do not overwrite each other's database environments, so that the number of segments created does not grow without bounds. See the DB_ENV->set_shm_key() method for more information. + + On Windows platforms, the use of the DB_SYSTEM_MEM flag is problematic because the operating system uses reference counting to clean up shared objects in the paging file automatically. In addition, the default access permissions for shared objects are different from files, which may cause problems when an environment is accessed by multiple processes running as different users. See the Windows Notes section in the Berkeley DB Installation and Build Guide for more information. + +3. If no memory-related flags are specified to DB_ENV->open(), memory backed by the filesystem is used to store the regions. On UNIX systems, the Berkeley DB library will use the POSIX mmap interface. If mmap is not available, the UNIX shmget interfaces may be used instead, if they are available. + +Any files created in the filesystem to back the regions are created in the environment home directory specified to the DB_ENV->open() call. These files are named \_\_db.### (for example, \_\_db.001, \_\_db.002 and so on). When region files are backed by the filesystem, one file per region is created. When region files are backed by system memory, a single file will still be created because there must be a well-known name in the filesystem so that multiple processes can locate the system shared memory that is being used by the environment. + +Statistics about the shared memory regions in the environment can be displayed using the **-e** option to the db_stat utility. diff --git a/docs-src/guides/programmer_reference/env_remote.md b/docs-src/guides/programmer_reference/env_remote.md new file mode 100644 index 000000000..fe108b37c --- /dev/null +++ b/docs-src/guides/programmer_reference/env_remote.md @@ -0,0 +1,18 @@ +--- +title: "Remote filesystems" +api-name: "Remote filesystems" +source: docs/programmer_reference/env_remote.html +--- +## Remote filesystems + +When Berkeley DB database environment shared memory regions are backed by the filesystem, it is a common application error to create database environments backed by remote filesystems such as the Network File System (NFS), Windows network shares (SMB/CIFS) or the Andrew File System (AFS). Remote filesystems rarely support mapping files into process memory, and even more rarely support correct semantics for mutexes if the mapping succeeds. For this reason, we recommend database environment directories be created in a local filesystem. + +For remote filesystems that do allow remote files to be mapped into process memory, database environment directories accessed via remote filesystems cannot be used simultaneously from multiple clients (that is, from multiple computers). No commercial remote filesystem of which we're aware supports coherent, distributed shared memory for remote-mounted files. As a result, different machines will see different versions of these shared region files, and the behavior is undefined. + +Databases, log files, and temporary files may be placed on remote filesystems, as long as the remote filesystem fully supports standard POSIX filesystem semantics (although the application may incur a performance penalty for doing so). Further, read-only databases on remote filesystems can be accessed from multiple systems simultaneously. However, it is difficult (or impossible) for modifiable databases on remote filesystems to be accessed from multiple systems simultaneously. The reason is the Berkeley DB library caches modified database pages, and when those modified pages are written to the backing file is not entirely under application control. If two systems were to write database pages to the remote filesystem at the same time, database corruption could result. If a system were to write a database page back to the remote filesystem at the same time as another system read a page, a core dump in the reader could result. + +FreeBSD note: +Some historic FreeBSD releases will return ENOLCK from fsync and close calls on NFS-mounted filesystems, even though the call has succeeded. To support Berkeley DB on these releases, the Berkeley DB code should be modified to ignore ENOLCK errors, or no Berkeley DB files should be placed on NFS-mounted filesystems on these systems. Note that current FreeBSD releases do not suffer from this problem. + +Linux note: +Some historic Linux releases do not support complete semantics for the POSIX fsync call on NFS-mounted filesystems. No Berkeley DB files should be placed on NFS-mounted filesystems on these systems. Note that current Linux releases do not suffer from this problem. diff --git a/docs-src/guides/programmer_reference/env_security.md b/docs-src/guides/programmer_reference/env_security.md new file mode 100644 index 000000000..52b33f666 --- /dev/null +++ b/docs-src/guides/programmer_reference/env_security.md @@ -0,0 +1,23 @@ +--- +title: "Security" +api-name: "Security" +source: docs/programmer_reference/env_security.html +--- +## Security + +The following are security issues that should be considered when writing Berkeley DB applications: + +Database environment permissions +The directory used as the Berkeley DB database environment should have its permissions set to ensure that files in the environment are not accessible to users without appropriate permissions. Applications that add to the user's permissions (for example, UNIX setuid or setgid applications), must be carefully checked to not permit illegal use of those permissions such as general file access in the environment directory. + +Environment variables +Setting the DB_USE_ENVIRON and DB_USE_ENVIRON_ROOT flags and allowing the use of environment variables during file naming can be dangerous. Setting those flags in Berkeley DB applications with additional permissions (for example, UNIX setuid or setgid applications) could potentially allow users to read and write databases to which they would not normally have access. + +File permissions +By default, Berkeley DB always creates files readable and writable by the owner and the group (that is, S_IRUSR, S_IWUSR, S_IRGRP and S_IWGRP; or octal mode 0660 on historic UNIX systems). The group ownership of created files is based on the system and directory defaults, and is not further specified by Berkeley DB. + +Temporary backing files +If an unnamed database is created and the cache is too small to hold the database in memory, Berkeley DB will create a temporary physical file to enable it to page the database to disk as needed. In this case, environment variables such as **TMPDIR** may be used to specify the location of that temporary file. Although temporary backing files are created readable and writable by the owner only (S_IRUSR and S_IWUSR, or octal mode 0600 on historic UNIX systems), some filesystems may not sufficiently protect temporary files created in random directories from improper access. To be absolutely safe, applications storing sensitive data in unnamed databases should use the DB_ENV->set_tmp_dir() method to specify a temporary directory with known permissions. + +Tcl API +The Berkeley DB Tcl API does not attempt to avoid evaluating input as Tcl commands. For this reason, it may be dangerous to pass unreviewed user input through the Berkeley DB Tcl API, as the input may subsequently be evaluated as a Tcl command. Additionally, the Berkeley DB Tcl API initialization routine resets process' effective user and group IDs to the real user and group IDs, to minimize the effectiveness of a Tcl injection attack. diff --git a/docs-src/guides/programmer_reference/env_size.md b/docs-src/guides/programmer_reference/env_size.md new file mode 100644 index 000000000..c962febcd --- /dev/null +++ b/docs-src/guides/programmer_reference/env_size.md @@ -0,0 +1,42 @@ +--- +title: "Sizing a database environment" +api-name: "Sizing a database environment" +source: docs/programmer_reference/env_size.html +--- +## Sizing a database environment + +The Berkeley DB environment allocates memory to hold shared structures, either in shared regions or in process data space (if the DB_PRIVATE flag is specified). There are three distinct memory regions: + +- The memory pool (also known as the database page cache), + +- the area containing mutexes, and + +- the main region which holds all other shared structures. + +The shared structures in the main region are used by the lock, transaction, logging, thread and replicatoin subsystems. + +Determining the amount of space allocated for each of these shared structures is dependent upon the structure in question. The sizing of the memory pool is discussed in Configuring the memory pool. The amount of memory needed for mutexes is calculated from the number of mutexes needed by various subsystems and can be adjusted using the DB_ENV->mutex_set_increment() method. + +For applications using shared memory (that is, they do not specify DB_PRIVATE), a maximum memory size for the main region must be specified or left to default. The maximum memory size is specified using the DB_ENV->set_memory_max() method. + +The amount of memory needed by an application is dependent on the resources that the application uses. For a very rough estimate, add all of the following together: + +1. The environment has an overhead of about 80 kilobytes without statistics enabled or 250 kilobytes with statistics enabled. + +2. Identify the amount of space you require for your locks: + + 1. Estimate the number of threads of control that will simultaneously access the environment. + + 2. Estimate the number of concurrency locks that, on average, will be required by each thread. For information on sizing concurrency locks, see Configuring locking: sizing the system. + + 3. Multiply these two numbers, then multiply by 1/2 to arrive at the number of kilobytes required to service your locks. + +3. Estimate the number of open database handles you will use at any given time. For each database handle, there is an overhead of about 1/2 kilobyte. + +4. Add 1 kilobyte for each active transaction. + +Note that these are very rough guidelines. It is best to overestimate the needs of your applications, because if the memory allocation is exhausted the application must be shutdown to increase the allocation. + +The estimate for maximum memory need not be exact. In most situations there is little penalty for over estimating. For systems using memory mapped files for the shared environment, this only allocates the address space in the process to hold the maximum memory. The backing file will only be extended as needed. For systems running with DB_PRIVATE specified, the maximum memory serves only as a limit and memory is allocated from the process data space as needed. No maximum need be set for private environments. + +For locking and thread information, groups of objects are allocated when needed so that there is less contention in the allocator during performance critical operations. Once allocated to a particular use, this memory will only be used for that structure. To avoid runtime contention, or to ensure a minimum number of a particular type of object, the DB_ENV->set_memory_init() method can be used. This method can set the initial numbers of particular types of structures to allocate at environment creation time. diff --git a/docs-src/guides/programmer_reference/ext.md b/docs-src/guides/programmer_reference/ext.md new file mode 100644 index 000000000..02bd783a2 --- /dev/null +++ b/docs-src/guides/programmer_reference/ext.md @@ -0,0 +1,67 @@ +--- +title: "Chapter 22.  Berkeley DB Extensions" +api-name: "Chapter 22.  Berkeley DB Extensions" +source: docs/programmer_reference/ext.html +--- +## Chapter 22.  Berkeley DB Extensions + +**Table of Contents** + + [Using Berkeley DB with Apache](ext.md#ext_mod) + + [Using Berkeley DB with Perl](ext_perl.md) + + [Using Berkeley DB with PHP](ext_php.md) + +## Using Berkeley DB with Apache + +A mod_db4 Apache module is included in the Berkeley DB distribution, providing a safe framework for running Berkeley DB applications in an Apache 1.3 environment. Apache natively provides no interface for communication between threads or processes, so the mod_db4 module exists to provide this communication. + +In general, it is dangerous to run Berkeley DB in a multiprocess system without some facility to coordinate database recovery between processes sharing the database environment after application or system failure. Failure to run recovery after failure can include process hangs and an inability to access the database environment. The mod_db4 Apache module oversees the proper management of Berkeley DB database environment resources. Developers building applications using Berkeley DB as the storage manager within an Apache module should employ this technique for proper resource management. + +Specifically, mod_db4 provides the following facilities: + +1. New constructors for DB_ENV and DB handles, which install replacement open/close methods. +2. Transparent caching of open DB_ENV and DB handles. +3. Reference counting on all structures, allowing the module to detect the initial opening of any managed database and automatically perform recovery. +4. Automatic detection of unexpected failures (segfaults, or a module actually calling exit() and avoiding shut down phases), and automatic termination of all child processes with open database resources to attempt consistency. + +mod_db4 is designed to be used as an alternative interface to Berkeley DB. To have another Apache module (for example, mod_foo) use mod_db4, do not link mod_foo against the Berkeley DB library. In your mod_foo makefile, you should: + +``` c +#include "mod_db4_export.h" +``` + +and add your Apache include directory to your CPPFLAGS. + +In mod_foo, to create a mod_db4 managed DB_ENV handle, use the following: + +``` c +int mod_db4_db_env_create(DB_ENV **dbenvp, u_int32_t flags); +``` + +which takes identical arguments to db_env_create(). + +To create a mod_db4 managed DB handle, use the following: + +``` c +int mod_db4_db_create(DB **dbp, DB_ENV *dbenv, u_int32_t flags); +``` + +which takes identical arguments to db_create(). + +Otherwise the API is completely consistent with the standard Berkeley DB API. + +The mod_db4 module requires the Berkeley DB library be compiled with C++ extensions and the MM library. (The MM library provides an abstraction layer which allows related processes to share data easily. On systems where shared memory or other inter-process communication mechanisms are not available, the MM library emulates them using temporary files. MM is used in several operating systems to provide shared memory pools to Apache modules.) + +To build this apache module, perform the following steps: + +``` c +% ./configure --with-apxs=[path to the apxs utility] \ + --with-db4=[Berkeley DB library installation directory] \ + --with-mm=[libmm installation directory] +% make +% make install +``` + +Post-installation, modules can use this extension via the functions documented in \$APACHE_INCLUDEDIR/mod_db4_export.h. diff --git a/docs-src/guides/programmer_reference/ext_perl.md b/docs-src/guides/programmer_reference/ext_perl.md new file mode 100644 index 000000000..fa5f66fe4 --- /dev/null +++ b/docs-src/guides/programmer_reference/ext_perl.md @@ -0,0 +1,14 @@ +--- +title: "Using Berkeley DB with Perl" +api-name: "Using Berkeley DB with Perl" +source: docs/programmer_reference/ext_perl.html +--- +## Using Berkeley DB with Perl + +The original Perl module for Berkeley DB was DB_File, which was written to interface to Berkeley DB version 1.85. The newer Perl module for Berkeley DB is BerkeleyDB, which was written to interface to version 2.0 and subsequent releases. Because Berkeley DB version 2.X has a compatibility API for version 1.85, you can (and should!) build DB_File using version 2.X of Berkeley DB, although DB_File will still only support the 1.85 functionality. + +DB_File is distributed with the standard Perl source distribution (look in the directory "ext/DB_File"). You can find both DB_File and BerkeleyDB on CPAN, the Comprehensive Perl Archive Network of mirrored FTP sites. The master CPAN site is ftp://ftp.funet.fi/. + +Versions of both BerkeleyDB and DB_File that are known to work correctly with each release of Berkeley DB are included in the distributed Berkeley DB source tree, in the subdirectories `perl.BerkeleyDB` and `perl.DB_File`. Each of those directories contains a `README` file with instructions on installing and using those modules. + +The Perl interface is not maintained by Oracle. Questions about the DB_File and BerkeleyDB modules are best asked on the Usenet newsgroup comp.lang.perl.modules. diff --git a/docs-src/guides/programmer_reference/ext_php.md b/docs-src/guides/programmer_reference/ext_php.md new file mode 100644 index 000000000..7fabe989c --- /dev/null +++ b/docs-src/guides/programmer_reference/ext_php.md @@ -0,0 +1,72 @@ +--- +title: "Using Berkeley DB with PHP" +api-name: "Using Berkeley DB with PHP" +source: docs/programmer_reference/ext_php.html +--- +## Using Berkeley DB with PHP + +A PHP 4 extension for this release of Berkeley DB is included in the distribution package. It can either either link directly against the installed Berkeley DB library (which is necessary for running in a non-Apache/mod_php4 environment), or against mod_db4, which provides additional safety when running under Apache/mod_php4. + +For installation instructions, see the `INSTALL` file that resides in the `lang/php_db4` directory in your Berkeley DB distribution. + +The PHP extension provides the following classes, which mirror the standard Berkeley DB C++ API. + +``` c +class Db4Env { + function Db4Env($flags = 0) {} + function close($flags = 0) {} + function dbremove($txn, $filename, $database = null, $flags = 0) {} + function dbrename($txn, $file, $database, $new_database, + $flags = 0) {} + function open($home, $flags = DB_CREATE | DB_INIT_LOCK | + DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN, + $mode = 0666) {} + function remove($home, $flags = 0) {} + function set_data_dir($directory) {} + function txn_begin($parent_txn = null, $flags = 0) {} + function txn_checkpoint($kbytes, $minutes, $flags = 0) {} +} + +class Db4 { + function Db4($dbenv = null) {} // create a new Db4 object using + // the optional DbEnv + function open($txn = null, $file = null, $database = null, + $flags = DB_CREATE, $mode = 0) {} + function close() {} + function del($key, $txn = null) {} + function get($key, $txn = null, $flags = 0) {} + function pget($key, &$pkey, $txn = null, $flags = 0) {} + function get_type() {} // returns the stringified + // database type name + function stat($txn = null, $flags = 0) {} // returns statistics as + // an as + function join($cursor_list, $flags = 0) {} + function sync() {} + function truncate($txn = null, $flags = 0) {} + function cursor($txn = null, flags = 0) {} +} + +class Db4Txn { + function abort() {} + function commit() {} + function discard() {} + function id() {} + function set_timeout($timeout, $flags = 0) {} +} + +class Db4Cursor { + function close() {} + function count() {} + function del() {} + function dup($flags = 0) {} + function get($key, $flags = 0) {} + function pget($key, &$primary_key, $flags = 0) {} + function put($key, $data, $flags = 0) {} +} +``` + +The PHP extension attempts to be "smart" for you by: + +1. Auto-committing operations on transactional databases if no explicit Db4Txn object is specified. +2. Performing reference and dependency checking to insure that all resources are closed in the correct order. +3. Supplying default values for flags. diff --git a/docs-src/guides/programmer_reference/general_am_conf.md b/docs-src/guides/programmer_reference/general_am_conf.md new file mode 100644 index 000000000..f1e826af0 --- /dev/null +++ b/docs-src/guides/programmer_reference/general_am_conf.md @@ -0,0 +1,109 @@ +--- +title: "General access method configuration" +api-name: "General access method configuration" +source: docs/programmer_reference/general_am_conf.html +--- +## General access method configuration + + [Selecting a page size](general_am_conf.md#am_conf_pagesize) + + [Selecting a cache size](general_am_conf.md#am_conf_cachesize) + + [Selecting a byte order](general_am_conf.md#am_conf_byteorder) + + [Duplicate data items](general_am_conf.md#am_conf_dup) + + [Non-local memory allocation](general_am_conf.md#am_conf_malloc) + +There are a series of configuration tasks which are common to all access methods. They are described in the following sections. + +### Selecting a page size + +The size of the pages used in the underlying database can be specified by calling the DB->set_pagesize() method. The minimum page size is 512 bytes and the maximum page size is 64K bytes, and must be a power of two. If no page size is specified by the application, a page size is selected based on the underlying filesystem I/O block size. (A page size selected in this way has a lower limit of 512 bytes and an upper limit of 16K bytes.) + +There are several issues to consider when selecting a pagesize: overflow record sizes, locking, I/O efficiency, and recoverability. + +First, the page size implicitly sets the size of an overflow record. Overflow records are key or data items that are too large to fit on a normal database page because of their size, and are therefore stored in overflow pages. Overflow pages are pages that exist outside of the normal database structure. For this reason, there is often a significant performance penalty associated with retrieving or modifying overflow records. Selecting a page size that is too small, and which forces the creation of large numbers of overflow pages, can seriously impact the performance of an application. + +Second, in the Btree, Hash and Recno access methods, the finest-grained lock that Berkeley DB acquires is for a page. (The Queue access method generally acquires record-level locks rather than page-level locks.) Selecting a page size that is too large, and which causes threads or processes to wait because other threads of control are accessing or modifying records on the same page, can impact the performance of your application. + +Third, the page size specifies the granularity of I/O from the database to the operating system. Berkeley DB will give a page-sized unit of bytes to the operating system to be scheduled for reading/writing from/to the disk. For many operating systems, there is an internal **block size** which is used as the granularity of I/O from the operating system to the disk. Generally, it will be more efficient for Berkeley DB to write filesystem-sized blocks to the operating system and for the operating system to write those same blocks to the disk. + +Selecting a database page size smaller than the filesystem block size may cause the operating system to coalesce or otherwise manipulate Berkeley DB pages and can impact the performance of your application. When the page size is smaller than the filesystem block size and a page written by Berkeley DB is not found in the operating system's cache, the operating system may be forced to read a block from the disk, copy the page into the block it read, and then write out the block to disk, rather than simply writing the page to disk. Additionally, as the operating system is reading more data into its buffer cache than is strictly necessary to satisfy each Berkeley DB request for a page, the operating system buffer cache may be wasting memory. + +Alternatively, selecting a page size larger than the filesystem block size may cause the operating system to read more data than necessary. On some systems, reading filesystem blocks sequentially may cause the operating system to begin performing read-ahead. If requesting a single database page implies reading enough filesystem blocks to satisfy the operating system's criteria for read-ahead, the operating system may do more I/O than is required. + +Fourth, when using the Berkeley DB Transactional Data Store product, the page size may affect the errors from which your database can recover See Berkeley DB recoverability for more information. + +### Note + +The db_tuner utility suggests a page size for btree databases that optimizes cache efficiency and storage space requirements. This utility works only when given a pre-populated database. So, it is useful when tuning an existing application and not when first implementing an application. + +### Selecting a cache size + +The size of the cache used for the underlying database can be specified by calling the DB->set_cachesize() method. Choosing a cache size is, unfortunately, an art. Your cache must be at least large enough for your working set plus some overlap for unexpected situations. + +When using the Btree access method, you must have a cache big enough for the minimum working set for a single access. This will include a root page, one or more internal pages (depending on the depth of your tree), and a leaf page. If your cache is any smaller than that, each new page will force out the least-recently-used page, and Berkeley DB will re-read the root page of the tree anew on each database request. + +If your keys are of moderate size (a few tens of bytes) and your pages are on the order of 4KB to 8KB, most Btree applications will be only three levels. For example, using 20 byte keys with 20 bytes of data associated with each key, a 8KB page can hold roughly 400 keys (or 200 key/data pairs), so a fully populated three-level Btree will hold 32 million key/data pairs, and a tree with only a 50% page-fill factor will still hold 16 million key/data pairs. We rarely expect trees to exceed five levels, although Berkeley DB will support trees up to 255 levels. + +The rule-of-thumb is that cache is good, and more cache is better. Generally, applications benefit from increasing the cache size up to a point, at which the performance will stop improving as the cache size increases. When this point is reached, one of two things have happened: either the cache is large enough that the application is almost never having to retrieve information from disk, or, your application is doing truly random accesses, and therefore increasing size of the cache doesn't significantly increase the odds of finding the next requested information in the cache. The latter is fairly rare -- almost all applications show some form of locality of reference. + +That said, it is important not to increase your cache size beyond the capabilities of your system, as that will result in reduced performance. Under many operating systems, tying down enough virtual memory will cause your memory and potentially your program to be swapped. This is especially likely on systems without unified OS buffer caches and virtual memory spaces, as the buffer cache was allocated at boot time and so cannot be adjusted based on application requests for large amounts of virtual memory. + +For example, even if accesses are truly random within a Btree, your access pattern will favor internal pages to leaf pages, so your cache should be large enough to hold all internal pages. In the steady state, this requires at most one I/O per operation to retrieve the appropriate leaf page. + +You can use the db_stat utility to monitor the effectiveness of your cache. The following output is excerpted from the output of that utility's **-m** option: + +``` c +prompt: db_stat -m +131072 Cache size (128K). +4273 Requested pages found in the cache (97%). +134 Requested pages not found in the cache. +18 Pages created in the cache. +116 Pages read into the cache. +93 Pages written from the cache to the backing file. +5 Clean pages forced from the cache. +13 Dirty pages forced from the cache. +0 Dirty buffers written by trickle-sync thread. +130 Current clean buffer count. +4 Current dirty buffer count. +``` + +The statistics for this cache say that there have been 4,273 requests of the cache, and only 116 of those requests required an I/O from disk. This means that the cache is working well, yielding a 97% cache hit rate. The db_stat utility will present these statistics both for the cache as a whole and for each file within the cache separately. + +### Selecting a byte order + +Database files created by Berkeley DB can be created in either little- or big-endian formats. The byte order used for the underlying database is specified by calling the DB->set_lorder() method. If no order is selected, the native format of the machine on which the database is created will be used. + +Berkeley DB databases are architecture independent, and any format database can be used on a machine with a different native format. In this case, as each page that is read into or written from the cache must be converted to or from the host format, and databases with non-native formats will incur a performance penalty for the run-time conversion. + +**It is important to note that the Berkeley DB access methods do no data conversion for application specified data. Key/data pairs written on a little-endian format architecture will be returned to the application exactly as they were written when retrieved on a big-endian format architecture.** + +### Duplicate data items + +The Btree and Hash access methods support the creation of multiple data items for a single key item. By default, multiple data items are not permitted, and each database store operation will overwrite any previous data item for that key. To configure Berkeley DB for duplicate data items, call the DB->set_flags() method with the DB_DUP flag. Only one copy of the key will be stored for each set of duplicate data items. If the Btree access method comparison routine returns that two keys compare equally, it is undefined which of the two keys will be stored and returned from future database operations. + +By default, Berkeley DB stores duplicates in the order in which they were added, that is, each new duplicate data item will be stored after any already existing data items. This default behavior can be overridden by using the DBC->put() method and one of the DB_AFTER, DB_BEFORE, DB_KEYFIRST or DB_KEYLAST flags. Alternatively, Berkeley DB may be configured to sort duplicate data items. + +When stepping through the database sequentially, duplicate data items will be returned individually, as a key/data pair, where the key item only changes after the last duplicate data item has been returned. For this reason, duplicate data items cannot be accessed using the DB->get() method, as it always returns the first of the duplicate data items. Duplicate data items should be retrieved using a Berkeley DB cursor interface such as the DBC->get() method. + +There is a flag that permits applications to request the following data item only if it **is** a duplicate data item of the current entry, see DB_NEXT_DUP for more information. There is a flag that permits applications to request the following data item only if it **is not** a duplicate data item of the current entry, see DB_NEXT_NODUP and DB_PREV_NODUP for more information. + +It is also possible to maintain duplicate records in sorted order. Sorting duplicates will significantly increase performance when searching them and performing equality joins — both of which are common operations when using secondary indices. To configure Berkeley DB to sort duplicate data items, the application must call the DB->set_flags() method with the DB_DUPSORT flag. Note that DB_DUPSORT automatically turns on the DB_DUP flag for you, so you do not have to also set that flag; however, it is not an error to also set DB_DUP when configuring for sorted duplicate records. + +When configuring sorted duplicate records, you can also specify a custom comparison function using the DB->set_dup_compare() method. If the DB_DUPSORT flag is given, but no comparison routine is specified, then Berkeley DB defaults to the same lexicographical sorting used for Btree keys, with shorter items collating before longer items. + +If the duplicate data items are unsorted, applications may store identical duplicate data items, or, for those that just like the way it sounds, *duplicate duplicates*. + +**It is an error to attempt to store identical duplicate data items when duplicates are being stored in a sorted order.** Any such attempt results in the error message "Duplicate data items are not supported with sorted data" with a `DB_KEYEXIST` return code. + +Note that you can suppress the error message "Duplicate data items are not supported with sorted data" by using the DB_NODUPDATA flag. Use of this flag does not change the database's basic behavior; storing duplicate data items in a database configured for sorted duplicates is still an error and so you will continue to receive the `DB_KEYEXIST` return code if you try to do that. + +For further information on how searching and insertion behaves in the presence of duplicates (sorted or not), see the DB->get() DB->put(), DBC->get() and DBC->put() documentation. + +### Non-local memory allocation + +Berkeley DB allocates memory for returning key/data pairs and statistical information which becomes the responsibility of the application. There are also interfaces where an application will allocate memory which becomes the responsibility of Berkeley DB. + +On systems in which there may be multiple library versions of the standard allocation routines (notably Windows NT), transferring memory between the library and the application will fail because the Berkeley DB library allocates memory from a different heap than the application uses to free it, or vice versa. To avoid this problem, the DB_ENV->set_alloc() and DB->set_alloc() methods can be used to give Berkeley DB references to the application's allocation routines. diff --git a/docs-src/guides/programmer_reference/group_membership.md b/docs-src/guides/programmer_reference/group_membership.md new file mode 100644 index 000000000..c9bee5597 --- /dev/null +++ b/docs-src/guides/programmer_reference/group_membership.md @@ -0,0 +1,108 @@ +--- +title: "Managing Replication Manager Group Membership" +api-name: "Managing Replication Manager Group Membership" +source: docs/programmer_reference/group_membership.html +--- +## Managing Replication Manager Group Membership + + [Adding Sites to a Replication Group](group_membership.md#group_mem_add) + + [Removing Sites from a Replication Group](group_membership.md#group_mem_remove) + + [Primordial Startups](group_membership.md#group_mem_primordialstartup) + + [Upgrading Groups](group_membership.md#group_mem_upgrade) + +A replication group is a collection of two or more database environments which are configured to replicate with one another. When operating normally, a replication group consists of a master site and one or more read-only sites. + +For Replication Manager applications, the sites comprising the replication group are recorded in an internal database, so even if a group member is not available, it counts towards the group's total site count. This matters for certain replication activities, such as holding elections and acknowledging replication messages that require some number of sites to participate in these activities. Replicated applications will often require all sites, or a majority of sites, to participate before the activity can be completed. + +### Note + +If you are configuring your application to keep replication metadata in-memory by specifying the DB_REP_CONF_INMEM flag to the DB_ENV->rep_set_config() method, then the internal database containing group site information is not stored persistently on disk. This severely limits Replication Manager's ability to automatically manage group membership. For more information, including some work-arounds, see Managing Replication Files. + +Because Replication Manager tracks group members, there are some administrative activities that you should know about when using BDB replication. + +### Adding Sites to a Replication Group + +To add a site to a replication group, you merely start up the site such that it knows where at least one site in the group is located. The site new site then joins the group. When this happens, the new site is recorded in the Replication Manager's group member database. + +Note that when you are starting the very first site in the group for the very first time (called *the primordial start up*), there are no other existing sites to help the new site join the group. In fact, a primordial start up actually creates the group. For this reason, there are some slight differences on how to perform a primordial start up. For a description of this, see Primordial Startups. + +When you add a site to a replication group, you use the following general procedure: + +- Make sure your replication group is operating well enough that write activity can occur. + +- Create and open the environment such that it is configured to use replication. + +- Use DB_ENV->repmgr_site() to obtain a DB_SITE handle. Configure this handle for the local site's host and port information when you create the handle. Then, use DB_SITE->set_config() to indicate that this is the local site by setting the `DB_LOCAL_SITE` parameter. + +- Use DB_ENV->repmgr_site() to obtain a second DB_SITE handle. Configure this handle with the host and port information for a site that already belongs to the replication group. Then, use DB_SITE->set_config() to indicate this site is a "helper" site by setting the `DB_BOOTSTRAP_HELPER` parameter. By configuring a DB_SITE handle in this way, your new site will know how to contact the replication group so that it can join the group. + +- Start replication as normal by configuring an acknowledgement policy, setting the site's replication priority, and then calling DB_ENV->repmgr_start(). + +Note that on subsequent start-ups of your replication code, any helper site information you might provide is ignored because the Replication Manager reads the group membership database in order to obtain this information. + +Also, be aware that if the new site cannot be added to the group for some reason (because a master site is not available, or because insufficient replicas are running to acknowledge the new site), the attempt to start the new site via DB_ENV->repmgr_start() will fail and return `DB_REP_UNAVAIL`. You can then pause and retry the start up attempt until it completes successfully. + +You must use the exact same host string and port number to refer to a given site throughout your application and on each of its sites. + +### Removing Sites from a Replication Group + +Elections and message acknowledgements require knowledge of the total number of sites in the group. If a site is shut down, or is otherwise unable to communicate with the rest of the group, it still counts towards the total number of sites in the group. In most cases, this is the desirable behavior. + +However, if you are shutting down a site permanently, then you should remove that site from the group. You might also want to remove a site from the group if you are shutting it down temporarily, but nevertheless for a very long period of time (days or weeks). In either case, you remove a site from the group by: + +- Make sure your replication group is operating well enough that write activity can occur. + +- On one of the sites in your replication group (this does not have to be the master site), use DB_ENV->repmgr_site() to obtain a DB_SITE handle. Configure this handle with the host and port information of the site that you want to remove. + + Note that this step can occur at any site — including the site that you are removing from the group. + +- Call the DB_SITE->remove() method. This removes the identified site from the replication group database. If this action is not performed on the master site, the client sends a request to the master to perform the operation and awaits confirmation. + +### Note + +Upon completing the above procedure, DO NOT call the DB_SITE->close() method. After removing (or even attempting to remove) a site from the group using a DB_SITE handle, the handle must never be accessed again. + +### Primordial Startups + +If you have never started a site in a replication group before, then the replication group membership database does not exist. In this situation, you must start the site and declare it to be the group creator. This causes the site to become the master, create the group membership database, and create a replication group of size 1. After that, subsequent sites can add themselves to the group as described in Adding Sites to a Replication Group. + +### Note + +It is never incorrect to declare a site the group creator. This is true even well-after the replication group has been established. This is because group creator information is ignored on any site start-up, except for the primoridial start-up; that is, a start-up where the group membership database does not exist. + +To declare a site as the group creator: + +- Create and open the environment such that it is configured to use replication. + +- Use DB_ENV->repmgr_site() to obtain a DB_SITE handle. Configure this handle for the local site's host and port information when you create the handle. Then, use DB_SITE->set_config() to indicate that this is the group creator site by setting the `DB_GROUP_CREATOR` parameter. + +- Start replication as normal by configuring acknowledgement policies, setting replication priorities for the site, and then calling DB_ENV->repmgr_start(). + +### Upgrading Groups + +Prior to the Berkeley DB 11.2.5.2 release, replication group membership was managed differently than in the way it is described in the previous sections. For this reason, when you upgrade from older releases of Berkeley DB to 11.2.5.2 or later, the upgrade procedure is different than when upgrading between other releases. + +To perform an upgrade that takes you from the old way of managing group membership to the new way of managing group membership (pre-11.2.5.2 to 11.2.5.2 and later), do the following: + +- Update your replication code to use the new DB_SITE handle and related methods. Recompile and thoroughly test your code to make sure it is production-ready. + +- Do the following one production machine at a time. Make sure to do this at the master site LAST. + + 1. Shut down the old replication code. + + 2. Install the new replication code. + + 3. Configure a DB_SITE handle for the local site. Use DB_SITE->set_config() to indicate that this is a legacy site by setting the `DB_LEGACY` parameter. + + 4. Configure a DB_SITE handle for *every other site* in the replication group. Set the `DB_LEGACY` parameter for each of these handles. + + Please pay careful attention to this step. To repeat: a DB_SITE handle MUST be configured for EVERY site in the replication group. + + 5. Start replication. The site is upgraded at this point. + + Once you have performed this procedure for each production site, making sure to upgrade the master only after every other site has been upgraded, you are done upgrading your replicated application to use the current group membership mechanism. + +On subsequent restarts of your replication code, you do not need to specify the `DB_LEGACY` parameter, nor do you need to identify all of the replication group members. However, it is not an error if you do specify this information on subsequent start ups. diff --git a/docs-src/guides/programmer_reference/hash_conf.md b/docs-src/guides/programmer_reference/hash_conf.md new file mode 100644 index 000000000..df89344f5 --- /dev/null +++ b/docs-src/guides/programmer_reference/hash_conf.md @@ -0,0 +1,38 @@ +--- +title: "Hash access method specific configuration" +api-name: "Hash access method specific configuration" +source: docs/programmer_reference/hash_conf.html +--- +## Hash access method specific configuration + + [Page fill factor](hash_conf.md#am_conf_h_ffactor) + + [Specifying a database hash](hash_conf.md#am_conf_h_hash) + + [Hash table size](hash_conf.md#am_conf_h_nelem) + +There are a series of configuration tasks which you can perform when using the Hash access method. They are described in the following sections. + +### Page fill factor + +The density, or page fill factor, is an approximation of the number of keys allowed to accumulate in any one bucket, determining when the hash table grows or shrinks. If you know the average sizes of the keys and data in your data set, setting the fill factor can enhance performance. A reasonable rule to use to compute fill factor is: + +``` c +(pagesize - 32) / (average_key_size + average_data_size + 8) +``` + +The desired density within the hash table can be specified by calling the DB->set_h_ffactor() method. If no density is specified, one will be selected dynamically as pages are filled. + +### Specifying a database hash + +The database hash determines in which bucket a particular key will reside. The goal of hashing keys is to distribute keys equally across the database pages, therefore it is important that the hash function work well with the specified keys so that the resulting bucket usage is relatively uniform. A hash function that does not work well can effectively turn into a sequential list. + +No hash performs equally well on all possible data sets. It is possible that applications may find that the default hash function performs poorly with a particular set of keys. The distribution resulting from the hash function can be checked using the db_stat utility. By comparing the number of hash buckets and the number of keys, one can decide if the entries are hashing in a well-distributed manner. + +The hash function for the hash table can be specified by calling the DB->set_h_hash() method. If no hash function is specified, a default function will be used. Any application-specified hash function must take a reference to a DB object, a pointer to a byte string and its length, as arguments and return an unsigned, 32-bit hash value. + +### Hash table size + +When setting up the hash database, knowing the expected number of elements that will be stored in the hash table is useful. This value can be used by the Hash access method implementation to more accurately construct the necessary number of buckets that the database will eventually require. + +The anticipated number of elements in the hash table can be specified by calling the DB->set_h_nelem() method. If not specified, or set too low, hash tables will expand gracefully as keys are entered, although a slight performance degradation may be noticed. In order for the estimated number of elements to be a useful value to Berkeley DB, the DB->set_h_ffactor() method must also be called to set the page fill factor. diff --git a/docs-src/guides/programmer_reference/heap_conf.md b/docs-src/guides/programmer_reference/heap_conf.md new file mode 100644 index 000000000..bf6245d9b --- /dev/null +++ b/docs-src/guides/programmer_reference/heap_conf.md @@ -0,0 +1,14 @@ +--- +title: "Heap access method specific configuration" +api-name: "Heap access method specific configuration" +source: docs/programmer_reference/heap_conf.html +--- +## Heap access method specific configuration + +Configuring the Heap access method is fairly simple. Beyond the general configuration required for any access method, you can configure how large the database will become, as well as the amount by which the database grows. + +If you provide no configuration relative to the heap size, then the database will grow without bound. Whether this is desirable depends on how much disk space is available to your application. + +You can limit the size of the on-disk database file by using the DB->set_heapsize() method. If the size specified on this method is reached, then further attempts to insert/update records will fail with a `DB_HEAP_FULL` error message. + +Heap databases are organized into regions, and each region is a constant size. The size of the region in a heap database is limited by the page size, the first page of the region contains a bitmap describing the available space on the remaining pages in the region. When the database experiences write contention, a region is added to reduce contention. This means heap databases can grow in size very quickly. In order to control the amount by which the database increases, the size of the region is configurable via DB->set_heap_regionsize(). diff --git a/docs-src/guides/programmer_reference/img/arch_bigpic.gif b/docs-src/guides/programmer_reference/img/arch_bigpic.gif new file mode 100644 index 000000000..48c52aed5 Binary files /dev/null and b/docs-src/guides/programmer_reference/img/arch_bigpic.gif differ diff --git a/docs-src/guides/programmer_reference/img/arch_smallpic.gif b/docs-src/guides/programmer_reference/img/arch_smallpic.gif new file mode 100644 index 000000000..5eb7ae8da Binary files /dev/null and b/docs-src/guides/programmer_reference/img/arch_smallpic.gif differ diff --git a/docs-src/guides/programmer_reference/index.md b/docs-src/guides/programmer_reference/index.md new file mode 100644 index 000000000..dbed63c48 --- /dev/null +++ b/docs-src/guides/programmer_reference/index.md @@ -0,0 +1,764 @@ +--- +title: "Berkeley DB Programmer's Reference Guide" +api-name: "Berkeley DB Programmer's Reference Guide" +source: docs/programmer_reference/index.html +--- +# Berkeley DB Programmer's Reference Guide + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction](intro.md) + + [An introduction to data management](intro.md#intro_data) + + [Mapping the terrain: theory and practice](intro_terrain.md) + + [Data access and data management](intro_terrain.md#idp50584200) + + [Relational databases](intro_terrain.md#idp50577368) + + [Object-oriented databases](intro_terrain.md#idp50621160) + + [Network databases](intro_terrain.md#idp50574144) + + [Clients and servers](intro_terrain.md#idp50647776) + + [What is Berkeley DB?](intro_dbis.md) + + [Data Access Services](intro_dbis.md#idp50588152) + + [Data management services](intro_dbis.md#idm1374112) + + [Design](intro_dbis.md#idp50659944) + + [What Berkeley DB is not](intro_dbisnot.md) + + [Berkeley DB is not a relational database](intro_dbisnot.md#idp50596256) + + [Berkeley DB is not an object-oriented database](intro_dbisnot.md#idp50675392) + + [Berkeley DB is not a network database](intro_dbisnot.md#idp50621304) + + [Berkeley DB is not a database server](intro_dbisnot.md#idp50657008) + + [Do you need Berkeley DB?](intro_need.md) + + [What other services does Berkeley DB provide?](intro_what.md) + + [What does the Berkeley DB distribution include?](intro_distrib.md) + + [Where does Berkeley DB run?](intro_where.md) + + [The Berkeley DB products](intro_products.md) + + [Berkeley DB Data Store](intro_products.md#idp50715960) + + [Berkeley DB Concurrent Data Store](intro_products.md#idp50715552) + + [Berkeley DB Transactional Data Store](intro_products.md#idp50708368) + + [Berkeley DB High Availability](intro_products.md#idp50712672) + + [2. Access Method Configuration](am_conf.md) + + [What are the available access methods?](am_conf.md#am_conf_intro) + + [Btree](am_conf.md#idp50599376) + + [Hash](am_conf.md#idp50705400) + + [Heap](am_conf.md#idp50708952) + + [Queue](am_conf.md#idm1385000) + + [Recno](am_conf.md#idp50715336) + + [Selecting an access method](am_conf_select.md) + + [Btree or Heap?](am_conf_select.md#idp50702528) + + [Hash or Btree?](am_conf_select.md#idp50755552) + + [Queue or Recno?](am_conf_select.md#idp50569200) + + [Logical record numbers](am_conf_logrec.md) + + [General access method configuration](general_am_conf.md) + + [Selecting a page size](general_am_conf.md#am_conf_pagesize) + + [Selecting a cache size](general_am_conf.md#am_conf_cachesize) + + [Selecting a byte order](general_am_conf.md#am_conf_byteorder) + + [Duplicate data items](general_am_conf.md#am_conf_dup) + + [Non-local memory allocation](general_am_conf.md#am_conf_malloc) + + [Btree access method specific configuration](bt_conf.md) + + [Btree comparison](bt_conf.md#am_conf_bt_compare) + + [Btree prefix comparison](bt_conf.md#am_conf_bt_prefix) + + [Minimum keys per page](bt_conf.md#am_conf_bt_minkey) + + [Retrieving Btree records by logical record number](bt_conf.md#am_conf_bt_recnum) + + [Compression](bt_conf.md#am_conf_bt_compress) + + [Hash access method specific configuration](hash_conf.md) + + [Page fill factor](hash_conf.md#am_conf_h_ffactor) + + [Specifying a database hash](hash_conf.md#am_conf_h_hash) + + [Hash table size](hash_conf.md#am_conf_h_nelem) + + [Heap access method specific configuration](heap_conf.md) + + [Queue and Recno access method specific configuration](rq_conf.md) + + [Managing record-based databases](rq_conf.md#am_conf_recno) + + [Selecting a Queue extent size](rq_conf.md#am_conf_extentsize) + + [Flat-text backing files](rq_conf.md#am_conf_re_source) + + [Logically renumbering records](rq_conf.md#am_conf_renumber) + + [3. Access Method Operations](am.md) + + [Database open](am.md#am_open) + + [Opening multiple databases in a single file](am_opensub.md) + + [Configuring databases sharing a file](am_opensub.md#idp50943544) + + [Caching databases sharing a file](am_opensub.md#idp50944288) + + [Locking in databases based on sharing a file](am_opensub.md#idp50944984) + + [Partitioning databases](am_partition.md) + + [Specifying partition keys](am_partition.md#am_partition_keys) + + [Partitioning callback](am_partition.md#am_partition_function) + + [Placing partition files](am_partition.md#partition_file_placement) + + [Retrieving records](am_get.md) + + [Storing records](am_put.md) + + [Deleting records](am_delete.md) + + [Database statistics](am_stat.md) + + [Database truncation](am_truncate.md) + + [Database upgrade](am_upgrade.md) + + [Database verification and salvage](am_verify.md) + + [Flushing the database cache](am_sync.md) + + [Database close](am_close.md) + + [Secondary indexes](am_second.md) + + [Error Handling With Secondary Indexes](am_second.md#idp51040080) + + [Foreign key indexes](am_foreign.md) + + [Cursor operations](am_cursor.md) + + [Retrieving records with a cursor](am_cursor.md#am_curget) + + [Storing records with a cursor](am_cursor.md#am_curput) + + [Deleting records with a cursor](am_cursor.md#am_curdel) + + [Duplicating a cursor](am_cursor.md#am_curdup) + + [Equality Join](am_cursor.md#am_join) + + [Data item count](am_cursor.md#am_count) + + [Cursor close](am_cursor.md#am_curclose) + + [4. Access Method Wrapup](am_misc.md) + + [Data alignment](am_misc.md#am_misc_align) + + [Retrieving and updating records in bulk](am_misc_bulk.md) + + [Bulk retrieval](am_misc_bulk.md#am_misc_bulk_get) + + [Bulk updates](am_misc_bulk.md#am_misc_bulk_put) + + [Bulk deletes](am_misc_bulk.md#am_misc_bulk_del) + + [Partial record storage and retrieval](am_misc_partial.md) + + [Storing C/C++ structures/objects](am_misc_struct.md) + + [Retrieved key/data permanence for C/C++](am_misc_perm.md) + + [Error support](am_misc_error.md) + + [Cursor stability](am_misc_stability.md) + + [Database limits](am_misc_dbsizes.md) + + [Disk space requirements](am_misc_diskspace.md) + + [Btree](am_misc_diskspace.md#idp51253016) + + [Hash](am_misc_diskspace.md#idp51253080) + + [Specifying a Berkeley DB schema using SQL DDL](am_misc_db_sql.md) + + [Access method tuning](am_misc_tune.md) + + [Access method FAQ](am_misc_faq.md) + + [5. Java API](java.md) + + [Java configuration](java.md#java_conf) + + [Compatibility](java_compat.md) + + [Java programming notes](java_program.md) + + [Java FAQ](java_faq.md) + + [6. C# API](csharp.md) + + [Compatibility](csharp.md#csharp_compat) + + [7. Standard Template Library API](stl.md) + + [Dbstl introduction](stl.md#stl_intro) + + [Standards compatible](stl.md#stl_intro_stdcompat) + + [Performance overhead](stl.md#stl_intro_performance) + + [Portability](stl.md#stl_intro_portability) + + [Dbstl typical use cases](stl_usecase.md) + + [Dbstl examples](stl_examples.md) + + [Berkeley DB configuration](stl_db_usage.md) + + [Registering database and environment handles](stl_db_usage.md#idp51381760) + + [Truncate requirements](stl_db_usage.md#idp51405208) + + [Auto commit support](stl_db_usage.md#idp51416888) + + [Database and environment identity checks](stl_db_usage.md#idp51379224) + + [Products, constructors and configurations](stl_db_usage.md#idp51415360) + + [Using advanced Berkeley DB features with dbstl](stl_db_advanced_usage.md) + + [Using bulk retrieval iterators](stl_db_advanced_usage.md#idp51421384) + + [Using the DB_RMW flag](stl_db_advanced_usage.md#idp51410312) + + [Using secondary index database and secondary containers](stl_db_advanced_usage.md#idp51398048) + + [Using transactions in dbstl](stl_txn_usage.md) + + [Using dbstl in multithreaded applications](stl_mt_usage.md) + + [Working with primitive types](stl_primitive_rw.md) + + [Storing strings](stl_primitive_rw.md#idp51467888) + + [Store and Retrieve data or objects of complex types](stl_complex_rw.md) + + [Storing varying length objects](stl_complex_rw.md#idp51458752) + + [Storing arbitrary sequences](stl_complex_rw.md#idp51477944) + + [Notes](stl_complex_rw.md#idp51524696) + + [Dbstl persistence](stl_persistence.md) + + [Direct database get](stl_persistence.md#directdbget) + + [Change persistence](stl_persistence.md#chg_persistence) + + [Object life time and persistence](stl_persistence.md#obj_life_persistence) + + [Dbstl container specific notes](stl_container_specific.md) + + [db_vector specific notes](stl_container_specific.md#idp51492808) + + [Associative container specific notes](stl_container_specific.md#idp51561456) + + [Using dbstl efficiently](stl_efficienct_use.md) + + [Using iterators efficiently](stl_efficienct_use.md#idp51530568) + + [Using containers efficiently](stl_efficienct_use.md#idp51530352) + + [Dbstl memory management](stl_memory_mgmt.md) + + [Freeing memory](stl_memory_mgmt.md#idp51564672) + + [Type specific notes](stl_memory_mgmt.md#idp51569240) + + [Dbstl miscellaneous notes](stl_misc.md) + + [Special notes about trivial methods](stl_misc.md#idp51587208) + + [Using correct container and iterator public types](stl_misc.md#idp51603304) + + [Dbstl known issues](stl_known_issues.md) + + [8. Berkeley DB Architecture](arch.md) + + [The big picture](arch.md#arch_bigpic) + + [Programming model](arch_progmodel.md) + + [Programmatic APIs](arch_apis.md) + + [C](arch_apis.md#idp51640232) + + [C++](arch_apis.md#idp51656168) + + [STL](arch_apis.md#idp51646944) + + [Java](arch_apis.md#idp51647760) + + [Dbm/Ndbm, Hsearch](arch_apis.md#idp51664896) + + [Scripting languages](arch_script.md) + + [Perl](arch_script.md#idp51640920) + + [PHP](arch_script.md#idp51639128) + + [Tcl](arch_script.md#idp51657264) + + [Supporting utilities](arch_utilities.md) + + [9. The Berkeley DB Environment](env.md) + + [Database environment introduction](env.md#env_intro) + + [Creating a database environment](env_create.md) + + [Sizing a database environment](env_size.md) + + [Opening databases within the environment](env_open.md) + + [Error support](env_error.md) + + [DB_CONFIG configuration file](env_db_config.md) + + [File naming](env_naming.md) + + [Specifying file naming to Berkeley DB](env_naming.md#idp51749352) + + [Filename resolution in Berkeley DB](env_naming.md#idp51763728) + + [Examples](env_naming.md#idp51756464) + + [Shared memory regions](env_region.md) + + [Security](env_security.md) + + [Encryption](env_encrypt.md) + + [Remote filesystems](env_remote.md) + + [Environment FAQ](env_faq.md) + + [10. Berkeley DB Concurrent Data Store Applications](cam.md) + + [Concurrent Data Store introduction](cam.md#cam_intro) + + [Handling failure in Data Store and Concurrent Data Store applications](cam_fail.md) + + [Architecting Data Store and Concurrent Data Store applications](cam_app.md) + + [11. Berkeley DB Transactional Data Store Applications](transapp.md) + + [Transactional Data Store introduction](transapp.md#transapp_intro) + + [Why transactions?](transapp_why.md) + + [Terminology](transapp_term.md) + + [Handling failure in Transactional Data Store applications](transapp_fail.md) + + [Architecting Transactional Data Store applications](transapp_app.md) + + [Opening the environment](transapp_env_open.md) + + [Opening the databases](transapp_data_open.md) + + [Recoverability and deadlock handling](transapp_put.md) + + [Atomicity](transapp_atomicity.md) + + [Isolation](transapp_inc.md) + + [Degrees of isolation](transapp_read.md) + + [Snapshot Isolation](transapp_read.md#snapshot_isolation) + + [Transactional cursors](transapp_cursor.md) + + [Nested transactions](transapp_nested.md) + + [Environment infrastructure](transapp_admin.md) + + [Deadlock detection](transapp_deadlock.md) + + [Checkpoints](transapp_checkpoint.md) + + [Database and log file archival](transapp_archival.md) + + [Log file removal](transapp_logfile.md) + + [Recovery procedures](transapp_recovery.md) + + [Hot failover](transapp_hotfail.md) + + [Using Recovery on Journaling Filesystems](transapp_journal.md) + + [Recovery and filesystem operations](transapp_filesys.md) + + [Berkeley DB recoverability](transapp_reclimit.md) + + [Transaction tuning](transapp_tune.md) + + [Transaction throughput](transapp_throughput.md) + + [Transaction FAQ](transapp_faq.md) + + [12. Berkeley DB Replication](rep.md) + + [Replication introduction](rep.md#rep_intro) + + [Replication environment IDs](rep_id.md) + + [Replication environment priorities](rep_pri.md) + + [Building replicated applications](rep_app.md) + + [Replication Manager methods](rep_mgr_meth.md) + + [Base API Methods](rep_base_meth.md) + + [Building the communications infrastructure](rep_comm.md) + + [Connecting to a new site](rep_newsite.md) + + [Managing Replication Manager Group Membership](group_membership.md) + + [Adding Sites to a Replication Group](group_membership.md#group_mem_add) + + [Removing Sites from a Replication Group](group_membership.md#group_mem_remove) + + [Primordial Startups](group_membership.md#group_mem_primordialstartup) + + [Upgrading Groups](group_membership.md#group_mem_upgrade) + + [Managing Replication Files](rep_filename.md) + + [Running Replication Manager in multiple processes](rep_mgrmulti.md) + + [One replication process and multiple subordinate processes](rep_mgrmulti.md#idp52420616) + + [Persistence of local site network address configuration](rep_mgrmulti.md#idp52417008) + + [Programming considerations](rep_mgrmulti.md#idp52400144) + + [Handling failure](rep_mgrmulti.md#idp52414488) + + [Other miscellaneous rules](rep_mgrmulti.md#idp52412256) + + [Running Replication using the db_replicate Utility](rep_replicate.md) + + [One Replication Process and Multiple Subordinate Processes](rep_replicate.md#idp52430544) + + [Common Use Case](rep_replicate.md#idp52447760) + + [Avoiding Rollback](rep_replicate.md#idp52457840) + + [When to Consider an Integrated HA Application](rep_replicate.md#idp52462952) + + [Choosing a Replication Manager Ack Policy](rep_mgr_ack.md) + + [Elections](rep_elect.md) + + [Synchronizing with a master](rep_mastersync.md) + + [Delaying client synchronization](rep_mastersync.md#rep_delay_sync) + + [Client-to-client synchronization](rep_mastersync.md#rep_c2c_sync) + + [Blocked client operations](rep_mastersync.md#idp52488504) + + [Clients too far out-of-date to synchronize](rep_mastersync.md#idp52510624) + + [Initializing a new site](rep_init.md) + + [Bulk transfer](rep_bulk.md) + + [Transactional guarantees](rep_trans.md) + + [Master Leases](rep_lease.md) + + [Changing Group Size](rep_lease.md#masterlease_change_groupsize) + + [Read your writes consistency](rep_ryw.md) + + [Getting a token](rep_ryw.md#gettoken) + + [Token handling](rep_ryw.md#tokenhandling) + + [Using a token to check or wait for a transaction](rep_ryw.md#usingtoken) + + [Clock Skew](rep_clock_skew.md) + + [Using Replication Manager message channels](repmgr_channels.md) + + [DB_CHANNEL](repmgr_channels.md#dbchannel_class) + + [Sending messages over a message channel](repmgr_channels.md#dbchannel_send) + + [Receiving messages](repmgr_channels.md#dbchannel_receive) + + [Special considerations for two-site replication groups](rep_twosite.md) + + [Network partitions](rep_partition.md) + + [Replication FAQ](rep_faq.md) + + [Ex_rep: a replication example](rep_ex.md) + + [Ex_rep_base: a TCP/IP based communication infrastructure](rep_ex_comm.md) + + [Ex_rep_base: putting it all together](rep_ex_rq.md) + + [Ex_rep_chan: a Replication Manager channel example](rep_ex_chan.md) + + [13. Distributed Transactions](xa.md) + + [Introduction](xa.md#xa_intro) + + [Berkeley DB XA Implementation](ch13s02.md) + + [Building a Global Transaction Manager](xa_build.md) + + [Communicating with multiple Berkeley DB environments](xa_build.md#idp52778488) + + [Recovering from GTM failure](xa_build.md#idp52779432) + + [Managing the Global Transaction ID (GID) name space](xa_build.md#idp52703176) + + [Maintaining state for each distributed transaction.](xa_build.md#idp52758336) + + [Recovering from the failure of a single environment](xa_build.md#idp52777008) + + [Recovering from GTM failure](xa_build.md#idp52779896) + + [XA Introduction](xa_xa_intro.md) + + [Configuring Berkeley DB with the Tuxedo System](xa_xa_config.md) + + [Update the Resource Manager File in Tuxedo](xa_xa_config.md#idp52786896) + + [Build the Transaction Manager Server](xa_xa_config.md#idp52812512) + + [Update the UBBCONFIG File](xa_xa_config.md#idp52759288) + + [Restrictions on XA Transactions](xa_xa_restrict.md) + + [XA: Frequently Asked Questions](xa_faq.md) + + [14. Application Specific Logging and Recovery](apprec.md) + + [Introduction to application specific logging and recovery](apprec.md#apprec_intro) + + [Defining application-specific log records](apprec_def.md) + + [Automatically generated functions](apprec_auto.md) + + [Application configuration](apprec_config.md) + + [15. Programmer Notes](program.md) + + [Signal handling](program.md#program_appsignals) + + [Error returns to applications](program_errorret.md) + + [Environment variables](program_environ.md) + + [Multithreaded applications](program_mt.md) + + [Berkeley DB handles](program_scope.md) + + [Name spaces](program_namespace.md) + + [C Language Name Space](program_namespace.md#idp52962960) + + [Filesystem Name Space](program_namespace.md#idp53001824) + + [Memory-only or Flash configurations](program_ram.md) + + [Disk drive caches](program_cache.md) + + [Copying or moving databases](program_copy.md) + + [Compatibility with historic UNIX interfaces](program_compatible.md) + + [Run-time configuration](program_runtime.md) + + [Performance Event Monitoring](program_perfmon.md) + + [Using the DTrace Provider](program_perfmon.md#program_perfmon_dtrace) + + [Using SystemTap](program_perfmon.md#program_perfmon_stap) + + [Example Scripts](program_perfmon.md#program_perfmon_examples) + + [Performance Events Reference](program_perfmon.md#program_perfmon_probes) + + [Programmer notes FAQ](program_faq.md) + + [16. The Locking Subsystem](lock.md) + + [Introduction to the locking subsystem](lock.md#lock_intro) + + [Configuring locking](lock_config.md) + + [Configuring locking: sizing the system](lock_max.md) + + [Standard lock modes](lock_stdmode.md) + + [Deadlock detection](lock_dead.md) + + [Deadlock detection using timers](lock_timeout.md) + + [Deadlock debugging](lock_deaddbg.md) + + [Locking granularity](lock_page.md) + + [Locking without transactions](lock_notxn.md) + + [Locking with transactions: two-phase locking](lock_twopl.md) + + [Berkeley DB Concurrent Data Store locking conventions](lock_cam_conv.md) + + [Berkeley DB Transactional Data Store locking conventions](lock_am_conv.md) + + [Locking and non-Berkeley DB applications](lock_nondb.md) + + [17. The Logging Subsystem](log.md) + + [Introduction to the logging subsystem](log.md#log_intro) + + [Configuring logging](log_config.md) + + [Log file limits](log_limits.md) + + [18. The Memory Pool Subsystem](mp.md) + + [Introduction to the memory pool subsystem](mp.md#mp_intro) + + [Configuring the memory pool](mp_config.md) + + [Warming the memory pool](mp_warm.md) + + [The warm_cache() function](mp_warm.md#warm_cache) + + [19. The Transaction Subsystem](txn.md) + + [Introduction to the transaction subsystem](txn.md#txn_intro) + + [Configuring transactions](txn_config.md) + + [Transaction limits](txn_limits.md) + + [Transaction IDs](txn_limits.md#idp53352320) + + [Cursors](txn_limits.md#idp53275624) + + [Multiple Threads of Control](txn_limits.md#idp53223368) + + [20. Sequences](sequence.md) + + [21. Berkeley DB Extensions: Tcl](tcl.md) + + [Loading Berkeley DB with Tcl](tcl.md#tcl_intro) + + [Installing as a Tcl Package](tcl.md#idp53366464) + + [Loading Berkeley DB with Tcl](tcl.md#idp53356912) + + [Using Berkeley DB with Tcl](tcl_using.md) + + [Tcl API programming notes](tcl_program.md) + + [Tcl error handling](tcl_error.md) + + [Tcl FAQ](tcl_faq.md) + + [22. Berkeley DB Extensions](ext.md) + + [Using Berkeley DB with Apache](ext.md#ext_mod) + + [Using Berkeley DB with Perl](ext_perl.md) + + [Using Berkeley DB with PHP](ext_php.md) + + [23. Dumping and Reloading Databases](dumpload.md) + + [The db_dump and db_load utilities](dumpload.md#dumpload_utility) + + [Dump output formats](dumpload_format.md) + + [Loading text into databases](dumpload_text.md) + + [24. Additional References](refs.md) + + [Additional references](refs.md#refs_refs) + + [Technical Papers on Berkeley DB](refs.md#idp53369464) + + [Background on Berkeley DB Features](refs.md#idp53449960) + + [Database Systems Theory](refs.md#idp53443200) diff --git a/docs-src/guides/programmer_reference/intro.md b/docs-src/guides/programmer_reference/intro.md new file mode 100644 index 000000000..e086cb1cd --- /dev/null +++ b/docs-src/guides/programmer_reference/intro.md @@ -0,0 +1,70 @@ +--- +title: "Chapter 1.  Introduction" +api-name: "Chapter 1.  Introduction" +source: docs/programmer_reference/intro.html +--- +## Chapter 1.  Introduction + +**Table of Contents** + + [An introduction to data management](intro.md#intro_data) + + [Mapping the terrain: theory and practice](intro_terrain.md) + + [Data access and data management](intro_terrain.md#idp50584200) + + [Relational databases](intro_terrain.md#idp50577368) + + [Object-oriented databases](intro_terrain.md#idp50621160) + + [Network databases](intro_terrain.md#idp50574144) + + [Clients and servers](intro_terrain.md#idp50647776) + + [What is Berkeley DB?](intro_dbis.md) + + [Data Access Services](intro_dbis.md#idp50588152) + + [Data management services](intro_dbis.md#idm1374112) + + [Design](intro_dbis.md#idp50659944) + + [What Berkeley DB is not](intro_dbisnot.md) + + [Berkeley DB is not a relational database](intro_dbisnot.md#idp50596256) + + [Berkeley DB is not an object-oriented database](intro_dbisnot.md#idp50675392) + + [Berkeley DB is not a network database](intro_dbisnot.md#idp50621304) + + [Berkeley DB is not a database server](intro_dbisnot.md#idp50657008) + + [Do you need Berkeley DB?](intro_need.md) + + [What other services does Berkeley DB provide?](intro_what.md) + + [What does the Berkeley DB distribution include?](intro_distrib.md) + + [Where does Berkeley DB run?](intro_where.md) + + [The Berkeley DB products](intro_products.md) + + [Berkeley DB Data Store](intro_products.md#idp50715960) + + [Berkeley DB Concurrent Data Store](intro_products.md#idp50715552) + + [Berkeley DB Transactional Data Store](intro_products.md#idp50708368) + + [Berkeley DB High Availability](intro_products.md#idp50712672) + +## An introduction to data management + +Cheap, powerful computing and networking have created countless new applications that could not have existed a decade ago. The advent of the World-Wide Web, and its influence in driving the Internet into homes and businesses, is one obvious example. Equally important, though, is the shift from large, general-purpose desktop and server computers toward smaller, special-purpose devices with built-in processing and communications services. + +As computer hardware has spread into virtually every corner of our lives, of course, software has followed. Software developers today are building applications not just for conventional desktop and server environments, but also for handheld computers, home appliances, networking hardware, cars and trucks, factory floor automation systems, cellphones, and more. + +While these operating environments are diverse, the problems that software engineers must solve in them are often strikingly similar. Most systems must deal with the outside world, whether that means communicating with users or controlling machinery. As a result, most need some sort of I/O system. Even a simple, single-function system generally needs to handle multiple tasks, and so needs some kind of operating system to schedule and manage control threads. Also, many computer systems must store and retrieve data to track history, record configuration settings, or manage access. + +Data management can be very simple. In some cases, just recording configuration in a flat text file is enough. More often, though, programs need to store and search a large amount of data, or structurally complex data. Database management systems are tools that programmers can use to do this work quickly and efficiently using off-the-shelf software. + +Of course, database management systems have been around for a long time. Data storage is a problem dating back to the earliest days of computing. Software developers can choose from hundreds of good, commercially-available database systems. The problem is selecting the one that best solves the problems that their applications face. diff --git a/docs-src/guides/programmer_reference/intro_dbis.md b/docs-src/guides/programmer_reference/intro_dbis.md new file mode 100644 index 000000000..ec79448c3 --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_dbis.md @@ -0,0 +1,66 @@ +--- +title: "What is Berkeley DB?" +api-name: "What is Berkeley DB?" +source: docs/programmer_reference/intro_dbis.html +--- +## What is Berkeley DB? + + [Data Access Services](intro_dbis.md#idp50588152) + + [Data management services](intro_dbis.md#idm1374112) + + [Design](intro_dbis.md#idp50659944) + +So far, we have discussed database systems in general terms. It is time now to consider Berkeley DB in particular and see how it fits into the framework we have introduced. The key question is, what kinds of applications should use Berkeley DB? + +Berkeley DB is an Open Source embedded database library that provides scalable, high-performance, transaction-protected data management services to applications. Berkeley DB provides a simple function-call API for data access and management. + +By "Open Source," we mean Berkeley DB is distributed under a license that conforms to the Open Source Definition. This license guarantees Berkeley DB is freely available for use and redistribution in other Open Source applications. Oracle Corporation sells commercial licenses allowing the redistribution of Berkeley DB in proprietary applications. In all cases the complete source code for Berkeley DB is freely available for download and use. + +Berkeley DB is "embedded" because it links directly into the application. It runs in the same address space as the application. As a result, no inter-process communication, either over the network or between processes on the same machine, is required for database operations. Berkeley DB provides a simple function-call API for a number of programming languages, including C, C++, Java, Perl, Tcl, Python, and PHP. All database operations happen inside the library. Multiple processes, or multiple threads in a single process, can all use the database at the same time as each uses the Berkeley DB library. Low-level services like locking, transaction logging, shared buffer management, memory management, and so on are all handled transparently by the library. + +The Berkeley DB library is extremely portable. It runs under almost all UNIX and Linux variants, Windows, and a number of embedded real-time operating systems. It runs on both 32-bit and 64-bit systems. It has been deployed on high-end Internet servers, desktop machines, and on palmtop computers, set-top boxes, in network switches, and elsewhere. Once Berkeley DB is linked into the application, the end user generally does not know that there is a database present at all. + +Berkeley DB is scalable in a number of respects. The database library itself is quite compact (under 300 kilobytes of text space on common architectures), which means it is small enough to run in tightly constrained embedded systems, but yet it can take advantage of gigabytes of memory and terabytes of disk if you are using hardware that has those resources. + +Each of Berkeley DB's database files can contain up to 256 terabytes of data, assuming the underlying filesystem is capable of supporting files of that size. Note that Berkeley DB applications often use multiple database files. This means that the amount of data your Berkeley DB application can manage is really limited only by the constraints imposed by your operating system, filesystem, and physical hardware. + +Berkeley DB also supports high concurrency, allowing thousands of users to operate on the same database files at the same time. + +Berkeley DB generally outperforms relational and object-oriented database systems in embedded applications for a couple of reasons. First, because the library runs in the same address space, no inter-process communication is required for database operations. The cost of communicating between processes on a single machine, or among machines on a network, is much higher than the cost of making a function call. Second, because Berkeley DB uses a simple function-call interface for all operations, there is no query language to parse, and no execution plan to produce. + +### Data Access Services + +Berkeley DB applications can choose the storage structure that best suits the application. Berkeley DB supports hash tables, Btrees, simple record-number-based storage, and persistent queues. Programmers can create tables using any of these storage structures, and can mix operations on different kinds of tables in a single application. + +Hash tables are generally good for very large databases that need predictable search and update times for random-access records. Hash tables allow users to ask, "Does this key exist?" or to fetch a record with a known key. Hash tables do not allow users to ask for records with keys that are close to a known key. + +Btrees are better for range-based searches, as when the application needs to find all records with keys between some starting and ending value. Btrees also do a better job of exploiting *locality of reference*. If the application is likely to touch keys near each other at the same time, the Btrees work well. The tree structure keeps keys that are close together near one another in storage, so fetching nearby values usually does not require a disk access. + +Record-number-based storage is natural for applications that need to store and fetch records, but that do not have a simple way to generate keys of their own. In a record number table, the record number is the key for the record. Berkeley DB will generate these record numbers automatically. + +Queues are well-suited for applications that create records, and then must deal with those records in creation order. A good example is on-line purchasing systems. Orders can enter the system at any time, but should generally be filled in the order in which they were placed. + +### Data management services + +Berkeley DB offers important data management services, including concurrency, transactions, and recovery. All of these services work on all of the storage structures. + +Many users can work on the same database concurrently. Berkeley DB handles locking transparently, ensuring that two users working on the same record do not interfere with one another. + +The library provides strict ACID transaction semantics, by default. However, applications are allowed to relax the isolation guarantees the database system makes. + +Multiple operations can be grouped into a single transaction, and can be committed or rolled back atomically. Berkeley DB uses a technique called *two-phase locking* to be sure that concurrent transactions are isolated from one another, and a technique called *write-ahead logging* to guarantee that committed changes survive application, system, or hardware failures. + +When an application starts up, it can ask Berkeley DB to run recovery. Recovery restores the database to a clean state, with all committed changes present, even after a crash. The database is guaranteed to be consistent and all committed changes are guaranteed to be present when recovery completes. + +An application can specify, when it starts up, which data management services it will use. Some applications need fast, single-user, non-transactional Btree data storage. In that case, the application can disable the locking and transaction systems, and will not incur the overhead of locking or logging. If an application needs to support multiple concurrent users, but does not need transactions, it can turn on locking without transactions. Applications that need concurrent, transaction-protected database access can enable all of the subsystems. + +In all these cases, the application uses the same function-call API to fetch and update records. + +### Design + +Berkeley DB was designed to provide industrial-strength database services to application developers, without requiring them to become database experts. It is a classic C-library style *toolkit*, providing a broad base of functionality to application writers. Berkeley DB was designed by programmers, for programmers: its modular design surfaces simple, orthogonal interfaces to core services, and it provides mechanism (for example, good thread support) without imposing policy (for example, the use of threads is not required). Just as importantly, Berkeley DB allows developers to balance performance against the need for crash recovery and concurrent use. An application can use the storage structure that provides the fastest access to its data and can request only the degree of logging and locking that it needs. + +Because of the tool-based approach and separate interfaces for each Berkeley DB subsystem, you can support a complete transaction environment for other system operations. Berkeley DB even allows you to wrap transactions around the standard UNIX file read and write operations! Further, Berkeley DB was designed to interact correctly with the native system's toolset, a feature no other database package offers. For example, on UNIX systems Berkeley DB supports hot backups (database backups while the database is in use), using standard UNIX system utilities, for example, dump, tar, cpio, pax or even cp. On other systems which do not support filesystems with read isolation, Berkeley DB provides a tool for safely copying files. + +Finally, because scripting language interfaces are available for Berkeley DB (notably Tcl and Perl), application writers can build incredibly powerful database engines with little effort. You can build transaction-protected database applications using your favorite scripting languages, an increasingly important feature in a world using CGI scripts to deliver HTML. diff --git a/docs-src/guides/programmer_reference/intro_dbisnot.md b/docs-src/guides/programmer_reference/intro_dbisnot.md new file mode 100644 index 000000000..a4b63f677 --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_dbisnot.md @@ -0,0 +1,65 @@ +--- +title: "What Berkeley DB is not" +api-name: "What Berkeley DB is not" +source: docs/programmer_reference/intro_dbisnot.html +--- +## What Berkeley DB is not + + [Berkeley DB is not a relational database](intro_dbisnot.md#idp50596256) + + [Berkeley DB is not an object-oriented database](intro_dbisnot.md#idp50675392) + + [Berkeley DB is not a network database](intro_dbisnot.md#idp50621304) + + [Berkeley DB is not a database server](intro_dbisnot.md#idp50657008) + +In contrast to most other database systems, Berkeley DB provides relatively simple data access services. + +Records in Berkeley DB are (*key*, *value*) pairs. Berkeley DB supports only a few logical operations on records. They are: + +- Insert a record in a table. +- Delete a record from a table. +- Find a record in a table by looking up its key. +- Update a record that has already been found. + +Notice that Berkeley DB never operates on the value part of a record. Values are simply payload, to be stored with keys and reliably delivered back to the application on demand. + +Both keys and values can be arbitrary byte strings, either fixed-length or variable-length. As a result, programmers can put native programming language data structures into the database without converting them to a foreign record format first. Storage and retrieval are very simple, but the application needs to know what the structure of a key and a value is in advance. It cannot ask Berkeley DB, because Berkeley DB doesn't know. + +This is an important feature of Berkeley DB, and one worth considering more carefully. On the one hand, Berkeley DB cannot provide the programmer with any information on the contents or structure of the values that it stores. The application must understand the keys and values that it uses. On the other hand, there is literally no limit to the data types that can be store in a Berkeley DB database. The application never needs to convert its own program data into the data types that Berkeley DB supports. Berkeley DB is able to operate on any data type the application uses, no matter how complex. + +Because both keys and values can be up to four gigabytes in length, a single record can store images, audio streams, or other large data values. Large values are not treated specially in Berkeley DB. They are simply broken into page-sized chunks, and reassembled on demand when the application needs them. Unlike some other database systems, Berkeley DB offers no special support for binary large objects (BLOBs). + +### Berkeley DB is not a relational database + +While Berkeley DB does provide a set of optional SQL APIs, usually all access to data stored in Berkeley DB is performed using the traditional Berkeley DB APIs. + +The traditional Berkeley DB APIs are the way that most Berkeley DB users will use Berkeley DB. Although the interfaces are fairly simple, they are non-standard in that they do not support SQL statements. + +That said, Berkeley DB does provide a set of SQL APIs that behave nearly identically to SQLite. By using these APIs you can interface with Berkeley DB using SQL statements. For Unix systems, these APIs are not available by default, while for Windows systems they are available by default. For more information, see the *Berkeley DB Getting Started with the SQL APIs* guide. + +Be aware that SQL support is a double-edged sword. One big advantage of relational databases is that they allow users to write simple declarative queries in a high-level language. The database system knows everything about the data and can carry out the command. This means that it's simple to search for data in new ways, and to ask new questions of the database. No programming is required. + +On the other hand, if a programmer can predict in advance how an application will access data, then writing a low-level program to get and store records can be faster. It eliminates the overhead of query parsing, optimization, and execution. The programmer must understand the data representation, and must write the code to do the work, but once that's done, the application can be very fast. + +Unless Berkeley DB is used with its SQL APIs, it has no notion of *schema* and data types in the way that relational systems do. Schema is the structure of records in tables, and the relationships among the tables in the database. For example, in a relational system the programmer can create a record from a fixed menu of data types. Because the record types are declared to the system, the relational engine can reach inside records and examine individual values in them. In addition, programmers can use SQL to declare relationships among tables, and to create indices on tables. Relational engines usually maintain these relationships and indices automatically. + +In Berkeley DB, the key and value in a record are opaque to Berkeley DB. They may have a rich internal structure, but the library is unaware of it. As a result, Berkeley DB cannot decompose the value part of a record into its constituent parts, and cannot use those parts to find values of interest. Only the application, which knows the data structure, can do that. Berkeley DB does support indices on tables and automatically maintain those indices as their associated tables are modified. + +Berkeley DB is not a relational system. Relational database systems are semantically rich and offer high-level database access. Compared to such systems, Berkeley DB is a high-performance, transactional library for record storage. It is possible to build a relational system on top of Berkeley DB (indeed, this is what the Berkeley DB SQL API really is). In fact, the popular MySQL relational system uses Berkeley DB for transaction-protected table management, and takes care of all the SQL parsing and execution. It uses Berkeley DB for the storage level, and provides the semantics and access tools. + +### Berkeley DB is not an object-oriented database + +Object-oriented databases are designed for very tight integration with object-oriented programming languages. Berkeley DB is written entirely in the C programming language. It includes language bindings for C++, Java, and other languages, but the library has no information about the objects created in any object-oriented application. Berkeley DB never makes method calls on any application object. It has no idea what methods are defined on user objects, and cannot see the public or private members of any instance. The key and value part of all records are opaque to Berkeley DB. + +Berkeley DB cannot automatically page in objects as they are accessed, as some object-oriented databases do. The object-oriented application programmer must decide what records are required, and must fetch them by making method calls on Berkeley DB objects. + +### Berkeley DB is not a network database + +Berkeley DB does not support network-style navigation among records, as network databases do. Records in a Berkeley DB table may move around over time, as new records are added to the table and old ones are deleted. Berkeley DB is able to do fast searches for records based on keys, but there is no way to create a persistent physical pointer to a record. Applications can only refer to records by key, not by address. + +### Berkeley DB is not a database server + +Berkeley DB is not a standalone database server. It is a library, and runs in the address space of the application that uses it. If more than one application links in Berkeley DB, then all can use the same database at the same time; the library handles coordination among the applications, and guarantees that they do not interfere with one another. + +It is possible to build a server application that uses Berkeley DB for data management. For example, many commercial and open source Lightweight Directory Access Protocol (LDAP) servers use Berkeley DB for record storage. LDAP clients connect to these servers over the network. Individual servers make calls through the Berkeley DB API to find records and return them to clients. On its own, however, Berkeley DB is not a server. diff --git a/docs-src/guides/programmer_reference/intro_distrib.md b/docs-src/guides/programmer_reference/intro_distrib.md new file mode 100644 index 000000000..fe30684f6 --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_distrib.md @@ -0,0 +1,8 @@ +--- +title: "What does the Berkeley DB distribution include?" +api-name: "What does the Berkeley DB distribution include?" +source: docs/programmer_reference/intro_distrib.html +--- +## What does the Berkeley DB distribution include? + +The Berkeley DB distribution includes complete source code for the Berkeley DB library, including all three Berkeley DB products and their supporting utilities, as well as complete documentation in HTML format. The distribution includes prebuilt binaries and libraries for a small number of platforms. The distribution does not include hard-copy documentation. diff --git a/docs-src/guides/programmer_reference/intro_need.md b/docs-src/guides/programmer_reference/intro_need.md new file mode 100644 index 000000000..bfef87c9e --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_need.md @@ -0,0 +1,18 @@ +--- +title: "Do you need Berkeley DB?" +api-name: "Do you need Berkeley DB?" +source: docs/programmer_reference/intro_need.html +--- +## Do you need Berkeley DB? + +Berkeley DB is an ideal database system for applications that need fast, scalable, and reliable embedded database management. For applications that need different services, however, it can be a poor choice. + +First, do you need the ability to access your data in ways you cannot predict in advance? If your users want to be able to enter SQL queries to perform complicated searches that you cannot program into your application to begin with, then you should consider a relational engine instead. Berkeley DB requires a programmer to write code in order to run a new kind of query. + +On the other hand, if you can predict your data access patterns up front — and in particular if you need fairly simple key/value lookups — then Berkeley DB is a good choice. The queries can be coded up once, and will then run very quickly because there is no SQL to parse and execute. + +Second, are there political arguments for or against a standalone relational server? If you're building an application for your own use and have a relational system installed with administrative support already, it may be simpler to use that than to build and learn Berkeley DB. On the other hand, if you'll be shipping many copies of your application to customers, and don't want your customers to have to buy, install, and manage a separate database system, then Berkeley DB may be a better choice. + +Third, are there any technical advantages to an embedded database? If you're building an application that will run unattended for long periods of time, or for end users who are not sophisticated administrators, then a separate server process may be too big a burden. It will require separate installation and management, and if it creates new ways for the application to fail, or new complexities to master in the field, then Berkeley DB may be a better choice. + +The fundamental question is, how closely do your requirements match the Berkeley DB design? Berkeley DB was conceived and built to provide fast, reliable, transaction-protected record storage. The library itself was never intended to provide interactive query support, graphical reporting tools, or similar services that some other database systems provide. We have tried always to err on the side of minimalism and simplicity. By keeping the library small and simple, we create fewer opportunities for bugs to creep in, and we guarantee that the database system stays fast, because there is very little code to execute. If your application needs that set of features, then Berkeley DB is almost certainly the best choice for you. diff --git a/docs-src/guides/programmer_reference/intro_products.md b/docs-src/guides/programmer_reference/intro_products.md new file mode 100644 index 000000000..a88fc21ef --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_products.md @@ -0,0 +1,55 @@ +--- +title: "The Berkeley DB products" +api-name: "The Berkeley DB products" +source: docs/programmer_reference/intro_products.html +--- +## The Berkeley DB products + + [Berkeley DB Data Store](intro_products.md#idp50715960) + + [Berkeley DB Concurrent Data Store](intro_products.md#idp50715552) + + [Berkeley DB Transactional Data Store](intro_products.md#idp50708368) + + [Berkeley DB High Availability](intro_products.md#idp50712672) + +Oracle provides four Berkeley DB products, each differing by the level of database support that they offer. + +- Berkeley DB Data Store +- Berkeley DB Concurrent Data Store +- Berkeley DB Transactional Data Store +- Berkeley DB High Availability + +Each product provides additional functionality to the product that precedes it in the list. As a result, you can download Berkeley DB and build an application that provides read-only database access for a single-user, and later add support for more complex database access patterns for multiple users. + +The single Open Source distribution of Berkeley DB from Oracle includes the four products and building the distribution automatically builds all four products. However, you must use the same Berkeley DB product throughout an application or group of applications. + +To redistribute Berkeley DB software, you must have a license for the Berkeley DB product you use. For further details, refer to the licensing informaion at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html + +A comparison of the four Berkeley DB product features is provided in the following table. + +|   | Berkeley DB Data Store | Berkeley DB Concurrent Data Store | Berkeley DB Transactional Data Store | Berkeley DB High Availability | +|:---|:---|:---|:---|:---| +| What is this product? | Provides indexed, single-reader/single-writer embedded data storage | Adds simple locking with multiple-reader/single-writer capabilities | Adds complete ACID transaction support, as well as recovery | Adds single-master data replication across multiple physical machines | +| Ensures recovery operation | No | No | Yes | Yes | +| Provides Locking feature | No | Yes | Yes | Yes | +| Provides concurrent read-write access | No | Yes | Yes | Yes | +| Provides transactional support | No | No | Yes | Yes | +| Supports SQL access | No | No | Yes | No | +| Provides replication support | No | No | No | Yes | + +### Berkeley DB Data Store + +The Berkeley DB Data Store product is an embeddable, high-performance data store. This product supports multiple concurrent threads of control, including multiple processes and multiple threads of control within a process. However, Berkeley DB Data Store does not support locking, and hence does not guarantee correct behavior if more than one thread of control is updating the database at a time. The Berkeley DB Data Store is intended for use in read-only applications or applications which can guarantee no more than one thread of control updates the database at a time. + +### Berkeley DB Concurrent Data Store + +The Berkeley DB Concurrent Data Store product adds multiple-reader, single writer capabilities to the Berkeley DB Data Store product. This product provides built-in concurrency and locking feature. Berkeley DB Concurrent Data Store is intended for applications that need support for concurrent updates to a database that is largely used for reading. + +### Berkeley DB Transactional Data Store + +The Berkeley DB Transactional Data Store product adds support for transactions and database recovery. Berkeley DB Transactional Data Store is intended for applications that require industrial-strength database services, including excellent performance under high-concurrency workloads of read and write operations, the ability to commit or roll back multiple changes to the database at a single instant, and the guarantee that in the event of a catastrophic system or hardware failure, all committed database changes are preserved. + +### Berkeley DB High Availability + +The Berkeley DB High Availability product adds support for data replication. A single master system handles all updates, and distributes these updates to multiple replicas. The number of replicas depends on the application requirements. All replicas can handle read requests during normal processing. If the master system fails for any reason, one of the replicas takes over as the new master system, and distributes updates to the remaining replicas. diff --git a/docs-src/guides/programmer_reference/intro_terrain.md b/docs-src/guides/programmer_reference/intro_terrain.md new file mode 100644 index 000000000..20b326a38 --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_terrain.md @@ -0,0 +1,95 @@ +--- +title: "Mapping the terrain: theory and practice" +api-name: "Mapping the terrain: theory and practice" +source: docs/programmer_reference/intro_terrain.html +--- +## Mapping the terrain: theory and practice + + [Data access and data management](intro_terrain.md#idp50584200) + + [Relational databases](intro_terrain.md#idp50577368) + + [Object-oriented databases](intro_terrain.md#idp50621160) + + [Network databases](intro_terrain.md#idp50574144) + + [Clients and servers](intro_terrain.md#idp50647776) + +The first step in selecting a database system is figuring out what the choices are. Decades of research and real-world deployment have produced countless systems. We need to organize them somehow to reduce the number of options. + +One obvious way to group systems is to use the common labels that vendors apply to them. The buzzwords here include "network," "relational," "object-oriented," and "embedded," with some cross-fertilization like "object-relational" and "embedded network". Understanding the buzzwords is important. Each has some grounding in theory, but has also evolved into a practical label for categorizing systems that work in a certain way. + +All database systems, regardless of the buzzwords that apply to them, provide a few common services. All of them store data, for example. We'll begin by exploring the common services that all systems provide, and then examine the differences among the different kinds of systems. + +### Data access and data management + +Fundamentally, database systems provide two services. + +The first service is *data access*. Data access means adding new data to the database (inserting), finding data of interest (searching), changing data already stored (updating), and removing data from the database (deleting). All databases provide these services. How they work varies from category to category, and depends on the record structure that the database supports. + +Each record in a database is a collection of values. For example, the record for a Web site customer might include a name, email address, shipping address, and payment information. Records are usually stored in tables. Each table holds records of the same kind. For example, the **customer** table at an e-commerce Web site might store the customer records for every person who shopped at the site. Often, database records have a different structure from the structures or instances supported by the programming language in which an application is written. As a result, working with records can mean: + +- using database operations like searches and updates on records; and +- converting between programming language structures and database record types in the application. + +The second service is *data management*. Data management is more complicated than data access. Providing good data management services is the hard part of building a database system. When you choose a database system to use in an application you build, making sure it supports the data management services you need is critical. + +Data management services include allowing multiple users to work on the database simultaneously (concurrency), allowing multiple records to be changed instantaneously (transactions), and surviving application and system crashes (recovery). Different database systems offer different data management services. Data management services are entirely independent of the data access services listed above. For example, nothing about relational database theory requires that the system support transactions, but most commercial relational systems do. + +Concurrency means that multiple users can operate on the database at the same time. Support for concurrency ranges from none (single-user access only) to complete (many readers and writers working simultaneously). + +Transactions permit users to make multiple changes appear at once. For example, a transfer of funds between bank accounts needs to be a transaction because the balance in one account is reduced and the balance in the other increases. If the reduction happened before the increase, than a poorly-timed system crash could leave the customer poorer; if the bank used the opposite order, then the same system crash could make the customer richer. Obviously, both the customer and the bank are best served if both operations happen at the same instant. + +Transactions have well-defined properties in database systems. They are *atomic*, so that the changes happen all at once or not at all. They are *consistent*, so that the database is in a legal state when the transaction begins and when it ends. They are typically *isolated*, which means that any other users in the database cannot interfere with them while they are in progress. And they are *durable*, so that if the system or application crashes after a transaction finishes, the changes are not lost. Together, the properties of *atomicity*, *consistency*, *isolation*, and *durability* are known as the ACID properties. + +As is the case for concurrency, support for transactions varies among databases. Some offer atomicity without making guarantees about durability. Some ignore isolatability, especially in single-user systems; there's no need to isolate other users from the effects of changes when there are no other users. + +Another important data management service is recovery. Strictly speaking, recovery is a procedure that the system carries out when it starts up. The purpose of recovery is to guarantee that the database is complete and usable. This is most important after a system or application crash, when the database may have been damaged. The recovery process guarantees that the internal structure of the database is good. Recovery usually means that any completed transactions are checked, and any lost changes are reapplied to the database. At the end of the recovery process, applications can use the database as if there had been no interruption in service. + +Finally, there are a number of data management services that permit copying of data. For example, most database systems are able to import data from other sources, and to export it for use elsewhere. Also, most systems provide some way to back up databases and to restore in the event of a system failure that damages the database. Many commercial systems allow *hot backups*, so that users can back up databases while they are in use. Many applications must run without interruption, and cannot be shut down for backups. + +A particular database system may provide other data management services. Some provide browsers that show database structure and contents. Some include tools that enforce data integrity rules, such as the rule that no employee can have a negative salary. These data management services are not common to all systems, however. Concurrency, recovery, and transactions are the data management services that most database vendors support. + +Deciding what kind of database to use means understanding the data access and data management services that your application needs. Berkeley DB is an embedded database that supports fairly simple data access with a rich set of data management services. To highlight its strengths and weaknesses, we can compare it to other database system categories. + +### Relational databases + +Relational databases are probably the best-known database variant, because of the success of companies like Oracle. Relational databases are based on the mathematical field of set theory. The term "relation" is really just a synonym for "set" -- a relation is just a set of records or, in our terminology, a table. One of the main innovations in early relational systems was to insulate the programmer from the physical organization of the database. Rather than walking through arrays of records or traversing pointers, programmers make statements about tables in a high-level language, and the system executes those statements. + +Relational databases operate on *tuples*, or records, composed of values of several different data types, including integers, character strings, and others. Operations include searching for records whose values satisfy some criteria, updating records, and so on. + +Virtually all relational databases use the Structured Query Language, or SQL. This language permits people and computer programs to work with the database by writing simple statements. The database engine reads those statements and determines how to satisfy them on the tables in the database. + +SQL is the main practical advantage of relational database systems. Rather than writing a computer program to find records of interest, the relational system user can just type a query in a simple syntax, and let the engine do the work. This gives users enormous flexibility; they do not need to decide in advance what kind of searches they want to do, and they do not need expensive programmers to find the data they need. Learning SQL requires some effort, but it's much simpler than a full-blown high-level programming language for most purposes. And there are a lot of programmers who have already learned SQL. + +### Object-oriented databases + +Object-oriented databases are less common than relational systems, but are still fairly widespread. Most object-oriented databases were originally conceived as persistent storage systems closely wedded to particular high-level programming languages like C++. With the spread of Java, most now support more than one programming language, but object-oriented database systems fundamentally provide the same class and method abstractions as do object-oriented programming languages. + +Many object-oriented systems allow applications to operate on objects uniformly, whether they are in memory or on disk. These systems create the illusion that all objects are in memory all the time. The advantage to object-oriented programmers who simply want object storage and retrieval is clear. They need never be aware of whether an object is in memory or not. The application simply uses objects, and the database system moves them between disk and memory transparently. All of the operations on an object, and all its behavior, are determined by the programming language. + +Object-oriented databases aren't nearly as widely deployed as relational systems. In order to attract developers who understand relational systems, many of the object-oriented systems have added support for query languages very much like SQL. In practice, though, object-oriented databases are mostly used for persistent storage of objects in C++ and Java programs. + +### Network databases + +The "network model" is a fairly old technique for managing and navigating application data. Network databases are designed to make pointer traversal very fast. Every record stored in a network database is allowed to contain pointers to other records. These pointers are generally physical addresses, so fetching the record to which it refers just means reading it from disk by its disk address. + +Network database systems generally permit records to contain integers, floating point numbers, and character strings, as well as references to other records. An application can search for records of interest. After retrieving a record, the application can fetch any record to which it refers, quickly. + +Pointer traversal is fast because most network systems use physical disk addresses as pointers. When the application wants to fetch a record, the database system uses the address to fetch exactly the right string of bytes from the disk. This requires only a single disk access in all cases. Other systems, by contrast, often must do more than one disk read to find a particular record. + +The key advantage of the network model is also its main drawback. The fact that pointer traversal is so fast means that applications that do it will run well. On the other hand, storing pointers all over the database makes it very hard to reorganize the database. In effect, once you store a pointer to a record, it is difficult to move that record elsewhere. Some network databases handle this by leaving forwarding pointers behind, but this defeats the speed advantage of doing a single disk access in the first place. Other network databases find, and fix, all the pointers to a record when it moves, but this makes reorganization very expensive. Reorganization is often necessary in databases, since adding and deleting records over time will consume space that cannot be reclaimed without reorganizing. Without periodic reorganization to compact network databases, they can end up with a considerable amount of wasted space. + +### Clients and servers + +Database vendors have two choices for system architecture. They can build a server to which remote clients connect, and do all the database management inside the server. Alternatively, they can provide a module that links directly into the application, and does all database management locally. In either case, the application developer needs some way of communicating with the database (generally, an Application Programming Interface (API) that does work in the process or that communicates with a server to get work done). + +Almost all commercial database products are implemented as servers, and applications connect to them as clients. Servers have several features that make them attractive. + +First, because all of the data is managed by a separate process, and possibly on a separate machine, it's easy to isolate the database server from bugs and crashes in the application. + +Second, because some database products (particularly relational engines) are quite large, splitting them off as separate server processes keeps applications small, which uses less disk space and memory. Relational engines include code to parse SQL statements, to analyze them and produce plans for execution, to optimize the plans, and to execute them. + +Finally, by storing all the data in one place and managing it with a single server, it's easier for organizations to back up, protect, and set policies on their databases. The enterprise databases for large companies often have several full-time administrators caring for them, making certain that applications run quickly, granting and denying access to users, and making backups. + +However, centralized administration can be a disadvantage in some cases. In particular, if a programmer wants to build an application that uses a database for storage of important information, then shipping and supporting the application is much harder. The end user needs to install and administer a separate database server, and the programmer must support not just one product, but two. Adding a server process to the application creates new opportunity for installation mistakes and run-time problems. diff --git a/docs-src/guides/programmer_reference/intro_what.md b/docs-src/guides/programmer_reference/intro_what.md new file mode 100644 index 000000000..5cd57b344 --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_what.md @@ -0,0 +1,21 @@ +--- +title: "What other services does Berkeley DB provide?" +api-name: "What other services does Berkeley DB provide?" +source: docs/programmer_reference/intro_what.html +--- +## What other services does Berkeley DB provide? + +Berkeley DB also provides core database services to developers. These services include: + +Page cache management: +The page cache provides fast access to a cache of database pages, handling the I/O associated with the cache to ensure that dirty pages are written back to the file system and that new pages are allocated on demand. Applications may use the Berkeley DB shared memory buffer manager to serve their own files and pages. + +Transactions and logging: +The transaction and logging systems provide recoverability and atomicity for multiple database operations. The transaction system uses two-phase locking and write-ahead logging protocols to ensure that database operations may be undone or redone in the case of application or system failure. Applications may use Berkeley DB transaction and logging subsystems to protect their own data structures and operations from application or system failure. + +Locking: +The locking system provides multiple reader or single writer access to objects. The Berkeley DB access methods use the locking system to acquire the right to read or write database pages. Applications may use the Berkeley DB locking subsystem to support their own locking needs. + +By combining the page cache, transaction, locking, and logging systems, Berkeley DB provides the same services found in much larger, more complex and more expensive database systems. Berkeley DB supports multiple simultaneous readers and writers and guarantees that all changes are recoverable, even in the case of a catastrophic hardware failure during a database update. + +Developers may select some or all of the core database services for any access method or database. Therefore, it is possible to choose the appropriate storage structure and the right degrees of concurrency and recoverability for any application. In addition, some of the subsystems (for example, the Locking subsystem) can be called separately from the Berkeley DB access method. As a result, developers can integrate non-database objects into their transactional applications using Berkeley DB. diff --git a/docs-src/guides/programmer_reference/intro_where.md b/docs-src/guides/programmer_reference/intro_where.md new file mode 100644 index 000000000..b5baf9f50 --- /dev/null +++ b/docs-src/guides/programmer_reference/intro_where.md @@ -0,0 +1,16 @@ +--- +title: "Where does Berkeley DB run?" +api-name: "Where does Berkeley DB run?" +source: docs/programmer_reference/intro_where.html +--- +## Where does Berkeley DB run? + +Berkeley DB requires only underlying IEEE/ANSI Std 1003.1 (POSIX) system calls and can be ported easily to new architectures by adding stub routines to connect the native system interfaces to the Berkeley DB POSIX-style system calls. See the Berkeley DB Porting Guide for more information. + +Berkeley DB will autoconfigure and run on almost any modern UNIX, POSIX or Linux systems, and on most historical UNIX platforms. Berkeley DB will autoconfigure and run on almost any GNU gcc toolchain-based embedded platform, including Cygwin, OpenLinux and others. See the Berkeley DB Installation and Build Guide for more information. + +The Berkeley DB distribution includes support for QNX Neutrino. See the Berkeley DB Installation and Build Guide for more information. + +The Berkeley DB distribution includes support for VxWorks. See the Berkeley DB Installation and Build Guide for more information. + +The Berkeley DB distribution includes support for Windows/NT, Windows/2000 and Windows/XP, via the Microsoft Visual C++ 6.0 and .NET development environments. See the Berkeley DB Installation and Build Guide for more information. diff --git a/docs-src/guides/programmer_reference/java.md b/docs-src/guides/programmer_reference/java.md new file mode 100644 index 000000000..00cb999b6 --- /dev/null +++ b/docs-src/guides/programmer_reference/java.md @@ -0,0 +1,86 @@ +--- +title: "Chapter 5.  Java API" +api-name: "Chapter 5.  Java API" +source: docs/programmer_reference/java.html +--- +## Chapter 5.  Java API + +**Table of Contents** + + [Java configuration](java.md#java_conf) + + [Compatibility](java_compat.md) + + [Java programming notes](java_program.md) + + [Java FAQ](java_faq.md) + +## Java configuration + +Building the Berkeley DB java classes, the examples and the native support library is integrated into the normal build process. See Configuring Berkeley DB and Building the Java API in the Berkeley DB Installation and Build Guide for more information. + +We expect that you already installed the Java JDK or equivalent on your system. For the sake of discussion, we assume that it is in a directory called db-VERSION; for example, you downloaded a Berkeley DB archive, and you did not change the top-level directory name. The files related to Java are in three subdirectories of db-VERSION: java (the java source files), libdb_java (the C++ files that provide the "glue" between java and Berkeley DB) and examples_java (containing all examples code). The directory tree looks like this: + +``` c +db-VERSION +|-- java +| `-- src +| `-- com +| `-- sleepycat +| |-- bind +| |-- db +| | `-- ... +| `-- util +|-- examples_java +| `-- src +| `-- db +| `-- ... +`-- libdb_java + `-- ... +``` + +This naming conforms to the de facto standard for naming java packages. When the java code is built, it is placed into two jar files: `db.jar`, containing the db package, and `dbexamples.jar`, containing the examples. + +For your application to use Berkeley DB successfully, you must set your `CLASSPATH` environment variable to include the full pathname of the db jar files as well as the classes in your java distribution. On UNIX, `CLASSPATH` is a colon-separated list of directories and jar files; on Windows, it is separated by semicolons. On UNIX, the jar files are put in your build directory, and when you do the make install step, they are copied to the lib directory of your installation tree. On Windows, the jar files are placed in the Release or Debug subdirectory with your other objects. + +The Berkeley DB Java classes are mostly implemented in native methods. Before you can use them, you need to make sure that the DLL or shared library containing the native methods can be found by your Java runtime. On Windows, you should set your PATH variable to include: + +``` c + db-VERSION\build_windows\Release + +``` + +On UNIX, you should set the `LD_LIBRARY_PATH` environment variable or local equivalent to include the Berkeley DB library installation directory. Of course, the standard install directory may have been changed for your site; see your system administrator for details. + +On other platforms, the path can be set on the command line as follows (assuming the shared library is in `/usr/local/BerkeleyDB/lib`:) + +``` c +% java -Djava.library.path=/usr/local/BerkeleyDB/lib ... +``` + +Regardless, if you get the following exception when you run, you probably do not have the library search path configured correctly: + +``` c +java.lang.UnsatisfiedLinkError +``` + +Different Java interpreters provide different error messages if the `CLASSPATH` value is incorrect, a typical error is the following: + +``` c +java.lang.NoClassDefFoundError +``` + +To ensure that everything is running correctly, you may want to try a simple test from the example programs in + +``` c + db-VERSION/examples_java/src/db + +``` + +For example, the following sample program will prompt for text input lines, which are then stored in a Btree database named `access.db` in your current directory: + +``` c +% java db.AccessExample +``` + +Try giving it a few lines of input text and then end-of-file. Before it exits, you should see a list of the lines you entered display with data items. This is a simple check to make sure the fundamental configuration is working correctly. diff --git a/docs-src/guides/programmer_reference/java_compat.md b/docs-src/guides/programmer_reference/java_compat.md new file mode 100644 index 000000000..d702fe612 --- /dev/null +++ b/docs-src/guides/programmer_reference/java_compat.md @@ -0,0 +1,8 @@ +--- +title: "Compatibility" +api-name: "Compatibility" +source: docs/programmer_reference/java_compat.html +--- +## Compatibility + +The Berkeley DB Java API has been tested with the Sun Microsystem's JDK 1.5 (Java 5) on Linux, Windows and OS X. It should work with any JDK 1.5- compatible environment. diff --git a/docs-src/guides/programmer_reference/java_faq.md b/docs-src/guides/programmer_reference/java_faq.md new file mode 100644 index 000000000..28c7ab344 --- /dev/null +++ b/docs-src/guides/programmer_reference/java_faq.md @@ -0,0 +1,59 @@ +--- +title: "Java FAQ" +api-name: "Java FAQ" +source: docs/programmer_reference/java_faq.html +--- +## Java FAQ + +1. **On what platforms is the Berkeley DB Java API supported?** + + All platforms supported by Berkeley DB that have a JVM compatible with J2SE 1.4 or above. + +2. **How does the Berkeley DB Java API relate to the J2EE standard?** + + The Berkeley DB Java API does not currently implement any part of the J2EE standard. That said, it does implement the implicit standard for Java Java Collections. The concept of a transaction exists in several Java packages (J2EE, XA, JINI to name a few). Support for these APIs will be added based on demand in future versions of Berkeley DB. + +3. **How should I incorporate db.jar and the db native library into a Tomcat or other J2EE application servers?** + + Tomcat and other J2EE application servers have the ability to rebuild and reload code automatically. When using Tomcat this is the case when "reloadable" is set to "true". If your WAR file includes the db.jar it too will be reloaded each time your code is reloaded. This causes exceptions as the native library can't be loaded more than once and there is no way to unload native code. The solution is to place the db.jar in \$TOMCAT_HOME/common/lib and let Tomcat load that library once at start time rather than putting it into the WAR that gets reloaded over and over. + +4. **Can I use the Berkeley DB Java API from within a EJB, a Servlet or a JSP page?** + + Yes. The Berkeley DB Java API can be used from within all the popular J2EE application servers in many different ways. + +5. **During one of the first calls to the Berkeley DB Java API, a DbException is thrown with a "Bad file number" or "Bad file descriptor" message.** + + There are known large-file support bugs under JNI in various releases of the JDK. Please upgrade to the latest release of the JDK, and, if that does not solve the problem, disable big file support using the --disable-largefile configuration option. + +6. **How can I use native methods from a debug build of the Java library?** + + Set Java's library path so that the debug version of Berkeley DB's Java library appears, but the release version does not. Berkeley DB tries to load the release library first, and if that fails tries the debug library. + +7. **Why is ClassNotFoundException thrown when adding a record to the database, when a SerialBinding is used?** + + This problem occurs if you copy the db.jar file into the Java extensions (ext) directory. This will cause the database code to run under the System class loader, and it won't be able to find your application classes. + + You'll have to actually remove db.jar from the Java extension directory. If you have more than one installation of Java, be sure to remove it from all of them. This is necessary even if db.jar is specified in the classpath. + + An example of the exception is: + + ``` c + collections.ship.basic.SupplierKey + at java.net.URLClassLoader$1.run(Unknown Source) + at java.security.AccessController.doPrivileged(Native Method) + at java.net.URLClassLoader.findClass(Unknown Source) + at java.lang.ClassLoader.loadClass(Unknown Source) + at java.lang.ClassLoader.loadClass(Unknown Source) + at java.lang.ClassLoader.loadClassInternal(Unknown Source) + at java.lang.Class.forName0(Native Method) + at java.lang.Class.forName(Unknown Source) + at com.sleepycat.bind.serial.StoredClassCatalog. + getClassInfo(StoredClassCatalog.java:211) + ... + ``` + +8. **I'm upgrading my Java application to Berkeley DB 4.3. Can I use the com.sleepycat.db.internal package rather than porting my code to the new API?** + + While it is possible to use the low-level API from applications, there are some caveats that should be considered when upgrading. The first is that the internal API depends on some classes in the public API such as DatabaseEntry. + + In addition, the internal API is closer to the C API and doesn't have some of the default settings that were part of the earlier Java API. For example, applications will need to set the DB_THREAD flag explicitly if handles are to be used from multiple threads, or subtle errors may occur. diff --git a/docs-src/guides/programmer_reference/java_program.md b/docs-src/guides/programmer_reference/java_program.md new file mode 100644 index 000000000..ee099d8b9 --- /dev/null +++ b/docs-src/guides/programmer_reference/java_program.md @@ -0,0 +1,26 @@ +--- +title: "Java programming notes" +api-name: "Java programming notes" +source: docs/programmer_reference/java_program.html +--- +## Java programming notes + +Although the Java API parallels the Berkeley DB C++/C interface in many ways, it differs where the Java language requires. For example, the handle method names are camel-cased and conform to Java naming patterns. (The C++/C method names are currently provided, but are deprecated.) + +1. The Java runtime does not automatically close Berkeley DB objects on finalization. There are several reasons for this. One is that finalization is generally run only when garbage collection occurs, and there is no guarantee that this occurs at all, even on exit. Allowing specific Berkeley DB actions to occur in ways that cannot be replicated seems wrong. Second, finalization of objects may happen in an arbitrary order, so we would have to do extra bookkeeping to make sure that everything was closed in the proper order. The best word of advice is to always do a close() for any matching open() call. Specifically, the Berkeley DB package requires that you explicitly call close on each individual Database and Cursor object that you opened. Your database activity may not be synchronized to disk unless you do so. + +2. Some methods in the Java API have no return type, and throw a DatabaseException when an severe error arises. There are some notable methods that do have a return value, and can also throw an exception. The "get" methods in Database and Cursor both return 0 when a get succeeds, DB_NOTFOUND when the key is not found, and throw an error when there is a severe error. This approach allows the programmer to check for typical data-driven errors by watching return values without special casing exceptions. + + An object of type MemoryException is thrown when a Dbt is too small to hold the corresponding key or data item. + + An object of type DeadlockException is thrown when a deadlock would occur. + + An object of type RunRecoveryException, a subclass of DatabaseException, is thrown when there is an error that requires a recovery of the database using db_recover utility. + + An object of type IllegalArgumentException a standard Java Language exception, is thrown when there is an error in method arguments. + + An object of type OutOfMemoryError is thrown when the system cannot provide enough memory to complete the operation (the ENOMEM system error on UNIX). + +3. If there are embedded nulls in the **curslist** argument for Database.join(com.sleepycat.db.Cursor[], com.sleepycat.db.JoinConfig), they will be treated as the end of the list of cursors, even if you may have allocated a longer array. Fill in all the cursors in your array unless you intend to cut it short. + +4. If you are using custom class loaders in your application, make sure that the Berkeley DB classes are loaded by the system class loader, not a custom class loader. This is due to a JVM bug that can cause an access violation during finalization (see the bug 4238486 in Sun Microsystem's Java Bug Database). diff --git a/docs-src/guides/programmer_reference/lock.md b/docs-src/guides/programmer_reference/lock.md new file mode 100644 index 000000000..e455c0313 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock.md @@ -0,0 +1,58 @@ +--- +title: "Chapter 16.  The Locking Subsystem" +api-name: "Chapter 16.  The Locking Subsystem" +source: docs/programmer_reference/lock.html +--- +## Chapter 16.  The Locking Subsystem + +**Table of Contents** + + [Introduction to the locking subsystem](lock.md#lock_intro) + + [Configuring locking](lock_config.md) + + [Configuring locking: sizing the system](lock_max.md) + + [Standard lock modes](lock_stdmode.md) + + [Deadlock detection](lock_dead.md) + + [Deadlock detection using timers](lock_timeout.md) + + [Deadlock debugging](lock_deaddbg.md) + + [Locking granularity](lock_page.md) + + [Locking without transactions](lock_notxn.md) + + [Locking with transactions: two-phase locking](lock_twopl.md) + + [Berkeley DB Concurrent Data Store locking conventions](lock_cam_conv.md) + + [Berkeley DB Transactional Data Store locking conventions](lock_am_conv.md) + + [Locking and non-Berkeley DB applications](lock_nondb.md) + +## Introduction to the locking subsystem + +The locking subsystem provides interprocess and intraprocess concurrency control mechanisms. Although the lock system is used extensively by the Berkeley DB access methods and transaction system, it may also be used as a standalone subsystem to provide concurrency control to any set of designated resources. + +The Lock subsystem is created, initialized, and opened by calls to DB_ENV->open() with the DB_INIT_LOCK or DB_INIT_CDB flags specified. + +The DB_ENV->lock_vec() method is used to acquire and release locks. The DB_ENV->lock_vec() method performs any number of lock operations atomically. It also provides the capability to release all locks held by a particular locker and release all the locks on a particular object. (Performing multiple lock operations atomically is useful in performing Btree traversals -- you want to acquire a lock on a child page and once acquired, immediately release the lock on its parent. This is traditionally referred to as *lock-coupling*). Two additional methods, DB_ENV->lock_get() and DB_ENV->lock_put(), are provided. These methods are simpler front-ends to the DB_ENV->lock_vec() functionality, where DB_ENV->lock_get() acquires a lock, and DB_ENV->lock_put() releases a lock that was acquired using DB_ENV->lock_get() or DB_ENV->lock_vec(). All locks explicitly requested by an application should be released via calls to DB_ENV->lock_put() or DB_ENV->lock_vec(). Using DB_ENV->lock_vec() instead of separate calls to DB_ENV->lock_put() and DB_ENV->lock_get() also reduces the synchronization overhead between multiple threads or processes. The three methods are fully compatible, and may be used interchangeably. + +Applications must specify lockers and lock objects appropriately. When used with the Berkeley DB access methods, lockers and objects are handled completely internally, but an application using the lock manager directly must either use the same conventions as the access methods or define its own convention to which it adheres. If an application is using the access methods with locking at the same time that it is calling the lock manager directly, the application must follow a convention that is compatible with the access methods' use of the locking subsystem. See Berkeley DB Transactional Data Store locking conventions for more information. + +The DB_ENV->lock_id() function returns a unique ID that may safely be used as the locker parameter to the DB_ENV->lock_vec() method. The access methods use DB_ENV->lock_id() to generate unique lockers for the cursors associated with a database. + +The DB_ENV->lock_detect() function provides the programmatic interface to the Berkeley DB deadlock detector. Whenever two threads of control issue lock requests concurrently, the possibility for deadlock arises. A deadlock occurs when two or more threads of control are blocked, waiting for actions that another one of the blocked threads must take. For example, assume that threads A and B have each obtained read locks on object X. Now suppose that both threads want to obtain write locks on object X. Neither thread can be granted its write lock (because of the other thread's read lock). Both threads block and will never unblock because the event for which they are waiting can never happen. + +The deadlock detector examines all the locks held in the environment, and identifies situations where no thread can make forward progress. It then selects one of the participants in the deadlock (according to the argument that was specified to DB_ENV->set_lk_detect()), and forces it to return the value DB_LOCK_DEADLOCK, which indicates that a deadlock occurred. The thread receiving such an error must release all of its locks and undo any incomplete modifications to the locked resource. Locks are typically released, and modifications undone, by closing any cursors involved in the operation and aborting any transaction enclosing the operation. The operation may optionally be retried. + +The DB_ENV->lock_stat() function returns information about the status of the lock subsystem. It is the programmatic interface used by the db_stat utility. + +The locking subsystem is closed by the call to DB_ENV->close(). + +Finally, the entire locking subsystem may be discarded using the DB_ENV->remove() method. + +For more information on the locking subsystem methods, see the Locking Subsystem and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/lock_am_conv.md b/docs-src/guides/programmer_reference/lock_am_conv.md new file mode 100644 index 000000000..b22633c40 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_am_conv.md @@ -0,0 +1,30 @@ +--- +title: "Berkeley DB Transactional Data Store locking conventions" +api-name: "Berkeley DB Transactional Data Store locking conventions" +source: docs/programmer_reference/lock_am_conv.html +--- +## Berkeley DB Transactional Data Store locking conventions + +All Berkeley DB access methods follow the same conventions for locking database objects. Applications that do their own locking and also do locking via the access methods must be careful to adhere to these conventions. + +Whenever a Berkeley DB database is opened, the DB handle is assigned a unique locker ID. Unless transactions are specified, that ID is used as the locker for all calls that the Berkeley DB methods make to the lock subsystem. In order to lock a file, pages in the file, or records in the file, we must create a unique ID that can be used as the object to be locked in calls to the lock manager. Under normal operation, that object is a 28-byte value created by the concatenation of a unique file identifier, a page or record number, and an object type (page or record). + +In a transaction-protected environment, database create and delete operations are recoverable and single-threaded. This single-threading is achieved using a single lock for the entire environment that must be acquired before beginning a create or delete operation. In this case, the object on which Berkeley DB will lock is a 4-byte unsigned integer with a value of 0. + +If applications are using the lock subsystem directly while they are also using locking via the access methods, they must take care not to inadvertently lock objects that happen to be equal to the unique file IDs used to lock files. This is most easily accomplished by using a lock object with a length different from the values used by Berkeley DB. + +All the access methods other than Queue use standard read/write locks in a simple multiple-reader/single writer page-locking scheme. An operation that returns data (for example, DB->get() or DBC->get()) obtains a read lock on all the pages accessed while locating the requested record. When an update operation is requested (for example, DB->put() or DBC->del()), the page containing the updated (or new) data is write-locked. As read-modify-write cycles are quite common and are deadlock-prone under normal circumstances, the Berkeley DB interfaces allow the application to specify the DB_RMW flag, which causes operations to immediately obtain a write lock, even though they are only reading the data. Although this may reduce concurrency somewhat, it reduces the probability of deadlock. In the presence of transactions, page locks are held until transaction commit. + +The Queue access method does not hold long-term page locks. Instead, page locks are held only long enough to locate records or to change metadata on a page, and record locks are held for the appropriate duration. In the presence of transactions, record locks are held until transaction commit. For DB operations, record locks are held until operation completion; for DBC operations, record locks are held until subsequent records are returned or the cursor is closed. + +Under non-transaction operations, the access methods do not normally hold locks across calls to the Berkeley DB interfaces. The one exception to this rule is when cursors are used. Because cursors maintain a position in a file, they must hold locks across calls; in fact, they will hold a lock until the cursor is closed. + +In this mode, the assignment of locker IDs to DB and cursor handles is complicated. If the DB_THREAD option was specified when the DB handle was opened, each use of DB has its own unique locker ID, and each cursor is assigned its own unique locker ID when it is created, so DB handle and cursor operations can all conflict with one another. (This is because when Berkeley DB handles may be shared by multiple threads of control the Berkeley DB library cannot identify which operations are performed by which threads of control, and it must ensure that two different threads of control are not simultaneously modifying the same data structure. By assigning each DB handle and cursor its own locker, two threads of control sharing a handle cannot inadvertently interfere with each other.) + +This has important implications. If a single thread of control opens two cursors, uses a combination of cursor and non-cursor operations, or begins two separate transactions, the operations are performed on behalf of different lockers. Conflicts that arise between these different lockers may not cause actual deadlocks, but can, in fact, permanently block the thread of control. For example, assume that an application creates a cursor and uses it to read record A. Now, assume a second cursor is opened, and the application attempts to write record A using the second cursor. Unfortunately, the first cursor has a read lock, so the second cursor cannot obtain its write lock. However, that read lock is held by the same thread of control, so the read lock can never be released if we block waiting for the write lock. This might appear to be a deadlock from the application's perspective, but Berkeley DB cannot identify it as such because it has no knowledge of which lockers belong to which threads of control. For this reason, application designers are encouraged to close cursors as soon as they are done with them. + +If the DB_THREAD option was not specified when the DB handle was opened, all uses of the DB handle and all cursors created using that handle will use the same locker ID for all operations. In this case, if a single thread of control opens two cursors or uses a combination of cursor and non-cursor operations, these operations are performed on behalf of the same locker, and so cannot deadlock or block the thread of control. + +Complicated operations that require multiple cursors (or combinations of cursor and non-cursor operations) can be performed in two ways. First, they may be performed within a transaction, in which case all operations lock on behalf of the designated transaction. Second, they may be performed using a local DB handle, although, as DB->open() operations are relatively slow, this may not be a good idea. Finally, the DBC->dup() function duplicates a cursor, using the same locker ID as the originating cursor. There is no way to achieve this duplication functionality through the DB handle calls, but any DB call can be implemented by one or more calls through a cursor. + +When the access methods use transactions, many of these problems disappear. The transaction ID is used as the locker ID for all operations performed on behalf of the transaction. This means that the application may open multiple cursors on behalf of the same transaction and these cursors will all share a common locker ID. This is safe because transactions cannot span threads of control, so the library knows that two cursors in the same transaction cannot modify the database concurrently. diff --git a/docs-src/guides/programmer_reference/lock_cam_conv.md b/docs-src/guides/programmer_reference/lock_cam_conv.md new file mode 100644 index 000000000..ce3f714c7 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_cam_conv.md @@ -0,0 +1,29 @@ +--- +title: "Berkeley DB Concurrent Data Store locking conventions" +api-name: "Berkeley DB Concurrent Data Store locking conventions" +source: docs/programmer_reference/lock_cam_conv.html +--- +## Berkeley DB Concurrent Data Store locking conventions + +The Berkeley DB Concurrent Data Store product has a simple set of conventions for locking. It provides multiple-reader/single-writer semantics, but not per-page locking or transaction recoverability. As such, it does its locking entirely in the Berkeley DB interface layer. + +The object it locks is the file, identified by its unique file number. The locking matrix is not one of the two standard lock modes, instead, we use a four-lock set, consisting of the following: + +DB_LOCK_NG +not granted (always 0) + +DB_LOCK_READ +read (shared) + +DB_LOCK_WRITE +write (exclusive) + +DB_LOCK_IWRITE +intention-to-write (shared with NG and READ, but conflicts with WRITE and IWRITE) + +The IWRITE lock is used for cursors that will be used for updating (IWRITE locks are implicitly obtained for write operations through the Berkeley DB handles, for example, DB->put() or DB->del()). While the cursor is reading, the IWRITE lock is held; but as soon as the cursor is about to modify the database, the IWRITE is upgraded to a WRITE lock. This upgrade blocks until all readers have exited the database. Because only one IWRITE lock is allowed at any one time, no two cursors can ever try to upgrade to a WRITE lock at the same time, and therefore deadlocks are prevented, which is essential because Berkeley DB Concurrent Data Store does not include deadlock detection and recovery. + +Applications that need to lock compatibly with Berkeley DB Concurrent Data Store must obey the following rules: + +1. Use only lock modes DB_LOCK_NG, DB_LOCK_READ, DB_LOCK_WRITE, DB_LOCK_IWRITE. +2. Never attempt to acquire a WRITE lock on an object that is already locked with a READ lock. diff --git a/docs-src/guides/programmer_reference/lock_config.md b/docs-src/guides/programmer_reference/lock_config.md new file mode 100644 index 000000000..20dcc10eb --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_config.md @@ -0,0 +1,16 @@ +--- +title: "Configuring locking" +api-name: "Configuring locking" +source: docs/programmer_reference/lock_config.html +--- +## Configuring locking + +The DB_ENV->set_lk_detect() method specifies that the deadlock detector should be run whenever a lock is about to block. This option provides for rapid detection of deadlocks at the expense of potentially frequent invocations of the deadlock detector. On a fast processor with a highly contentious application where response time is critical, this is a good choice. An option argument to the DB_ENV->set_lk_detect() method indicates which lock requests should be rejected. + +The application can limit how long it blocks on a contested resource. The DB_ENV->set_timeout() method specifies the length of the timeout. This value is checked whenever deadlock detection is performed, so the accuracy of the timeout depends upon the frequency of deadlock detection. + +In general, when applications are not specifying lock and transaction timeout values, the DB_LOCK_DEFAULT option is probably the correct first choice, and other options should only be selected based on evidence that they improve transaction throughput. If an application has long-running transactions, DB_LOCK_YOUNGEST will guarantee that transactions eventually complete, but it may do so at the expense of a large number of lock request rejections (and therefore, transaction aborts). + +The alternative to using the DB_ENV->set_lk_detect() method is to explicitly perform deadlock detection using the Berkeley DB DB_ENV->lock_detect() method. + +The DB_ENV->set_lk_conflicts() method allows you to specify your own locking conflicts matrix. This is an advanced configuration option, and is almost never necessary. diff --git a/docs-src/guides/programmer_reference/lock_dead.md b/docs-src/guides/programmer_reference/lock_dead.md new file mode 100644 index 000000000..d8da1545f --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_dead.md @@ -0,0 +1,16 @@ +--- +title: "Deadlock detection" +api-name: "Deadlock detection" +source: docs/programmer_reference/lock_dead.html +--- +## Deadlock detection + +Practically any application that uses locking may deadlock. The exceptions to this rule are when all the threads of control accessing the database are read-only or when the Berkeley DB Concurrent Data Store product is used; the Berkeley DB Concurrent Data Store product guarantees deadlock-free operation at the expense of reduced concurrency. While there are data access patterns that are deadlock free (for example, an application doing nothing but overwriting fixed-length records in an already existing database), they are extremely rare. + +When a deadlock exists in the system, all the threads of control involved in the deadlock are, by definition, waiting on a lock. The deadlock detector examines the state of the lock manager and identifies a deadlock, and selects one of the lock requests to reject. (See Configuring locking for a discussion of how a participant is selected). The DB_ENV->lock_get() or DB_ENV->lock_vec() call for which the selected participant is waiting then returns a DB_LOCK_DEADLOCK error. When using the Berkeley DB access methods, this error return is propagated back through the Berkeley DB database handle method to the calling application. + +The deadlock detector identifies deadlocks by looking for a cycle in what is commonly referred to as its "waits-for" graph. More precisely, the deadlock detector reads through the lock table, and reviews each lock object currently locked. Each object has lockers that currently hold locks on the object and possibly a list of lockers waiting for a lock on the object. Each object's list of waiting lockers defines a partial ordering. That is, for a particular object, every waiting locker comes after every holding locker because that holding locker must release its lock before the waiting locker can make forward progress. Conceptually, after each object has been examined, the partial orderings are topologically sorted. If this topological sort reveals any cycles, the lockers forming the cycle are involved in a deadlock. One of the lockers is selected for rejection. + +It is possible that rejecting a single lock request involved in a deadlock is not enough to allow other lockers to make forward progress. Unfortunately, at the time a lock request is selected for rejection, there is not enough information available to determine whether rejecting that single lock request will allow forward progress or not. Because most applications have few deadlocks, Berkeley DB takes the conservative approach, rejecting as few requests as may be necessary to resolve the existing deadlocks. In particular, for each unique cycle found in the waits-for graph described in the previous paragraph, only one lock request is selected for rejection. However, if there are multiple cycles, one lock request from each cycle is selected for rejection. Only after the enclosing transactions have received the lock request rejection return and aborted their transactions can it be determined whether it is necessary to reject additional lock requests in order to allow forward progress. + +The db_deadlock utility performs deadlock detection by calling the underlying Berkeley DB DB_ENV->lock_detect() method at regular intervals (DB_ENV->lock_detect() runs a single iteration of the Berkeley DB deadlock detector). Alternatively, applications can create their own deadlock utility or thread by calling the DB_ENV->lock_detect() method directly, or by using the DB_ENV->set_lk_detect() method to configure Berkeley DB to automatically run the deadlock detector whenever there is a conflict over a lock. The tradeoffs between using the DB_ENV->lock_detect() and DB_ENV->set_lk_detect() methods is that automatic deadlock detection will resolve deadlocks more quickly (because the deadlock detector runs as soon as the lock request blocks), however, automatic deadlock detection often runs the deadlock detector when there is no need for it, and for applications with large numbers of locks and/or where many operations block temporarily on locks but are soon able to proceed, automatic detection can decrease performance. diff --git a/docs-src/guides/programmer_reference/lock_deaddbg.md b/docs-src/guides/programmer_reference/lock_deaddbg.md new file mode 100644 index 000000000..a0b25b357 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_deaddbg.md @@ -0,0 +1,102 @@ +--- +title: "Deadlock debugging" +api-name: "Deadlock debugging" +source: docs/programmer_reference/lock_deaddbg.html +--- +## Deadlock debugging + +An occasional debugging problem in Berkeley DB applications is unresolvable deadlock. The output of the **-Co** flags of the db_stat utility can be used to detect and debug these problems. The following is a typical example of the output of this utility: + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- + 1 READ 1 HELD a.db handle 0 +80000004 WRITE 1 HELD a.db page 3 +``` + +In this example, we have opened a database and stored a single key/data pair in it. Because we have a database handle open, we have a read lock on that database handle. The database handle lock is the read lock labeled *handle*. (We can normally ignore handle locks for the purposes of database debugging, as they will only conflict with other handle operations, for example, an attempt to remove the database will block because we are holding the handle locked, but reading and writing the database will not conflict with the handle lock.) + +It is important to note that locker IDs are 32-bit unsigned integers, and are divided into two name spaces. Locker IDs with the high bit set (that is, values 80000000 or higher), are locker IDs associated with transactions. Locker IDs without the high bit set are locker IDs that are not associated with a transaction. Locker IDs associated with transactions map one-to-one with the transaction, that is, a transaction never has more than a single locker ID, and all of the locks acquired by the transaction will be acquired on behalf of the same locker ID. + +We also hold a write lock on the database page where we stored the new key/data pair. The page lock is labeled *page* and is on page number 3. If we were to put an additional key/data pair in the database, we would see the following output: + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- +80000004 WRITE 2 HELD a.db page 3 + 1 READ 1 HELD a.db handle 0 +``` + +That is, we have acquired a second reference count to page number 3, but have not acquired any new locks. If we add an entry to a different page in the database, we would acquire additional locks: + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- + 1 READ 1 HELD a.db handle 0 +80000004 WRITE 2 HELD a.db page 3 +80000004 WRITE 1 HELD a.db page 2 +``` + +Here's a simple example of one lock blocking another one: + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- +80000004 WRITE 1 HELD a.db page 2 +80000005 WRITE 1 WAIT a.db page 2 + 1 READ 1 HELD a.db handle 0 +80000004 READ 1 HELD a.db page 1 +``` + +In this example, there are two different transactional lockers (80000004 and 80000005). Locker 80000004 is holding a write lock on page 2, and locker 80000005 is waiting for a write lock on page 2. This is not a deadlock, because locker 80000004 is not blocked on anything. Presumably, the thread of control using locker 80000004 will proceed, eventually release its write lock on page 2, at which point the thread of control using locker 80000005 can also proceed, acquiring a write lock on page 2. + +If lockers 80000004 and 80000005 are not in different threads of control, the result would be *self deadlock*. Self deadlock is not a true deadlock, and won't be detected by the Berkeley DB deadlock detector. It's not a true deadlock because, if work could continue to be done on behalf of locker 80000004, then the lock would eventually be released, and locker 80000005 could acquire the lock and itself proceed. So, the key element is that the thread of control holding the lock cannot proceed because it is the same thread as is blocked waiting on the lock. + +Here's an example of three transactions reaching true deadlock. First, three different threads of control opened the database, acquiring three database handle read locks. + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- + 1 READ 1 HELD a.db handle 0 + 3 READ 1 HELD a.db handle 0 + 5 READ 1 HELD a.db handle 0 +``` + +The three threads then each began a transaction, and put a key/data pair on a different page: + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- +80000008 WRITE 1 HELD a.db page 4 + 1 READ 1 HELD a.db handle 0 + 3 READ 1 HELD a.db handle 0 + 5 READ 1 HELD a.db handle 0 +80000006 READ 1 HELD a.db page 1 +80000007 READ 1 HELD a.db page 1 +80000008 READ 1 HELD a.db page 1 +80000006 WRITE 1 HELD a.db page 2 +80000007 WRITE 1 HELD a.db page 3 +``` + +The thread using locker 80000006 put a new key/data pair on page 2, the thread using locker 80000007, on page 3, and the thread using locker 80000008 on page 4. Because the database is a 2-level Btree, the tree was searched, and so each transaction acquired a read lock on the Btree root page (page 1) as part of this operation. + +The three threads then each attempted to put a second key/data pair on a page currently locked by another thread. The thread using locker 80000006 tried to put a key/data pair on page 3, the thread using locker 80000007 on page 4, and the thread using locker 80000008 on page 2: + +``` c +Locks grouped by object +Locker Mode Count Status ----------- Object ---------- +80000008 WRITE 1 HELD a.db page 4 +80000007 WRITE 1 WAIT a.db page 4 + 1 READ 1 HELD a.db handle 0 + 3 READ 1 HELD a.db handle 0 + 5 READ 1 HELD a.db handle 0 +80000006 READ 2 HELD a.db page 1 +80000007 READ 2 HELD a.db page 1 +80000008 READ 2 HELD a.db page 1 +80000006 WRITE 1 HELD a.db page 2 +80000008 WRITE 1 WAIT a.db page 2 +80000007 WRITE 1 HELD a.db page 3 +80000006 WRITE 1 WAIT a.db page 3 +``` + +Now, each of the threads of control is blocked, waiting on a different thread of control. The thread using locker 80000007 is blocked by the thread using locker 80000008, due to the lock on page 4. The thread using locker 80000008 is blocked by the thread using locker 80000006, due to the lock on page 2. And the thread using locker 80000006 is blocked by the thread using locker 80000007, due to the lock on page 3. Since none of the threads of control can make progress, one of them will have to be killed in order to resolve the deadlock. diff --git a/docs-src/guides/programmer_reference/lock_max.md b/docs-src/guides/programmer_reference/lock_max.md new file mode 100644 index 000000000..841bc425e --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_max.md @@ -0,0 +1,49 @@ +--- +title: "Configuring locking: sizing the system" +api-name: "Configuring locking: sizing the system" +source: docs/programmer_reference/lock_max.html +--- +## Configuring locking: sizing the system + +The amount of memory available to the locking system is specified using the DB_ENV->set_memory_max() method. Sizing of the enviroment using the DB_ENV->set_memory_max() method is discussed in Sizing a database environment. Here we will discuss how to estimate the number of objects your application is likely to lock. Since running out of memory for locking structures is a fatal error requiring reconfiguration and restarting the environment it is best to overestimate the numbers. + +When configuring a Berkeley DB Concurrent Data Store application, the number of lock objects needed is two per open database (one for the database lock, and one for the cursor lock when the DB_CDB_ALLDB option is not specified). The number of locks needed is one per open database handle plus one per simultaneous cursor or non-cursor operation. + +Configuring a Berkeley DB Transactional Data Store application is more complicated. The recommended algorithm for selecting the number of locks, lockers, and lock objects is to run the application under stressful conditions and then review the lock system's statistics to determine the number of locks, lockers, and lock objects that were used. Then, double these values for safety. However, in some large applications, finer granularity of control is necessary in order to minimize the size of the Lock subsystem. + +The number of lockers can be estimated as follows: + +- If the database environment is using transactions, the number of lockers can be estimated by adding the number of simultaneously active non-transactional cursors and open database handles to the number of simultaneously active transactions and child transactions (where a child transaction is active until it commits or aborts, not until its parent commits or aborts). +- If the database environment is not using transactions, the number of lockers can be estimated by adding the number of simultaneously active non-transactional cursors and open database handles to the number of simultaneous non-cursor operations. + +The number of lock objects needed for a transaction can be estimated as follows: + +- For each access to a non-Queue database, one lock object is needed for each page that is read or updated. + +- For the Queue access method you will need one lock object per record that is read or updated. Deleted records skipped by a DB_NEXT or DB_PREV operation do not require a separate lock object. + +- For Btree and Recno databases additional lock objects may be needed for each node in the btree that has to be split due to an update. + +- For Hash and Queue databases, every access must obtain a lock on the metadata page for the duration of the access. This is not held to the end of the transaction. + +- If the transaction performs an update that needs to allocate a page to the database then a lock object for the metadata page will be needed to the end of the transaction. + +Note that transactions accumulate locks over the transaction lifetime, and the lock objects required by a single transaction is the total lock objects required by all of the database operations in the transaction. However, a database page (or record, in the case of the Queue access method), that is accessed multiple times within a transaction only requires a single lock object for the entire transaction. So if a transaction in your application typically accesses 10 records, that transaction will require about 10 lock objects (it may be a few more if it splits btree nodes). If you have up to 10 concurrent threads in your application, then you need to configure your system to have about 100 lock objects. It is always better to configure more than you need so that you don't run out of lock objects. The memory overhead of over-allocating lock objects is minimal as they are small structures. + +The number of locks required by an application cannot be easily estimated. It is possible to calculate a number of locks by multiplying the number of lockers, times the number of lock objects, times two (two for the two possible lock modes for each object, read and write). However, this is a pessimal value, and real applications are unlikely to actually need that many locks. Reviewing the Lock subsystem statistics is the best way to determine this value. + +By default a minimum number of locking objects are allocated at startup. To avoid contention due to allocation the application may use the DB_ENV->set_memory_init() method to preallocate and initialize the following lock structures: + +- `DB_MEM_LOCK` + + Specifies the number of locks that can be simultaneously requested in the system. + +- `DB_MEM_LOCKER` + + Specifies the number of lockers that can simultaneously request locks in the system. + +- `DB_MEM_LOCKOBJECTS` + + Specifies the number of objects that can simultaneously be locked in the system. + +In addition to the above structures, sizing your locking subsystem also requires specifying the number of lock table partitions. You do this using the DB_ENV->set_lk_partitions() method. Each partition may be accessed independently by a thread. More partitions can lead to higher levels of concurrency. The default is to set the number of partitions to be 10 times the number of cpus that the operating system reports at the time the environment is created. Having more than one partition when there is only one cpu is not beneficial because the locking system is more efficient when there is a single partition. Some operating systems (Linux, Solaris) may report thread contexts as cpus, and so it may be necessary to override the default to force a single partition on a single hyperthreaded cpu system. Objects and locks are divided among the partitions so it is best to allocate several locks and objects per partition. The system will force there to be at least one per partition. If a partition runs out of locks or objects it will steal what is needed from the other partitions. This operation could impact performance if it occurs too often. The final values specified for the locks and lock objects should be more than or equal to the number of lock table partitions. diff --git a/docs-src/guides/programmer_reference/lock_nondb.md b/docs-src/guides/programmer_reference/lock_nondb.md new file mode 100644 index 000000000..cc6df9308 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_nondb.md @@ -0,0 +1,14 @@ +--- +title: "Locking and non-Berkeley DB applications" +api-name: "Locking and non-Berkeley DB applications" +source: docs/programmer_reference/lock_nondb.html +--- +## Locking and non-Berkeley DB applications + +The Lock subsystem is useful outside the context of Berkeley DB. It can be used to manage concurrent access to any collection of either ephemeral or persistent objects. That is, the lock region can persist across invocations of an application, so it can be used to provide long-term locking (for example, conference room scheduling). + +In order to use the locking subsystem in such a general way, the applications must adhere to a convention for identifying objects and lockers. Consider a conference room scheduling problem, in which there are three conference rooms scheduled in half-hour intervals. The scheduling application must then select a way to identify each conference room/time slot combination. In this case, we could describe the objects being locked as bytestrings consisting of the conference room name, the date when it is needed, and the beginning of the appropriate half-hour slot. + +Lockers are 32-bit numbers, so we might choose to use the User ID of the individual running the scheduling program. To schedule half-hour slots, all the application needs to do is issue a DB_ENV->lock_get() call for the appropriate locker/object pair. To schedule a longer slot, the application needs to issue a DB_ENV->lock_vec() call, with one DB_ENV->lock_get() operation per half-hour — up to the total length. If the DB_ENV->lock_vec() call fails, the application would have to release the parts of the time slot that were obtained. + +To cancel a reservation, the application would make the appropriate DB_ENV->lock_put() calls. To reschedule a reservation, the DB_ENV->lock_get() and DB_ENV->lock_put() calls could all be made inside of a single DB_ENV->lock_vec() call. The output of DB_ENV->lock_stat() could be post-processed into a human-readable schedule of conference room use. diff --git a/docs-src/guides/programmer_reference/lock_notxn.md b/docs-src/guides/programmer_reference/lock_notxn.md new file mode 100644 index 000000000..ce0c2303c --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_notxn.md @@ -0,0 +1,10 @@ +--- +title: "Locking without transactions" +api-name: "Locking without transactions" +source: docs/programmer_reference/lock_notxn.html +--- +## Locking without transactions + +If an application runs with locking specified, but not transactions (for example, DB_ENV->open() is callsed with DB_INIT_LOCK or DB_INIT_CDB specified, but not DB_INIT_TXN), locks are normally acquired during each Berkeley DB operation and released before the operation returns to the caller. The only exception is in the case of cursor operations. Cursors identify a particular position in a file. For this reason, cursors must retain read locks across cursor calls to make sure that the position is uniquely identifiable during a subsequent cursor call, and so that an operation using DB_CURRENT will always refer to the same record as a previous cursor call. These cursor locks cannot be released until the cursor is either repositioned and a new cursor lock established (for example, using the DB_NEXT or DB_SET flags), or the cursor is closed. As a result, application writers are encouraged to close cursors as soon as possible. + +It is important to realize that concurrent applications that use locking must ensure that two concurrent threads do not block each other. However, because Btree and Hash access method page splits can occur at any time, there is virtually no way to guarantee that an application that writes the database cannot deadlock. Applications running without the protection of transactions may deadlock, and can leave the database in an inconsistent state when they do so. Applications that need concurrent access, but not transactions, are more safely implemented using the Berkeley DB Concurrent Data Store Product. diff --git a/docs-src/guides/programmer_reference/lock_page.md b/docs-src/guides/programmer_reference/lock_page.md new file mode 100644 index 000000000..ce2cdff6a --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_page.md @@ -0,0 +1,24 @@ +--- +title: "Locking granularity" +api-name: "Locking granularity" +source: docs/programmer_reference/lock_page.html +--- +## Locking granularity + +With the exception of the Queue access method, the Berkeley DB access methods do page-level locking. The size of pages in a database may be set when the database is created by calling the DB->set_pagesize() method. If not specified by the application, Berkeley DB selects a page size that will provide the best I/O performance by setting the page size equal to the block size of the underlying file system. Selecting a smaller page size can result in increased concurrency for some applications. + +In the Btree access method, Berkeley DB uses a technique called lock coupling to improve concurrency. The traversal of a Btree requires reading a page, searching that page to determine which page to search next, and then repeating this process on the next page. Once a page has been searched, it will never be accessed again for this operation, unless a page split is required. To improve concurrency in the tree, once the next page to read/search has been determined, that page is locked and then the original page lock is released atomically (that is, without relinquishing control of the lock manager). When page splits become necessary, write locks are reacquired. + +Because the Recno access method is built upon Btree, it also uses lock coupling for read operations. However, because the Recno access method must maintain a count of records on its internal pages, it cannot lock-couple during write operations. Instead, it retains write locks on all internal pages during every update operation. For this reason, it is not possible to have high concurrency in the Recno access method in the presence of write operations. + +The Queue access method uses only short-term page locks. That is, a page lock is released prior to requesting another page lock. Record locks are used for transaction isolation. The provides a high degree of concurrency for write operations. A metadata page is used to keep track of the head and tail of the queue. This page is never locked during other locking or I/O operations. + +The Hash access method does not have such traversal issues, but it must always refer to its metadata while computing a hash function because it implements dynamic hashing. This metadata is stored on a special page in the hash database. This page must therefore be read-locked on every operation. Fortunately, it needs to be write-locked only when new pages are allocated to the file, which happens in three cases: + +- a hash bucket becomes full and needs to split +- a key or data item is too large to fit on a normal page +- the number of duplicate items for a fixed key becomes so large that they are moved to an auxiliary page + +In this case, the access method must obtain a write lock on the metadata page, thus requiring that all readers be blocked from entering the tree until the update completes. + +Finally, when traversing duplicate data items for a key, the lock on the key value also acts as a lock on all duplicates of that key. Therefore, two conflicting threads of control cannot access the same duplicate set simultaneously. diff --git a/docs-src/guides/programmer_reference/lock_stdmode.md b/docs-src/guides/programmer_reference/lock_stdmode.md new file mode 100644 index 000000000..d64753b03 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_stdmode.md @@ -0,0 +1,46 @@ +--- +title: "Standard lock modes" +api-name: "Standard lock modes" +source: docs/programmer_reference/lock_stdmode.html +--- +## Standard lock modes + +The Berkeley DB locking protocol is described by a conflict matrix. A conflict matrix is an NxN array in which N is the number of different lock modes supported, and the (i, j)th entry of the array indicates whether a lock of mode i conflicts with a lock of mode j. In addition, Berkeley DB defines the type **db_lockmode_t**, which is the type of a lock mode within a conflict matrix. + +The following is an example of a conflict matrix. The actual conflict matrix used by Berkeley DB to support the underlying access methods is more complicated, but this matrix shows the lock mode relationships available to applications using the Berkeley DB Locking subsystem interfaces directly. + +DB_LOCK_NG +not granted (always 0) + +DB_LOCK_READ +read (shared) + +DB_LOCK_WRITE +write (exclusive) + +DB_LOCK_IWRITE +intention to write (shared) + +DB_LOCK_IREAD +intention to read (shared) + +DB_LOCK_IWR +intention to read and write (shared) + +In a conflict matrix, the rows indicate the lock that is held, and the columns indicate the lock that is requested. A 1 represents a conflict (that is, do not grant the lock if the indicated lock is held), and a 0 indicates that it is OK to grant the lock. + +``` c + Notheld Read Write IWrite IRead IRW +Notheld 0 0 0 0 0 0 +Read* 0 0 1 1 0 1 +Write** 0 1 1 1 1 1 +Intent Write 0 1 1 0 0 0 +Intent Read 0 0 1 0 0 0 +Intent RW 0 1 1 0 0 0 +``` + +\* +In this case, suppose that there is a read lock held on an object. A new request for a read lock would be granted, but a request for a write lock would not. + +\*\* +In this case, suppose that there is a write lock held on an object. A new request for either a read or write lock would be denied. diff --git a/docs-src/guides/programmer_reference/lock_timeout.md b/docs-src/guides/programmer_reference/lock_timeout.md new file mode 100644 index 000000000..5e481bbe9 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_timeout.md @@ -0,0 +1,18 @@ +--- +title: "Deadlock detection using timers" +api-name: "Deadlock detection using timers" +source: docs/programmer_reference/lock_timeout.html +--- +## Deadlock detection using timers + +Lock and transaction timeouts may be used in place of, or in addition to, regular deadlock detection. If lock timeouts are set, lock requests will return DB_LOCK_NOTGRANTED from a lock call when it is detected that the lock's timeout has expired, that is, the lock request has blocked, waiting, longer than the specified timeout. If transaction timeouts are set, lock requests will return DB_LOCK_NOTGRANTED from a lock call when it has been detected that the transaction has been active longer than the specified timeout. + +If lock or transaction timeouts have been set, database operations will return DB_LOCK_DEADLOCK when the lock timeout has expired or the transaction has been active longer than the specified timeout. Applications wanting to distinguish between true deadlock and timeout can use the DB_ENV->set_flags() configuration flag, which causes database operations to instead return DB_LOCK_NOTGRANTED in the case of timeout. + +As lock and transaction timeouts are only checked when lock requests first block or when deadlock detection is performed, the accuracy of the timeout depends on how often deadlock detection is performed. More specifically, transactions will continue to run after their timeout has expired if they do not block on a lock request after that time. A separate deadlock detection thread (or process) should always be used if the application depends on timeouts; otherwise, if there are no new blocked lock requests a pending timeout will never trigger. + +If the database environment deadlock detector has been configured with the DB_LOCK_EXPIRE option, timeouts are the only mechanism by which deadlocks will be broken. If the deadlock detector has been configured with a different option, then regular deadlock detection will be performed, and in addition, if timeouts have also been specified, lock requests and transactions will time out as well. + +Lock and transaction timeouts may be specified on a database environment wide basis using the DB_ENV->set_timeout() method. Lock timeouts may be specified on a per-lock request basis using the DB_ENV->lock_vec() method. Lock and transaction timeouts may be specified on a per-transaction basis using the DB_TXN->set_timeout() method. Per-lock and per-transaction timeouts supersede environment wide timeouts. + +For example, consider that the environment wide transaction timeout has been set to 20ms, the environment wide lock timeout has been set to 10ms, a transaction has been created in this environment and its timeout value set to 8ms, and a specific lock request has been made on behalf of this transaction where the lock timeout was set to 4ms. By default, transactions in this environment will be timed out if they block waiting for a lock after 20ms. The specific transaction described will be timed out if it blocks waiting for a lock after 8ms. By default, any lock request in this system will be timed out if it blocks longer than 10ms, and the specific lock described will be timed out if it blocks longer than 4ms. diff --git a/docs-src/guides/programmer_reference/lock_twopl.md b/docs-src/guides/programmer_reference/lock_twopl.md new file mode 100644 index 000000000..3292b9883 --- /dev/null +++ b/docs-src/guides/programmer_reference/lock_twopl.md @@ -0,0 +1,16 @@ +--- +title: "Locking with transactions: two-phase locking" +api-name: "Locking with transactions: two-phase locking" +source: docs/programmer_reference/lock_twopl.html +--- +## Locking with transactions: two-phase locking + +Berkeley DB uses a locking protocol called *two-phase locking (2PL)*. This is the traditional protocol used in conjunction with lock-based transaction systems. + +In a two-phase locking system, transactions are divided into two distinct phases. During the first phase, the transaction only acquires locks; during the second phase, the transaction only releases locks. More formally, once a transaction releases a lock, it may not acquire any additional locks. Practically, this translates into a system in which locks are acquired as they are needed throughout a transaction and retained until the transaction ends, either by committing or aborting. In Berkeley DB, locks are released during DB_TXN->abort() or DB_TXN->commit(). The only exception to this protocol occurs when we use lock-coupling to traverse a data structure. If the locks are held only for traversal purposes, it is safe to release locks before transactions commit or abort. + +For applications, the implications of 2PL are that long-running transactions will hold locks for a long time. When designing applications, lock contention should be considered. In order to reduce the probability of deadlock and achieve the best level of concurrency possible, the following guidelines are helpful. + +1. When accessing multiple databases, design all transactions so that they access the files in the same order. +2. If possible, access your most hotly contested resources last (so that their locks are held for the shortest time possible). +3. If possible, use nested transactions to protect the parts of your transaction most likely to deadlock. diff --git a/docs-src/guides/programmer_reference/log.md b/docs-src/guides/programmer_reference/log.md new file mode 100644 index 000000000..dab66e35e --- /dev/null +++ b/docs-src/guides/programmer_reference/log.md @@ -0,0 +1,46 @@ +--- +title: "Chapter 17.  The Logging Subsystem" +api-name: "Chapter 17.  The Logging Subsystem" +source: docs/programmer_reference/log.html +--- +## Chapter 17.  The Logging Subsystem + +**Table of Contents** + + [Introduction to the logging subsystem](log.md#log_intro) + + [Configuring logging](log_config.md) + + [Log file limits](log_limits.md) + +## Introduction to the logging subsystem + +The Logging subsystem is the logging facility used by Berkeley DB. It is largely Berkeley DB-specific, although it is potentially useful outside of the Berkeley DB package for applications wanting write-ahead logging support. Applications wanting to use the log for purposes other than logging file modifications based on a set of open file descriptors will almost certainly need to make source code modifications to the Berkeley DB code base. + +A log can be shared by any number of threads of control. The DB_ENV->open() method is used to open a log. When the log is no longer in use, it should be closed using the DB_ENV->close() method. + +Individual log entries are identified by log sequence numbers. Log sequence numbers are stored in an opaque object, an DB_LSN. + +The DB_ENV->log_cursor() method is used to allocate a log cursor. Log cursors have two methods: DB_LOGC->get() method to retrieve log records from the log, and DB_LOGC->close() method to destroy the cursor. + +There are additional methods for integrating the log subsystem with a transaction processing system: + + DB_ENV->log_flush() +Flushes the log up to a particular log sequence number. + + DB_ENV->log_compare() +Allows applications to compare any two log sequence numbers. + + DB_ENV->log_file() +Maps a log sequence number to the specific log file that contains it. + + DB_ENV->log_archive() +Returns various sets of log filenames. These methods are used for database administration; for example, to determine if log files may safely be removed from the system. + + DB_ENV->log_stat() +The display db_stat utility used the DB_ENV->log_stat() method to display statistics about the log. + + DB_ENV->remove() +The log meta-information (but not the log files themselves) may be removed using the DB_ENV->remove() method. + +For more information on the logging subsystem methods, see the Logging Subsystem and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/log_config.md b/docs-src/guides/programmer_reference/log_config.md new file mode 100644 index 000000000..b9126fe9e --- /dev/null +++ b/docs-src/guides/programmer_reference/log_config.md @@ -0,0 +1,18 @@ +--- +title: "Configuring logging" +api-name: "Configuring logging" +source: docs/programmer_reference/log_config.html +--- +## Configuring logging + +The aspects of logging that may be configured are the size of the logging subsystem's region, the size of the log files on disk and the size of the log buffer in memory. The DB_ENV->set_lg_regionmax() method specifies the size of the logging subsystem's region, in bytes. The logging subsystem's default size is approximately 60KB. This value may need to be increased if a large number of files are registered with the Berkeley DB log manager, for example, by opening a large number of Berkeley DB database files in a transactional application. + +The DB_ENV->set_lg_max() method specifies the individual log file size for all the applications sharing the Berkeley DB environment. Setting the log file size is largely a matter of convenience and a reflection of the application's preferences in backup media and frequency. However, setting the log file size too low can potentially cause problems because it would be possible to run out of log sequence numbers, which requires a full archival and application restart to reset. See Log file limits for more information. + +The DB_ENV->set_lg_bsize() method specifies the size of the in-memory log buffer, in bytes. Log information is stored in memory until the buffer fills up or transaction commit forces the buffer to be written to disk. Larger buffer sizes can significantly increase throughput in the presence of long-running transactions, highly concurrent applications, or transactions producing large amounts of data. By default, the buffer is approximately 32KB. + +The DB_ENV->set_lg_dir() method specifies the directory in which log files will be placed. By default, log files are placed in the environment home directory. + +The DB_ENV->set_lg_filemode() method specifies the absolute file mode for created log files. This method is only useful for the rare Berkeley DB application that does not control its umask value. + +The DB_ENV->log_set_config() method configures several boolean parameters that control the use of file system controls such as O_DIRECT and O_DSYNC, automatic removal of log files, in-memory logging, and pre-zeroing of logfiles. diff --git a/docs-src/guides/programmer_reference/log_limits.md b/docs-src/guides/programmer_reference/log_limits.md new file mode 100644 index 000000000..e199ca97f --- /dev/null +++ b/docs-src/guides/programmer_reference/log_limits.md @@ -0,0 +1,23 @@ +--- +title: "Log file limits" +api-name: "Log file limits" +source: docs/programmer_reference/log_limits.html +--- +## Log file limits + +Log filenames and sizes impose a limit on how long databases may be used in a Berkeley DB database environment. It is quite unlikely that an application will reach this limit; however, if the limit is reached, the Berkeley DB environment's databases must be dumped and reloaded. + +The log filename consists of **log.** followed by 10 digits, with a maximum of 2,000,000,000 log files. Consider an application performing 6000 transactions per second for 24 hours a day, logged into 10MB log files, in which each transaction is logging approximately 500 bytes of data. The following calculation: + +``` c +(10 * 2^20 * 2000000000) / (6000 * 500 * 365 * 60 * 60 * 24) = ~221 +``` + +indicates that the system will run out of log filenames in roughly 221 years. + +There is no way to reset the log filename space in Berkeley DB. If your application is reaching the end of its log filename space, you must do the following: + +1. Archive your databases as if to prepare for catastrophic failure (see Database and log file archival for more information). +2. Reset the database's log sequence numbers (see the **-r** option to the db_load utility for more information). +3. Remove all of the log files from the database environment. (This is the only situation in which all the log files are removed from an environment; in all other cases, at least a single log file is retained.) +4. Restart your application. diff --git a/docs-src/guides/programmer_reference/moreinfo.md b/docs-src/guides/programmer_reference/moreinfo.md new file mode 100644 index 000000000..7994f4b94 --- /dev/null +++ b/docs-src/guides/programmer_reference/moreinfo.md @@ -0,0 +1,38 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/programmer_reference/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Getting Started with Transaction Processing for C + +- Berkeley DB Getting Started with Replicated Applications for C + +- Berkeley DB C API Reference Guide + +- Berkeley DB C++ API Reference Guide + +- Berkeley DB STL API Reference Guide + +- Berkeley DB TCL API Reference Guide + +- Berkeley DB Installation and Build Guide + +- Berkeley DB Upgrade Guide + +- Berkeley DB Getting Started with the SQL APIs + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/programmer_reference/mp.md b/docs-src/guides/programmer_reference/mp.md new file mode 100644 index 000000000..ad80d14da --- /dev/null +++ b/docs-src/guides/programmer_reference/mp.md @@ -0,0 +1,42 @@ +--- +title: "Chapter 18.  The Memory Pool Subsystem" +api-name: "Chapter 18.  The Memory Pool Subsystem" +source: docs/programmer_reference/mp.html +--- +## Chapter 18.  The Memory Pool Subsystem + +**Table of Contents** + + [Introduction to the memory pool subsystem](mp.md#mp_intro) + + [Configuring the memory pool](mp_config.md) + + [Warming the memory pool](mp_warm.md) + + [The warm_cache() function](mp_warm.md#warm_cache) + +## Introduction to the memory pool subsystem + +The Memory Pool subsystem is the general-purpose shared memory buffer pool used by Berkeley DB. This module is useful outside of the Berkeley DB package for processes that require page-oriented, shared and cached file access. (However, such "use outside of Berkeley DB" is not supported in replicated environments.) + +A *memory pool* is a memory cache shared among any number of threads of control. The DB_INIT_MPOOL flag to the DB_ENV->open() method opens and optionally creates a memory pool. When that pool is no longer in use, it should be closed using the DB_ENV->close() method. + +The DB_ENV->memp_fcreate() method returns a DB_MPOOLFILE handle on an underlying file within the memory pool. The file may be opened using the DB_MPOOLFILE->open() method. The DB_MPOOLFILE->get() method is used to retrieve pages from files in the pool. All retrieved pages must be subsequently returned using the DB_MPOOLFILE->put() method. At the time pages are returned, they may be marked **dirty**, which causes them to be written to the underlying file before being discarded from the pool. If there is insufficient room to bring a new page in the pool, a page is selected to be discarded from the pool using a least-recently-used algorithm. All dirty pages in the pool from the file may be flushed using the DB_MPOOLFILE->sync() method. When the file handle is no longer in use, it should be closed using the DB_MPOOLFILE->close() method. + +There are additional configuration interfaces that apply when opening a new file in the memory pool: + +- The DB_MPOOLFILE->set_clear_len() method specifies the number of bytes to clear when creating a new page in the memory pool. +- The DB_MPOOLFILE->set_fileid() method specifies a unique ID associated with the file. +- The DB_MPOOLFILE->set_ftype() method specifies the type of file for the purposes of page input and output processing. +- The DB_MPOOLFILE->set_lsn_offset() method specifies the byte offset of each page's log sequence number (DB_LSN) for the purposes of transaction checkpoints. +- The DB_MPOOLFILE->set_pgcookie() method specifies an application provided argument for the purposes of page input and output processing. + +There are additional interfaces for the memory pool as a whole: + +- It is possible to gradually flush buffers from the pool in order to maintain a consistent percentage of clean buffers in the pool using the DB_ENV->memp_trickle() method. +- Because special-purpose processing may be necessary when pages are read or written (for example, endian conversion, or page checksums), the DB_ENV->memp_register() function allows applications to specify automatic input and output processing in these cases. +- The db_stat utility uses the DB_ENV->memp_stat() method to display statistics about the efficiency of the pool. +- All dirty pages in the pool may be flushed using the DB_ENV->memp_sync() method. In addition, DB_ENV->memp_sync() takes an argument that is specific to database systems, and which allows the memory pool to be flushed up to a specified log sequence number (DB_LSN). +- The entire pool may be discarded using the DB_ENV->remove() method. + +For more information on the memory pool subsystem methods, see the Memory Pools and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/mp_config.md b/docs-src/guides/programmer_reference/mp_config.md new file mode 100644 index 000000000..62acdfc06 --- /dev/null +++ b/docs-src/guides/programmer_reference/mp_config.md @@ -0,0 +1,18 @@ +--- +title: "Configuring the memory pool" +api-name: "Configuring the memory pool" +source: docs/programmer_reference/mp_config.html +--- +## Configuring the memory pool + +There are two issues to consider when configuring the memory pool. + +The first issue, the most important tuning parameter for Berkeley DB applications, is the size of the memory pool. There are two ways to specify the pool size. First, calling the DB_ENV->set_cachesize() method specifies the pool size for all of the applications sharing the Berkeley DB environment. Second, the DB->set_cachesize() method only specifies a pool size for the specific database. Note: It is meaningless to call DB->set_cachesize() for a database opened inside of a Berkeley DB environment because the environment pool size will override any pool size specified for a single database. For information on tuning the Berkeley DB cache size, see Selecting a cache size. + +Note the memory pool defaults to assuming that the average page size is 4k. This factor is used to determine the size of the hash table used to locate pages in the memory pool. The size of the hash table is calculated to so that on average 2.5 pages will be in each hash table entry. Each page requires a mutex be allocated to it and the average page size is used to determine the number of mutexes to allocate to the memory pool. + +Normally you should see good results by using the default values for the page size, but in some cases you may be able to achieve better performance by manually configuring the page size. The expected page size, hash table size and mutex count can be set via the methods: DB_ENV->set_mp_pagesize(), DB_ENV->set_mp_tablesize(), and DB_ENV->set_mp_mtxcount(). + +The second memory pool configuration issue is the maximum size an underlying file can be and still be mapped into the process address space (instead of reading the file's pages into the cache). Mapping files into the process address space can result in better performance because available virtual memory is often much larger than the local cache, and page faults are faster than page copying on many systems. However, in the presence of limited virtual memory, it can cause resource starvation; and in the presence of large databases, it can result in immense process sizes. In addition, because of the requirements of the Berkeley DB transactional implementation, only read-only files can be mapped into process memory. + +To specify that no files are to be mapped into the process address space, specify the DB_NOMMAP flag to the DB_ENV->set_flags() method. To specify that any individual file should not be mapped into the process address space, specify the DB_NOMMAP flag to the DB_MPOOLFILE->open() interface. To limit the size of files mapped into the process address space, use the DB_ENV->set_mp_mmapsize() method. diff --git a/docs-src/guides/programmer_reference/mp_warm.md b/docs-src/guides/programmer_reference/mp_warm.md new file mode 100644 index 000000000..7ce11177b --- /dev/null +++ b/docs-src/guides/programmer_reference/mp_warm.md @@ -0,0 +1,268 @@ +--- +title: "Warming the memory pool" +api-name: "Warming the memory pool" +source: docs/programmer_reference/mp_warm.html +--- +## Warming the memory pool + + [The warm_cache() function](mp_warm.md#warm_cache) + +Some applications find it is useful to pre-load the memory pool upon application startup. This is a strictly optional activity that provides faster initial access to your data at the expense of longer application startup times. + +To warm the cache, you simply have to read the records that your application will operate on most frequently. You can do this with normal database reads, and you can also use cursors. But the most efficient way to warm the cache is to use memory pool APIs to get the pages that contain your most frequently accessed records. + +You read pages into the memory pool using the `DB_MPOOLFILE->get()` method. This method acquires locks on the page, so immediately upon getting the page you need to put it so as to release the locks. + +Also, you obtain a memory pool file handle using a database handle. This means that if your data is contained in more than one Berkeley DB database, you must operate on each database handle in turn. + +The following example code illustrates this. It does the following: + +- Opens an environment and two database handles. + +- Determines how many database pages can fit into the memory pool. + +- Uses `DB_MPOOLFILE->get()` and `DB_MPOOLFILE->put()` to load that number of pages into the memory pool. + +First, we include the libraries that we need, forward declare some functions, and intialize some variables. + +``` c +#include +#include +#include +#include + +/* Forward declarations */ +int warm_cache(DB *, int *, int); +int open_db(DB_ENV *, DB **, const char *); + +int +main(void) +{ + DB *dbp1 = 0, *dbp2 = 0; + DB_ENV *envp = 0; + u_int32_t env_flags, pagesize, gbytes, bytes; + int ret = 0, ret_t = 0, numcachepages, pagecount; +``` + +Then we open the environment and our databases. The `open_db()` function that we use here simply opens a database. We will provide that code at the end of this example, but it should hold no surprises for you. We only use the function so as to reuse the code. + +``` c + /* + * Open the environment and the databases + */ + ret = db_env_create(&envp, 0); + if (ret != 0) { + fprintf(stderr, "Error creating environment handle: %s\n", + db_strerror(ret)); + goto err; + } + + env_flags = + DB_CREATE | /* Create the environment if it does + not exist */ + DB_RECOVER | /* Run normal recovery. */ + DB_INIT_LOCK | /* Initialize the locking subsystem */ + DB_INIT_LOG | /* Initialize the logging subsystem */ + DB_INIT_TXN | /* Initialize the transactional subsystem. This + * also turns on logging. */ + DB_INIT_MPOOL; /* Initialize the memory pool */ + + /* Now actually open the environment */ + ret = envp->open(envp, "./env", env_flags, 0); + if (ret != 0) { + fprintf(stderr, "Error opening environment: %s\n", + db_strerror(ret)); + goto err; + } + + ret = open_db(envp, &dbp1, "mydb1.db"); + if (ret != 0) + goto err; + + ret = open_db(envp, &dbp2, "mydb2.db"); + if (ret != 0) + goto err; +``` + +Next we determine how many database pages we can fit into the cache. We do this by finding out how large our pages are, and then finding out how large our cache can be. + +``` c + /* Find out how many pages can fit at most in the cache */ + ret = envp->get_mp_pagesize(envp, &pagesize); + if (ret != 0) { + fprintf(stderr, "Error retrieving the cache pagesize: %s\n", + db_strerror(ret)); + goto err; + } + + ret = envp->get_cache_max(envp, &gbytes, &bytes); + if (ret != 0) { + fprintf(stderr, "Error retrieving maximum cache size: %s\n", + db_strerror(ret)); + goto err; + } + /* Avoid an overflow by first calculating pages per gigabyte. */ + numcachepages = gbytes * ((1024 * 1024 * 1024) / pagesize); + numcachepages += bytes / pagesize; +``` + +Now we call our `warm_cache()` function. We will describe this function in a little while, but note that we call `warm_cache()` twice. This is because our example uses two databases, and the memory pool methods operate on a per-handle basis. + +``` c + /* + * Warm the cache by loading pages from each of the databases + * in turn. + */ + pagecount = 0; + ret = warm_cache(dbp1, &pagecount, numcachepages); + if (ret != 0) { + fprintf(stderr, "Error warming the cache: %s\n", + db_strerror(ret)); + goto err; + } + + ret = warm_cache(dbp2, &pagecount, numcachepages); + if (ret != 0) { + fprintf(stderr, "Error warming the cache: %s\n", + db_strerror(ret)); + goto err; + } +``` + +Now we close all our handles and finish our `main()` function. Again, this is straight-forward boilerplate code that we provide simply to be complete. + +``` c +err: + /* Close our database handles, if they were opened. */ + if (dbp1 != NULL) { + ret_t = dbp1->close(dbp1, 0); + if (ret_t != 0) { + fprintf(stderr, "dbp1 close failed: %s\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + if (dbp2 != NULL) { + ret_t = dbp2->close(dbp2, 0); + if (ret_t != 0) { + fprintf(stderr, "dbp2 close failed: %s\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Close our environment, if it was opened. */ + if (envp != NULL) { + ret_t = envp->close(envp, 0); + if (ret_t != 0) { + fprintf(stderr, "environment close failed: %s\n", + db_strerror(ret_t)); + ret = ret_t; + } + } + + /* Final status message and return. */ + printf("I'm all done.\n"); + return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} +``` + +As noted above, this example uses an `open_db()` function, which opens a database handle inside the provided environment. To be complete, this is the implementation of that function: + +``` c +/* Open a database handle */ +int +open_db(DB_ENV *envp, DB **dbpp, const char *file_name) +{ + int ret = 0; + u_int32_t db_flags = 0; + DB *dbp; + + /* Open the database */ + ret = db_create(&dbp, envp, 0); + if (ret != 0) { + fprintf(stderr, "Error opening database: %s : %s\n", + file_name, db_strerror(ret)); + return ret; + } + + /* Point to the memory malloc'd by db_create() */ + *dbpp = dbp; + + db_flags = DB_CREATE | /* Create the database if it does + not exist */ + DB_AUTO_COMMIT; /* Allow autocommit */ + + ret = dbp->open(dbp, /* Pointer to the database */ + 0, /* Txn pointer */ + file_name, /* File name */ + 0, /* Logical db name */ + DB_BTREE, /* Database type (using btree) */ + db_flags, /* Open flags */ + 0); /* File mode. Using defaults */ + if (ret != 0) { + dbp->err(dbp, ret, "Database open failed: %s : %s\n", + file_name, db_strerror(ret)); + return ret; + } + return 0; +} +``` + +### The warm_cache() function + +In this section we provide the implementation of the `warm_cache()` function. This example function simply loads all the database pages that will fit into the memory pool. It starts from the first database page and continues until it either runs out of database pages or it runs out of room in the memory pool. + +``` c +/* Warm the cache */ +int +warm_cache(DB *dbp, int *pagecountp, int numcachepages) +{ + DB_MPOOLFILE *mpf = 0; + void *page_addrp = 0; + db_pgno_t page_number = 0; + int ret = 0; + int pagecount = *pagecountp; + + /* + * Get the mpool handle + */ + mpf = dbp->get_mpf(dbp); + + /* Load pages until there are no more pages in the database, + * or until we've put as many pages into the cache as will fit. + */ + while (ret != DB_PAGE_NOTFOUND && pagecount < numcachepages) { + /* + * Get the page from the cache. This causes DB to retrieve + * the page from disk if it isn't already in the cache. + */ + ret = mpf->get(mpf, &page_number, 0, 0, &page_addrp); + if (ret && ret != DB_PAGE_NOTFOUND) { + fprintf(stderr, "Error retrieving db page: %i : %s\n", + page_number, db_strerror(ret)); + return ret; + } + + /* + * If a page was retrieved, put it back into the cache. This + * releases the page latch so that the page can be evicted + * if DB needs more room in the cache at some later time. + */ + if (ret != DB_PAGE_NOTFOUND) { + ret = mpf->put(mpf, page_addrp, DB_PRIORITY_UNCHANGED, 0); + if (ret) { + fprintf(stderr, "Error putting db page: %i : %s\n", + page_number, db_strerror(ret)); + return ret; + } + } + ++page_number; + ++pagecount; + *pagecountp = pagecount; + } + + return 0; +} +``` diff --git a/docs-src/guides/programmer_reference/preface.md b/docs-src/guides/programmer_reference/preface.md new file mode 100644 index 000000000..d0ac29b64 --- /dev/null +++ b/docs-src/guides/programmer_reference/preface.md @@ -0,0 +1,44 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/programmer_reference/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +Welcome to Berkeley DB (DB). This document provides an introduction and usage notes for skilled programmers who wish to use the Berkeley DB APIs. + +This document reflects Berkeley DB 11*g* Release 2, which provides DB library version 11.2.5.3. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Structure names are represented in `monospaced font`, as are `method names`. For example: "`DB->open()` is a method on a `DB` handle." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/guides/programmer_reference/program.md b/docs-src/guides/programmer_reference/program.md new file mode 100644 index 000000000..793e1f247 --- /dev/null +++ b/docs-src/guides/programmer_reference/program.md @@ -0,0 +1,54 @@ +--- +title: "Chapter 15.  Programmer Notes" +api-name: "Chapter 15.  Programmer Notes" +source: docs/programmer_reference/program.html +--- +## Chapter 15.  Programmer Notes + +**Table of Contents** + + [Signal handling](program.md#program_appsignals) + + [Error returns to applications](program_errorret.md) + + [Environment variables](program_environ.md) + + [Multithreaded applications](program_mt.md) + + [Berkeley DB handles](program_scope.md) + + [Name spaces](program_namespace.md) + + [C Language Name Space](program_namespace.md#idp52962960) + + [Filesystem Name Space](program_namespace.md#idp53001824) + + [Memory-only or Flash configurations](program_ram.md) + + [Disk drive caches](program_cache.md) + + [Copying or moving databases](program_copy.md) + + [Compatibility with historic UNIX interfaces](program_compatible.md) + + [Run-time configuration](program_runtime.md) + + [Performance Event Monitoring](program_perfmon.md) + + [Using the DTrace Provider](program_perfmon.md#program_perfmon_dtrace) + + [Using SystemTap](program_perfmon.md#program_perfmon_stap) + + [Example Scripts](program_perfmon.md#program_perfmon_examples) + + [Performance Events Reference](program_perfmon.md#program_perfmon_probes) + + [Programmer notes FAQ](program_faq.md) + +## Signal handling + +When applications using Berkeley DB receive signals, it is important that they exit gracefully, discarding any Berkeley DB locks that they may hold. This is normally done by setting a flag when a signal arrives and then checking for that flag periodically within the application. Because Berkeley DB is not re-entrant, the signal handler should not attempt to release locks and/or close the database handles itself. Re-entering Berkeley DB is not guaranteed to work correctly, and the results are undefined. + +If an application exits holding a lock, the situation is no different than if the application crashed, and all applications participating in the database environment must be shut down, and then recovery must be performed. If this is not done, databases may be left in an inconsistent state, or locks the application held may cause unresolvable deadlocks inside the environment, causing applications to hang. + +Berkeley DB restarts all system calls interrupted by signals, that is, any underlying system calls that return failure with errno set to EINTR will be restarted rather than failing. diff --git a/docs-src/guides/programmer_reference/program_cache.md b/docs-src/guides/programmer_reference/program_cache.md new file mode 100644 index 000000000..bb780be54 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_cache.md @@ -0,0 +1,10 @@ +--- +title: "Disk drive caches" +api-name: "Disk drive caches" +source: docs/programmer_reference/program_cache.html +--- +## Disk drive caches + +Many disk drives contain onboard caches. Some of these drives include battery-backup or other functionality that guarantees that all cached data will be completely written if the power fails. These drives can offer substantial performance improvements over drives without caching support. However, some caching drives rely on capacitors or other mechanisms that guarantee only that the write of the current sector will complete. These drives can endanger your database and potentially cause corruption of your data. + +To avoid losing your data, make sure the caching on your disk drives is properly configured so the drive will never report that data has been written unless the data is guaranteed to be written in the face of a power failure. Many times, this means that write-caching on the disk drive must be disabled. diff --git a/docs-src/guides/programmer_reference/program_compatible.md b/docs-src/guides/programmer_reference/program_compatible.md new file mode 100644 index 000000000..5dc2d5d25 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_compatible.md @@ -0,0 +1,10 @@ +--- +title: "Compatibility with historic UNIX interfaces" +api-name: "Compatibility with historic UNIX interfaces" +source: docs/programmer_reference/program_compatible.html +--- +## Compatibility with historic UNIX interfaces + +The Berkeley DB version 2 library provides backward-compatible interfaces for the historic UNIX dbm, ndbm and hsearch interfaces. It also provides a backward-compatible interface for the historic Berkeley DB 1.85 release. + +Berkeley DB version 2 does not provide database compatibility for any of the previous interfaces, and existing databases must be converted manually. To convert existing databases from the Berkeley DB 1.85 format to the Berkeley DB version 2 format, review the db_dump185 utility and the db_load utility information. No utilities are provided to convert UNIX dbm, ndbm or hsearch databases. diff --git a/docs-src/guides/programmer_reference/program_copy.md b/docs-src/guides/programmer_reference/program_copy.md new file mode 100644 index 000000000..3ed87bd0b --- /dev/null +++ b/docs-src/guides/programmer_reference/program_copy.md @@ -0,0 +1,22 @@ +--- +title: "Copying or moving databases" +api-name: "Copying or moving databases" +source: docs/programmer_reference/program_copy.html +--- +## Copying or moving databases + +There are two issues with copying or moving databases: database page log sequence numbers (LSNs), and database file identification strings. + +Because database pages contain references to the database environment log records (LSNs), databases cannot be copied or moved from one transactional database environment to another without first clearing the LSNs. Note that this is not a concern for non-transactional database environments and applications, and can be ignored if the database is not being used transactionally. Specifically, databases created and written non-transactionally (for example, as part of a bulk load procedure), can be copied or moved into a transactional database environment without resetting the LSNs. The database's LSNs may be reset in one of three ways: the application can call the DB_ENV->lsn_reset() method to reset the LSNs in place, or a system administrator can reset the LSNs in place using the **-r** option to the db_load utility, or by dumping and reloading the database (using the db_dump utility and the db_load utility). + +Because system file identification information (for example, filenames, device and inode numbers, volume and file IDs, and so on) are not necessarily unique or maintained across system reboots, each Berkeley DB database file contains a unique 20-byte file identification bytestring. When multiple processes or threads open the same database file in Berkeley DB, it is this bytestring that is used to ensure the same underlying pages are updated in the database environment cache, no matter which Berkeley DB handle is used for the operation. + +The database file identification string is not a concern when moving databases, and databases may be moved or renamed without resetting the identification string. However, when copying a database, you must ensure there are never two databases with the same file identification bytestring in the same cache at the same time. Copying databases is further complicated because Berkeley DB caches do not discard cached database pages when database handles are closed. Cached pages are only discarded when the database is removed by calling the DB_ENV->remove() or DB->remove() methods. + +Before physically copying a database file, first ensure that all modified pages have been written from the cache to the backing database file. This is done using the DB->sync() or DB->close() methods. + +Before using a copy of a database file in a database environment, you must ensure that all pages from any other database with the same bytestring have been removed from the memory pool cache. If the environment in which you will open the copy of the database has pages from files with identical bytestrings to the copied database, there are a few possible solutions: + +1. Remove the environment, either using system utilities or by calling the DB_ENV->remove() method. Obviously, this will not allow you to access both the original database and the copy of the database at the same time. +2. Create a new file that will have a new bytestring. The simplest way to create a new file that will have a new bytestring is to call the db_dump utility to dump out the contents of the database and then use the db_load utility to load the dumped output into a new file. This allows you to access both the original and copy of the database at the same time. +3. If your database is too large to be dumped and reloaded, you can copy the database by other means, and then reset the bytestring in the copied database to a new bytestring. There are two ways to reset the bytestring in the copy: the application can call the DB_ENV->fileid_reset() method, or a system administrator can use the **-r** option to the db_load utility. This allows you to access both the original and copy of the database at the same time. diff --git a/docs-src/guides/programmer_reference/program_environ.md b/docs-src/guides/programmer_reference/program_environ.md new file mode 100644 index 000000000..0fd805b52 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_environ.md @@ -0,0 +1,14 @@ +--- +title: "Environment variables" +api-name: "Environment variables" +source: docs/programmer_reference/program_environ.html +--- +## Environment variables + +The Berkeley DB library uses the following environment variables: + +DB_HOME +If the environment variable DB_HOME is set, it is used as part of File naming. Note: For the DB_HOME variable to take effect, either the DB_USE_ENVIRON or DB_USE_ENVIRON_ROOT flags must be specified to DB_ENV->open(). + +TMPDIR, TEMP, TMP, TempFolder +The TMPDIR, TEMP, TMP, and TempFolder environment variables are all checked as locations in which to create temporary files. See DB_ENV->set_tmp_dir() for more information. diff --git a/docs-src/guides/programmer_reference/program_errorret.md b/docs-src/guides/programmer_reference/program_errorret.md new file mode 100644 index 000000000..446444fc8 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_errorret.md @@ -0,0 +1,57 @@ +--- +title: "Error returns to applications" +api-name: "Error returns to applications" +source: docs/programmer_reference/program_errorret.html +--- +## Error returns to applications + +Except for the historic dbm, ndbm and hsearch interfaces, Berkeley DB does not use the global variable `errno` to return error values. The return values for all Berkeley DB functions are grouped into the following three categories: + +0 +A return value of 0 indicates that the operation was successful. + +\> 0 +A return value that is greater than 0 indicates that there was a system error. The **errno** value returned by the system is returned by the function; for example, when a Berkeley DB function is unable to allocate memory, the return value from the function will be ENOMEM. + +\< 0 +A return value that is less than 0 indicates a condition that was not a system failure, but was not an unqualified success, either. For example, a routine to retrieve a key/data pair from the database may return DB_NOTFOUND when the key/data pair does not appear in the database; as opposed to the value of 0, which would be returned if the key/data pair were found in the database. + +All values returned by Berkeley DB functions are less than 0 in order to avoid conflict with possible values of **errno**. Specifically, Berkeley DB reserves all values from -30,800 to -30,999 to itself as possible error values. There are a few Berkeley DB interfaces where it is possible for an application function to be called by a Berkeley DB function and subsequently fail with an application-specific return. Such failure returns will be passed back to the function that originally called a Berkeley DB interface. To avoid ambiguity about the cause of the error, error values separate from the Berkeley DB error name space should be used. + +Although possible error returns are specified by each individual function's manual page, there are a few error returns that deserve general mention: + +**DB_NOTFOUND and DB_KEYEMPTY** + +There are two special return values that are similar in meaning and that are returned in similar situations, and therefore might be confused: DB_NOTFOUND and DB_KEYEMPTY. + +The DB_NOTFOUND error return indicates that the requested key/data pair did not exist in the database or that start-of- or end-of-file has been reached by a cursor. + +The DB_KEYEMPTY error return indicates that the requested key/data pair logically exists but was never explicitly created by the application (the Recno and Queue access methods will automatically create key/data pairs under some circumstances; see DB->open() for more information), or that the requested key/data pair was deleted and never re-created. In addition, the Queue access method will return DB_KEYEMPTY for records that were created as part of a transaction that was later aborted and never re-created. + +**DB_KEYEXIST** + +The DB_KEYEXIST error return indicates the DB_NOOVERWRITE option was specified when inserting a key/data pair into the database and the key already exists in the database, or the DB_NODUPDATA option was specified and the key/data pair already exists in the data. + +**DB_LOCK_DEADLOCK** + +When multiple threads of control are modifying the database, there is normally the potential for deadlock. In Berkeley DB, deadlock is signified by an error return from the Berkeley DB function of the value DB_LOCK_DEADLOCK. Whenever a Berkeley DB function returns DB_LOCK_DEADLOCK, the enclosing transaction should be aborted. + +Any Berkeley DB function that attempts to acquire locks can potentially return DB_LOCK_DEADLOCK. Practically speaking, the safest way to deal with applications that can deadlock is to anticipate a DB_LOCK_DEADLOCK return from any DB or DBC handle method call, or any DB_ENV handle method call that references a database, including the database's backing physical file. + +**DB_LOCK_NOTGRANTED** + +If a lock is requested from the DB_ENV->lock_get() or DB_ENV->lock_vec() methods with the DB_LOCK_NOWAIT flag specified, the method will return DB_LOCK_NOTGRANTED if the lock is not immediately available. + +If the DB_TIME_NOTGRANTED flag is specified to the DB_ENV->set_flags() method, database calls timing out based on lock or transaction timeout values will return DB_LOCK_NOTGRANTED instead of DB_LOCK_DEADLOCK. + +**DB_RUNRECOVERY** + +There exists a class of errors that Berkeley DB considers fatal to an entire Berkeley DB environment. An example of this type of error is a corrupted database page. The only way to recover from these failures is to have all threads of control exit the Berkeley DB environment, run recovery of the environment, and re-enter Berkeley DB. (It is not strictly necessary that the processes exit, although that is the only way to recover system resources, such as file descriptors and memory, allocated by Berkeley DB.) + +When this type of error is encountered, the error value DB_RUNRECOVERY is returned. This error can be returned by any Berkeley DB interface. Once DB_RUNRECOVERY is returned by any interface, it will be returned from all subsequent Berkeley DB calls made by any threads of control participating in the environment. + +Applications can handle such fatal errors in one of two ways: first, by checking for DB_RUNRECOVERY as part of their normal Berkeley DB error return checking, similarly to DB_LOCK_DEADLOCK or any other error. Alternatively, applications can specify a fatal-error callback function using the DB_ENV->set_event_notify() method. Applications with no cleanup processing of their own should simply exit from the callback function. + +**DB_SECONDARY_BAD** + +The DB_SECONDARY_BAD error is returned if a secondary index has been corrupted. This may be the result of an application operating on related databases without first associating them. diff --git a/docs-src/guides/programmer_reference/program_faq.md b/docs-src/guides/programmer_reference/program_faq.md new file mode 100644 index 000000000..7734e3359 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_faq.md @@ -0,0 +1,20 @@ +--- +title: "Programmer notes FAQ" +api-name: "Programmer notes FAQ" +source: docs/programmer_reference/program_faq.html +--- +## Programmer notes FAQ + +1. **What priorities should threads/tasks executing Berkeley DB functions be given?** + + Tasks executing Berkeley DB functions should have the same, or roughly equivalent, system priorities. For example, it can be dangerous to give tasks of control performing checkpoints a lower priority than tasks of control doing database lookups, and starvation can sometimes result. + +2. **Why isn't the C++ API exception safe?** + + The Berkeley DB C++ API is a thin wrapper around the C API that maps most return values to exceptions, and gives the C++ handles the same lifecycles as their C counterparts. One consequence is that if an exception occurs while a cursor or transaction handle is open, the application must explicitly close the cursor or abort the transaction. + + Applications can be simplified and bugs avoided by creating wrapper classes around DBC and TXN that call the appropriate cleanup method in the wrapper's destructor. By creating an instance of the wrappers on the stack, C++ scoping rules will ensure that the destructor is called before exception handling unrolls the block that contains the wrapper object. + +3. **How do I handle a "pass 4" error when trying to run one of the example Performance Event Monitoring scripts on my Linux system? The library configured with --enable-dtrace and built without error.** + + A Linux installation can have SystemTap support for kernel probe points without including the kernel "utrace" module needed to use userspace probes. Pass 4 errors can occur when this required userspace support is not present. diff --git a/docs-src/guides/programmer_reference/program_mt.md b/docs-src/guides/programmer_reference/program_mt.md new file mode 100644 index 000000000..c545bb1c4 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_mt.md @@ -0,0 +1,34 @@ +--- +title: "Multithreaded applications" +api-name: "Multithreaded applications" +source: docs/programmer_reference/program_mt.html +--- +## Multithreaded applications + +Berkeley DB fully supports multithreaded applications. The Berkeley DB library is not itself multithreaded, and was deliberately architected to not use threads internally because of the portability problems that would introduce. Database environment and database object handles returned from Berkeley DB library functions are free-threaded. No other object handles returned from the Berkeley DB library are free-threaded. The following rules should be observed when using threads to access the Berkeley DB library: + +1. The DB_THREAD flag must be specified to the DB_ENV->open() and DB->open() methods if the Berkeley DB handles returned by those interfaces will be used in the context of more than one thread. Setting the DB_THREAD flag inconsistently may result in database corruption. + + Threading is assumed in the Java API, so no special flags are required; and Berkeley DB functions will always behave as if the DB_THREAD flag was specified. + + Only a single thread may call the DB_ENV->close() or DB->close() methods for a returned environment or database handle. + + No other Berkeley DB handles are free-threaded. + +2. When using the non-cursor Berkeley DB calls to retrieve key/data items (for example, DB->get()), the memory to which the pointer stored into the Dbt refers is valid only until the next call using the DB handle returned by DB->open(). This includes **any** use of the returned DB handle, including by another thread within the process. + + For this reason, if the DB_THREAD handle was specified to the DB->open() method, either DB_DBT_MALLOC, DB_DBT_REALLOC or DB_DBT_USERMEM must be specified in the DBT when performing any non-cursor key or data retrieval. + +3. Cursors may not span transactions. Each cursor must be allocated and deallocated within the same transaction. + + Transactions and cursors may span threads, but only serially, that is, the application must serialize access to the TXN and DBC handles. In the case of nested transactions, since all child transactions are part of the same parent transaction, they must observe the same constraints. That is, children may execute in different threads only if each child executes serially. + +4. User-level synchronization mutexes must have been implemented for the compiler/architecture combination. Attempting to specify the DB_THREAD flag will fail if fast mutexes are not available. + + If blocking mutexes are available (for example POSIX pthreads), they will be used. Otherwise, the Berkeley DB library will make a system call to pause for some amount of time when it is necessary to wait on a lock. This may not be optimal, especially in a thread-only environment, in which it is usually more efficient to explicitly yield the processor to another thread. + + It is possible to specify a yield function on an per-application basis. See db_env_set_func_yield for more information. + + It is possible to specify the number of attempts that will be made to acquire the mutex before waiting. See DB_ENV->mutex_set_tas_spins() for more information. + +When creating multiple databases in a single physical file, multithreaded programs may have additional requirements. For more information, see Opening multiple databases in a single file diff --git a/docs-src/guides/programmer_reference/program_namespace.md b/docs-src/guides/programmer_reference/program_namespace.md new file mode 100644 index 000000000..e6ad0f6aa --- /dev/null +++ b/docs-src/guides/programmer_reference/program_namespace.md @@ -0,0 +1,26 @@ +--- +title: "Name spaces" +api-name: "Name spaces" +source: docs/programmer_reference/program_namespace.html +--- +## Name spaces + + [C Language Name Space](program_namespace.md#idp52962960) + + [Filesystem Name Space](program_namespace.md#idp53001824) + +### C Language Name Space + +The Berkeley DB library is careful to avoid C language programmer name spaces, but there are a few potential areas for concern, mostly in the Berkeley DB include file db.h. The db.h include file defines a number of types and strings. Where possible, all of these types and strings are prefixed with "DB\_" or "db\_". There are a few notable exceptions. + +The Berkeley DB library uses a macro named "\_\_P" to configure for systems that do not provide ANSI C function prototypes. This could potentially collide with other systems using a "\_\_P" macro for similar or different purposes. + +The Berkeley DB library needs information about specifically sized types for each architecture. If they are not provided by the system, they are typedef'd in the db.h include file. The types that may be typedef'd by db.h include the following: u_int8_t, int16_t, u_int16_t, int32_t, u_int32_t, u_char, u_short, u_int, and u_long. + +The Berkeley DB library declares a few external routines. All these routines are prefixed with the strings "db\_". All internal Berkeley DB routines are prefixed with the strings "\_\_XXX\_", where "XXX" is the subsystem prefix (for example, "\_\_db_XXX\_" and "\_\_txn_XXX\_"). + +### Filesystem Name Space + +Berkeley DB environments create or use some number of files in environment home directories. These files are named DB_CONFIG, "log.NNNNN" (for example, log.0000000003, where the number of digits following the dot is unspecified), or with the string prefix "\_\_db" (for example, \_\_db.001). Applications should never create files or databases in database environment home directories with names beginning with the characters "log" or "\_\_db". + +In some cases, applications may choose to remove Berkeley DB files as part of their cleanup procedures, using system utilities instead of Berkeley DB interfaces (for example, using the UNIX rm utility instead of the DB_ENV->remove() method). This is not a problem, as long as applications limit themselves to removing only files named "\_\_db.###", where "###" are the digits 0 through 9. Applications should never remove any files named with the prefix "\_\_db" or "log", other than "\_\_db.###" files. diff --git a/docs-src/guides/programmer_reference/program_perfmon.md b/docs-src/guides/programmer_reference/program_perfmon.md new file mode 100644 index 000000000..70e5ca635 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_perfmon.md @@ -0,0 +1,288 @@ +--- +title: "Performance Event Monitoring" +api-name: "Performance Event Monitoring" +source: docs/programmer_reference/program_perfmon.html +--- +## Performance Event Monitoring + + [Using the DTrace Provider](program_perfmon.md#program_perfmon_dtrace) + + [Using SystemTap](program_perfmon.md#program_perfmon_stap) + + [Example Scripts](program_perfmon.md#program_perfmon_examples) + + [Performance Events Reference](program_perfmon.md#program_perfmon_probes) + +The Performance Event Monitoring feature uses Solaris DTrace or Linux SystemTap to "publish" interesting events as they occur inside of Berkeley DB. The operating system utilities **dtrace** or **stap** run scripts which select, analyze, and display events. There is no need to modify the application. Any application which uses that Berkeley DB library can be monitored. For more information about these instrumentation tools refer to the following pages: + +DTrace +http://www.oracle.com/technetwork/systems/dtrace/dtrace/index-jsp-137532.html + +SystemTap +http://sourceware.org/systemtap/ + +Performance Event Monitoring is available for operating systems where DTrace or SystemTap supports static probe points in user applications, such as Solaris 10 and OpenSolaris, some versions of Linux, and Mac OS X 10.6 and later. By including --enable-dtrace to the configuration options, the resulting libraries will include probe points for these event categories: + +- database operations: opening a database or cursor, get, put, delete. +- internal operations regarding disk allocation, transactions, concurrency control, and caching. +- the beginning and ending of waiting periods due to conflicting transactional consistency locks (for example, page locks), mutexes, or shared latches. +- timing dependent code paths which are expected to be infrequent. These could raise concerns if they were to happen too often. + +These probe points are implemented as user-level statically defined traces (USDT's) for DTrace, and static userspace markers for SystemTap. + +To monitor the statistics values as they are updated include the --enable-perfmon-statistics configuration option. This option generates probe points for updates to many of the counters whose values are displayed by the db_stat utility or returned by the various statistics functions. The "cache" example script uses a few of these probe points. + +Performance Event Monitoring is intended to be suitable for production applications. Running Berkeley DB with DTrace or SystemTap support built in has little effect on execution speed until probes are enabled at runtime by the **dtrace** or **stap** programs. + +The list of available events may be displayed by running 'make listprobes' after building the libdb-5.3 shared library. + +### Using the DTrace Provider + +The DTrace probe provider for Berkeley DB is named 'bdb'. A simple dtrace command to monitor all dtrace-enabled Berkeley DB activity in the system is: + +` dtrace -Zn 'bdb*::: { printf("%s", probename); }' ` + +DTrace requires elevated privileges in order to run. On Solaris you can avoid running as root by giving any users who need to run **dtrace** the dtrace_proc or dtrace_user privilege in **/etc/user_attr**. + +DTrace works on both 32 and 64 bit applications. However, when tracing a 32-bit application on a 64-bit processor it might be necessary to pass a "32 bit" option to **dtrace**. Without this option the D language might use a pointer size of 8 bytes, which could cause pointer values (and structures containing them) to be processed incorrectly. Use the `-32` option on Solaris; on Mac OS X use `-arch i386`. + +### Using SystemTap + +SystemTap looks up its static userspace markers in the library name specified in the **stap** script. A simple **stap** command to list the probes of the default Berkeley DB installation is: + +` stap -l 'process("/usr/local/BerkeleyDB.5.3/lib/libdb-5.3.so").mark("*")' ` + +Berkeley DB supports SystemTap version 1.1 or later. Building with userspace marker support requires `sys/sdt.h`, which is often available in the package `systemtap-sdt-devel`. Running **stap** with userspace markers requires that the kernel have "utrace" support; see http://sourceware.org/systemtap/wiki/utrace for more information. + +SystemTap needs elevated privileges in order to run. You can avoid running as root by adding the users who need to run **stap** to the group stapdev. + +### Example Scripts + +Berkeley DB includes several example scripts, in both DTrace and SystemTap versions. The DTrace examples end with a `.d` suffix and are located in `util/dtrace`. The SystemTap examples have a `.stp` suffix and can found be in `util/systemtap`. The Berkeley DB shared library name, including any necessary path, is expected as the first parameter to the SystemTap examples. + +apicalls +This script graphs the count of the main API calls. The result is grouped by thread of the target process. + +apitimes +This script graphs the time spent in the main API calls, grouped by thread. + + apitrace +This script displays the entry to and return from each of the main API calls. + + cache +This script displays overall and per-file buffer cache statistics every *N* (default: 1) seconds for *M* (default: 60) intervals. It prints the number of cache hits, misses, and evictions for each file with any activity during the interval. + + dbdefs +This contains DTrace-compatible declarations of Berkeley DB data structures returned from probe points. There is no Linux equivalent; SystemTap can obtain type information directly from the debugging symbols compiled into the libdb\*-5.3.so shared library. + + locktimes +This script graphs the time spent waiting for DB page locks. The result times in nanoseconds are grouped by filename, pgno, and lock_mode. The optional integer maxcount parameter directs the script to exit once that many page lock waits have been measured. + + locktimesid +This is similar to the locktimes script above, except that it displays the 20 byte file identifier rather than the file name. This can be useful when there are several environments involved, or when database files are recreated during the monitoring period. + + mutex +This script measures mutex wait periods, summarizing the results two ways. + +- The first grouping is by mutex, mode (exclusive or shared), and thread id. +- The second grouping is by the mutex category and mode (exclusive or shared). The mutex categories are the MTX_XXX definitions in `dbinc/mutex.h`. + + showerror +This script displays the application stack when the basic error routines are called. It provides additional information about an error, beyond the string sent to the diagnostic output. + +These examples are designed to trace a single process. They run until interrupted, or the monitored process exits, or the limit as given in an optional 'maximum count' argument has been reached. + +### Performance Events Reference + +The events are described below as if they were functions with ANSI C-style signatures. The values each event provides to DTrace or SystemTap are the arguments to the functions. + + alloc +The alloc class covers the allocation of "on disk" database pages. + +alloc-new (char \*file, char \*db, unsigned pgno, unsigned type, struct \_db_page \*pg, int ret); +An attempt to allocate a database page of type 'type' for database 'db' returned 'ret'. If the allocation succeeded then ret is 0, pgno is the location of the new page, and pg is the address of the new page. Details of the page can be extracted from the pg pointer. + + alloc-free (char \*file, char \*db, unsigned pgno, unsigned ret); +An attempt to free the page 'pgno' of 'db' returned 'ret'. When successful the page is returned to the free list or the file is truncated. + +alloc-btree_split (char \*file, char \*db, unsigned pgno, unsigned parent, unsigned level); +A btree split of pgno in db is being attempted. The parent page number and the level in the btree are also provided. + + db +These DB API calls provide the name of the file and database being accessed. In-memory databases will have a NULL (0) file name address. The db name will be null unless subdatabases are in use. + + db-open (char \*file, char \*db, unsigned flags, uint8_t \*fileid); +The database or file name was opened. The 20 byte unique fileid can be used to keep track of databases as they are created and destroyed. + + db-close (char \*file, char \*db, unsigned flags, uint8_t \*fileid); +The database or file name was closed. + +db-cursor (char \*file, char \*db, unsigned txnid, unsigned flags, uint8_t \*fileid); +An attempt is being made to open a cursor on the database or file. + +db-get (char \*file, char \*db, unsigned txnid, DBT \*key, DBT \*data, unsigned flags); +An attempt is being made to get data from a db. + +db-put (char \*file, char \*db, unsigned txnid, DBT \*key, DBT \*data, unsigned flags); +An attempt is being made to put data to a db. + + db-del (char \*file, char \*db, unsigned txnid, DBT \*key, unsigned flags); +An attempt is being made to delete data from a db. + + lock +The lock class monitors the transactional consistency locks: page, record, and database. It also monitors the non-transactional file handle locks. + + lock-suspend (DBT \*lock, db_lockmode_t lock_mode); +The thread is about to suspend itself because another locker already has a conflicting lock on object 'lock'. The lock DBT's data points to a \_\_db_ilock structure, except for the atypical program which uses application specific locking. + + lock-resume (DBT \*lock, db_lockmode_t lock_mode); +The thread is awakening from a suspend. + + lock-put (struct \_\_sh_dbt \*lock, unsigned flags); +The lock is being freed. + + lock-put_reduce_count (struct \_\_sh_dbt \*lock, unsigned flags); +The lock would have been freed except that its refcount was greater than 1. + +These lock counters are included by --enable-perfmon-statistics. + +lock-deadlock (unsigned st_ndeadlocks, unsigned locker_id, struct \_\_sh_dbt \*lock_obj); +The locker_id's lock request in lock_obj is about to be aborted in order to resolve a deadlock. The lock region's st_ndeadlocks has been incremented. + + lock-nowait_notgranted (unsigned count, DBT \*lock, unsigned locker_id); +A DB_LOCK_NOWAIT lock request by locker_id would have had to wait. The lock regions's st_lock_nowait has been incremented and the request returns DB_LOCK_NOTGRANTED. + + lock-steal (unsigned st_locksteals, unsigned from, unsigned to); +A lock is being stolen from one partition for another one. The 'from' lock partition's st_locksteals has been incremented. + + lock-object_steal (unsigned st_objectsteals, unsigned from, unsigned to); +A lock object is being stolen from one partition for another one. The 'from' lock partition's st_objectsteals has been incremented. + + lock-locktimeout (unsigned st_nlocktimeouts, const DBT \*lock); +A lock wait expired due to the lock request timeout. + + lock-txntimeout (unsigned st_ntxntimeouts, const DBT \*lock); +A lock wait expired due to the transaction's timeout. + + lock-nlockers (unsigned active, unsigned locker_id); +The allocation or deallocation of the locker id changed the number of active locker identifiers. + + lock-maxnlockers (unsigned new_max_active, unsigned locker_id); +The allocation of the locker id set a new maximum number of active locker identifiers. + + mpool +The mpool class monitors the allocation and management of memory, including the cache. + + mpool-read (char \*file, unsigned pgno, struct \_\_bh \*buf); +Read a page from file into buf. + + mpool-write (char \*file, unsigned pgno, struct \_\_bh \*buf); +Write a page from buf to file. + + mpool-env_alloc (unsigned size, unsigned region_id, unsigned reg_type); +This is an attempt to allocate size bytes from region_id. The reg_type is one of the reg_type_t enum values. + + mpool-evict (char \*file, unsigned pgno, struct \_\_bh \*buf); +The page is about to be removed from the cache. + +mpool-alloc_wrap (unsigned alloc_len, int region_id, int wrap_count, int put_counter); +The memory allocator has incremented wrap_count after searching through the entire region without being able to fulfill the request for alloc_len bytes. As wrap_count increases the library makes more effort to allocate space. + +These mpool counters are included by --enable-perfmon-statistics. + + mpool-clean_eviction (unsigned st_ro_evict, unsigned region_id); +The eviction of a clean page from a cache incremented st_ro_evict. + + mpool-dirty_eviction (unsigned st_rw_evict, unsigned region_id); +The eviction of a dirty page from a cache incremented st_rw_evict. The page has already been written out. + + mpool-fail (unsigned failure_count, unsigned alloc_len, unsigned region_id); +An attempt to allocate memory from region_id failed. + + mpool-hash_search (unsigned st_hash_searches, char \*file, unsigned pgno); +A search for pgno of file incremented st_hash_searches. + + mpool-hash_examined (unsigned st_hash_examined, char \*file, unsigned pgno); +A search for pgno of file increased st_hash_examined by the number of hash buckets examined. + + mpool-hash_longest (unsigned st_hash_longest, char \*file, unsigned pgno); +A search for pgno of file set a new maximum st_hash_longest value. + + mpool-map (unsigned st_map, char \*file, unsigned pgno); +A file's st_map count was incremented after a page was mapped into memory. The mapping might have caused disk I/O. + + mpool-hit (unsigned st_cache_hit, char \*file, unsigned pgno); +The hit count was incremented because pgno from file was found in the cache. + + mpool-miss (unsigned st_cache_miss, char \*file, unsigned pgno); +The miss count was incremented because pgno from file was not already present in the cache. + + mpool-page_create (unsigned st_page_create, char \*file, unsigned pgno); +The st_page_create field was incremented because the pgno of file was created in the cache. + + mpool-page_in (unsigned st_page_in, char \*file, unsigned pgno); +The st_page_in field was incremented because the pgno from file was read into the cache. + + mpool-page_out (unsigned st_page_out, char \*file, unsigned pgno); +The st_page_out field was incremented because the pgno from file was written out. + + mutex +The mutex category monitors includes shared latches. The alloc_id value is one of the MTX_XXX definitions from dbinc/mutex.h + +mutex-suspend (unsigned mutex, unsigned excl, unsigned alloc_id, struct \_\_db_mutex_t \*mutexp); +This thread is about to suspend itself because a thread has the mutex or shared latch locked in a mode which conflicts with the this request. + +mutex-resume (unsigned mutex, unsigned excl, unsigned alloc_id, struct \_\_db_mutex_t \*mutexp); +The thread is returning from a suspend and will attempt to obtain the mutex or shared latch again. It might need to suspend again. + +These mutex counters are included by --enable-perfmon-statistics. + + mutex-set_nowait (unsigned mutex_set_nowait, unsigned mutex); +Increment the count of times that the mutex was free when trying to lock it. + + mutex-set_wait (unsigned mutex_set_wait, unsigned mutex); +Increment the count of times that the mutex was busy when trying to lock it. + + mutex-set_rd_nowait (unsigned mutex_set_rd_nowait, unsigned mutex); +Increment the count of times that the shared latch was free when trying to get a shared lock on it. + + mutex-set_rd_wait (unsigned mutex_set_rd_wait, unsigned mutex); +Increment the count of times that the shared latch was already exclusively latched when trying to get a shared lock on it. + + mutex-hybrid_wait (unsigned hybrid_wait, unsigned mutex); +Increment the count of times that a hybrid mutex had to block on its condition variable. n a busy system this might happen several times before the corresponding hybrid_wakeup. + + mutex-hybrid_wakeup (unsigned hybrid_wakeup, unsigned mutex); +Increment the count of times that a hybrid mutex finished one or more waits for its condition variable. + + txn +The txn category covers the basic transaction operations. + + txn-begin (unsigned txnid, unsigned flags); +A transaction was successfully begun. + + txn-commit (unsigned txnid, unsigned flags); +A transaction is starting to commit. + + txn-prepare (unsigned txnid, uint8_t \*gid); +The transaction is starting to prepare, flushing the log so that a future commit can be guaranteed to succeed. The global identifier field is 128 bytes long. + + txn-abort (unsigned txnid); +The transaction is about to abort. + +These txn counters are included by --enable-perfmon-statistics. + + txn-nbegins (unsigned st_nbegins, unsigned txnid); +Beginning the transaction incremented st_nbegins. + + txn-naborts (unsigned st_nbegins, unsigned txnid); +Aborting the transaction incremented st_naborts. + + txn-ncommits (unsigned st_ncommits, unsigned txnid); +Committing the transaction incremented st_ncommits. + + txn-nactive (unsigned st_nactive, unsigned txnid); +Beginning or ending the transaction updated the number of active transactions. + + txn-maxnactive (unsigned st_maxnactive, unsigned txnid); +The creation of the transaction set a new maximum number of active transactions. diff --git a/docs-src/guides/programmer_reference/program_ram.md b/docs-src/guides/programmer_reference/program_ram.md new file mode 100644 index 000000000..71b4f0879 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_ram.md @@ -0,0 +1,52 @@ +--- +title: "Memory-only or Flash configurations" +api-name: "Memory-only or Flash configurations" +source: docs/programmer_reference/program_ram.html +--- +## Memory-only or Flash configurations + +Berkeley DB supports a variety of memory-based configurations for systems where filesystem space is either limited in availability or entirely replaced by some combination of memory and Flash. In addition, Berkeley DB can be configured to minimize writes to the filesystem when the filesystem is backed by Flash memory. + +There are four parts of the Berkeley DB database environment normally written to the filesystem: the database environment region files, the database files, the database environment log files and the replication internal files. Each of these items can be configured to live in memory rather than in the filesystem: + +The database environment region files: +Each of the Berkeley DB subsystems in a database environment is described by one or more regions, or chunks of memory. The regions contain all of the per-process and per-thread shared information (including mutexes), that comprise a Berkeley DB environment. By default, these regions are backed by the filesystem. In situations where filesystem backed regions aren't optimal, applications can create memory-only database environments in two different types of memory: either in the application's heap memory or in system shared memory. + +To create the database environment in heap memory, specify the DB_PRIVATE flag to the DB_ENV->open() method. Note that database environments created in heap memory are only accessible to the threads of a single process, however. + +To create the database environment in system shared memory, specify the DB_SYSTEM_MEM flag to the DB_ENV->open() method. Database environments created in system memory are accessible to multiple processes, but note that database environments created in system shared memory do create a small (roughly 8 byte) file in the filesystem, read by the processes to identify which system shared memory segments to use. + +For more information, see Shared memory regions. + +The database files: +By default, databases are periodically flushed from the Berkeley DB memory cache to backing physical files in the filesystem. To keep databases from being written to backing physical files, pass the DB_MPOOL_NOFILE flag to the DB_MPOOLFILE->set_flags() method. This flag implies the application's databases must fit entirely in the Berkeley DB cache, of course. To avoid a database file growing to consume the entire cache, applications can limit the size of individual databases in the cache by calling the DB_MPOOLFILE->set_maxsize() method. + +The database environment log files: +If a database environment is not intended to be transactionally recoverable after application or system failure (that is, if it will not exhibit the transactional attribute of "durability"), applications should not configure the database environment for logging or transactions, in which case no log files will be created. If the database environment is intended to be durable, log files must either be written to stable storage and recovered after application or system failure, or they must be replicated to other systems. + +In applications running on systems without any form of stable storage, durability must be accomplished through replication. In this case, database environments should be configured to maintain database logs in memory, rather than in the filesystem, by specifying the DB_LOG_IN_MEMORY flag to the DB_ENV->log_set_config() method. + +The replication internal files: +By default, Berkeley DB replication stores a small amount of internal data in the filesystem. To store this replication internal data in memory only and not in the filesystem, specify the DB_REP_CONF_INMEM flag to the DB_ENV->rep_set_config() method before opening the database environment. + +In systems where the filesystem is backed by Flash memory, the number of times the Flash memory is written may be a concern. Each of the four parts of the Berkeley DB database environment normally written to the filesystem can be configured to minimize the number of times the filesystem is written: + +The database environment region files: +On a Flash-based filesystem, the database environment should be placed in heap or system memory, as described previously. + +The database files: +The Berkeley DB library maintains a cache of database pages. The database pages are only written to backing physical files when the application checkpoints the database environment with the DB_ENV->txn_checkpoint() method, when database handles are closed with the DB->close() method, or when the application explicitly flushes the cache with the DB->sync() or DB_ENV->memp_sync() methods. + +To avoid unnecessary writes of Flash memory due to checkpoints, applications should decrease the frequency of their checkpoints. This is especially important in applications which repeatedly modify a specific database page, as repeatedly writing a database page to the backing physical file will repeatedly update the same blocks of the filesystem. + +To avoid unnecessary writes of the filesystem due to closing a database handle, applications should specify the DB_NOSYNC flag to the DB->close() method. + +To avoid unnecessary writes of the filesystem due to flushing the cache, applications should not explicitly flush the cache under normal conditions – flushing the cache is rarely if ever needed in a normally-running application. + +The database environment log files: +The Berkeley DB log files do not repeatedly overwrite the same blocks of the filesystem as the Berkeley DB log files are not implemented as a circular buffer and log files are not re-used. For this reason, the Berkeley DB log files should not cause any difficulties for Flash memory configurations. + +However, as Berkeley DB does not write log records in filesystem block sized pieces, it is probable that sequential transaction commits (each of which flush the log file to the backing filesystem), will write a block of Flash memory twice, as the last log record from the first commit will write the same block of Flash memory as the first log record from the second commit. Applications not requiring absolute durability should specify the DB_TXN_WRITE_NOSYNC or DB_TXN_NOSYNC flags to the DB_ENV->set_flags() method to avoid this overwrite of a block of Flash memory. + +The replication internal files: +On a Flash-based filesystem, the replication internal data should be stored in memory only, as described previously. diff --git a/docs-src/guides/programmer_reference/program_runtime.md b/docs-src/guides/programmer_reference/program_runtime.md new file mode 100644 index 000000000..3b07cd994 --- /dev/null +++ b/docs-src/guides/programmer_reference/program_runtime.md @@ -0,0 +1,54 @@ +--- +title: "Run-time configuration" +api-name: "Run-time configuration" +source: docs/programmer_reference/program_runtime.html +--- +## Run-time configuration + +It is possible for applications to configure Berkeley DB at run-time to redirect Berkeley DB library and system calls to alternate interfaces. For example, an application might want Berkeley DB to call debugging memory allocation routines rather than the standard C library interfaces. The following interfaces support this functionality: + +- db_env_set_func_close + +- db_env_set_func_dirfree + +- db_env_set_func_dirlist + +- db_env_set_func_exists + +- db_env_set_func_file_map + +- db_env_set_func_free + +- db_env_set_func_fsync + +- db_env_set_func_ftruncate + +- db_env_set_func_ioinfo + +- db_env_set_func_malloc + +- db_env_set_func_open + +- db_env_set_func_pread + +- db_env_set_func_pwrite + +- db_env_set_func_read + +- db_env_set_func_realloc + +- db_env_set_func_region_map + +- db_env_set_func_rename + +- db_env_set_func_seek + +- db_env_set_func_unlink + +- db_env_set_func_write + +- db_env_set_func_yield + +These interfaces are available only on POSIX platforms and from the Berkeley DB C language API. + +A not-uncommon problem for applications is the new API in Solaris 2.6 for manipulating large files. Because this API was not part of Solaris 2.5, it is difficult to create a single binary that takes advantage of the large file functionality in Solaris 2.6, but still runs on Solaris 2.5. Example code that supports this is included in the Berkeley DB distribution, however, the example code was written using previous versions of the Berkeley DB APIs, and is only useful as an example. diff --git a/docs-src/guides/programmer_reference/program_scope.md b/docs-src/guides/programmer_reference/program_scope.md new file mode 100644 index 000000000..77d4c2b1a --- /dev/null +++ b/docs-src/guides/programmer_reference/program_scope.md @@ -0,0 +1,28 @@ +--- +title: "Berkeley DB handles" +api-name: "Berkeley DB handles" +source: docs/programmer_reference/program_scope.html +--- +## Berkeley DB handles + +The Berkeley DB library has a number of object handles. The following table lists those handles, their scope, and whether they are free-threaded (that is, whether multiple threads within a process can share them). + + DB_ENV +The DB_ENV handle, created by the db_env_create() method, refers to a Berkeley DB database environment — a collection of Berkeley DB subsystems, log files and databases. DB_ENV handles are free-threaded if the DB_THREAD flag is specified to the DB_ENV->open() method when the environment is opened. The handle should not be closed while any other handle remains open that is using it as a reference (for example, DB, TXN). Once either the DB_ENV->close() or DB_ENV->remove() methods are called, the handle may not be accessed again, regardless of the method's return. + + TXN +The TXN handle, created by the DB_ENV->txn_begin() method, refers to a single transaction. The handle is not free-threaded. Transactions may span threads, but only serially, that is, the application must serialize access to the TXN handles. In the case of nested transactions, since all child transactions are part of the same parent transaction, they must observe the same constraints. That is, children may execute in different threads only if each child executes serially. + +Once the DB_TXN->abort() or DB_TXN->commit() methods are called, the handle may not be accessed again, regardless of the method's return. In addition, parent transactions may not issue any Berkeley DB operations while they have active child transactions (child transactions that have not yet been committed or aborted) except for DB_ENV->txn_begin(), DB_TXN->abort() and DB_TXN->commit(). + + DB_LOGC +The DB_LOGC handle refers to a cursor into the log files. The handle is not free-threaded. Once the DB_LOGC->close() method is called, the handle may not be accessed again, regardless of the method's return. + + DB_MPOOLFILE +The DB_MPOOLFILE handle refers to an open file in the shared memory buffer pool of the database environment. The handle is not free-threaded. Once the DB_MPOOLFILE->close() method is called, the handle may not be accessed again, regardless of the method's return. + + DB +The DB handle, created by the db_create() method, refers to a single Berkeley DB database, which may or may not be part of a database environment. DB handles are free-threaded if the DB_THREAD flag is specified to the DB->open() method when the database is opened or if the database environment in which the database is opened is free-threaded. The handle should not be closed while any other handle that refers to the database is in use; for example, database handles should be left open while cursor handles into the database remain open, or transactions that include operations on the database have not yet been committed or aborted. Once the DB->close(), DB->remove() or DB->rename() methods are called, the handle may not be accessed again, regardless of the method's return. + + DBC +The DBC handle refers to a cursor into a Berkeley DB database. The handle is not free-threaded. Cursors may span threads, but only serially, that is, the application must serialize access to the DBC handles. If the cursor is to be used to perform operations on behalf of a transaction, the cursor must be opened and closed within the context of that single transaction. Once DBC->close() has been called, the handle may not be accessed again, regardless of the method's return. diff --git a/docs-src/guides/programmer_reference/refs.md b/docs-src/guides/programmer_reference/refs.md new file mode 100644 index 000000000..1711245d6 --- /dev/null +++ b/docs-src/guides/programmer_reference/refs.md @@ -0,0 +1,80 @@ +--- +title: "Chapter 24.  Additional References" +api-name: "Chapter 24.  Additional References" +source: docs/programmer_reference/refs.html +--- +## Chapter 24.  Additional References + +**Table of Contents** + + [Additional references](refs.md#refs_refs) + + [Technical Papers on Berkeley DB](refs.md#idp53369464) + + [Background on Berkeley DB Features](refs.md#idp53449960) + + [Database Systems Theory](refs.md#idp53443200) + +## Additional references + + [Technical Papers on Berkeley DB](refs.md#idp53369464) + + [Background on Berkeley DB Features](refs.md#idp53449960) + + [Database Systems Theory](refs.md#idp53443200) + +For more information on Berkeley DB or on database systems theory in general, we recommend the following sources: + +### Technical Papers on Berkeley DB + +These papers have appeared in refereed conference proceedings, and are subject to copyrights held by the conference organizers and the authors of the papers. Oracle makes them available here as a courtesy with the permission of the copyright holders. + + *Berkeley DB* (Postscript) +Michael Olson, Keith Bostic, and Margo Seltzer, Proceedings of the 1999 Summer Usenix Technical Conference, Monterey, California, June 1999. This paper describes recent commercial releases of Berkeley DB, its most important features, the history of the software, and Sleepycat Software's Open Source licensing policies. + + *Challenges in Embedded Database System Administration* (HTML) +Margo Seltzer and Michael Olson, First Workshop on Embedded Systems, Cambridge, Massachusetts, March 1999. This paper describes the challenges that face embedded systems developers, and how Berkeley DB has been designed to address them. + + *LIBTP: Portable Modular Transactions for UNIX* (Postscript) +Margo Seltzer and Michael Olson, USENIX Conference Proceedings, Winter 1992. This paper describes an early prototype of the transactional system for Berkeley DB. + + *A New Hashing Package for UNIX* (Postscript) +Margo Seltzer and Oz Yigit, USENIX Conference Proceedings, Winter 1991. This paper describes the Extended Linear Hashing techniques used by Berkeley DB. + +### Background on Berkeley DB Features + +These papers, although not specific to Berkeley DB, give a good overview of the way different Berkeley DB features were implemented. + + *Operating System Support for Database Management* +Michael Stonebraker, Communications of the ACM 24(7), 1981, pp. 412-418. + + *Dynamic Hash Tables* +Per-Ake Larson, Communications of the ACM, April 1988. + + *Linear Hashing: A New Tool for File and Table Addressing* +Witold Litwin, Proceedings of the 6th International Conference on Very Large Databases (VLDB), 1980 + + *The Ubiquitous B-tree* +Douglas Comer, ACM Comput. Surv. 11, 2 (June 1979), pp. 121-138. + + *Prefix B-trees* +Bayer and Unterauer, ACM Transactions on Database Systems, Vol. 2, 1 (March 1977), pp. 11-26. + + *The Art of Computer Programming Vol. 3: Sorting and Searching* +D.E. Knuth, 1968, pp. 471-480. + + *Document Processing in a Relational Database System* +Michael Stonebraker, Heidi Stettner, Joseph Kalash, Antonin Guttman, Nadene Lynn, Memorandum No. UCB/ERL M82/32, May 1982. + +### Database Systems Theory + +These publications are standard reference works on the design and implementation of database systems. Berkeley DB uses many of the ideas they describe. + + *Transaction Processing Concepts and Techniques* +by Jim Gray and Andreas Reuter, Morgan Kaufmann Publishers. We recommend chapters 1, 4 (skip 4.6, 4.7, 4.9, 4.10 and 4.11), 7, 9, 10.3, and 10.4. + + *An Introduction to Database Systems, Volume 1* +by C.J. Date, Addison Wesley Longman Publishers. In the 5th Edition, we recommend chapters 1, 2, 3, 16 and 17. + + *Concurrency Control and Recovery in Database Systems* +by Bernstein, Goodman, Hadzilaco. Currently out of print, but available from http://research.microsoft.com/en-us/people/philbe/ccontrol.aspx. diff --git a/docs-src/guides/programmer_reference/rep.md b/docs-src/guides/programmer_reference/rep.md new file mode 100644 index 000000000..feb8c35e7 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep.md @@ -0,0 +1,139 @@ +--- +title: "Chapter 12.  Berkeley DB Replication" +api-name: "Chapter 12.  Berkeley DB Replication" +source: docs/programmer_reference/rep.html +--- +## Chapter 12.  Berkeley DB Replication + +**Table of Contents** + + [Replication introduction](rep.md#rep_intro) + + [Replication environment IDs](rep_id.md) + + [Replication environment priorities](rep_pri.md) + + [Building replicated applications](rep_app.md) + + [Replication Manager methods](rep_mgr_meth.md) + + [Base API Methods](rep_base_meth.md) + + [Building the communications infrastructure](rep_comm.md) + + [Connecting to a new site](rep_newsite.md) + + [Managing Replication Manager Group Membership](group_membership.md) + + [Adding Sites to a Replication Group](group_membership.md#group_mem_add) + + [Removing Sites from a Replication Group](group_membership.md#group_mem_remove) + + [Primordial Startups](group_membership.md#group_mem_primordialstartup) + + [Upgrading Groups](group_membership.md#group_mem_upgrade) + + [Managing Replication Files](rep_filename.md) + + [Running Replication Manager in multiple processes](rep_mgrmulti.md) + + [One replication process and multiple subordinate processes](rep_mgrmulti.md#idp52420616) + + [Persistence of local site network address configuration](rep_mgrmulti.md#idp52417008) + + [Programming considerations](rep_mgrmulti.md#idp52400144) + + [Handling failure](rep_mgrmulti.md#idp52414488) + + [Other miscellaneous rules](rep_mgrmulti.md#idp52412256) + + [Running Replication using the db_replicate Utility](rep_replicate.md) + + [One Replication Process and Multiple Subordinate Processes](rep_replicate.md#idp52430544) + + [Common Use Case](rep_replicate.md#idp52447760) + + [Avoiding Rollback](rep_replicate.md#idp52457840) + + [When to Consider an Integrated HA Application](rep_replicate.md#idp52462952) + + [Choosing a Replication Manager Ack Policy](rep_mgr_ack.md) + + [Elections](rep_elect.md) + + [Synchronizing with a master](rep_mastersync.md) + + [Delaying client synchronization](rep_mastersync.md#rep_delay_sync) + + [Client-to-client synchronization](rep_mastersync.md#rep_c2c_sync) + + [Blocked client operations](rep_mastersync.md#idp52488504) + + [Clients too far out-of-date to synchronize](rep_mastersync.md#idp52510624) + + [Initializing a new site](rep_init.md) + + [Bulk transfer](rep_bulk.md) + + [Transactional guarantees](rep_trans.md) + + [Master Leases](rep_lease.md) + + [Changing Group Size](rep_lease.md#masterlease_change_groupsize) + + [Read your writes consistency](rep_ryw.md) + + [Getting a token](rep_ryw.md#gettoken) + + [Token handling](rep_ryw.md#tokenhandling) + + [Using a token to check or wait for a transaction](rep_ryw.md#usingtoken) + + [Clock Skew](rep_clock_skew.md) + + [Using Replication Manager message channels](repmgr_channels.md) + + [DB_CHANNEL](repmgr_channels.md#dbchannel_class) + + [Sending messages over a message channel](repmgr_channels.md#dbchannel_send) + + [Receiving messages](repmgr_channels.md#dbchannel_receive) + + [Special considerations for two-site replication groups](rep_twosite.md) + + [Network partitions](rep_partition.md) + + [Replication FAQ](rep_faq.md) + + [Ex_rep: a replication example](rep_ex.md) + + [Ex_rep_base: a TCP/IP based communication infrastructure](rep_ex_comm.md) + + [Ex_rep_base: putting it all together](rep_ex_rq.md) + + [Ex_rep_chan: a Replication Manager channel example](rep_ex_chan.md) + +## Replication introduction + +Berkeley DB includes support for building highly available applications based on replication. Berkeley DB replication groups consist of some number of independently configured database environments. There is a single *master* database environment and one or more *client* database environments. Master environments support both database reads and writes; client environments support only database reads. If the master environment fails, applications may upgrade a client to be the new master. The database environments might be on separate computers, on separate hardware partitions in a non-uniform memory access (NUMA) system, or on separate disks in a single server. As always with Berkeley DB environments, any number of concurrent processes or threads may access a database environment. In the case of a master environment, any number of threads of control may read and write the environment, and in the case of a client environment, any number of threads of control may read the environment. + +Applications may be written to provide various degrees of consistency between the master and clients. The system can be run synchronously such that replicas are guaranteed to be up-to-date with all committed transactions, but doing so may incur a significant performance penalty. Higher performance solutions sacrifice total consistency, allowing the clients to be out of date for an application-controlled amount of time. + +There are two ways to build replicated applications. The simpler way is to use the Berkeley DB Replication Manager. The Replication Manager provides a standard communications infrastructure, and it creates and manages the background threads needed for processing replication messages. + +The Replication Manager implementation is based on TCP/IP sockets, and uses POSIX 1003.1 style networking and thread support. (On Windows systems, it uses standard Windows thread support.) As a result, it is not as portable as the rest of the Berkeley DB library itself. + +The alternative is to use the lower-level replication "Base APIs". This approach affords more flexibility, but requires the application to provide some critical components: + +1. A communication infrastructure. Applications may use whatever wire protocol is appropriate for their application (for example, RPC, TCP/IP, UDP, VI or message-passing over the backplane). +2. The application is responsible for naming. Berkeley DB refers to the members of a replication group using an application-provided ID, and applications must map that ID to a particular database environment or communication channel. +3. The application is responsible for monitoring the status of the master and clients, and identifying any unavailable database environments. +4. The application must provide whatever security policies are needed. For example, the application may choose to encrypt data, use a secure socket layer, or do nothing at all. The level of security is left to the sole discretion of the application. + +(Note that Replication Manager does not provide wire security for replication messages.) + +The following pages present various programming considerations, many of which are directly relevant only for Base API applications. However, even when using Replication Manager it is important to understand the concepts. + +Finally, the Berkeley DB replication implementation has one other additional feature to increase application reliability. Replication in Berkeley DB is implemented to perform database updates using a different code path than the standard ones. This means operations that manage to crash the replication master due to a software bug will not necessarily also crash replication clients. + +For more information on the replication manager operations, see the Replication and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/rep_app.md b/docs-src/guides/programmer_reference/rep_app.md new file mode 100644 index 000000000..dc2345e91 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_app.md @@ -0,0 +1,34 @@ +--- +title: "Building replicated applications" +api-name: "Building replicated applications" +source: docs/programmer_reference/rep_app.html +--- +## Building replicated applications + +The simplest way to build a replicated Berkeley DB application is to first build (and debug!) the transactional version of the same application. Then, add a thin replication layer: application initialization must be changed and the application's communication infrastructure must be added. + +The application initialization changes are relatively simple. Replication Manager provides a communication infrastructure, but in order to use the replication Base APIs you must provide your own. + +For implementation reasons, all replicated databases must reside in the data directories set from DB_ENV->set_data_dir() (or in the default environment home directory, if not using DB_ENV->set_data_dir()), rather than in a subdirectory below the specified directory. Care must be taken in applications using relative pathnames and changing working directories after opening the environment. In such applications the replication initialization code may not be able to locate the databases, and applications that change their working directories may need to use absolute pathnames. + +During application initialization, the application performs three additional tasks: first, it must specify the DB_INIT_REP flag when opening its database environment and additionally, a Replication Manager application must also specify the DB_THREAD flag; second, it must provide Berkeley DB information about its communications infrastructure; and third, it must start the Berkeley DB replication system. Generally, a replicated application will do normal Berkeley DB recovery and configuration, exactly like any other transactional application. + +Replication Manager applications configure the built-in communications infrastructure by calling obtaining a DB_SITE handle, and then using it to configure the local site. It can optionally obtain one or more DB_SITE handles to configure remote sites. Once the environment has been opened, the application starts the replication system by calling the DB_ENV->repmgr_start() method. + +A Base API application calls the DB_ENV->rep_set_transport() method to configure the entry point to its own communications infrastructure, and then calls the DB_ENV->rep_start() method to join or create the replication group. + +When starting the replication system, an application has two choices: it may choose the group master site explicitly, or alternatively it may configure all group members as clients and then call for an election, letting the clients select the master from among themselves. Either is correct, and the choice is entirely up to the application. + +Replication Manager applications make this choice simply by setting the flags parameter to the DB_ENV->repmgr_start() method. + +For a Base API application, the result of calling DB_ENV->rep_start() is usually the discovery of a master, or the declaration of the local environment as the master. If a master has not been discovered after a reasonable amount of time, the application should call DB_ENV->rep_elect() to call for an election. + +Consider a Base API application with multiple processes or multiple environment handles that modify databases in the replicated environment. All modifications must be done on the master environment. The first process to join or create the master environment must call both the DB_ENV->rep_set_transport() and the DB_ENV->rep_start() method. Subsequent replication processes must at least call the DB_ENV->rep_set_transport() method. Those processes may call the DB_ENV->rep_start() method (as long as they use the same master or client argument). If multiple processes are modifying the master environment there must be a unified communication infrastructure such that messages arriving at clients have a single master ID. Additionally the application must be structured so that all incoming messages are able to be processed by a single DB_ENV handle. + +Note that not all processes running in replicated environments need to call DB_ENV->repmgr_start(), DB_ENV->rep_set_transport() or DB_ENV->rep_start(). Read-only processes running in a master environment do not need to be configured for replication in any way. Processes running in a client environment are read-only by definition, and so do not need to be configured for replication either (although, in the case of clients that may become masters, it is usually simplest to configure for replication on process startup rather than trying to reconfigure when the client becomes a master). Obviously, at least one thread of control on each client must be configured for replication as messages must be passed between the master and the client. + +Any site in a replication group may have its own private transactional databases in the environment as well. A site may create a local database by specifying the DB_TXN_NOT_DURABLE flag to the DB->set_flags() method. The application must never create a private database with the same name as a database replicated across the entire environment as data corruption can result. + +For implementation reasons, Base API applications must process all incoming replication messages using the same DB_ENV handle. It is not required that a single thread of control process all messages, only that all threads of control processing messages use the same handle. + +No additional calls are required to shut down a database environment participating in a replication group. The application should shut down the environment in the usual manner, by calling the DB_ENV->close() method. For Replication Manager applications, this also terminates all network connections and background processing threads. diff --git a/docs-src/guides/programmer_reference/rep_base_meth.md b/docs-src/guides/programmer_reference/rep_base_meth.md new file mode 100644 index 000000000..62255a085 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_base_meth.md @@ -0,0 +1,37 @@ +--- +title: "Base API Methods" +api-name: "Base API Methods" +source: docs/programmer_reference/rep_base_meth.html +--- +## Base API Methods + +Base API applications use the following Berkeley DB methods. + + DB_ENV->rep_set_transport() +The DB_ENV->rep_set_transport() method configures the replication system's communications infrastructure. + + DB_ENV->rep_start() +The DB_ENV->rep_start() method configures (or reconfigures) an existing database environment to be a replication master or client. + + DB_ENV->rep_process_message() +The DB_ENV->rep_process_message() method is used to process incoming messages from other environments in the replication group. For clients, it is responsible for accepting log records and updating the local databases based on messages from the master. For both the master and the clients, it is responsible for handling administrative functions (for example, the protocol for dealing with lost messages), and permitting new clients to join an active replication group. This method should only be called after the replication system's communications infrastructure has been configured via DB_ENV->rep_set_transport(). + + DB_ENV->rep_elect() +The DB_ENV->rep_elect() method causes the replication group to elect a new master; it is called whenever contact with the master is lost and the application wants the remaining sites to select a new master. + + DB_ENV->set_event_notify() +The DB_ENV->set_event_notify() method is needed for applications to discover important replication-related events, such as the result of an election and appointment of a new master. + + DB_ENV->rep_set_priority() +The DB_ENV->rep_set_priority() method configures the local site's priority for the purpose of elections. + + DB_ENV->rep_set_timeout() +This method optionally configures various timeout values. Otherwise default timeout values as specified in DB_ENV->rep_set_timeout() are used. + + DB_ENV->rep_set_limit() +The DB_ENV->rep_set_limit() method imposes an upper bound on the amount of data that will be sent in response to a single call to DB_ENV->rep_process_message(). During client recovery, that is, when a replica site is trying to synchronize with the master, clients may ask the master for a large number of log records. If it is going to harm an application for the master message loop to remain busy for an extended period transmitting records to the replica, then the application will want to use DB_ENV->rep_set_limit() to limit the amount of data the master will send before relinquishing control and accepting other messages. + + DB_ENV->rep_set_request() +This method sets a threshold for the minimum and maximum time that a client waits before requesting retransmission of a missing message. + +In addition to the methods previously described, Base API applications may also call the following methods, as needed: DB_ENV->rep_stat(), DB_ENV->rep_sync() and DB_ENV->rep_set_config(). diff --git a/docs-src/guides/programmer_reference/rep_bulk.md b/docs-src/guides/programmer_reference/rep_bulk.md new file mode 100644 index 000000000..737f885a4 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_bulk.md @@ -0,0 +1,14 @@ +--- +title: "Bulk transfer" +api-name: "Bulk transfer" +source: docs/programmer_reference/rep_bulk.html +--- +## Bulk transfer + +Sites in a replication group may be configured to use bulk transfer by calling the DB_ENV->rep_set_config() method with the DB_REP_CONF_BULK flag. When configured for bulk transfer, sites will accumulate records in a buffer and transfer them to another site in a single network transfer. Configuring bulk transfer makes sense for master sites, of course. Additionally, applications using client-to-client synchronization may find it helpful to configure bulk transfer for client sites as well. + +When a master is generating new log records, or any information request is made of a master, and bulk transfer has been configured, records will accumulate in a bulk buffer. The bulk buffer will be sent to the client if either the buffer is full or if a permanent record (for example, a transaction commit or checkpoint record) is queued for the client. + +When a client is responding to another client's request for information, and bulk transfer has been configured, records will accumulate in a bulk buffer. The bulk buffer will be sent to the client when the buffer is full or when the client's request has been satisfied; no particular type of record will cause the buffer to be sent. + +The size of the bulk buffer itself is internally determined and cannot be configured. However, the overall size of a transfer may be limited using the DB_ENV->rep_set_limit() method. diff --git a/docs-src/guides/programmer_reference/rep_clock_skew.md b/docs-src/guides/programmer_reference/rep_clock_skew.md new file mode 100644 index 000000000..c089dbead --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_clock_skew.md @@ -0,0 +1,18 @@ +--- +title: "Clock Skew" +api-name: "Clock Skew" +source: docs/programmer_reference/rep_clock_skew.html +--- +## Clock Skew + +Since master leases take into account a timeout that is used across all sites in a replication group, leases must also take into account any known skew (or drift) between the clocks on different machines in the group. The guarantees provided by master leases take clock skew into account. Consider a replication group where a client's clock is running faster than the master's clock and the group has a lease timeout of 5 seconds. If clock skew is not taken into account, eventually, the client will believe that 5 seconds have passed faster than the master and that client may then grant its lease to another site. Meanwhile, the master site does not believe 5 seconds have passed because its clock is slower, and it believes it still holds a valid lease grant. For this reason, Berkeley DB compensates for clock skew. + +The master of a group using leases must account for skew in case that site has the slowest clock in the group. This computation avoids the problem of a master site believing a lease grant is valid too long. Clients in a group must account for skew in case they have the fastest clock in the group. This computation avoids the problem of a client site expiring its grant too soon and potentially granting a lease to a different site. Berkeley DB uses a conservative computation and accounts for clock skew on both sides, yielding a double compensation. + +The DB_ENV->rep_set_clockskew() method takes the values for both the fastest and slowest clocks in the entire replication group as parameters. The values passed in must be the same for all sites in the group. If the user knows the maximum clock drift of their sites, then those values can be expressed as a relative percentage. Or, if the user runs an experiment then the actual values can be used. + +For example, suppose the user knows that there is a maximum drift rate of 2% among sites in the group. The user should pass in 102 and 100 for the fast and slow clock values respectively. That is an unusually large value, so suppose, for example, the rate is 0.03% among sites in the group. The user should pass in 10003 and 10000 for the fast and slow clock values. Those values can be used to express the level of precision the user needs. + +An example of an experiment a user can run to help determine skew would be to write a program that started simultaneously on all sites in the group. Suppose, after 1 day (86400 seconds), one site shows 86400 seconds and the other site shows it ran faster and it indicates 86460 seconds has passed. The user can use 86460 and 86400 for their parameters for the fast and slow clock values. + +Since Berkeley DB is using those fast and slow clock values to compute a ratio internally, if the user cannot detect or measure any clock skew, then the same value should be passed in for both parameters, such as 1 and 1. diff --git a/docs-src/guides/programmer_reference/rep_comm.md b/docs-src/guides/programmer_reference/rep_comm.md new file mode 100644 index 000000000..902352337 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_comm.md @@ -0,0 +1,34 @@ +--- +title: "Building the communications infrastructure" +api-name: "Building the communications infrastructure" +source: docs/programmer_reference/rep_comm.html +--- +## Building the communications infrastructure + +Replication Manager provides a built-in communications infrastructure. + +Base API applications must provide their own communications infrastructure, which is typically written with one or more threads of control looping on one or more communication channels, receiving and sending messages. These threads accept messages from remote environments for the local database environment, and accept messages from the local environment for remote environments. Messages from remote environments are passed to the local database environment using the DB_ENV->rep_process_message() method. Messages from the local environment are passed to the application for transmission using the callback function specified to the DB_ENV->rep_set_transport() method. + +Processes establish communication channels by calling the DB_ENV->rep_set_transport() method, regardless of whether they are running in client or server environments. This method specifies the **send** function, a callback function used by Berkeley DB for sending messages to other database environments in the replication group. The **send** function takes an environment ID and two opaque data objects. It is the responsibility of the **send** function to transmit the information in the two data objects to the database environment corresponding to the ID, with the receiving application then calling the DB_ENV->rep_process_message() method to process the message. + +The details of the transport mechanism are left entirely to the application; the only requirement is that the data buffer and size of each of the control and rec DBTs passed to the **send** function on the sending site be faithfully copied and delivered to the receiving site by means of a call to DB_ENV->rep_process_message() with corresponding arguments. Messages that are broadcast (whether by broadcast media or when directed by setting the DB_ENV->rep_set_transport() method's envid parameter DB_EID_BROADCAST), should not be processed by the message sender. In all cases, the application's transport media or software must ensure that DB_ENV->rep_process_message() is never called with a message intended for a different database environment or a broadcast message sent from the same environment on which DB_ENV->rep_process_message() will be called. The DB_ENV->rep_process_message() method is free-threaded; it is safe to deliver any number of messages simultaneously, and from any arbitrary thread or process in the Berkeley DB environment. + +There are a number of informational returns from the DB_ENV->rep_process_message() method: + + DB_REP_DUPMASTER +When DB_ENV->rep_process_message() returns DB_REP_DUPMASTER, it means that another database environment in the replication group also believes itself to be the master. The application should complete all active transactions, close all open database handles, reconfigure itself as a client using the DB_ENV->rep_start() method, and then call for an election by calling the DB_ENV->rep_elect() method. + + DB_REP_HOLDELECTION +When DB_ENV->rep_process_message() returns DB_REP_HOLDELECTION, it means that another database environment in the replication group has called for an election. The application should call the DB_ENV->rep_elect() method. + + DB_REP_IGNORE +When DB_ENV->rep_process_message() returns DB_REP_IGNORE, it means that this message cannot be processed. This is normally an indication that this message is irrelevant to the current replication state, such as a message from an old generation that arrived late. + + DB_REP_ISPERM +When DB_ENV->rep_process_message() returns DB_REP_ISPERM, it means a permanent record, perhaps a message previously returned as DB_REP_NOTPERM, was successfully written to disk. This record may have filled a gap in the log record that allowed additional records to be written. The **ret_lsnp** contains the maximum LSN of the permanent records written. + + DB_REP_NEWSITE +When DB_ENV->rep_process_message() returns DB_REP_NEWSITE, it means that a message from a previously unknown member of the replication group has been received. The application should reconfigure itself as necessary so it is able to send messages to this site. + + DB_REP_NOTPERM +When DB_ENV->rep_process_message() returns DB_REP_NOTPERM, it means a message marked as DB_REP_PERMANENT was processed successfully but was not written to disk. This is normally an indication that one or more messages, which should have arrived before this message, have not yet arrived. This operation will be written to disk when the missing messages arrive. The **ret_lsnp** argument will contain the LSN of this record. The application should take whatever action is deemed necessary to retain its recoverability characteristics. diff --git a/docs-src/guides/programmer_reference/rep_elect.md b/docs-src/guides/programmer_reference/rep_elect.md new file mode 100644 index 000000000..1e49459ca --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_elect.md @@ -0,0 +1,41 @@ +--- +title: "Elections" +api-name: "Elections" +source: docs/programmer_reference/rep_elect.html +--- +## Elections + +Replication Manager automatically conducts elections when necessary, based on configuration information supplied to the DB_ENV->rep_set_priority() method, unless the application turns off automatic elections using the DB_ENV->rep_set_config() method. + +It is the responsibility of a Base API application to initiate elections if desired. It is never dangerous to hold an election, as the Berkeley DB election process ensures there is never more than a single master database environment. Clients should initiate an election whenever they lose contact with the master environment, whenever they see a return of DB_REP_HOLDELECTION from the DB_ENV->rep_process_message() method, or when, for whatever reason, they do not know who the master is. It is not necessary for applications to immediately hold elections when they start, as any existing master will be discovered after calling DB_ENV->rep_start(). If no master has been found after a short wait period, then the application should call for an election. + +For a client to win an election, the replication group must currently have no master, and the client must have the most recent log records. In the case of clients having equivalent log records, the priority of the database environments participating in the election will determine the winner. The application specifies the minimum number of replication group members that must participate in an election for a winner to be declared. We recommend at least ((N/2) + 1) members. If fewer than the simple majority are specified, a warning will be given. + +If an application's policy for what site should win an election can be parameterized in terms of the database environment's information (that is, the number of sites, available log records and a relative priority are all that matter), then Berkeley DB can handle all elections transparently. However, there are cases where the application has more complete knowledge and needs to affect the outcome of elections. For example, applications may choose to handle master selection, explicitly designating master and client sites. Applications in these cases may never need to call for an election. Alternatively, applications may choose to use DB_ENV->rep_elect()'s arguments to force the correct outcome to an election. That is, if an application has three sites, A, B, and C, and after a failure of C determines that A must become the winner, the application can guarantee an election's outcome by specifying priorities appropriately after an election: + +``` c +on A: priority 100, nsites 2 +on B: priority 0, nsites 2 +``` + +It is dangerous to configure more than one master environment using the DB_ENV->rep_start() method, and applications should be careful not to do so. Applications should only configure themselves as the master environment if they are the only possible master, or if they have won an election. An application knows it has won an election when it receives the DB_EVENT_REP_ELECTED event. + +Normally, when a master failure is detected it is desired that an election finish quickly so the application can continue to service updates. Also, participating sites are already up and can participate. However, in the case of restarting a whole group after an administrative shutdown, it is possible that a slower booting site had later logs than any other site. To cover that case, an application would like to give the election more time to ensure all sites have a chance to participate. Since it is intractable for a starting site to determine which case the whole group is in, the use of a long timeout gives all sites a reasonable chance to participate. If an application wanting full participation sets the DB_ENV->rep_elect() method's **nvotes** argument to the number of sites in the group and one site does not reboot, a master can never be elected without manual intervention. + +In those cases, the desired action at a group level is to hold a full election if all sites crashed and a majority election if a subset of sites crashed or rebooted. Since an individual site cannot know which number of votes to require, a mechanism is available to accomplish this using timeouts. By setting a long timeout (perhaps on the order of minutes) using the **DB_REP_FULL_ELECTION_TIMEOUT** flag to the DB_ENV->rep_set_timeout() method, an application can allow Berkeley DB to elect a master even without full participation. Sites may also want to set a normal election timeout for majority based elections using the **DB_REP_ELECTION_TIMEOUT** flag to the DB_ENV->rep_set_timeout() method. + +Consider 3 sites, A, B, and C where A is the master. In the case where all three sites crash and all reboot, all sites will set a timeout for a full election, say 10 minutes, but only require a majority for **nvotes** to the DB_ENV->rep_elect() method. Once all three sites are booted the election will complete immediately if they reboot within 10 minutes of each other. Consider if all three sites crash and only two reboot. The two sites will enter the election, but after the 10 minute timeout they will elect with the majority of two sites. Using the full election timeout sets a threshold for allowing a site to reboot and rejoin the group. + +To add a database environment to the replication group with the intent of it becoming the master, first add it as a client. Since it may be out-of-date with respect to the current master, allow it to update itself from the current master. Then, shut the current master down. Presumably, the added client will win the subsequent election. If the client does not win the election, it is likely that it was not given sufficient time to update itself with respect to the current master. + +If a client is unable to find a master or win an election, it means that the network has been partitioned and there are not enough environments participating in the election for one of the participants to win. In this case, the application should repeatedly call DB_ENV->rep_start() and DB_ENV->rep_elect(), alternating between attempting to discover an existing master, and holding an election to declare a new one. In desperate circumstances, an application could simply declare itself the master by calling DB_ENV->rep_start(), or by reducing the number of participants required to win an election until the election is won. Neither of these solutions is recommended: in the case of a network partition, either of these choices can result in there being two masters in one replication group, and the databases in the environment might irretrievably diverge as they are modified in different ways by the masters. + +Note that this presents a special problem for a replication group consisting of only two environments. If a master site fails, the remaining client can never comprise a majority of sites in the group. If the client application can reach a remote network site, or some other external tie-breaker, it may be able to determine whether it is safe to declare itself master. Otherwise it must choose between providing availability of a writable master (at the risk of duplicate masters), or strict protection against duplicate masters (but no master when a failure occurs). Replication Manager offers this choice via the DB_ENV->rep_set_config() method DB_REPMGR_CONF_2SITE_STRICT flag. Base API applications can accomplish this by judicious setting of the **nvotes** and **nsites** parameters to the DB_ENV->rep_elect() method. + +It is possible for a less-preferred database environment to win an election if a number of systems crash at the same time. Because an election winner is declared as soon as enough environments participate in the election, the environment on a slow booting but well-connected machine might lose to an environment on a badly connected but faster booting machine. In the case of a number of environments crashing at the same time (for example, a set of replicated servers in a single machine room), applications should bring the database environments on line as clients initially (which will allow them to process read queries immediately), and then hold an election after sufficient time has passed for the slower booting machines to catch up. + +If, for any reason, a less-preferred database environment becomes the master, it is possible to switch masters in a replicated environment. For example, the preferred master crashes, and one of the replication group clients becomes the group master. In order to restore the preferred master to master status, take the following steps: + +1. The preferred master should reboot and re-join the replication group as a client. +2. Once the preferred master has caught up with the replication group, the application on the current master should complete all active transactions and reconfigure itself as a client using the DB_ENV->rep_start() method. +3. Then, the current or preferred master should call for an election using the DB_ENV->rep_elect() method. diff --git a/docs-src/guides/programmer_reference/rep_ex.md b/docs-src/guides/programmer_reference/rep_ex.md new file mode 100644 index 000000000..63417726f --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_ex.md @@ -0,0 +1,73 @@ +--- +title: "Ex_rep: a replication example" +api-name: "Ex_rep: a replication example" +source: docs/programmer_reference/rep_ex.html +--- +## Ex_rep: a replication example + +Ex_rep, found in the `examples_c/ex_rep` subdirectory of the Berkeley DB distribution, is a simple but complete demonstration of a replicated application. The application is a mock stock ticker. The master accepts a stock symbol and a numerical value as input, and stores this information into a replicated database; either master or clients can display the contents of the database, given an empty input line. + +There are two versions of the application: ex_rep_mgr uses Replication Manager, while ex_rep_base uses the replication Base API. This is intended to demonstrate that, while the basic function of the application is the same in either case, the replication support infrastructure differs markedly. + +The communication infrastructure demonstrated with ex_rep_base has the same dependencies on system networking and threading support as does the Replication Manager (see the Replication introduction). The Makefile created by the standard UNIX configuration will build the ex_rep examples on most platforms. Enter "make ex_rep_mgr" and/or "make ex_rep_base" to build them. + +The synopsis for both programs is as follows: + +`ex_rep_xxx `**`-h home`**` `**`-l host:port`**` [`**`-MC`**`] [`**`-r host:port`**`] [`**`-R host:port`**`] [`**`-a all|quorum`**`] [`**`-b`**`] [`**`-n sites`**`] [`**`-p priority`**`] [`**`-v`**`]` + +where "ex_rep_xxx" is either "ex_rep_mgr" or "ex_rep_base". The only difference is that: + +- specifying **-M** or **-C** is optional for ex_rep_mgr, but one of these options must be specified for ex_rep_base. + +- The **-n** option is not supported supported by ex_rep_mgr. That option specifies the number of nodes in the replication group. When you use the Replication Manager, this number is automatically determined for you. + +The options apply to either version of the program except where noted. They are as follows: + + **-h** +Specify a home directory for the database environment. + + **-l** +Listen on local host "host" at port "port" for incoming connections. + + **-M** +Configure this process as a master. + + **-C** +Configure this process as a client. + + **-r** +Identifies the helper site used for joining the group. + + **-R** +Identifies a remote peer to be used for joining the group. This peer is used for syncing purposes. See Client-to-client synchronization for more information. + + **-a** +Specify repmgr acknowledgement policy of all or quorum. See DB_ENV->repmgr_set_ack_policy() for more information (ex_rep_mgr only.) + + **-b** +Indicates that bulk transfer should be used. See Bulk transfer for more information. + + **-n** +Specify the total number of sites in the replication group (ex_rep_base only). + + **-p** +Set the election priority. See Elections for more information. + + **-v** +Indicates that additional informational and debugging output should be enabled. + +A typical ex_rep_mgr session begins with a command such as the following, to start a master: + +``` c +ex_rep_mgr -M -p 100 -h DIR1 -l localhost:30100 +``` + +and several clients: + +``` c +ex_rep_mgr -C -p 50 -h DIR2 -l localhost:30101 -r localhost:30100 +ex_rep_mgr -C -p 10 -h DIR3 -l localhost:30102 -r localhost:30100 +ex_rep_mgr -C -p 0 -h DIR4 -l localhost:30103 -r localhost:30100 +``` + +In this example, the client with home directory DIR4 can never become a master (its priority is 0). Both of the other clients can become masters, but the one with home directory DIR2 is preferred. Priorities are assigned by the application and should reflect the desirability of having particular clients take over as master in the case that the master fails. diff --git a/docs-src/guides/programmer_reference/rep_ex_chan.md b/docs-src/guides/programmer_reference/rep_ex_chan.md new file mode 100644 index 000000000..d62438525 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_ex_chan.md @@ -0,0 +1,60 @@ +--- +title: "Ex_rep_chan: a Replication Manager channel example" +api-name: "Ex_rep_chan: a Replication Manager channel example" +source: docs/programmer_reference/rep_ex_chan.html +--- +## Ex_rep_chan: a Replication Manager channel example + +Ex_rep_chan, found in the `examples/c/ex_rep_chan` subdirectory of the Berkeley DB distribution, is a simple but complete demonstration of a replicated application that uses the Replication Manager feature of channels to perform write forwarding. The application is a mock stock ticker. Although similar to the ex_rep_mgr example program, this example differs in that it provides an example of using Replication Manager message channels. Any site accepts a command to write data to the database. If the site is a client, then, using the channels feature, the application forwards the request to the master site. If the site is a master then the request is automatically handled locally. You can read and write stock values at any site without needing to know what site is currently the master. + +The set of supported commands can be viewed with either the **help** or the **?** command. Several commands work with key/data pairs where the key is a stock symbol and the data is its value. + +The command to retrieve and print the current site's database contents is **print** or simply an empty input line. To read the contents of the master's database from any site use the **get key key ...** command. That command will forward the read request to the master if necessary and return the key/data pairs for all given keys. + +There are two commands to put data into the database. Both commands take one or more key/data pairs, all of which are written into the database in a single transaction at the master site. The **put** command sends the data to the master site, and simply waits for a status response. The **put_sync** command sends the data to the master site, and uses a transaction token returned by the master to wait for the contents of that put to be available on the local site. This serves as a demonstration of the read your writes consistency feature. + +The Makefile created by the standard UNIX configuration will build the ex_rep_chan example on most platforms. Enter "make ex_rep_chan" to build it. + +The synopsis for the program is as follows: + +`ex_rep_chan `**`-h home`**` `**`-l host:port`**` [`**`-MC`**`] [`**`-r host:port`**`] [`**`-R host:port`**`] [`**`-p priority`**`] [`**`-v`**`]` + +The options are as follows: + + **-h** +Specify a home directory for the database environment. + + **-l** +Listen on local host "host" at port "port" for incoming connections. + + **-M** +Configure this process as a master. + + **-C** +Configure this process as a client. + + **-r** +Identifies the helper site used for joining the group. + + **-R** +Identifies a remote peer to be used for joining the group. This peer is used for syncing purposes. See Client-to-client synchronization for more information. + + **-p** +Set the election priority. See Elections for more information. + + **-v** +Indicates that additional informational and debugging output should be enabled. + +A typical ex_rep_chan session begins with a command such as the following, to start a master: + +``` c +ex_rep_chan -M -h DIR1 -l localhost:30100 +``` + +and several clients: + +``` c +ex_rep_chan -C -h DIR2 -l localhost:30101 -r localhost:30100 +ex_rep_chan -C -h DIR3 -l localhost:30102 -r localhost:30100 +ex_rep_chan -C -h DIR4 -l localhost:30103 -r localhost:30100 +``` diff --git a/docs-src/guides/programmer_reference/rep_ex_comm.md b/docs-src/guides/programmer_reference/rep_ex_comm.md new file mode 100644 index 000000000..e39a2e8cb --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_ex_comm.md @@ -0,0 +1,22 @@ +--- +title: "Ex_rep_base: a TCP/IP based communication infrastructure" +api-name: "Ex_rep_base: a TCP/IP based communication infrastructure" +source: docs/programmer_reference/rep_ex_comm.html +--- +## Ex_rep_base: a TCP/IP based communication infrastructure + +Base API applications must implement a communication infrastructure. The communication infrastructure consists of three parts: a way to map environment IDs to particular sites, the functions to send and receive messages, and the application architecture that supports the particular communication infrastructure used (for example, individual threads per communicating site, a shared message handler for all sites, a hybrid solution). The communication infrastructure for ex_rep_base is implemented in the file `ex_rep/base/rep_net.c`, and each part of that infrastructure is described as follows. + +Ex_rep_base maintains a table of environment ID to TCP/IP port mappings. A pointer to this table is stored in a structure pointed to by the app_private field of the DB_ENV object so it can be accessed by any function that has the database environment handle. The table is represented by a machtab_t structure which contains a reference to a linked list of member_t's, both of which are defined in `ex_rep/base/rep_net.c`. Each member_t contains the host and port identification, the environment ID, and a file descriptor. + +This design is particular to this application and communication infrastructure, but provides an indication of the sort of functionality that is needed to maintain the application-specific state for a TCP/IP-based infrastructure. The goal of the table and its interfaces is threefold: First, it must guarantee that given an environment ID, the send function can send a message to the appropriate place. Second, when given the special environment ID DB_EID_BROADCAST, the send function can send messages to all the machines in the group. Third, upon receipt of an incoming message, the receive function can correctly identify the sender and pass the appropriate environment ID to the DB_ENV->rep_process_message() method. + +Mapping a particular environment ID to a specific port is accomplished by looping through the linked list until the desired environment ID is found. Broadcast communication is implemented by looping through the linked list and sending to each member found. Since each port communicates with only a single other environment, receipt of a message on a particular port precisely identifies the sender. + +This is implemented in the quote_send, quote_send_broadcast and quote_send_one functions, which can be found in `ex_rep/base/rep_net.c`. + +The example provided is merely one way to satisfy these requirements, and there are alternative implementations as well. For instance, instead of associating separate socket connections with each remote environment, an application might instead label each message with a sender identifier; instead of looping through a table and sending a copy of a message to each member of the replication group, the application could send a single message using a broadcast protocol. + +The quote_send function is passed as the callback to DB_ENV->rep_set_transport(); Berkeley DB automatically sends messages as needed for replication. The receive function is a mirror to the quote_send_one function. It is not a callback function (the application is responsible for collecting messages and calling DB_ENV->rep_process_message() on them as is convenient). In the sample application, all messages transmitted are Berkeley DB messages that get handled by DB_ENV->rep_process_message(), however, this is not always going to be the case. The application may want to pass its own messages across the same channels, distinguish between its own messages and those of Berkeley DB, and then pass only the Berkeley DB ones to DB_ENV->rep_process_message(). + +The final component of the communication infrastructure is the process model used to communicate with all the sites in the replication group. Each site creates a thread of control that listens on its designated socket (as specified by the **-l** command line argument) and then creates a new channel for each site that contacts it. In addition, each site explicitly connects to the sites specified in the **-r** and **-R** command line arguments. This is a fairly standard TCP/IP process architecture and is implemented by the connect_thread, connect_all and connect_site functions in `ex_rep/base/rep_msg.c` and supporting functions in `ex_rep/base/rep_net.c`. diff --git a/docs-src/guides/programmer_reference/rep_ex_rq.md b/docs-src/guides/programmer_reference/rep_ex_rq.md new file mode 100644 index 000000000..3ab957fb6 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_ex_rq.md @@ -0,0 +1,18 @@ +--- +title: "Ex_rep_base: putting it all together" +api-name: "Ex_rep_base: putting it all together" +source: docs/programmer_reference/rep_ex_rq.html +--- +## Ex_rep_base: putting it all together + +Beyond simply initializing a replicated environment, a Base API application must set up its communication infrastructure, and then make sure that incoming messages are received and processed. + +To initialize replication, ex_rep_base creates a Berkeley DB environment and calls DB_ENV->rep_set_transport() to establish a send function. (See the main function in `ex_rep/base/rep_base.c`, including its calls to the create_env and env_init functions in `ex_rep/common/rep_common.c`.) + +ex_rep_base opens a listening socket for incoming connections and opens an outgoing connection to every machine that it knows about (that is, all the sites listed in the **-r** and **-R** command line arguments). Applications can structure the details of this in different ways, but ex_rep_base creates a user-level thread to listen on its socket, plus a thread to loop and handle messages on each socket, in addition to the threads needed to manage the user interface, update the database on the master, and read from the database on the client (in other words, in addition to the normal functionality of any database application). + +Once the initial threads have all been started and the communications infrastructure is initialized, the application signals that it is ready for replication and joins a replication group by calling DB_ENV->rep_start(). (Again, see the main function in `ex_rep/base/rep_base.c`.) + +Note the use of the optional second argument to DB_ENV->rep_start() in the client initialization code. The argument "local" is a piece of data, opaque to Berkeley DB, that will be broadcast to each member of a replication group; it allows new clients to join a replication group, without knowing the location of all its members; the new client will be contacted by the members it does not know about, who will receive the new client's contact information that was specified in "myaddr." See Connecting to a new site for more information. + +The final piece of a replicated application is the code that loops, receives, and processes messages from a given remote environment. ex_rep_base runs one of these loops in a parallel thread for each socket connection (see the hm_loop function in `ex_rep/base/rep_msg.c`). Other applications may want to queue messages somehow and process them asynchronously, or select() on a number of sockets and either look up the correct environment ID for each or encapsulate the ID in the communications protocol. diff --git a/docs-src/guides/programmer_reference/rep_faq.md b/docs-src/guides/programmer_reference/rep_faq.md new file mode 100644 index 000000000..ca270c43c --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_faq.md @@ -0,0 +1,36 @@ +--- +title: "Replication FAQ" +api-name: "Replication FAQ" +source: docs/programmer_reference/rep_faq.html +--- +## Replication FAQ + +1. **Does Berkeley DB provide support for forwarding write queries from clients to masters?** + + No, it does not. In general this protocol is left entirely to the application. Note, there is no reason not to use the communications channels a Base API application establishes for replication support to forward database update messages to the master, since Berkeley DB does not require those channels to be used exclusively for replication messages. Replication Manager does not currently offer this service to the application. + +2. **Can I use replication to partition my environment across multiple sites?** + + No, this is not possible. All replicated databases must be equally shared by all environments in the replication group. + +3. **I'm running with replication but I don't see my databases on the client.** + + This problem may be the result of the application using absolute path names for its databases, and the pathnames are not valid on the client system. + +4. **How can I distinguish Berkeley DB messages from application messages?** + + There is no way to distinguish Berkeley DB messages from application-specific messages, nor does Berkeley DB offer any way to wrap application messages inside of Berkeley DB messages. Distributed applications exchanging their own messages should either enclose Berkeley DB messages in their own wrappers, or use separate network connections to send and receive Berkeley DB messages. The one exception to this rule is connection information for new sites; Berkeley DB offers a simple method for sites joining replication groups to send connection information to the other database environments in the group (see Connecting to a new site for more information). + +5. **How should I build my ****send** function?**** + + This depends on the specifics of the application. One common way is to write the **rec** and **control** arguments' sizes and data to a socket connected to each remote site. On a fast, local area net, the simplest method is likely to be to construct broadcast messages. Each Berkeley DB message would be encapsulated inside an application specific message, with header information specifying the intended recipient(s) for the message. This will likely require a global numbering scheme, however, as the Berkeley DB library has to be able to send specific log records to clients apart from the general broadcast of new log records intended for all members of a replication group. + +6. **Does every one of my threads of control on the master have to set up its own connection to every client? And, does every one of my threads of control on the client have to set up its own connection to every master?** + + This is not always necessary. In the Berkeley DB replication model, any thread of control which modifies a database in the master environment must be prepared to send a message to the client environments, and any thread of control which delivers a message to a client environment must be prepared to send a message to the master. There are many ways in which these requirements can be satisfied. + + The simplest case is probably a single, multithreaded process running on the master and clients. The process running on the master would require a single write connection to each client and a single read connection from each client. A process running on each client would require a single read connection from the master and a single write connection to the master. Threads running in these processes on the master and clients would use the same network connections to pass messages back and forth. + + A common complication is when there are multiple processes running on the master and clients. A straight-forward solution is to increase the numbers of connections on the master — each process running on the master has its own write connection to each client. However, this requires only one additional connection for each possible client in the master process. The master environment still requires only a single read connection from each client (this can be done by allocating a separate thread of control which does nothing other than receive client messages and forward them into the database). Similarly, each client still only requires a single thread of control that receives master messages and forwards them into the database, and which also takes database messages and forwards them back to the master. This model requires the networking infrastructure support many-to-one writers-to-readers, of course. + + If the number of network connections is a problem in the multiprocess model, and inter-process communication on the system is inexpensive enough, an alternative is have a single process which communicates between the master and each client, and whenever a process' **send** function is called, the process passes the message to the communications process which is responsible for forwarding the message to the appropriate client. Alternatively, a broadcast mechanism will simplify the entire networking infrastructure, as processes will likely no longer have to maintain their own specific network connections. diff --git a/docs-src/guides/programmer_reference/rep_filename.md b/docs-src/guides/programmer_reference/rep_filename.md new file mode 100644 index 000000000..87d4cee1c --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_filename.md @@ -0,0 +1,30 @@ +--- +title: "Managing Replication Files" +api-name: "Managing Replication Files" +source: docs/programmer_reference/rep_filename.html +--- +## Managing Replication Files + +Whether you use the Base API or the Replication Manager, replication creates a set of internal files that are normally stored on-disk in your environment home directory. These files contain metadata which is necessary for replication operations, and so you should never delete these files. + +You can cause these files to not be stored on disk, but instead to be held entirely in-memory, by specifying the DB_REP_CONF_INMEM flag to the DB_ENV->rep_set_config() method. Doing this can improve your application's data throughput by avoiding the disk I/O associated with these metadata files. However, in the event that your application is shut down, the contents of these files are lost. This results in some loss of functionality, including an increased chance that elections will fail, or that the wrong site will win an election. See the DB_REP_CONF_INMEM flag description for more information. + +Note that turning on DB_REP_CONF_INMEM means that Replication Manager cannot store group membership changes persistently. This is because Replication Manager stores group membership information in an internal database, which is held in memory when DB_REP_CONF_INMEM is turned on. For this reason, if your Replication Manager application requires replication metadata to be stored in memory, then you must manually identify all the sites in your replication group using the `DB_LEGACY` site configuration attribute. Be aware that this configuration needs to be made permanent. (Normally, `DB_LEGACY` is used only on a temporary basis for the purpose of upgrading old Replication Manager applications.) + +Do the following: + +1. Shut down all the sites in your replication group. + +2. For every site in your replication group: + + 1. Configure a DB_SITE handle for the local site. Use DB_SITE->set_config() to indicate that this is a legacy site by setting the `DB_LEGACY` parameter. + + 2. Configure a DB_SITE handle for *every other site* in the replication group. Set the `DB_LEGACY` parameter for each of these handles. + + Please pay careful attention to this step. To repeat: a DB_SITE handle MUST be configured for EVERY site in the replication group. + +3. Restart all the sites in the replication group. + +Alternatively, you can store persistent environment metadata files, including those required by replication, in a location other than your environment home directory. Doing so can help improve I/O throughput by placing these files on a spindle that is not being used for other environment data I/O. You do this using the DB_ENV->set_metadata_dir() method. + +Note that you must configure the handling of your environment metadata consistently across your entire replication group. That is, if you place your replication metadata in-memory on one site, then it must be placed in-memory on all the sites in the group. Similarly, if you place your replication metadata files in a non-standard directory location on one site, then they must be placed in the exact same directory location on all the sites in your group. diff --git a/docs-src/guides/programmer_reference/rep_id.md b/docs-src/guides/programmer_reference/rep_id.md new file mode 100644 index 000000000..6428a58f1 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_id.md @@ -0,0 +1,20 @@ +--- +title: "Replication environment IDs" +api-name: "Replication environment IDs" +source: docs/programmer_reference/rep_id.html +--- +## Replication environment IDs + +Each database environment included in a replication group must have a unique identifier for itself and for the other members of the replication group. The identifiers do not need to be global, that is, each database environment can assign local identifiers to members of the replication group as it encounters them. For example, given three sites: A, B and C, site A might assign the identifiers 1 and 2 to sites B and C respectively, while site B might assign the identifiers 301 and 302 to sites A and C respectively. Note that it is not wrong to have global identifiers, it is just not a requirement. + +Replication Manager assigns and manages environment IDs on behalf of the application. + +It is the responsibility of a Base API application to label each incoming replication message passed to DB_ENV->rep_process_message() method with the appropriate identifier. Subsequently, Berkeley DB will label outgoing messages to the **send** function with those same identifiers. + +Negative identifiers are reserved for use by Berkeley DB, and should never be assigned to environments by the application. Two of these reserved identifiers are intended for application use, as follows: + + DB_EID_BROADCAST +The DB_EID_BROADCAST identifier indicates a message should be broadcast to all members of a replication group. + +DB_EID_INVALID +The DB_EID_INVALID identifier is an invalid environment ID, and may be used to initialize environment ID variables that are subsequently checked for validity. diff --git a/docs-src/guides/programmer_reference/rep_init.md b/docs-src/guides/programmer_reference/rep_init.md new file mode 100644 index 000000000..ba8927e69 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_init.md @@ -0,0 +1,21 @@ +--- +title: "Initializing a new site" +api-name: "Initializing a new site" +source: docs/programmer_reference/rep_init.html +--- +## Initializing a new site + +By default, adding a new site to a replication group only requires the client to join. Berkeley DB will automatically perform internal initialization from the master to the client, bringing the client into sync with the master. + +However, depending on the network and infrastructure, it can be advantageous in a few instances to use a "hot backup" to initialize a client into a replication group. Clients not wanting to automatically perform internal initialization should call the DB_ENV->rep_set_config() method to turn off the DB_REP_CONF_AUTOINIT flag. Turning off this configuration flag causes Berkeley DB to return DB_REP_JOIN_FAILURE to the application's DB_ENV->rep_process_message() method instead of performing internal initialization. + +To use a hot backup to initialize a client into a replication group, perform the following steps: + +1. Do an archival backup of the master's environment, as described in Database and log file archival. The backup can either be a conventional backup or a hot backup. +2. Copy the archival backup into a clean environment directory on the client. +3. Run catastrophic recovery on the client's new environment, as described in Recovery procedures. +4. Reconfigure and reopen the environment as a client member of the replication group. + +If copying the backup to the client takes a long time relative to the frequency with which log files are reclaimed using the db_archive utility or the DB_ENV->log_archive() method, it may be necessary to suppress log reclamation until the newly restarted client has "caught up" and applied all log records generated during its downtime. + +As with any Berkeley DB application, the database environment must be in a consistent state at application startup. This is most easily assured by running recovery at startup time in one thread or process; it is harmless to do this on both clients and masters even when not strictly necessary. diff --git a/docs-src/guides/programmer_reference/rep_lease.md b/docs-src/guides/programmer_reference/rep_lease.md new file mode 100644 index 000000000..ff67f1656 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_lease.md @@ -0,0 +1,100 @@ +--- +title: "Master Leases" +api-name: "Master Leases" +source: docs/programmer_reference/rep_lease.html +--- +## Master Leases + + [Changing Group Size](rep_lease.md#masterlease_change_groupsize) + +Some applications have strict requirements about the consistency of data read on a master site. Berkeley DB provides a mechanism called master leases to provide such consistency. Without master leases, it is sometimes possible for Berkeley DB to return old data to an application when newer data is available due to unfortunate scheduling as illustrated below: + +1. **Application on master site**: Read data item *foo* via Berkeley DB DB->get() or DBC->get() call. +2. **Application on master site**: sleep, get descheduled, etc. +3. **System**: Master changes role, becomes a client. +4. **System**: New site is elected master. +5. **System**: New master modifies data item *foo*. +6. **Application**: Berkeley DB returns old data for *foo* to application. + +By using master leases, Berkeley DB can provide guarantees about the consistency of data read on a master site. The master site can be considered a recognized authority for the data and consequently can provide authoritative reads. Clients grant master leases to a master site. By doing so, clients acknowledge the right of that site to retain the role of master for a period of time. During that period of time, clients cannot elect a new master, become master, nor grant their lease to another site. + +By holding a collection of granted leases, a master site can guarantee to the application that the data returned is the current, authoritative value. As a master performs operations, it continually requests updated grants from the clients. When a read operation is required, the master guarantees that it holds a valid collection of lease grants from clients before returning data to the application. By holding leases, Berkeley DB provides several guarantees to the application: + +1. Authoritative reads: A guarantee that the data being read by the application is the current value. + +2. Durability from rollback: A guarantee that the data being written or read by the application is permanent across a majority of client sites and will never be rolled back. + + The rollback guarantee also depends on the DB_TXN_NOSYNC flag. The guarantee is effective as long as there isn't total replication group failure while clients have granted leases but are holding the updates in their cache. The application must weigh the performance impact of synchronous transactions against the risk of total replication group failure. If clients grant a lease while holding updated data in cache, and total failure occurs, then the data is no longer present on the clients and rollback can occur if the master also crashes. + + The guarantee that data will not be rolled back applies only to data successfully committed on a master. Data read on a client, or read while ignoring leases can be rolled back. + +3. Freshness: A guarantee that the data being read by the application on the *master* is up-to-date and has not been modified or removed during the read. + + The read authority is only on the master. Read operations on a client always ignore leases and consequently, these operations can return stale data. + +4. Master viability: A guarantee that a current master with valid leases cannot encounter a duplicate master situation. + + Leases remove the possibility of a duplicate master situation that forces the current master to downgrade to a client. However, it is still possible that old masters with expired leases can discover a later master and return DB_REP_DUPMASTER to the application. + +There are several requirements of the application using leases: + +1. Replication Manager applications must configure a majority (or larger) acknowledgement policy via the DB_ENV->repmgr_set_ack_policy() method. Base API applications must implement and enforce such a policy on their own. +2. Base API applications must return an error from the send callback function when the majority acknowledgement policy is not met for permanent records marked with DB_REP_PERMANENT. Note that the Replication Manager automatically fulfills this requirement. +3. Base API applications must set the number of sites in the group using the DB_ENV->rep_set_nsites() method before starting replication and cannot change it during operation. +4. Using leases in a replication group is all or none. Behavior is undefined when some sites configure leases and others do not. Use the DB_ENV->rep_set_config() method to turn on leases. +5. The configured lease timeout value must be the same on all sites in a replication group, set via the DB_ENV->rep_set_timeout() method. +6. The configured clock_scale_factor value must be the same on all sites in a replication group. This value defaults to no skew, but can be set via the DB_ENV->rep_set_clockskew() method. +7. Applications that care about read guarantees must perform all read operations on the master. Reading on a client does not guarantee freshness. +8. The application must use elections to choose a master site. It must never simply declare a master without having won an election (as is allowed without Master Leases). + +Master leases are based on timeouts. Berkeley DB assumes that time always runs forward. Users who change the system clock on either client or master sites when leases are in use void all guarantees and can get undefined behavior. See the DB_ENV->rep_set_timeout() method for more information. + +Applications using master leases should be prepared to handle `DB_REP_LEASE_EXPIRED` errors from read operations on a master and from the DB_TXN->commit() method. + +Read operations on a master that should not be subject to leases can use the DB_IGNORE_LEASE flag to the DB->get() method. Read operations on a client always imply leases are ignored. + +Master lease checks cannot succeed until a majority of sites have completed client synchronization. Read operations on a master performed before this condition is met can use the DB_IGNORE_LEASE flag to avoid errors. + +Clients are forbidden from participating in elections while they have an outstanding lease granted to a master. Therefore, if the DB_ENV->rep_elect() method is called, then Berkeley DB will block, waiting until its lease grant expires before participating in any election. While it waits, the client attempts to contact the current master. If the client finds a current master, then it returns from the DB_ENV->rep_elect() method. When leases are configured and the lease has never yet been granted (on start-up), clients must wait a full lease timeout before participating in an election. + +### Changing Group Size + +If you are using master leases and you change the size of your replication group, there is a remote possibility that you can lose some data previously thought to be durable. This is only true for users of the Base API. + +The problem can arise if you are removing sites from your replication group. (You might be increasing the size of your site overall, but if you remove all of the wrong sites you can lose data.) + +Suppose you have a replication group with five sites; A, B, C, D and E; and you are using a quorum acknowledgement policy. Then: + +1. Master A replicates a transaction to replicas B and C. Those sites acknowledge the write activity. + +2. Sites D and E do not receive the transaction. However, B and C have acknowledged the transaction, which means the acknowledgement policy is met and so the transaction is considered durable. + +3. You shutdown sites B and C. Now only A has the transaction. + +4. You increase the size of your replication group to 3 using DB_ENV->rep_set_nsites(). + +5. You shutdown or otherwise lose site A. + +6. Sites D and E hold an election. Because the size of the replication group is 3, they have enough sites to successfully hold an election. However, neither site has the transaction in question. In this way, the transaction can become lost. + +An alternative scenario exists where you do not change the size of your replication group, or you actually increase the size of your replication group, but in the process you happen to remove the exact wrong sites: + +1. Master A replicates a transaction to replicas B and C. Those sites acknowledge the write activity. + +2. Sites D and E do not receive the transaction. However, B and C have acknowledged the transaction, which means the acknowledgement policy is met and so the transaction is considered durable. + +3. You shutdown sites B and C. Now only A has the transaction. + +4. You add three new sites to your replication group: F, G and H, increasing the size of your replication group to 6 using DB_ENV->rep_set_nsites(). + +5. You shutdown or otherwise lose site A before F, G and H can be fully populated with data. + +6. Sites D, E, F, G and H hold an election. Because the size of the replication group is 6, they have enough sites to successfully hold an election. However, none of these sites has the transaction in question. In this way, the transaction can become lost. + +This scenario represents a race condition that would be highly unlikely to be seen outside of a lab environment. To minimize the chance of this race condition occurring to the absolute minimum, do one or more of the following when using master leases with the Base API: + +1. Require all sites to acknowledge transaction commits. + +2. Never change the size of your replication group unless all sites in the group are running and communicating normally with one another. + +3. Don't remove (or replace) a large percentage of your sites from your replication group unless all sites in the group are running and communicating normally with one another. If you are going to remove a large percentage of your sites from your replication group, try removing just one site at a time, pausing in between each removal to give the replication group a chance to fully distribute all writes before removing the next site. diff --git a/docs-src/guides/programmer_reference/rep_mastersync.md b/docs-src/guides/programmer_reference/rep_mastersync.md new file mode 100644 index 000000000..755ee33fb --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_mastersync.md @@ -0,0 +1,48 @@ +--- +title: "Synchronizing with a master" +api-name: "Synchronizing with a master" +source: docs/programmer_reference/rep_mastersync.html +--- +## Synchronizing with a master + + [Delaying client synchronization](rep_mastersync.md#rep_delay_sync) + + [Client-to-client synchronization](rep_mastersync.md#rep_c2c_sync) + + [Blocked client operations](rep_mastersync.md#idp52488504) + + [Clients too far out-of-date to synchronize](rep_mastersync.md#idp52510624) + +When a client detects a new replication group master, the client must synchronize with the new master before the client can process new database changes. Synchronizing is a heavyweight operation which can place a burden on both the client and the master. There are several controls an application can use to reduce the synchronization burden. + +### Delaying client synchronization + +When a replication group has a new master, either as specified by the application or as a result of winning an election, all clients in the replication group must synchronize with the new master. This can strain the resources of the new master since a large number of clients may be attempting to communicate with and transfer records from the master. Client applications wanting to delay client synchronization should call the DB_ENV->rep_set_config() method with the DB_REP_CONF_DELAYCLIENT flag. The application will be notified of the establishment of the new master as usual, but the client will not proceed to synchronize with the new master. + +Applications learn of a new master via the DB_EVENT_REP_NEWMASTER event. + +Client applications choosing to delay synchronization in this manner are responsible for synchronizing the client environment at some future time using the DB_ENV->rep_sync() method. + +### Client-to-client synchronization + +Instead of synchronizing with the new master, it is sometimes possible for a client to synchronize with another client. Berkeley DB initiates synchronization at the client by sending a request message via the transport call-back function of the communication infrastructure. The message is destined for the master site, but is also marked with a DB_REP_ANYWHERE flag. The application may choose to send such a request to another client, or to ignore the flag, sending it to its indicated destination. + +Furthermore, when the other client receives such a request it may be unable to satisfy it. In this case it will reply to the requesting client, telling it that it is unable to provide the requested information. The requesting client will then re-issue the request. Additionally, if the original request never reaches the other client, the requesting client will again re-issue the request. In either of these cases the message will be marked with the DB_REP_REREQUEST flag. The application may continue trying to find another client to service the request, or it may give up and simply send it to the master (that is, the environment ID explicitly specified to the transport function). + +Replication Manager allows an application to designate one or more remote sites (called its "peers") to receive client-to-client requests. You do this by setting the `DB_REPMGR_PEER` parameter using the DB_SITE->set_config() method. Replication Manager always tries to send requests marked with the DB_REP_ANYWHERE flag to a peer, if available. However, it always sends a DB_REP_REREQUEST to the master site. + +Base API applications have complete freedom in choosing where to send these DB_REP_ANYWHERE requests, and in deciding how to handle DB_REP_REREQUEST. + +The delayed synchronization and client-to-client synchronization features allow applications to do load balancing within replication groups. For example, consider a replication group with 5 sites, A, B, C, D and E. Site E just crashed, and site A was elected master. Sites C and D have been configured for delayed synchronization. When site B is notified that site A is a new master, it immediately synchronizes. When B finishes synchronizing with the master, the application calls the DB_ENV->rep_sync() method on sites C and D to cause them to synchronize as well. Sites C and D (and E, when it has finished rebooting) can send their requests to site B, and B then bears the brunt of the work and network traffic for synchronization, making master site A available to handle the normal application load and any write requests paused by the election. + +### Blocked client operations + +Clients in the process of synchronizing with the master block access to Berkeley DB operations during some parts of that process. By default, most Berkeley DB methods will block until client synchronization is complete, and then the method call proceeds. + +Client applications which cannot wait and would prefer an immediate error return instead of blocking, should call the DB_ENV->rep_set_config() method with the DB_REP_CONF_NOWAIT flag. This configuration causes DB method calls to immediately return a DB_REP_LOCKOUT error instead of blocking, if the client is currently synchronizing with the master. + +### Clients too far out-of-date to synchronize + +Clients attempting to synchronize with the master may discover that synchronization is not possible because the client no longer has any overlapping information with the master site. By default, the master and client automatically detect this state and perform an internal initialization of the client. Because internal initialization requires transfer of entire databases to the client, it can take a relatively long period of time and may require database handles to be reopened in the client applications. + +Client applications which cannot wait or would prefer to do a hot backup instead of performing internal initialization, should call the DB_ENV->rep_set_config() method to turn off the DB_REP_CONF_AUTOINIT flag. Turning off this configuration flag causes Berkeley DB to return DB_REP_JOIN_FAILURE to the application instead of performing internal initialization. diff --git a/docs-src/guides/programmer_reference/rep_mgr_ack.md b/docs-src/guides/programmer_reference/rep_mgr_ack.md new file mode 100644 index 000000000..97f840fa6 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_mgr_ack.md @@ -0,0 +1,18 @@ +--- +title: "Choosing a Replication Manager Ack Policy" +api-name: "Choosing a Replication Manager Ack Policy" +source: docs/programmer_reference/rep_mgr_ack.html +--- +## Choosing a Replication Manager Ack Policy + +Replication Manager allows the user to choose from a variety of acknowledgement policies. There are two characteristics that should be considered when choosing the policy: consistency and durability. Consistency means making sure some number of clients have applied all available master transactions. Durability, in this context, means only indicating success only if enough clients have applied a transaction. The issue of how many is enough depends on the application's requirements and varies per acknowledgement policy. For example, DB_REPMGR_ACKS_QUORUM means the data will survive a change in master or a network partition. In most cases, the number of sites for consistency is equal to the number of sites for durability. Replication Manager uses the consistency value to decide whether or not to wait for acknowledgements. Replication manager uses the durability value to decide either the transaction was successfully processed or that a DB_EVENT_REP_PERM_FAILED event should be generated. + +Replication Manager also strives to give the application the answer and return to the application as quickly as possible. Therefore, if it knows that the number of sites connected is insufficient to meet the consistency value, then it does not wait for any acknowledgements and if it knows that the durability value cannot be met, it returns DB_EVENT_REP_PERM_FAILED immediately to the user. + +With one exception, discussed below, all acknowledgement policies combine the consistency and durability values. For most policies the primary purpose is the durability of the data. For example, the DB_REPMGR_ACKS_QUORUM policy ensures that, if successful, the transaction's data is safe in the event of a network partition so that a majority of the sites in the group have the data. The DB_REPMGR_ACKS_NONE policy does not consider either consistency or durability, and it is very fast because it does not wait for any acknowledgements and it does not ever trigger the DB_EVENT_REP_PERM_FAILED event. Other policies, DB_REPMGR_ACKS_ALL and DB_REPMGR_ACKS_ALL_PEERS, have a primary purpose of consistency. These two policies wait for acknowledgements from all (or all electable) sites in the group. + +In the face of failure, however, the DB_REPMGR_ACKS_ALL and DB_REPMGR_ACKS_ALL_PEERS policies can result in a surprising lack of consistency due to the fact that Replication Manager strives to give the answer back to the application as fast as it can. So, for example, with DB_REPMGR_ACKS_ALL, and one site down, Replication Manager knows that disconnected site can never acknowledge, so it immediately triggers DB_EVENT_REP_PERM_FAILED. An unfortunate side effect of this policy is that existing, running sites may fall further and further behind the master if the master site is sending a fast, busy stream of transactions and never waiting for any site to send an acknowledgement. The master does not wait because the consistency value cannot be met, and it does trigger the DB_EVENT_REP_PERM_FAILED event because the durability value cannot be met, but those actions now affect the consistency of the other running sites. + +In order to counteract this unfortunate side effect, the DB_REPMGR_ACKS_ALL_AVAILABLE acknowledgement policy focuses on the consistency aspect, but also considers durability. This policy uses all sites for consistency, and a quorum of sites for its decision about durability. As long as there is a non-zero number of client replicas to send to, the master will wait for all available sites to acknowledge the transaction. As long as any client site is connected, this policy will prevent the master from racing ahead if one or more sites is down. On the master, this policy will then consider the transaction durable if the number of acknowledgements meets quorum for the group. + +The following acknowledgement policies determine durability using acknowledgements from electable peers only: DB_REPMGR_ACKS_QUORUM, DB_REPMGR_ACKS_ONE_PEER, DB_REPMGR_ACKS_ALL_PEERS. An electable peer is a site where the priority value is greater than zero. In replication groups using these policies, an unelectable site does not send acknowledgements and cannot contribute to transaction durability. diff --git a/docs-src/guides/programmer_reference/rep_mgr_meth.md b/docs-src/guides/programmer_reference/rep_mgr_meth.md new file mode 100644 index 000000000..2f50b85d6 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_mgr_meth.md @@ -0,0 +1,38 @@ +--- +title: "Replication Manager methods" +api-name: "Replication Manager methods" +source: docs/programmer_reference/rep_mgr_meth.html +--- +## Replication Manager methods + +Applications which use the Replication Manager support generally call the following Berkeley DB methods. The general pattern is to call various methods to configure Replication Manager, and then start it by calling DB_ENV->repmgr_start(). Once this initialization is complete, the application rarely needs to call any of these methods. (A prime example of an exception to this rule would be the DB_ENV->rep_sync() method, if the application is Delaying client synchronization.) + + DB_SITE +The DB_SITE handle is used to configure a site that belongs to the replication group. You can obtain a DB_SITE handle by calling the DB_ENV->repmgr_site() method. When you do this, you provide the TCP/IP host name and port that the replication site uses for incoming connections. + +Once you have the DB_SITE handle, you use the DB_SITE->set_config() method to configure the handle. One of the things you can configure about the handle is whether it is the local site (using the `DB_LOCAL_SITE` parameter). You must configure one and only one DB_SITE handle to be a local site before you start replication. + +You can also optionally configure DB_SITE handles for remote sites to help Replication Manager startup more efficiently. Note that it is usually not necessary for each site in the replication group initially to know about all other sites in the group. Sites can discover each other dynamically, as described in Connecting to a new site. + +Once you have configured your DB_SITE handles, you start replication using DB_ENV->repmgr_start(). + +When you are shutting down your application, you must use the DB_SITE->close() method to close all your open DB_SITE handles before you close your environment handles. + + DB_ENV->repmgr_set_ack_policy() +The DB_ENV->repmgr_set_ack_policy() method configures the acknowledgement policy to be used in the replication group, in other words, the behavior of the master with respect to acknowledgements for "permanent" messages, which implements the application's requirements for Transactional guarantees. The current implementation requires all sites in the replication group to configure the same acknowledgement policy. + + DB_ENV->rep_set_priority() +The DB_ENV->rep_set_priority() method configures the local site's priority for the purpose of elections. + + DB_ENV->rep_set_timeout() +This method optionally configures various timeout values. Otherwise default timeout values as specified in DB_ENV->rep_set_timeout() are used. In particular, Replication Manager client sites can be configured to monitor the health of the TCP/IP connection to the master site using heartbeat messages. If the client receives no messages from the master for a certain amount of time, it considers the connection to be broken, and calls for an election to choose a new master. Heartbeat messages also help clients request missing master changes in the absence of master activity. + + DB_ENV->set_event_notify() +Once configured and started, Replication Manager does virtually all of its work in the background, usually without the need for any direct communication with the application. However, occasionally events occur which the application may be interested in knowing about. The application can request notification of these events by calling the DB_ENV->set_event_notify() method. + + DB_ENV->repmgr_start() +The DB_ENV->repmgr_start() method starts the replication system. It opens the listening TCP/IP socket and creates all the background processing threads that will be needed. + +In addition to the methods previously described, Replication Manager applications may also call the following methods, as needed: DB_ENV->rep_set_config(), DB_ENV->rep_set_limit(), DB_ENV->rep_set_request(), DB_ENV->rep_sync() and DB_ENV->rep_stat(). + +Finally, Replication Manager applications can also make use of the Replication Manager's message channels. This allows the various sites in the replication group to pass messages that are tailored to the application's requirements. For more information, see Using Replication Manager message channels. diff --git a/docs-src/guides/programmer_reference/rep_mgrmulti.md b/docs-src/guides/programmer_reference/rep_mgrmulti.md new file mode 100644 index 000000000..af807912f --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_mgrmulti.md @@ -0,0 +1,55 @@ +--- +title: "Running Replication Manager in multiple processes" +api-name: "Running Replication Manager in multiple processes" +source: docs/programmer_reference/rep_mgrmulti.html +--- +## Running Replication Manager in multiple processes + + [One replication process and multiple subordinate processes](rep_mgrmulti.md#idp52420616) + + [Persistence of local site network address configuration](rep_mgrmulti.md#idp52417008) + + [Programming considerations](rep_mgrmulti.md#idp52400144) + + [Handling failure](rep_mgrmulti.md#idp52414488) + + [Other miscellaneous rules](rep_mgrmulti.md#idp52412256) + +Replication Manager supports shared access to a database environment from multiple processes. + +### One replication process and multiple subordinate processes + +Each site in a replication group has just one network address (TCP/IP host name and port number). This means that only one process can accept incoming connections. At least one application process must invoke the DB_ENV->repmgr_start() method to initiate communications and management of the replication state. + +If it is convenient, multiple processes may issue calls to the Replication Manager configuration methods, and multiple processes may call DB_ENV->repmgr_start(). Replication Manager automatically opens the TCP/IP listening socket in the first process to do so (we'll call it the "replication process" here), and ignores this step in any subsequent processes ("subordinate processes"). + +### Persistence of local site network address configuration + +The local site network address is stored in shared memory, and remains intact even when (all) processes close their environment handles gracefully and terminate. A process which opens an environment handle without running recovery automatically inherits the existing local site network address configuration. Such a process may not change the local site address (although it is allowed to redundantly specify a local site configuration matching that which is already in effect). + +In order to change the local site network address, the application must run recovery. The application can then specify a new local site address before restarting Replication Manager. The application should also remove the old local site address from the replication group if it is no longer needed. + +### Programming considerations + +Note that Replication Manager applications must follow all the usual rules for Berkeley DB multi-threaded and/or multi-process applications, such as ensuring that the recovery operation occurs single-threaded, only once, before any other thread or processes operate in the environment. Since Replication Manager creates its own background threads which operate on the environment, all environment handles must be opened with the DB_THREAD flag, even if the application is otherwise single-threaded per process. + +At the replication master site, each Replication Manager process opens outgoing TCP/IP connections to all clients in the replication group. It uses these direct connections to send to clients any log records resulting from update transactions that the process executes. But all other replication activity —message processing, elections, etc.— takes place only in the "replication process". + +Replication Manager notifies the application of certain events, using the callback function configured with the DB_ENV->set_event_notify() method. These notifications occur only in the process where the event itself occurred. Generally this means that most notifications occur only in the "replication process". Currently the only replication notification that can occur in a "subordinate process" is DB_EVENT_REP_PERM_FAILED. + +It is not supported for a process running Replication Manager to spawn a subprocess. + +### Handling failure + +Multi-process Replication Manager applications should handle failures in a manner consistent with the rules described in Handling failure in Transactional Data Store applications. To summarize, there are two ways to handle failure of a process: + +1. The simple way is to kill all remaining processes, run recovery, and then restart all processes from the beginning. But this can be a bit drastic. + +2. Using the DB_ENV->failchk() method, it is sometimes possible to leave surviving processes running, and just restart the failed process. + + Multi-process Replication Manager applications using this technique must start a new process when an old process fails. It is not possible for a "subordinate process" to take over the duties of a failed "replication process". If the failed process happens to be the replication process, then after a failchk() call the next process to call DB_ENV->repmgr_start() will become the new replication process. + +### Other miscellaneous rules + +1. A database environment may not be shared between a Replication Manager application process and a Base API application process. +2. It is not possible to run multiple Replication Manager processes during mixed-version live upgrades from Berkeley DB versions prior to 4.8. diff --git a/docs-src/guides/programmer_reference/rep_newsite.md b/docs-src/guides/programmer_reference/rep_newsite.md new file mode 100644 index 000000000..2976f6cf4 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_newsite.md @@ -0,0 +1,14 @@ +--- +title: "Connecting to a new site" +api-name: "Connecting to a new site" +source: docs/programmer_reference/rep_newsite.html +--- +## Connecting to a new site + +To add a new site to the replication group all that is needed is for the client member to join. Berkeley DB will perform an internal initialization from the master to the client automatically and will run recovery on the client to bring it up to date with the master. + +For Base API applications, connecting to a new site in the replication group happens whenever the DB_ENV->rep_process_message() method returns DB_REP_NEWSITE. The application should assign the new site a local environment ID number, and all future messages from the site passed to DB_ENV->rep_process_message() should include that environment ID number. It is possible, of course, for the application to be aware of a new site before the return of DB_ENV->rep_process_message() (for example, applications using connection-oriented protocols are likely to detect new sites immediately, while applications using broadcast protocols may not). + +Regardless, in applications supporting the dynamic addition of database environments to replication groups, environments joining an existing replication group may need to provide contact information. (For example, in an application using TCP/IP sockets, a DNS name or IP address might be a reasonable value to provide.) This can be done using the **cdata** parameter to the DB_ENV->rep_start() method. The information referenced by **cdata** is wrapped in the initial contact message sent by the new environment, and is provided to the existing members of the group using the **rec** parameter returned by DB_ENV->rep_process_message(). If no additional information was provided for Berkeley DB to forward to the existing members of the group, the **data** field of the **rec** parameter passed to the DB_ENV->rep_process_message() method will be NULL after DB_ENV->rep_process_message() returns DB_REP_NEWSITE. + +Replication Manager automatically distributes contact information using the mechanisms previously described. diff --git a/docs-src/guides/programmer_reference/rep_partition.md b/docs-src/guides/programmer_reference/rep_partition.md new file mode 100644 index 000000000..abbae5f03 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_partition.md @@ -0,0 +1,28 @@ +--- +title: "Network partitions" +api-name: "Network partitions" +source: docs/programmer_reference/rep_partition.html +--- +## Network partitions + +The Berkeley DB replication implementation can be affected by network partitioning problems. + +For example, consider a replication group with N members. The network partitions with the master on one side and more than N/2 of the sites on the other side. The sites on the side with the master will continue forward, and the master will continue to accept write queries for the databases. Unfortunately, the sites on the other side of the partition, realizing they no longer have a master, will hold an election. The election will succeed as there are more than N/2 of the total sites participating, and there will then be two masters for the replication group. Since both masters are potentially accepting write queries, the databases could diverge in incompatible ways. + +If multiple masters are ever found to exist in a replication group, a master detecting the problem will return DB_REP_DUPMASTER. If the application sees this return, it should reconfigure itself as a client (by calling DB_ENV->rep_start()), and then call for an election (by calling DB_ENV->rep_elect()). The site that wins the election may be one of the two previous masters, or it may be another site entirely. Regardless, the winning system will bring all of the other systems into conformance. + +As another example, consider a replication group with a master environment and two clients A and B, where client A may upgrade to master status and client B cannot. Then, assume client A is partitioned from the other two database environments, and it becomes out-of-date with respect to the master. Then, assume the master crashes and does not come back on-line. Subsequently, the network partition is restored, and clients A and B hold an election. As client B cannot win the election, client A will win by default, and in order to get back into sync with client B, possibly committed transactions on client B will be unrolled until the two sites can once again move forward together. + +In both of these examples, there is a phase where a newly elected master brings the members of a replication group into conformance with itself so that it can start sending new information to them. This can result in the loss of information as previously committed transactions are unrolled. + +In architectures where network partitions are an issue, applications may want to implement a heart-beat protocol to minimize the consequences of a bad network partition. As long as a master is able to contact at least half of the sites in the replication group, it is impossible for there to be two masters. If the master can no longer contact a sufficient number of systems, it should reconfigure itself as a client, and hold an election. Replication Manager does not currently implement such a feature, so this technique is only available to Base API applications. + +There is another tool applications can use to minimize the damage in the case of a network partition. By specifying an **nsites** argument to DB_ENV->rep_elect() that is larger than the actual number of database environments in the replication group, Base API applications can keep systems from declaring themselves the master unless they can talk to a large percentage of the sites in the system. For example, if there are 20 database environments in the replication group, and an argument of 30 is specified to the DB_ENV->rep_elect() method, then a system will have to be able to talk to at least 16 of the sites to declare itself the master. + +Replication Manager uses the value of **nsites** (configured by the DB_ENV->rep_set_nsites() method) for elections as well as in calculating how many acknowledgements to wait for when sending a DB_REP_PERMANENT message. So this technique may be useful here as well, unless the application uses the DB_REPMGR_ACKS_ALL or DB_REPMGR_ACKS_ALL_PEERS acknowledgement policies. + +Specifying a **nsites** argument to DB_ENV->rep_elect() that is smaller than the actual number of database environments in the replication group has its uses as well. For example, consider a replication group with 2 environments. If they are partitioned from each other, neither of the sites could ever get enough votes to become the master. A reasonable alternative would be to specify a **nsites** argument of 2 to one of the systems and a **nsites** argument of 1 to the other. That way, one of the systems could win elections even when partitioned, while the other one could not. This would allow one of the systems to continue accepting write queries after the partition. + +In a 2-site group, Replication Manager by default reacts to the loss of communication with the master by observing a strict majority rule that prevents the survivor from taking over. Thus it avoids multiple masters and the need to unroll some transactions if both sites are running but cannot communicate. But it does leave the group in a read-only state until both sites are available. If application availability while one site is down is a priority and it is acceptable to risk unrolling some transactions, there is a configuration option to turn off the strict majority rule and allow the surviving client to declare itself to be master. See the DB_ENV->rep_set_config() method DB_REPMGR_CONF_2SITE_STRICT flag for more information. + +These scenarios stress the importance of good network infrastructure in Berkeley DB replicated environments. When replicating database environments over sufficiently lossy networking, the best solution may well be to pick a single master, and only hold elections when human intervention has determined the selected master is unable to recover at all. diff --git a/docs-src/guides/programmer_reference/rep_pri.md b/docs-src/guides/programmer_reference/rep_pri.md new file mode 100644 index 000000000..b00be399b --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_pri.md @@ -0,0 +1,12 @@ +--- +title: "Replication environment priorities" +api-name: "Replication environment priorities" +source: docs/programmer_reference/rep_pri.html +--- +## Replication environment priorities + +Each database environment included in a replication group must have a priority, which specifies a relative ordering among the different environments in a replication group. This ordering is a factor in determining which environment will be selected as a new master in case the existing master fails. Both Replication Manager applications and Base API applications should specify environment priorities. + +Priorities are an unsigned integer, but do not need to be unique throughout the replication group. A priority of 0 means the system can never become a master. Otherwise, larger valued priorities indicate a more desirable master. For example, if a replication group consists of three database environments, two of which are connected by an OC3 and the third of which is connected by a T1, the third database environment should be assigned a priority value which is lower than either of the other two. + +Desirability of the master is first determined by the client having the most recent log records. Ties in log records are broken with the client priority. If both sites have the same log and the same priority, one is selected at random. diff --git a/docs-src/guides/programmer_reference/rep_replicate.md b/docs-src/guides/programmer_reference/rep_replicate.md new file mode 100644 index 000000000..e9e91dabd --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_replicate.md @@ -0,0 +1,98 @@ +--- +title: "Running Replication using the db_replicate Utility" +api-name: "Running Replication using the db_replicate Utility" +source: docs/programmer_reference/rep_replicate.html +--- +## Running Replication using the db_replicate Utility + + [One Replication Process and Multiple Subordinate Processes](rep_replicate.md#idp52430544) + + [Common Use Case](rep_replicate.md#idp52447760) + + [Avoiding Rollback](rep_replicate.md#idp52457840) + + [When to Consider an Integrated HA Application](rep_replicate.md#idp52462952) + +Replication Manager supports shared access to a database environment from multiple processes. Berkeley DB provides a replication-aware utility, db_replicate, that enables you to upgrade an existing Transactional Data Store application, as discussed in the Transactional Data Store introduction section, to an HA application with minor modifications. While the db_replicate utility simplifies the use of replication with a TDS application, you must still understand replication and its impact on the application. + +### One Replication Process and Multiple Subordinate Processes + +Based on the terminology introduced in the Running Replication Manager in multiple processes section, application processes are "subordinate processes" and the db_replicate utility is the "primary replication process". + +You must consider the following items when planning to use the db_replicate utility in combination with a TDS application. + +- Memory regions + + The db_replicate utility requires shared memory access among separate processes, and therefore cannot be used with DB_PRIVATE. + +- Multi-process implications + + You must understand and accept all of the TDS implications of multi-process use as specified in Architecting Transactional Data Store applications. Special attention should be paid to the coordination needed for unrelated processes to start up correctly. + +- Replication configuration + + Several configuration settings are required for replication. You must set the DB_INIT_REP and DB_THREAD flags for the DB_ENV->open() method. Another required configuration item is the local address. You identify this by creating a DB_SITE handle and then setting the `DB_LOCAL_SITE` parameter using the DB_SITE->set_config() method. You also tell sites how to contact other sites by creating DB_SITE handles for those sites. Most replication configuration options start with reasonable defaults, but applications have to customize at least some of them. You can set all replication related configuration options either programmatically or in the DB_CONFIG file. + +- Starting the application and replication + + The db_replicate utility assumes that an environment exists and that the application has run recovery, if necessary, and created and configured the environment. The startup flow of a typical TDS application may not be the best flow for a replication application and you must understand the issues involved. For instance, if an application starts, runs recovery, and performs update operations before starting the db_replicate utility, then if that site becomes a client when replication starts, those update operations will be rolled back. + +- Handling events + + Almost all of the replication-specific events are handled by the db_replicate utility process, and therefore the application process does not see them. If the application needs to know the information from those replication-specific events, such as role changes, the application must call the rep_stat() method method. The one replication-specific event the application can now receive is the DB_EVENT_REP_PERM_FAILED event. See Choosing a Replication Manager Ack Policy for additional information about this event. + +- Handling errors + + There are some error return values that relate only to replication. Specifically, the `DB_REP_HANDLE_DEAD` error should now be handled by the application. Also, if master leases are in use, then the application also needs to consider the `DB_REP_LEASE_EXPIRED` error. + +- Flexibility tradeoff + + You are giving up flexibility for the ease of use of the utility. Application complexity or requirements may eventually dictate integrating HA calls into the application over using the db_replicate utility. + +- Read-only client application + + The application requires additional changes to manage the read-only status when the application takes on the role of a client. + +### Common Use Case + +This section lists the steps needed to get replication running for a common use case of the db_replicate utility. The use case presented is an existing TDS application that already has its environment and databases created and is up and running. At some point, HA is considered because failover protection or balancing the read load may now be desired. + +1. To understand the issues involved in a replication/HA application, see the db_replicate utility section in the *API Reference Guide*, the Replication Chapter in the *Programmer's Reference Guide*, and the source code of the ex_rep_mgr example program. + +2. Make a local hot backup of the current application environment to a new location to use as a testing area. + +3. Add the DB_INIT_REP and DB_THREAD flags (if not already being used) to the application or the DB_CONFIG file. + +4. Modify the DB_CONFIG file to add the necessary replication configuration values. At a minimum, the local host and port information must be added using the repmgr_site method parameter. As more sites are added to the group, remote host and port information can optionally also be added by adding more repmgr_site method parameters to the DB_CONFIG file file. + +5. Rebuild the application and restart it in the current testing directory. + +6. Start the db_replicate utility on the master site with the -M option and any other options needed such as -h for the home directory. At this point you have a lone master site running in an environment with no other replicated sites in the group. + +7. Optionally, prepare to start a client site by performing a manual hot backup of the running master environment to initialize a client target directory. While replication can make its own copy, the hot backup will expedite the synchronization process. Also, if the application assumes the existence of a database and the client site is started without data, the application may have errors or incorrectly attempt to create the database. + +8. Copy the application to the client target. + +9. Modify the client environment's DB_CONFIG file to set the client's local host and port values and to add remote site information for the master site and any other replication configuration choices necessary. + +10. Start the application on the client. The client application should not update data at this point, as explained previously. + +11. Start the db_replicate utility specifying the client environment's home directory using the -h option. Omit the -M option in this case, because the utility defaults to starting in the client role. + +Once the initial replication group is established, do not use the -M option with the db_replicate utility. After the initial start, db_replicate utility assumes the use of elections. If a site crashes, it should rejoin the group as a client so that it can synchronize with the rest of the group. + +### Avoiding Rollback + +Depending on how an application is structured, transactional rollback can occur. If this is possible, then you must make application changes or be prepared for successful transactions to disappear. Consider a common program flow where the application first creates and opens the environment with recovery. Then, immediately after that, the application opens up the databases it expects to use. Often an application will use the DB_CREATE flag so that if the database does not exist it is created, otherwise the existing one is used automatically. Then the application begins servicing transactions to write and read data. + +When replication is introduced, particularly via the db_replicate utility, the possibility of rollback exists unless the application takes steps to prevent it. In the situation described above, if all of the above steps occur before the db_replicate utility process starts, and the site is started as a client, then all the operations will be rolled back when the site finds the master. The client site will synchronize with the log and operations on the master site, so any operations that occurred in the client application before it knew it was a client will be discarded. + +One way to reduce the possibility of rollback is to modify the application so that it only performs update operations (including creation of a database) if it is the master site. If the application refrains from updating until it is the master, then it will not perform operations when it is in the undefined state before replication has been started. The event indicating a site is master will be delivered to the db_replicate utility process, so the application process must look for that information via the rep_stat() method. A site that is expecting to perform updates may need to poll via the rep_stat() method to see the state change from an undefined role to either the master or client role. Similarly, since a client site cannot create a database, it may need to poll for the database's existence while the client synchronizes with the master until the database is created at the client site. + +### When to Consider an Integrated HA Application + +The db_replicate utility provides the means to achieve a replicated application quickly. However, the trade-off for this rapid implementation is that the full flexibility of replication is not available. Some applications may eventually need to consider integrating directly with replication rather than using the db_replicate utility if greater flexibility is desired. + +One likely reason for considering integration would be the convenience of receiving all replication-related events in the application process and gaining direct knowledge of such things as role changes. Using the event callback is cleaner and easier than polling for state changes via the rep_stat() method. + +A second likely reason for integrating replication directly into the application is the multi-process aspect of the utility program. The developer may find it easier to insert the start of replication directly into the code once the environment is created, recovered, or opened, and avoid the scenario where the application is running in the undefined state. Also it may simply be easier to start the application once than to coordinate different processes and their startup order in the system. diff --git a/docs-src/guides/programmer_reference/rep_ryw.md b/docs-src/guides/programmer_reference/rep_ryw.md new file mode 100644 index 000000000..1fbdf8be1 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_ryw.md @@ -0,0 +1,48 @@ +--- +title: "Read your writes consistency" +api-name: "Read your writes consistency" +source: docs/programmer_reference/rep_ryw.html +--- +## Read your writes consistency + + [Getting a token](rep_ryw.md#gettoken) + + [Token handling](rep_ryw.md#tokenhandling) + + [Using a token to check or wait for a transaction](rep_ryw.md#usingtoken) + +Some applications require the ability to read replicated data at a client site, and determine whether it is consistent with data that has been written previously at the master site. + +For example, a web application may be backed by multiple database environments, linked to form a replication group, in order to share the workload. Web requests that update data must be served by the replication master, but any site in the group may serve a read-only request. Consider a work flow of a series of web requests from one specific user at a web browser: the first request generates a database update, but the second request merely reads data. If the read-only request is served by a replication client database environment, it may be important to make sure that the updated data has been replicated to the client before performing the read (or to wait until it has been replicated) in order to show this user a consistent view of the data. + +Berkeley DB supports this requirement through the use of transaction "tokens". A token is a form of identification for a transaction within the scope of the replication group. The application may request a copy of the transaction's token at the master site during the execution of the transaction. Later, the application running on a client site can use a copy of the token to determine whether the transaction has been applied at that site. + +It is the application's responsibility to keep track of the token during the interim. In the web example, the token might be sent to the browser as a "cookie", or stored on the application server in the user's session context. + +The operations described here are supported both for Replication Manager applications and for applications that use the replication Base API. + +### Getting a token + +In order to get a token, the application must supply a small memory buffer, using the DB_TXN->set_commit_token() method. + +Note that a token is generated only upon a successful commit operation, and therefore the token buffer content is valid only after a successful commit. Also, if a transaction does not perform any update operations it does not generate a useful token. + +In the Berkeley DB Java and C# API, getting a token is simpler. The application need only invoke the Transaction.getCommitToken() method, after the transaction has committed. + +### Token handling + +The application should not try to interpret the content of the token buffer, but may store and/or transmit it freely between systems. However, since the buffer contains binary data it may be necessary to apply some encoding for transmission (e.g., base 64). + +The data is resilient to differences in byte order between different systems. It does not expire: it may be retained indefinitely for later use, even across Berkeley DB version upgrades. + +### Using a token to check or wait for a transaction + +The DB_ENV->txn_applied() method takes a copy of a token, and determines whether the corresponding transaction is currently applied at the local site. The timeout argument allows the application to block for a bounded amount of time for cases where the transaction has not yet been applied. + +Depending on the transaction durability levels implemented or configured by the application, it is sometimes possible for a transaction to disappear from a replication group if an original master site fails and a different site becomes the new master without having received the transaction. When the DB_ENV->txn_applied() method discovers this, it produces the `DB_NOTFOUND` return code. + +This means that the results of DB_ENV->txn_applied() are not guaranteed forever. Even after a successful call to DB_ENV->txn_applied(), it is possible that by the time the application tries to read the data, the transaction and its data could have disappeared. + +To avoid this problem the application should do the read operations in the context of a transaction, and hold the transaction handle open during the DB_ENV->txn_applied() call. The DB_ENV->txn_applied() method itself does not actually execute in the context of the transaction; but no rollbacks due to new master synchronization ever occur while a transaction is active, even a read-only transaction at a client site. + +Note that the DB_ENV->txn_applied() method can return `DB_LOCK_DEADLOCK`. The application should respond to this situation just as it does for any other normal operation: abort any existing transaction, and then pause briefly before retrying. diff --git a/docs-src/guides/programmer_reference/rep_trans.md b/docs-src/guides/programmer_reference/rep_trans.md new file mode 100644 index 000000000..1f7b0929c --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_trans.md @@ -0,0 +1,62 @@ +--- +title: "Transactional guarantees" +api-name: "Transactional guarantees" +source: docs/programmer_reference/rep_trans.html +--- +## Transactional guarantees + +It is important to consider replication in the context of the overall database environment's transactional guarantees. To briefly review, transactional guarantees in a non-replicated application are based on the writing of log file records to "stable storage", usually a disk drive. If the application or system then fails, the Berkeley DB logging information is reviewed during recovery, and the databases are updated so that all changes made as part of committed transactions appear, and all changes made as part of uncommitted transactions do not appear. In this case, no information will have been lost. + +If a database environment does not require the log be flushed to stable storage on transaction commit (using the DB_TXN_NOSYNC flag to increase performance at the cost of sacrificing transactional durability), Berkeley DB recovery will only be able to restore the system to the state of the last commit found on stable storage. In this case, information may have been lost (for example, the changes made by some committed transactions may not appear in the databases after recovery). + +Further, if there is database or log file loss or corruption (for example, if a disk drive fails), then catastrophic recovery is necessary, and Berkeley DB recovery will only be able to restore the system to the state of the last archived log file. In this case, information may also have been lost. + +Replicating the database environment extends this model, by adding a new component to "stable storage": the client's replicated information. If a database environment is replicated, there is no lost information in the case of database or log file loss, because the replicated system can be configured to contain a complete set of databases and log records up to the point of failure. A database environment that loses a disk drive can have the drive replaced, and it can then rejoin the replication group. + +Because of this new component of stable storage, specifying DB_TXN_NOSYNC in a replicated environment no longer sacrifices durability, as long as one or more clients have acknowledged receipt of the messages sent by the master. Since network connections are often faster than local synchronous disk writes, replication becomes a way for applications to significantly improve their performance as well as their reliability. + +The return status from the application's **send** function must be set by the application to ensure the transactional guarantees the application wants to provide. Whenever the **send** function returns failure, the local database environment's log is flushed as necessary to ensure that any information critical to database integrity is not lost. Because this flush is an expensive operation in terms of database performance, applications should avoid returning an error from the **send** function, if at all possible. + +The only interesting message type for replication transactional guarantees is when the application's **send** function was called with the DB_REP_PERMANENT flag specified. There is no reason for the **send** function to ever return failure unless the DB_REP_PERMANENT flag was specified -- messages without the DB_REP_PERMANENT flag do not make visible changes to databases, and the **send** function can return success to Berkeley DB as soon as the message has been sent to the client(s) or even just copied to local application memory in preparation for being sent. + +When a client receives a DB_REP_PERMANENT message, the client will flush its log to stable storage before returning (unless the client environment has been configured with the DB_TXN_NOSYNC option). If the client is unable to flush a complete transactional record to disk for any reason (for example, there is a missing log record before the flagged message), the call to the DB_ENV->rep_process_message() method on the client will return DB_REP_NOTPERM and return the LSN of this record to the application in the **ret_lsnp** parameter. The application's client or master message handling loops should take proper action to ensure the correct transactional guarantees in this case. When missing records arrive and allow subsequent processing of previously stored permanent records, the call to the DB_ENV->rep_process_message() method on the client will return DB_REP_ISPERM and return the largest LSN of the permanent records that were flushed to disk. Client applications can use these LSNs to know definitively if any particular LSN is permanently stored or not. + +An application relying on a client's ability to become a master and guarantee that no data has been lost will need to write the **send** function to return an error whenever it cannot guarantee the site that will win the next election has the record. Applications not requiring this level of transactional guarantees need not have the **send** function return failure (unless the master's database environment has been configured with DB_TXN_NOSYNC), as any information critical to database integrity has already been flushed to the local log before **send** was called. + +To sum up, the only reason for the **send** function to return failure is when the master database environment has been configured to not synchronously flush the log on transaction commit (that is, DB_TXN_NOSYNC was configured on the master), the DB_REP_PERMANENT flag is specified for the message, and the **send** function was unable to determine that some number of clients have received the current message (and all messages preceding the current message). How many clients need to receive the message before the **send** function can return success is an application choice (and may not depend as much on a specific number of clients reporting success as one or more geographically distributed clients). + +If, however, the application does require on-disk durability on the master, the master should be configured to synchronously flush the log on commit. If clients are not configured to synchronously flush the log, that is, if a client is running with DB_TXN_NOSYNC configured, then it is up to the application to reconfigure that client appropriately when it becomes a master. That is, the application must explicitly call DB_ENV->set_flags() to disable asynchronous log flushing as part of re-configuring the client as the new master. + +Of course, it is important to ensure that the replicated master and client environments are truly independent of each other. For example, it does not help matters that a client has acknowledged receipt of a message if both master and clients are on the same power supply, as the failure of the power supply will still potentially lose information. + +Configuring your replication-based application to achieve the proper mix of performance and transactional guarantees can be complex. In brief, there are a few controls an application can set to configure the guarantees it makes: specification of DB_TXN_NOSYNC for the master environment, specification of DB_TXN_NOSYNC for the client environment, the priorities of different sites participating in an election, and the behavior of the application's **send** function. + +Applications using Replication Manager are free to use DB_TXN_NOSYNC at the master and/or clients as they see fit. The behavior of the **send** function that Replication Manager provides on the application's behalf is determined by an "acknowledgement policy", which is configured by the DB_ENV->repmgr_set_ack_policy() method. Clients always send acknowledgements for DB_REP_PERMANENT messages (unless the acknowledgement policy in effect indicates that the master doesn't care about them). For a DB_REP_PERMANENT message, the master blocks the sending thread until either it receives the proper number of acknowledgements, or the DB_REP_ACK_TIMEOUT expires. In the case of timeout, Replication Manager returns an error code from the **send** function, causing Berkeley DB to flush the transaction log before returning to the application, as previously described. The default acknowledgement policy is DB_REPMGR_ACKS_QUORUM, which ensures that the effect of a permanent record remains durable following an election. + +First, it is rarely useful to write and synchronously flush the log when a transaction commits on a replication client. It may be useful where systems share resources and multiple systems commonly fail at the same time. By default, all Berkeley DB database environments, whether master or client, synchronously flush the log on transaction commit or prepare. Generally, replication masters and clients turn log flush off for transaction commit using the DB_TXN_NOSYNC flag. + +Consider two systems connected by a network interface. One acts as the master, the other as a read-only client. The client takes over as master if the master crashes and the master rejoins the replication group after such a failure. Both master and client are configured to not synchronously flush the log on transaction commit (that is, DB_TXN_NOSYNC was configured on both systems). The application's **send** function never returns failure to the Berkeley DB library, simply forwarding messages to the client (perhaps over a broadcast mechanism), and always returning success. On the client, any DB_REP_NOTPERM returns from the client's DB_ENV->rep_process_message() method are ignored, as well. This system configuration has excellent performance, but may lose data in some failure modes. + +If both the master and the client crash at once, it is possible to lose committed transactions, that is, transactional durability is not being maintained. Reliability can be increased by providing separate power supplies for the systems and placing them in separate physical locations. + +If the connection between the two machines fails (or just some number of messages are lost), and subsequently the master crashes, it is possible to lose committed transactions. Again, transactional durability is not being maintained. Reliability can be improved in a couple of ways: + +1. Use a reliable network protocol (for example, TCP/IP instead of UDP). + +2. Increase the number of clients and network paths to make it less likely that a message will be lost. In this case, it is important to also make sure a client that did receive the message wins any subsequent election. If a client that did not receive the message wins a subsequent election, data can still be lost. + +Further, systems may want to guarantee message delivery to the client(s) (for example, to prevent a network connection from simply discarding messages). Some systems may want to ensure clients never return out-of-date information, that is, once a transaction commit returns success on the master, no client will return old information to a read-only query. Some of the following changes to a Base API application may be used to address these issues: + +1. Write the application's **send** function to not return to Berkeley DB until one or more clients have acknowledged receipt of the message. The number of clients chosen will be dependent on the application: you will want to consider likely network partitions (ensure that a client at each physical site receives the message) and geographical diversity (ensure that a client on each coast receives the message). + +2. Write the client's message processing loop to not acknowledge receipt of the message until a call to the DB_ENV->rep_process_message() method has returned success. Messages resulting in a return of DB_REP_NOTPERM from the DB_ENV->rep_process_message() method mean the message could not be flushed to the client's disk. If the client does not acknowledge receipt of such messages to the master until a subsequent call to the DB_ENV->rep_process_message() method returns DB_REP_ISPERM and the LSN returned is at least as large as this message's LSN, then the master's **send** function will not return success to the Berkeley DB library. This means the thread committing the transaction on the master will not be allowed to proceed based on the transaction having committed until the selected set of clients have received the message and consider it complete. + + Alternatively, the client's message processing loop could acknowledge the message to the master, but with an error code indicating that the application's **send** function should not return to the Berkeley DB library until a subsequent acknowledgement from the same client indicates success. + + The application send callback function invoked by Berkeley DB contains an LSN of the record being sent (if appropriate for that record). When DB_ENV->rep_process_message() method returns indicators that a permanent record has been written then it also returns the maximum LSN of the permanent record written. + +There is one final pair of failure scenarios to consider. First, it is not possible to abort transactions after the application's **send** function has been called, as the master may have already written the commit log records to disk, and so abort is no longer an option. Second, a related problem is that even though the master will attempt to flush the local log if the **send** function returns failure, that flush may fail (for example, when the local disk is full). Again, the transaction cannot be aborted as one or more clients may have committed the transaction even if **send** returns failure. Rare applications may not be able to tolerate these unlikely failure modes. In that case the application may want to: + +1. Configure the master to do always local synchronous commits (turning off the DB_TXN_NOSYNC configuration). This will decrease performance significantly, of course (one of the reasons to use replication is to avoid local disk writes.) In this configuration, failure to write the local log will cause the transaction to abort in all cases. + +2. Do not return from the application's **send** function under any conditions, until the selected set of clients has acknowledged the message. Until the **send** function returns to the Berkeley DB library, the thread committing the transaction on the master will wait, and so no application will be able to act on the knowledge that the transaction has committed. diff --git a/docs-src/guides/programmer_reference/rep_twosite.md b/docs-src/guides/programmer_reference/rep_twosite.md new file mode 100644 index 000000000..2720f0c81 --- /dev/null +++ b/docs-src/guides/programmer_reference/rep_twosite.md @@ -0,0 +1,24 @@ +--- +title: "Special considerations for two-site replication groups" +api-name: "Special considerations for two-site replication groups" +source: docs/programmer_reference/rep_twosite.html +--- +## Special considerations for two-site replication groups + +One of the benefits of replication is that it helps your application remain available for writes even when a site crashes. Another benefit is the added durability achieved by storing multiple copies of your application data at different sites. However, if your replication group contains only two sites, you must prioritize which of these benefits is more important to your application. + +A two-site replication group is particularly vulnerable to duplicate masters if there is a loss of communication between the sites. The original master continues to accept new transactions. If the original client detects the loss of the master and elects itself master, it also starts accepting new transactions. When communications are restored, there are duplicate masters and one site's new transactions will be rolled back. + +If it is unacceptable to your application for any new transactions to be rolled back, the alternative in a two-site replication group is to require both sites to be present in order to elect a master. This stops a client from electing itself master when it loses contact with the master and prevents creation of parallel sets of transactions, one of which must be rolled back. + +However, requiring both sites to be present to elect a master results in a loss of write availability when the master crashes. The client cannot take over as master and the replication group exists in a read-only state until the original master site rejoins the replication group. + +Replication Manager applications use the DB_ENV->rep_set_config() method DB_REPMGR_CONF_2SITE_STRICT flag to make this tradeoff between write availability and transaction durability. When this flag is turned on, Replication Manager favors transaction durability. When it is turned off, Replication Manager favors write availability. + +A two-site Replication Manager application that uses heartbeats in an environment with frequent communications disruptions generally should operate with the DB_REPMGR_CONF_2SITE_STRICT flag turned on. Otherwise, frequent heartbeat failures will cause frequent duplicate masters and the resulting elections and client synchronizations will make one or both sites unavailable for extended periods of time. + +Base API applications use the values of the **nvotes** and **nsites** parameters in calls to the DB_ENV->rep_elect() method to make this tradeoff. For more information, see Elections. + +A replication group containing only two electable sites is subject to duplicate masters and rollback of one site's new transactions even when it contains additional unelectable sites. The DB_REPMGR_CONF_2SITE_STRICT does not apply in this case because the replication group is larger than two sites. + +If both write availability and transaction durability are important to your application, you should strongly consider having three or more electable sites in your replication group. You should also carefully choose an acknowledgement policy that requires at least a quorum of sites. It is best to have an odd number of electable sites to provide a clear majority in the event of a network partition. diff --git a/docs-src/guides/programmer_reference/repmgr_channels.md b/docs-src/guides/programmer_reference/repmgr_channels.md new file mode 100644 index 000000000..3721f26ed --- /dev/null +++ b/docs-src/guides/programmer_reference/repmgr_channels.md @@ -0,0 +1,68 @@ +--- +title: "Using Replication Manager message channels" +api-name: "Using Replication Manager message channels" +source: docs/programmer_reference/repmgr_channels.html +--- +## Using Replication Manager message channels + + [DB_CHANNEL](repmgr_channels.md#dbchannel_class) + + [Sending messages over a message channel](repmgr_channels.md#dbchannel_send) + + [Receiving messages](repmgr_channels.md#dbchannel_receive) + +The various sites comprising a replication group frequently need to communicate with one another. Mostly, these messages are handled for you internally by the Replication Manager. However, your application may have a requirement to pass messages beyond what the Replication Manager requires in order to satisfy its own internal workings. + +For this reason, you can access and use the Replication Manager's internal message channels. You do this by using the `DB_CHANNEL` class, and by implementing a message handling function on each of your sites. + +Note that an example of using Replication Manager message channels is available in the distribution. See Ex_rep_chan: a Replication Manager channel example for more information. + +### DB_CHANNEL + +The `DB_CHANNEL` class provides a series of methods which allow you to send messages to the other sites in your replication group. You create a `DB_CHANNEL` handle using the DB_ENV->repmgr_channel() method. When you are done with the handle, close it using the DB_CHANNEL->close() method. A closed handle must never be accessed again. Note that all channel handles should be closed before the associated environment handle is closed. Also, allow all message operations to complete on the channel before closing the handle. + +When you create a `DB_CHANNEL` handle, you indicate what channel you want to use. Possibilities are: + +- The numerical env ID of a remote site in the replication group. + +- `DB_EID_MASTER` + + Messages sent on this channel are sent only to the master site. Note that messages are always sent to the current master, even if the master has changed since the channel was opened. + + If the local site is the master, then sending messages on this channel will result in the local site receiving those messages echoed back to itself. + +### Sending messages over a message channel + +You can send any message you want over a message channel. The message can be as simple as a character string and as complex as a large data structure. However, before you can send the message, you must encapsulate it within one or more DBTs. This means marshaling the message if it is contained within a complex data structure. + +The methods that you use to send messages all accept an array of DBTs. This means that in most circumstances it is perfectly acceptable to send multi-part messages. + +Messages may be sent either asynchronously or synchronously. To send a message asynchronously, use the DB_CHANNEL->send_msg() method. This method sends its message and then immediately returns without waiting for any sort of a response. + +To send a message synchronously, use the DB_CHANNEL->send_request() method. This method blocks until it receives a response from the site to which it sent the message (or until a timeout threshold is reached). + +#### Message Responses + +Message responses are required if a message is sent on a channel using the DB_CHANNEL->send_request() method. That method accepts the address of a single DBT which is used to receive the response from the remote site. + +Message responses are encapsulated in a single DBT. The response can be anything from a complex data structure, to a string, to a simple type, to no information at all. In the latter case, receipt of the DBT is sufficient to indicate that the request was received at the remote site. + +Responses are sent back from the remote system using its message handling function. Usually that function calls DB_CHANNEL->send_msg() to send a single response. + +The response must be contained in a single DBT. If a multi-part response is required by the application, you can configure the response DBT that you provide to DB_CHANNEL->send_request() for bulk operations. + +### Receiving messages + +Messages received at a remote site are handled using a callback function. This function is configured for the local environment using the DB_ENV->repmgr_msg_dispatch() method. For best results, the message dispatch function should be configured for the local environment before replication is started. In this way, you do not run the risk of missing messages sent after replication has started but before the message dispatch function is configured for the environment. + +The callback configured by DB_ENV->repmgr_msg_dispatch() accepts four parameters of note: + +- A response channel. This is the channel the function will use to response to the message, if a response is required. To respond to the message, the function uses the DB_CHANNEL->send_msg() method. + +- An array of DBTs. These hold the message that this function must handle. + +- A numerical value that indicates how many elements the previously described array holds. + +- A flag that indicates whether the message requires a response. If the flag is set to `DB_REPMGR_NEED_RESPONSE`, then the function should send a single DBT in response using the channel provided to this function, and the DB_CHANNEL->send_msg() method. + +For an example of using this callback, see the `operation_dispatch()` function, which is available with the ex_rep_chan example in your product distribution. diff --git a/docs-src/guides/programmer_reference/rq_conf.md b/docs-src/guides/programmer_reference/rq_conf.md new file mode 100644 index 000000000..4ff07b8bf --- /dev/null +++ b/docs-src/guides/programmer_reference/rq_conf.md @@ -0,0 +1,72 @@ +--- +title: "Queue and Recno access method specific configuration" +api-name: "Queue and Recno access method specific configuration" +source: docs/programmer_reference/rq_conf.html +--- +## Queue and Recno access method specific configuration + + [Managing record-based databases](rq_conf.md#am_conf_recno) + + [Selecting a Queue extent size](rq_conf.md#am_conf_extentsize) + + [Flat-text backing files](rq_conf.md#am_conf_re_source) + + [Logically renumbering records](rq_conf.md#am_conf_renumber) + +There are a series of configuration tasks which you can perform when using the Queue and Recno access methods. They are described in the following sections. + +### Managing record-based databases + +When using fixed- or variable-length record-based databases, particularly with flat-text backing files, there are several items that the user can control. The Recno access method can be used to store either variable- or fixed-length data items. By default, the Recno access method stores variable-length data items. The Queue access method can only store fixed-length data items. + +#### Record Delimiters + +When using the Recno access method to store variable-length records, records read from any backing source file are separated by a specific byte value which marks the end of one record and the beginning of the next. This delimiting value is ignored except when reading records from a backing source file, that is, records may be stored into the database that include the delimiter byte. However, if such records are written out to the backing source file and the backing source file is subsequently read into a database, the records will be split where delimiting bytes were found. + +For example, UNIX text files can usually be interpreted as a sequence of variable-length records separated by ASCII newline characters. This byte value (ASCII 0x0a) is the default delimiter. Applications may specify a different delimiting byte using the DB->set_re_delim() method. If no backing source file is being used, there is no reason to set the delimiting byte value. + +#### Record Length + +When using the Recno or Queue access methods to store fixed-length records, the record length must be specified. Since the Queue access method always uses fixed-length records, the user must always set the record length prior to creating the database. Setting the record length is what causes the Recno access method to store fixed-length, not variable-length, records. + +The length of the records is specified by calling the DB->set_re_len() method. The default length of the records is 0 bytes. Any record read from a backing source file or otherwise stored in the database that is shorter than the declared length will automatically be padded as described for the DB->set_re_pad() method. Any record stored that is longer than the declared length results in an error. For further information on backing source files, see Flat-text backing files. + +#### Record Padding Byte Value + +When storing fixed-length records in a Queue or Recno database, a pad character may be specified by calling the DB->set_re_pad() method. Any record read from the backing source file or otherwise stored in the database that is shorter than the expected length will automatically be padded with this byte value. If fixed-length records are specified but no pad value is specified, a space character (0x20 in the ASCII character set) will be used. For further information on backing source files, see Flat-text backing files. + +### Selecting a Queue extent size + +In Queue databases, records are allocated sequentially and directly mapped to an offset within the file storage for the database. As records are deleted from the Queue, pages will become empty and will not be reused in normal queue operations. To facilitate the reclamation of disk space a Queue may be partitioned into extents. Each extent is kept in a separate physical file. + +Extent files are automatically created as needed and marked for deletion when the head of the queue moves off the extent. The extent will not be deleted until all processes close the extent. In addition, Berkeley DB caches a small number of extents that have been recently used; this may delay when an extent will be deleted. The number of extents left open depends on queue activity. + +The extent size specifies the number of pages that make up each extent. By default, if no extent size is specified, the Queue resides in a single file and disk space is not reclaimed. In choosing an extent size there is a tradeoff between the amount of disk space used and the overhead of creating and deleting files. If the extent size is too small, the system will pay a performance penalty, creating and deleting files frequently. In addition, if the active part of the queue spans many files, all those files will need to be open at the same time, consuming system and process file resources. + +You can set the Queue extent size using the DB->set_q_extentsize() method. You can see the current extent size using the DB->get_q_extentsize() method. + +### Flat-text backing files + +It is possible to back any Recno database (either fixed or variable length) with a flat-text source file. This provides fast read (and potentially write) access to databases that are normally created and stored as flat-text files. The backing source file may be specified by calling the DB->set_re_source() method. + +The backing source file will be read to initialize the database. In the case of variable length records, the records are assumed to be separated as described for the DB->set_re_delim() method. For example, standard UNIX byte stream files can be interpreted as a sequence of variable length records separated by ASCII newline characters. This is the default. + +When cached data would normally be written back to the underlying database file (for example, when the DB->close() or DB->sync() methods are called), the in-memory copy of the database will be written back to the backing source file. + +The backing source file must already exist (but may be zero-length) when DB->open() is called. By default, the backing source file is read lazily, that is, records are not read from the backing source file until they are requested by the application. If multiple processes (not threads) are accessing a Recno database concurrently and either inserting or deleting records, the backing source file must be read in its entirety before more than a single process accesses the database, and only that process should specify the backing source file as part of the DB->open() call. This can be accomplished by calling the DB->set_flags() method with the DB_SNAPSHOT flag. + +Reading and writing the backing source file cannot be transactionally protected because it involves filesystem operations that are not part of the Berkeley DB transaction methodology. For this reason, if a temporary database is used to hold the records (a NULL was specified as the file argument to DB->open()), **it is possible to lose the contents of the backing source file if the system crashes at the right instant**. If a permanent file is used to hold the database (a filename was specified as the file argument to DB->open()), normal database recovery on that file can be used to prevent information loss. It is still possible that the contents of the backing source file itself will be corrupted or lost if the system crashes. + +For all of the above reasons, the backing source file is generally used to specify databases that are read-only for Berkeley DB applications, and that are either generated on the fly by software tools, or modified using a different mechanism such as a text editor. + +### Logically renumbering records + +Records stored in the Queue and Recno access methods are accessed by logical record number. In all cases in Btree databases, and optionally in Recno databases (see the DB->set_flags() method and the DB_RENUMBER flag for more information), record numbers are mutable. This means that the record numbers may change as records are added to and deleted from the database. The deletion of record number 4 causes any records numbered 5 and higher to be renumbered downward by 1; the addition of a new record after record number 4 causes any records numbered 5 and higher to be renumbered upward by 1. In all cases in Queue databases, and by default in Recno databases, record numbers are not mutable, and the addition or deletion of records to the database will not cause already-existing record numbers to change. For this reason, new records cannot be inserted between already-existing records in databases with immutable record numbers. + +Cursors pointing into a Btree database or a Recno database with mutable record numbers maintain a reference to a specific record, rather than a record number, that is, the record they reference does not change as other records are added or deleted. For example, if a database contains three records with the record numbers 1, 2, and 3, and the data items "A", "B", and "C", respectively, the deletion of record number 2 ("B") will cause the record "C" to be renumbered downward to record number 2. A cursor positioned at record number 3 ("C") will be adjusted and continue to point to "C" after the deletion. Similarly, a cursor previously referring to the now deleted record number 2 will be positioned between the new record numbers 1 and 2, and an insertion using that cursor will appear between those records. In this manner records can be added and deleted to a database without disrupting the sequential traversal of the database by a cursor. + +Only cursors created using a single DB handle can adjust each other's position in this way, however. If multiple DB handles have a renumbering Recno database open simultaneously (as when multiple processes share a single database environment), a record referred to by one cursor could change underfoot if a cursor created using another DB handle inserts or deletes records into the database. For this reason, applications using Recno databases with mutable record numbers will usually make all accesses to the database using a single DB handle and cursors created from that handle, or will otherwise single-thread access to the database, for example, by using the Berkeley DB Concurrent Data Store product. + +In any Queue or Recno databases, creating new records will cause the creation of multiple records if the record number being created is more than one greater than the largest record currently in the database. For example, creating record number 28, when record 25 was previously the last record in the database, will implicitly create records 26 and 27 as well as 28. All first, last, next and previous cursor operations will automatically skip over these implicitly created records. So, if record number 5 is the only record the application has created, implicitly creating records 1 through 4, the DBC->get() method with the DB_FIRST flag will return record number 5, not record number 1. Attempts to explicitly retrieve implicitly created records by their record number will result in a special error return, DB_KEYEMPTY. + +In any Berkeley DB database, attempting to retrieve a deleted record, using a cursor positioned on the record, results in a special error return, DB_KEYEMPTY. In addition, when using Queue databases or Recno databases with immutable record numbers, attempting to retrieve a deleted record by its record number will also result in the DB_KEYEMPTY return. diff --git a/docs-src/guides/programmer_reference/sequence.md b/docs-src/guides/programmer_reference/sequence.md new file mode 100644 index 000000000..4722986ef --- /dev/null +++ b/docs-src/guides/programmer_reference/sequence.md @@ -0,0 +1,16 @@ +--- +title: "Chapter 20.  Sequences" +api-name: "Chapter 20.  Sequences" +source: docs/programmer_reference/sequence.html +--- +## Chapter 20.  Sequences + +Sequences provide an arbitrary number of persistent objects that return an increasing or decreasing sequence of integers. Opening a sequence handle associates it with a record in a database. The handle can maintain a cache of values from the database so that a database update is not needed as the application allocates a value. + +A sequence is stored as a record pair in a database. The database may be of any type, but may not have been configured to support duplicate data items. The sequence is referenced by the key used when the sequence is created, therefore the key must be compatible with the underlying access method. If the database stores fixed-length records, the record size must be at least 64 bytes long. + +Since a sequence handle is opened using a database handle, the use of transactions with the sequence must follow how the database handle was opened. In other words, if the database handle was opened within a transaction, operations on the sequence handle must use transactions. Of course, if sequences are cached, not all operations will actually trigger a transaction. + +For the highest concurrency, caching should be used and the DB_AUTO_COMMIT and DB_TXN_NOSYNC flags should be specified to the DB_SEQUENCE->get() method call. If the allocation of the sequence value must be part of a transaction, and rolled back if the transaction aborts, then no caching should be specified and the transaction handle must be passed to the DB_SEQUENCE->get() method. + +For more information on the operations supported by the sequence handle, see the Sequences and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/stl.md b/docs-src/guides/programmer_reference/stl.md new file mode 100644 index 000000000..c6badf5e6 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl.md @@ -0,0 +1,144 @@ +--- +title: "Chapter 7. Standard Template Library API" +api-name: "Chapter 7. Standard Template Library API" +source: docs/programmer_reference/stl.html +--- +## Chapter 7. Standard Template Library API + +**Table of Contents** + + [Dbstl introduction](stl.md#stl_intro) + + [Standards compatible](stl.md#stl_intro_stdcompat) + + [Performance overhead](stl.md#stl_intro_performance) + + [Portability](stl.md#stl_intro_portability) + + [Dbstl typical use cases](stl_usecase.md) + + [Dbstl examples](stl_examples.md) + + [Berkeley DB configuration](stl_db_usage.md) + + [Registering database and environment handles](stl_db_usage.md#idp51381760) + + [Truncate requirements](stl_db_usage.md#idp51405208) + + [Auto commit support](stl_db_usage.md#idp51416888) + + [Database and environment identity checks](stl_db_usage.md#idp51379224) + + [Products, constructors and configurations](stl_db_usage.md#idp51415360) + + [Using advanced Berkeley DB features with dbstl](stl_db_advanced_usage.md) + + [Using bulk retrieval iterators](stl_db_advanced_usage.md#idp51421384) + + [Using the DB_RMW flag](stl_db_advanced_usage.md#idp51410312) + + [Using secondary index database and secondary containers](stl_db_advanced_usage.md#idp51398048) + + [Using transactions in dbstl](stl_txn_usage.md) + + [Using dbstl in multithreaded applications](stl_mt_usage.md) + + [Working with primitive types](stl_primitive_rw.md) + + [Storing strings](stl_primitive_rw.md#idp51467888) + + [Store and Retrieve data or objects of complex types](stl_complex_rw.md) + + [Storing varying length objects](stl_complex_rw.md#idp51458752) + + [Storing arbitrary sequences](stl_complex_rw.md#idp51477944) + + [Notes](stl_complex_rw.md#idp51524696) + + [Dbstl persistence](stl_persistence.md) + + [Direct database get](stl_persistence.md#directdbget) + + [Change persistence](stl_persistence.md#chg_persistence) + + [Object life time and persistence](stl_persistence.md#obj_life_persistence) + + [Dbstl container specific notes](stl_container_specific.md) + + [db_vector specific notes](stl_container_specific.md#idp51492808) + + [Associative container specific notes](stl_container_specific.md#idp51561456) + + [Using dbstl efficiently](stl_efficienct_use.md) + + [Using iterators efficiently](stl_efficienct_use.md#idp51530568) + + [Using containers efficiently](stl_efficienct_use.md#idp51530352) + + [Dbstl memory management](stl_memory_mgmt.md) + + [Freeing memory](stl_memory_mgmt.md#idp51564672) + + [Type specific notes](stl_memory_mgmt.md#idp51569240) + + [Dbstl miscellaneous notes](stl_misc.md) + + [Special notes about trivial methods](stl_misc.md#idp51587208) + + [Using correct container and iterator public types](stl_misc.md#idp51603304) + + [Dbstl known issues](stl_known_issues.md) + +## Dbstl introduction + + [Standards compatible](stl.md#stl_intro_stdcompat) + + [Performance overhead](stl.md#stl_intro_performance) + + [Portability](stl.md#stl_intro_portability) + +Dbstl is a C++ STL style API that provides for Berkeley DB usage. It allows for the storage and retrieval of data/objects of any type using Berkeley DB databases, but with an interface that mimics that of C++ STL containers. Dbstl provides access to all of the functionality of Berkeley DB available via this STL-style API. + +With proper configuration, dbstl is able to store/retrieve any complex data types. There is no need to perform repetitive marshalling and unmarshalling of data. Dbstl also properly manages the life-cycle of all Berkeley DB structures and objects. All example methods referred to in this chapter can be found in the StlAdvancedFeaturesExample class in the \$DbSrc/examples_stl/StlAdvancedFeatures.cpp file, and you can build the example in \$DbSrc/build_unix directory like this: make exstl_advancedfeatures, where DbSrc is the source directory for Berkeley DB. + +### Standards compatible + +Dbstl is composed of many container and iterator class templates. These containers and iterators correspond exactly to each container and iterator available in the C++ STL API, including identical sets of methods. This allows existing algorithms, functions and container-adapters for C++ STL to use dbstl containers through its standard iterators. This means that existing STL code can manipulate Berkeley DB databases. As a result, existing C++ STL code can very easily use dbstl to gain persistence and transaction guarantees. + +### Performance overhead + +Because dbstl uses C++ template technologies, its performance overhead is minimal. + +The dbstl API performs almost equally to the C API, as measured by two different implementations of the TPC-B benchmark: `ex_tpcb` and `exstl_tpcb`. + +### Portability + +The degree to which dbstl is portable to a new platform is determined by whether Berkeley DB is available on the platform, as well as whether an appropriate C++ compiler is available on the platform. + +For information on porting Berkeley DB to new platforms, see the *Berkeley DB Porting Guide*. + +Almost all the advanced C++ template features are used in dbstl, including: + +- member function templates + +- member function template overloads + +- partial specialization + +- default template parameters. + +For this reason, you need a standards-compatible C++ compiler to build dbstl. As of this writing, the following compilers are known to build dbstl successfully: + +- MSVC8 + +- gcc3.4.4 and above + +- Intel C++ 9 and above + +For \*nix platforms, if you can successfully configure your Berkeley DB build script with `--enable-stl`, then you should be able to successfully build dbstl library and application code using it. + +Besides its own test suite, dbstl has also been tested against, and passes, the following test suites: + +- MS STL test suite + +- SGI STL test suite diff --git a/docs-src/guides/programmer_reference/stl_complex_rw.md b/docs-src/guides/programmer_reference/stl_complex_rw.md new file mode 100644 index 000000000..59d9ca52d --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_complex_rw.md @@ -0,0 +1,133 @@ +--- +title: "Store and Retrieve data or objects of complex types" +api-name: "Store and Retrieve data or objects of complex types" +source: docs/programmer_reference/stl_complex_rw.html +--- +## Store and Retrieve data or objects of complex types + + [Storing varying length objects](stl_complex_rw.md#idp51458752) + + [Storing arbitrary sequences](stl_complex_rw.md#idp51477944) + + [Notes](stl_complex_rw.md#idp51524696) + +### Storing varying length objects + +A structure like this: + +``` c +class SMSMsg +{ +public: + size_t mysize; + time_t when; + size_t szmsg; + int to; + char msg[1]; +}; +``` + +with a varying length string in `msg` cannot simply be stored in a `db_vector` without some configuration on your part. This is because, by default, dbstl uses the **sizeof()** operator to get the size of an object and then `memcpy()` to copy the object. This process is not suitable for this use-case as it will fail to capture the variable length string contained in `msg`. + +There are currently two ways to store these kind of objects: + +1. Register callback functions with dbstl that are used to measure an object's size, and then marshal/unmarshal the object. + +2. Use a `DbstlDbt` wrapper object. + +#### Storing by marshaling objects + +One way to store an object that contains variable-sized fields is to marshall all of the object's data into a single contiguous area in memory, and then store the contents of that buffer. This means that upon retrieval, the contents of the buffer must be unmarshalled. To do these things, you must register three callback functions: + +- `typedef void (*ElemRstoreFunct)(T& dest, const void *srcdata);` + + This callback is used to unmarshal an object, updating **dest** using data found in **srcdata**. The data in **srcdata** contains the chunk of memory into which the object was originally marshalled. The default unmarshalling function simply performs a cast (for example, `dest = *((T*)srcdata)`), which assumes the **srcdata** simply points to the memory layout of the object. + +- `typedef size_t (*ElemSizeFunct)(const T& elem);` + + This callback returns the size in bytes needed to store the **elem** object. By default this function simply uses **sizeof(elem)** to determine the size of **elem**. + +- `typedef void (*ElemCopyFunct)(void *dest, const T&elem);` + + This callback is used to arrange all data contained by **elem** into the chunk of memory to which **dest** refers. The size of **dest** is set by the `ElemSizeFunct` function, discussed above. The default marshalling function simply uses `memcpy()` to copy **elem** to **dest**. + +The `DbstlElemTraits::instance()->set_size_function()`, `set_copy_function()` and `set_restore_function()` methods are used to register these callback functions. If a callback is not registered, its default function is used. + +By providing non-default implementations of the callbacks described here, you can store objects of varying length and/or objects which do not reside in a continuous memory chunk — for example, objects containing a pointer which refers another object, or a string, and so forth. As a result, containers/iterators can manage variable length objects in the same as they would manage objects that reside in continuous chunks of memory and are of identical size. + +#### Using a `DbstlDbt` wrapper object + +To use a `DbstlDbt` wrapper object to store objects of variable length, a `db_vector` container is used to store complex objects in a `db_vector`. `DbstlDbt` derives from DB C++ API's `Dbt`class, but can manage its referenced memory properly and release it upon destruction. The memory referenced by `DbstlDbt` objects is required to be allocated using the `malloc()`/`realloc()` functions from the standard C library. + +Note that the use of `DbstlDbt` wrapper class is not ideal. It exists only to allow raw bytes of no specific type to be stored in a container. + +To store an `SMSMsg` object into a `db_vector` container using a `DbstlDbt` object: + +1. Wrap the `SMSMSg` object into a `DbstlDbt` object, then marshal the SMSMsg object properly into the memory chunk referenced by `DbstlDbt::data`. +2. Store the `DbstlDbt` object into a `db_vector` container. The bytes in the memory chunk referenced by the `DbstlDbt` object's **data** member are stored in the `db_vector` container. +3. Reading from the container returns a `DbstlDbt` object whose **data** field points to the `SMSMsg` object located in a continuous chunk of memory. The application needs to perform its own unmarshalling. +4. The memory referenced by `DbstlDbt::data` is freed automatically, and so the application should not attempt to free the memory. + +`ElementHolder` should not be used to store objects of a class because it doesn't support access to object members using **(\*iter).member** or **iter-\>member** expressions. In this case, the default `ElementRef` is used automatically. + +`ElementRef` inherits from `ddt`, which allows **\*iter** to return the object stored in the container. (Technically it is an `ElementRef object`, whose "base class" part is the object you stored). There are a few data members and member functions in `ElementRef`, which all start with `_DB_STL_`. To avoid potential name clashes, applications should not use names prefixing `_DB_STL_` in classes whose instances may be stored into dbstl containers. + +Example code demonstrating this feature can be found in the `StlAdvancedFeaturesExample::arbitrary_object_storage` method. + +### Storing arbitrary sequences + +A sequence is a group of related objects, such as an array, a string, and so forth. You can store sequences of any structure using dbstl, so long as you implement and register the proper callback functions. By using these callbacks, each object in the sequence can be a complex object with data members that are all not stored in a continuous memory chunk. + +Note that when using these callbacks, when you retrieve a stored sequence from the database, the entire sequence will reside in a single continuous block of memory with the same layout as that constructed by your sequence copy function. + +For example, given a type RGB: + +``` c +struct RGB{char r, g, b, bright;}; +``` + +and an array of RGB objects, the following steps describe how to store an array into one key/data pair of a `db_map` container. + +1. Use a `db_map >` container. + +2. Define two functions. The first returns the number of objects in a sequence, the second that copies objects from a sequence to a defined destination in memory: + + ``` c + typedef size_t (*SequenceLenFunct)(const RGB*); + ``` + + and + + ``` c + typedef void (*SequenceCopyFunct)(RGB*dest, const RGB*src); + ``` + +3. Call DbstlElemTraits\::set_sequence_len_function()/set_sequence_copy_function() to register them as callbacks. + +#### The `SequenceLenFunct` function + +``` c +typedef size_t (*SequenceLenFunct)(const RGB*); +``` + +A `SequenceLenFunct` function returns the number of objects in a sequence. It is called when inserting into or reading from the database, so there must be enough information in the sequence itself to enable the `SequenceLenFunct` function to tell how many objects the sequence contains. The `char*` and `wchar_t*` strings use a `'\0'` special character to do this. For example, RGB(0, 0, 0, 0) could be used to denote the end of the sequence. Note that for your implementation of this callback, you are not required to use a trailing object with a special value like `'\0'` or `RGB(0, 0, 0, 0)` to denote the end of the sequence. You are free to use what mechanism you want in your `SequenceLenFunct` function implementation to figure out the length of the sequence. + +#### The `SequenceCopyFunct` function + +``` c + typedef void (*SequenceCopyFunct)(RGB*dest, const RGB*src); +``` + +`SequenceCopyFunct` copies objects from the sequence **src** into memory chunk **dest**. If the objects in the sequence do not reside in a continuous memory chunk, this function must marshal each object in the sequence into the **dest** memory chunk. + +The sequence objects will reside in the continuous memory chunk referred to by **dest**, which has been sized by `SequenceLenFunct` and `ElemSizeFunct` if available (which is when objects in the sequence are of varying lengths). `ElemSizeFunct` function is not needed in this example because **RGB** is a simple fixed length type, the `sizeof()` operator is sufficient to return the size of the sequence. + +### Notes + +- The get and set functions of this class are not protected by any mutexes. When using multiple threads to access the function pointers, the callback functions must be registered to the singleton of this class before any retrieval of the callback function pointers. Isolation may also be required among multiple threads. The best way is to register all callback function pointers in a single thread before making use of the any containers. + +- If objects in a sequence are not of identical sizes, or are not located in a consecutive chunk of memory, you also need to implement and register the `DbstlElemTraits<>::ElemSizeFunct` callback function to measure the size of each object. When this function is registered, it is also used when allocating memory space. + + There is example code demonstrating the use this feature in the `StlAdvancedFeaturesExample::arbitray_sequence_storage()` method. + +- A consequence of this dbstl feature is that you can not store a pointer value directly because dbstl will think it is a sequence head pointer. Instead, you need to convert the pointer into a `long` and then store it into a `long` container. And please note that pointer values are probably meaningless if the stored value is to be used across different application run times. diff --git a/docs-src/guides/programmer_reference/stl_container_specific.md b/docs-src/guides/programmer_reference/stl_container_specific.md new file mode 100644 index 000000000..549450241 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_container_specific.md @@ -0,0 +1,32 @@ +--- +title: "Dbstl container specific notes" +api-name: "Dbstl container specific notes" +source: docs/programmer_reference/stl_container_specific.html +--- +## Dbstl container specific notes + + [db_vector specific notes](stl_container_specific.md#idp51492808) + + [Associative container specific notes](stl_container_specific.md#idp51561456) + +### db_vector specific notes + +- Set the DB_RENUMBER flag in the database handle if you want `db_vector<>` to work like `std::vector` or `std::deque`. Do not set DB_RENUMBER if you want `db_vector<>` to work like `std::list`. Note that without DB_RENUMBER set, `db_vector<>` can work faster. + + For example, to construct a fast std::queue/std::stack object, you only need a `db_vector<>` object whose database handle does not have DB_RENUMBER set. Of course, if the database handle has DB_RENUMBER set, it still works for this kind of scenario, just not as fast. + + `db_vector` does not check whether DB_RENUMBER is set. If you do not set it, `db_vector<>` will not work like std::vector\<\>/std::deque\<\> with regard to operator\[\], because the indices are not maintained in that case. + + You can find example code showing how to use this feature in the `StlAdvancedFeaturesExample::queue_stack()` method. + +- Just as is the case with `std::vector`, inserting/deleting in the middle of a `db_vector` is slower than doing the same action at the end of the sequence. This is because the underlying DB_RECNO DB (with the DB_RENUMBER flag set) is relatively slow when inserting/deleting in the middle or the head — it has to update the index numbers of all the records following the one that was inserted/deleted. If you do not need to keep the index ordered on insert/delete, you can use `db_map` instead. + + `db_vector` also contains methods inherited from `std::list` and `std::deque`, including `std::list<>'s` unique methods `remove()`, `remove_if()`, `unique()`, `merge()`, `sort()`, `reverse()`, and `splice()`. These use the identical semantics/behaviors of the `std::list<>` methods, although pushing/deleting at the head is slower than the `std::deque` and `std::list` equivalent when there are quite a lot of elements in the database. + +- You can use `std::queue`, `std::priority_queue` and `std::stack` container adapters with `db_vector`; they work with db_vector even without DB_RENUMBER set. + +### Associative container specific notes + +`db_map` contains the union of method set from `std::map` and `hash_map`, but there are some methods that can only be called on containers backed by `DB_BTREE` or `DB_HASH` databases. You can call `db_map<>::is_hash()` to figure out the type of the backing database. If you call unsupported methods then an InvalidFunctionCall exception is thrown. + +These are the `DB_BTREE` specific methods: `upper_bound()`, `lower_bound()`, `key_comp()`, and `value_comp()`. The `DB_HASH` specific methods are `key_eq()`, `hash_funct()`. diff --git a/docs-src/guides/programmer_reference/stl_db_advanced_usage.md b/docs-src/guides/programmer_reference/stl_db_advanced_usage.md new file mode 100644 index 000000000..7b4c28f68 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_db_advanced_usage.md @@ -0,0 +1,50 @@ +--- +title: "Using advanced Berkeley DB features with dbstl" +api-name: "Using advanced Berkeley DB features with dbstl" +source: docs/programmer_reference/stl_db_advanced_usage.html +--- +## Using advanced Berkeley DB features with dbstl + + [Using bulk retrieval iterators](stl_db_advanced_usage.md#idp51421384) + + [Using the DB_RMW flag](stl_db_advanced_usage.md#idp51410312) + + [Using secondary index database and secondary containers](stl_db_advanced_usage.md#idp51398048) + +This section describes advanced Berkeley DB features that are available through dbstl. + +### Using bulk retrieval iterators + +Bulk retrieval is an optimization option for const iterators and nonconst but read-only iterators. Bulk retrieval can minimize the number of database accesses performed by your application. It does this by reading multiple entries at a time, which reduces read overhead. Note that non-sequential reads will benefit less from, or even be hurt by, this behavior, because it might result in unneeded data being read from the database. Also, non-serializable reads may read obsolete data, because part of the data read from the bulk read buffer may have been updated since the retrieval. + +When using the default transaction isolation, iterators will perform serializable reads. In this situation, the bulk-retrieved data cannot be updated until the iterator's cursor is closed. + +Iterators using a different isolation levels, such as DB_READ_COMMITTED or DB_READ_UNCOMMITTED will not perform serializable reads. The same is true for any iterators that do not use transactions. + +A bulk retrieval iterator can only move in a singled direction, from beginning to end. This means that iterators only support operator++, and reverse iterators only support operator--. + +Iterator objects that use bulk retrieval might contain hundreds of kilobytes of data, which makes copying the iterator object an expensive operation. If possible, use ++iterator rather than iterator++. This can save a useless copy construction of the iterator, as well as an unnecessary dup/close of the cursor. + +You can configure bulk retrieval for each container using both in the const and non-const version of the `begin()` method. The non-const version of `begin()` will return a read-only cursor. Note that read-only means something different in C++ than it does when referring to an iterator. The latter only means that it cannot be used to update the database. + +To configure the bulk retrieval buffer for an iterator when calling the `begin()` method, use the `BulkRetrievelItrOpt::bulk_retrieval(u_int32_t bulk_buffer_size)` function. + +If you move a `db_vector_iterator` randomly rather than sequentially, then dbstl will not perform bulk retrieval because there is little performance gain from bulk retrieval in such an access pattern. + +You can call `iterator::set_bulk_buffer()` to modify the iterator's bulk buffer size. Note that once bulk read is enabled, only the bulk buffer size can be modified. This means that bulk read cannot be disabled. Also, if bulk read was not enabled when you created the iterator, you can't enable it after creation. + +Example code using this feature can be found in the `StlAdvancedFeaturesExample::bulk_retrieval_read()` method. + +### Using the DB_RMW flag + +The DB_RMW flag is an optimization for non-const (read-write) iterators. This flag causes the underlying cursor to acquire a write lock when reading so as to avoid deadlocks. Passing `ReadModifyWriteOption::read_modify_write()` to a container's `begin()` method creates an iterator whose cursor has this behavior. + +### Using secondary index database and secondary containers + +Because duplicate keys are forbidden in primary databases, only `db_map`, `db_set` and `db_vector` are allowed to use primary databases. For this reason, they are called **primary containers**. A secondary database that supports duplicate keys can be used with `db_multimap` containers. These are called **secondary containers**. Finally, a secondary database that forbids duplicate keys can back a `db_map` container. + +The **data_type** of this `db_multimap` secondary container is the **data_type** for the primary container. For example, a `db_map` object where the `Person` class has an `age` property of type `size_t`, a `db_multimap` using a secondary database allows access to a person by age. + +A container created from a secondary database can only be used to iterate, search or delete. It can not be used to update or insert. While dbstl does expose the update and insert operations, Berkeley DB does not, and an exception will be thrown if attempts are made to insert objects into or update objects of a secondary container. + +Example code demonstrating this feature is available in the `StlAdvancedFeaturesExample::secondary_containers()` method. diff --git a/docs-src/guides/programmer_reference/stl_db_usage.md b/docs-src/guides/programmer_reference/stl_db_usage.md new file mode 100644 index 000000000..817149808 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_db_usage.md @@ -0,0 +1,72 @@ +--- +title: "Berkeley DB configuration" +api-name: "Berkeley DB configuration" +source: docs/programmer_reference/stl_db_usage.html +--- +## Berkeley DB configuration + + [Registering database and environment handles](stl_db_usage.md#idp51381760) + + [Truncate requirements](stl_db_usage.md#idp51405208) + + [Auto commit support](stl_db_usage.md#idp51416888) + + [Database and environment identity checks](stl_db_usage.md#idp51379224) + + [Products, constructors and configurations](stl_db_usage.md#idp51415360) + +While dbstl behaves like the C++ STL APIs in most situations, there are some Berkeley DB configuration activities that you can and should perform using dbstl. These activities are described in the following sections. + +### Registering database and environment handles + +Remember the following things as you use Berkeley DB Database and Environment handles with dbstl: + +- If you share environment or database handles among multiple threads, remember to specify the DB_THREAD flag in the open call to the handle. + +- If you create or open environment and/or database handles without using the dbstl helper functions, `dbstl::open_db()` or `dbstl::open_env()`, remember that your environment and database handles should be: + + 1. Allocated in the heap via "new" operator. + + 2. Created using the DB_CXX_NO_EXCEPTIONS flag. + + 3. In each thread sharing the handles, the handles are registered using either `dbstl::register_db()` or `dbstl::register_dbenv()`. + +- If you opened the database or environment handle using the `open_db()` or `open_env()` functions, the thread opening the handles should not call `register_db()` or `register_env()` again. This is because they have already been registered by the `open_db()` or `open_env()` functions. However, other threads sharing these handles still must register them locally. + +### Truncate requirements + +Some Berkeley DB operations require there to be no open cursors on the database handle at the time the operation occurs. Dbstl is aware of these requirements, and will attempt to close the cursors opened in the current thread when it performs these operations. However, the scope of dbstl's activities in this regard are limited to the current thread; it makes no attempt to close cursors opened in other threads. So you are required to ensure there are no open cursors on database handles shared across threads when operations are performed that require all cursors on that handle to be closed. + +There are only a a few operations which require all open cursors to be closed. This include all container `clear()` and `swap()` functions, and all versions of `db_vection<>::assign()` functions. These functions require all cursors to be closed for the database because by default they remove all key/data pairs from the database by truncating it. + +When a function removes all key/data pairs from a database, there are two ways it can perform this activity: + +- The default method is to truncate the database, which is an operation that requires all cursors to be closed. As mentioned above, it is your responsibility to close cursors opened in other threads before performing this operation. Otherwise, the operation will fail. + +- Alternatively, you can specify that the database not be truncated. Instead, you can cause dbstl to delete all key/data pairs individually, one after another. In this situation, open cursors in the database will not cause the delete operations to fail. However, due to lock contention, the delete operations might not complete until all cursors are closed, which is when all their read locks are released. + +### Auto commit support + +Dbstl supports auto commit for some of its container's operations. When a dbstl container is created using a `Db` or `DbEnv` object, if that object was opened using the DB_AUTO_COMMIT flag, then every operation subsequently performed on that object will be automatically enclosed in a unique transaction (unless the operation is already in an external transaction). This is identical to how the Berkeley DB C, C++ and Java APIs behave. + +Note that only a subset of a container's operations support auto commit. This is because those operations that accept or return an iterator have to exist in an external transactional context and so cannot support auto commit. + +The dbstl API documentation identifies when a method supports auto commit transactions. + +### Database and environment identity checks + +When a container member function involves another container (for example, `db_vector::swap(self& v2)`), the two containers involved in the operation must not use the same database. Further, if the function is in an external or internal transaction context, then both containers must belong to the same transactional database environment; Otherwise, the two containers can belong to the same database environment, or two different ones. + +For example, if `db_vector::swap(self& v2)` is an auto commit method or it is in an external transaction context, then `v2` must be in the same transactional database environment as this container, because a transaction is started internally that must be used by both `v2` and this container. If this container and the `v2` container have different database environments, and either of them are using transactions, an exception is thrown. This condition is checked in every such member function. + +However, if the function is not in a transactional context, then the databases used by these containers can be in different environments because in this situation dbstl makes no attempt to wrap container operations in a common transaction context. + +### Products, constructors and configurations + +You can use dbstl with all Berkeley DB products (DS, CDS, TDS, and HA). Because dbstl is a Berkeley DB interface, all necessary configurations for these products are performed using Berkeley DB's standard create/open/set APIs. + +As a result, the dbstl container constructors differ from those of C++ STL because in dbstl no configuration is supported using the container constructors. On the other hand, dbstl container constructors accept already opened and configured environment and database handles. They also provide functions to retrieve some handle configuration, such as key comparison and hash functions, as required by the C++ STL specifications. + +The constructors verify that the handles passed to them are well configured. This means they ensure that no banned settings are used, as well as ensuring that all required setting are performed. If the handles are not well configured, an `InvalidArgumentException` is thrown. + +If a container constructor is not passed a database or environment handle, an internal anonymous database is created for you by dbstl. This anonymous database does not provide data persistence. diff --git a/docs-src/guides/programmer_reference/stl_efficienct_use.md b/docs-src/guides/programmer_reference/stl_efficienct_use.md new file mode 100644 index 000000000..5aa7c83af --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_efficienct_use.md @@ -0,0 +1,86 @@ +--- +title: "Using dbstl efficiently" +api-name: "Using dbstl efficiently" +source: docs/programmer_reference/stl_efficienct_use.html +--- +## Using dbstl efficiently + + [Using iterators efficiently](stl_efficienct_use.md#idp51530568) + + [Using containers efficiently](stl_efficienct_use.md#idp51530352) + +### Using iterators efficiently + +To make the most efficient possible use of iterators: + +- Close an iterator's cursor as soon as possible. + + Each iterator has an open cursor associated with it, so when you are finished using the iterator it is a good habit to explicitly close its cursor. This can potentially improve performance by avoiding locking issues, which will enhanced concurrency. Dbstl will close the cursor when the iterator is destroyed, but you can close the cursor before that time. If the cursor is closed, the associated iterator cannot any longer be used. + + In some functions of container classes, an iterator is used to access the database, and its cursor is internally created by dbstl. So if you want to specify a non-zero flag for the `Db::cursor()` call, you need to call the container's `set_cursor_open_flag()` function to do so. + +- Use const iterators where applicable. + + If your data access is read only, you are strongly recommended to use a const iterator. In order to create a const iterator, you must use a const reference to the container object. For example, supposed we have: + + ``` c + db_vector intv(10); + ``` + + then we must use a: + + ``` c + const db_vector& intv_ref = intv; + ``` + + reference to invoke the const begin/end functions. `intv_ref.begin()` will give you a const iterator. You can use a const iterator only to read its referenced data elements, not update them. However, you should have better performance with this iterator using, for example, either `iterator::operator*` or `iterator::operator->member`. Also, using array indices like `intv_ref[i]` will also perform better. + + All functions in dbstl's containers which return an iterator or data element reference have two versions — one returns a const iterator/reference, the other returns an iterator/reference. If your access is read only, choose the version returning const iterators/references. + + Remember that you can only use a const reference to a container object to call the const versions of `operator*` and `operator[]`. + + You can also use the non-const container object or its non-const reference to create a read only iterator by passing `true` to the **readonly** parameter in the container's `begin()` method. + +- Use pre-increment/pre-decrement rather than post-increment/post-decrement where possible + + Pre-increment operations are more efficient because the `++iterator` avoids two iterator copy constructions. This is true when you are using C++ standard STL iterators as well. + +- Use bulk retrieval in iterators + + If your access pattern is to go through the entire database read only, or if you are reading a continuous range of the database, bulk retrieval can be very useful because it returns multiple key/data pairs in one database call. But be aware that you can only read the returned data, you can not update it. Also, if you do a bulk retrieval and read the data, and simultaneously some other thread of control updates that same data, then unless you are using a serializable transaction, you will now be working with old data. + +### Using containers efficiently + +To make the most efficient possible use of containers: + +- Avoid using container methods that return references. These because they are a little more expensive. + + To implement reference semantics, dbstl has to wrap the data element with the current key/data pair, and must invoke two iterator copy constructions and two Berkeley DB cursor duplications for each such a call. This is true of non-const versions of these functions: + + | | + |------------------------------| + | `db_vector::operator[]()` | + | `db_vector::front()` | + | `db_vector::back()` | + | `db_vector::at()` | + | `db_map<>::operator[]()` | + + There are alternatives to these functions, mainly through explicit use of iterators. + +- Use const containers where possible. + + The const versions of the functions listed above have less overhead than their non-const counterparts. Using const containers and iterators can bring more performance when you call the const version of the overloaded container/iterator methods. To do so, you define a const container reference to an existing container, and then use this reference to call the methods. For example, if you have: + + ``` c + db_vector container int_vec + ``` + + then you can define a const reference to `int_vec`: + + ``` c + const db_vector& int_vec_ref; + ``` + + Then you use `int_vec_ref.begin()` to create a const iterator, `citr`. You can now can use `int_vec_ref` to call the const versions of the container's member functions, and then use `citr` to access the data read only. By using `int_vec_ref` and `citr`, we can gain better performance. + + It is acceptable to call the non-const versions of container functions that return non-const iterators, and then assign these return values to const iterator objects. But if you are using Berkeley DB concurrent data store (CDS), be sure to set the **readonly** parameter for each container method that returns an iterator to `true`. This is because each iterator corresponds to a Berkeley DB cursor, and so for best performance you should specify that the returned iterator be read-only so that the underlying cursor is also read-only. Otherwise, the cursor will be a writable cursor, and performance might be somewhat degraded. If you are not using CDS, but instead TDS or DS or HA, there is no distinction between read-only cursors and read-write cursors. Consequently, you do not need to specify the **readonly** parameter at all. diff --git a/docs-src/guides/programmer_reference/stl_examples.md b/docs-src/guides/programmer_reference/stl_examples.md new file mode 100644 index 000000000..e5d736344 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_examples.md @@ -0,0 +1,86 @@ +--- +title: "Dbstl examples" +api-name: "Dbstl examples" +source: docs/programmer_reference/stl_examples.html +--- +## Dbstl examples + +Because dbstl is so much like C++ STL, its usage exactly mirrors that of C++ STL, with the exception of a few optional Berkeley DB specific configurations. In fact, the only difference between a program using dbstl and one using C++ STL is the class names. That is, `vector` becomes `db_vector`, and `map` becomes `db_map`. + +The typical procedure for using dbstl is: + +1. Optionally create and open your own Berkeley DB environment and database handles using the DB C++ API. If you perform these opens using the C++ API, make sure to perform necessary environment and database configurations at that time. + +2. Optionally pass environment and database handles to dbstl container constructors when you create dbstl container objects. Note that you can create a dbstl container without passing it an environment and database object. When you do this, an internal anonymous database is created for you. In this situation, dbstl provides no data persistence guarantees. + +3. Perform dbstl-specific configurations. For example, you can configure cursor open flags, as well as database access for autocommit. You can also configure callback functions. + +4. Interact with the data contained in your Berkeley DB databases using dbstl containers and iterators. This usage of dbstl is identical to C++ STL container and iterator usage. + +5. At this time, you can also use dbstl calls that are specific to Berkeley DB. For example, you can use Berkeley DB specific calls that manage transaction begin/commit/abort, handle registration, and so forth. While these calls are part of dbstl, they have no equivalence in the C++ STL APIs. + +6. When your application is done using Berkeley DB, you do not need to explicitly close any Berkeley DB handles (environments, database, cursors, and so forth). Dbstl automatically closes all such handles for you. + +For examples of dbstl usage, see the example programs in the `$db/examples_stl` directory. + +The following program listing provides two code fragments. You can find more example code in the `dbstl/examples/` and `dbstl/test` directories. + +``` c +//////////////// Code Snippet 1 //////////////// +db_vector > vctr(100); +for (int i = 0; i < 100; i++) + vctr[i] = i; + +for (int i = 0; i < 100; i++) { + cout<<"\nvctr["< > + strmap_t2; +strmap_t2 strmap; +char str[2], str2[2]; +str[1] = str2[1] = '\0'; +for (char c = 0; c < 26; c++) { + str[0] = c + 'a'; + str2[0] = 'z' - c; + strmap[str] = str2; +} +for (strmap_t2::iterator itr = strmap.begin(); itr != strmap.end(); ++itr) + cout<first<<" : "<second; + +using namespace dbstl; +dbstl::db_map v; +v['i'] = 1; +cout< name_addr_map; +// The strings rather than the memory pointers are stored into DB. +name_addr_map["Alex"] = "Sydney Australia"; +name_addr_map["David"] = "Shenzhen China"; +cout<<"Alex's address:"< vi; +// Some callback configurations follow here. + +// The strings and objects rather than pointers are stored into DB. + +Person obj("David Zhao", "Oracle", new Office("Boston", "USA")); +vi.push_back(obj); // More person storage. +for (int I = 0; I < vi.size(); I++) + cout<`*. The `ElementHolder` class template should be used for every type of dbstl container that will store C++ primitive data types, such as `int`, `float`, `char *`, `wchar_t *`, and so forth. But these class templates should not be used for class types for reasons that we explain in the following chapters. + +In the second code snippet, the assignment: + +``` c +strmap[str] = str2; +``` + +is used to store a string pair (`(str, str2)`) instead of pointers to the underlying database. + +The rest of the code used in these snippets is identical to the code you would use for C++ STL containers. However, by using dbstl, you are storing data into a Berkeley DB database. If you create your own database with backing files on disk, your data or objects can persist and be restored when the program runs again. diff --git a/docs-src/guides/programmer_reference/stl_known_issues.md b/docs-src/guides/programmer_reference/stl_known_issues.md new file mode 100644 index 000000000..8444b6314 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_known_issues.md @@ -0,0 +1,16 @@ +--- +title: "Dbstl known issues" +api-name: "Dbstl known issues" +source: docs/programmer_reference/stl_known_issues.html +--- +## Dbstl known issues + +Three algorithm functions of gcc's C++ STL test suite do not work with dbstl. They are `find_end()`, `inplace_merge()` and `stable_sort()`. + +The reason for the incompatibility of `find_end()` is that it assumes the data an iterator refers to is located at a shared place (owned by its container). This assumption is not correct in that it is part of the C++ STL standards specification. However, this assumption can not be true for dbstl because each dbstl container iterator caches its referenced value. + +Consequently, please do not use `find_end()` for dbstl container iterators if you are using gcc's STL library. + +The reason for the incompatibility with `inplace_merge()` and `stable_sort()` is that their implementation in gcc requires the **value_type** for a container to be default constructible. This requirement is not a part of the the C++ STL standard specification. Dbstl's value type wrappers (such as `ElementHolder`) do not support it. + +These issues do not exist for any function available with the Microsoft Visual C++ 8 STL library. There are two algorithm functions of Microsoft Visual C++ 10 STL library that do have an issue: `partial_sort()` and `partial_sort_copy()`. These are not compatible because they require the dbstl `vector` iterator to create a new element when updating the current element. Dbstl `vector` iterator can copy the new content to the current element, but it cannot create a new one. This requirement is not a part of the C++ STL standard specification, and so dbstl's `vector` iterator does not support it. diff --git a/docs-src/guides/programmer_reference/stl_memory_mgmt.md b/docs-src/guides/programmer_reference/stl_memory_mgmt.md new file mode 100644 index 000000000..4ce9be139 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_memory_mgmt.md @@ -0,0 +1,42 @@ +--- +title: "Dbstl memory management" +api-name: "Dbstl memory management" +source: docs/programmer_reference/stl_memory_mgmt.html +--- +## Dbstl memory management + + [Freeing memory](stl_memory_mgmt.md#idp51564672) + + [Type specific notes](stl_memory_mgmt.md#idp51569240) + +### Freeing memory + +When using dbstl, make sure memory allocated in the heap is released after use. The rules for this are: + +- dbstl will free/delete any memory allocated by dbstl itself. + +- You are responsible for freeing/deleting any memory allocated by your code outside of dbstl. + +### Type specific notes + +#### DbEnv/Db + +When you open a `DbEnv` or `Db` object using `dbstl::open_env()` or `dbstl::open_db()`, you do not need to delete that object. However, if you new'd that object and then opened it without using the `dbstl::open_env()` or `dbstl::open_db()` methods, you are responsible for deleting the object. + +Note that you must `new` the `Db` or `DbEnv` object, which allocates it on the heap. You can not allocate it on the stack. If you do, the order of destruction is uncontrollable, which makes dbstl unable to work properly. + +You can call `dbstl_exit()` before the process exits, to release any memory allocated by dbstl that has to live during the entire process lifetime. Releasing the memory explicitly will not make much difference, because the process is about to exit and so all memory allocated on the heap is going to be returned to the operating system anyway. The only real difference is that your memory leak checker will not report false memory leaks. + +`dbstl_exit()` releases any memory allocated by dbstl on the heap. It also performs other required shutdown operations, such as closing any databases and environments registered to dbstl and shared across the process. + +If you are calling the `dbstl_exit()` function, and your `DbEnv` or `Db` objects are new'd by your code, the `dbstl_exit()` function should be called before deleting the `DbEnv` or `Db` objects, because they need to be closed before being deleted. Alternatively, you can call the `dbstl::close_env()` or `dbstl::close_db()` functions before deleting the `DbEnv` or `Db` objects in order to explicitly close the databases or environments. If you do this, can then delete these objects, and then call `dbstl_exit()`. + +In addition, before exiting a thread that uses dbstl API, you can call the `dbstl_thread_exit() `function to release any Berkeley DB handles if they are not used by other threads. If you do not call the `dbstl_thread_exit() `function or call this function only in some threads, all open Berkeley DB handles will be closed by the `dbstl_exit()`function. You must call the `dbstl_exit() `function before the process exits, to avoid memory leak and database update loss, if you do not have transactions and persistent log files. + +#### DbstlDbt + +Only when you are storing raw bytes (such as a bitmap) do you have to store and retrieve data by using the `DbstlDbt` helper class. Although you also can do so simply by using the Berkeley DB `Dbt` class, the `DbstlDbt` class offers more convenient memory management behavior. + +When you are storing `DbstlDbt` objects (such as `db_vector`), you *must* allocate heap memory explicitly using the `malloc()` function for the `DbstlDbt` object to reference, but you do not need to free the memory – it is automatically freed by the `DbstlDbt` object that owns it by calling the standard C library `free()` function. + +However, because dbstl supports storing any type of object or primitive data, it is rare that you would have to store data using `DbstlDbt` objects while using dbstl. Examples of storing `DbstlDbt` objects can be found in the `StlAdvancedFeaturesExample::arbitrary_object_storage()` and `StlAdvancedFeaturesExample::char_star_string_storage()` methods. diff --git a/docs-src/guides/programmer_reference/stl_misc.md b/docs-src/guides/programmer_reference/stl_misc.md new file mode 100644 index 000000000..f8cc2badc --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_misc.md @@ -0,0 +1,67 @@ +--- +title: "Dbstl miscellaneous notes" +api-name: "Dbstl miscellaneous notes" +source: docs/programmer_reference/stl_misc.html +--- +## Dbstl miscellaneous notes + + [Special notes about trivial methods](stl_misc.md#idp51587208) + + [Using correct container and iterator public types](stl_misc.md#idp51603304) + +### Special notes about trivial methods + +There are some standard STL methods which are meaningless in dbstl, but they are kept in dbstl as no-ops so as to stay consistent with the standard. These are: + +| | +|--------------------------| +| `db_vecter::reserve();` | +| `db_vector::max_size();` | +| `db_vector::capacity();` | +| `db_map::reserve();` | +| `db_map::max_size();` | + +`db_vector<>::max_size()` and `db_map<>::max_size()` both return 2^30. This does not mean that Berkeley DB can only hold that much data. This value is returned to conform to some compilers' overflow rules — if we set bigger numbers like 2^32 or 2^31, some compilers complain that the number has overflowed. + +See the Berkeley DB documentation for information about limitations on how much data a database can store. + +There are also some read-only functions. You set the configuration for these using the Berkeley DB API. You access them using the container's methods. Again, this is to keep consistent with C++ standard STL containers, such as: + +| | +|-------------------------| +| `db_map::key_comp();` | +| `db_map::value_comp();` | +| `db_map::hash_funct();` | +| `db_map::key_eq();` | + +### Using correct container and iterator public types + +All public types defined by the C++ STL specification are present in dbstl. One thing to note is the **value_type**. dbstl defines the **value_type** for each iterator and container class to be the raw type without the `ElementRef`/`ElementHolder` wrapper, so this type of variable can not be used to store data in a database. There is a **value_type_wrap** type for each container and iterator type, with the raw type wrapped by the `ElementRef`/`ElementHolder`. + +For example, when type `int_vector_t` is defined as + +``` c +db_vector > +``` + +its **value_type** is `int`, its **value_type_wrap** is `ElementHolder`, and its reference and pointer types are `ElementHolder&` and `ElementHolder*` respectively. If you need to store data, use **value_type_wrap** to make use of the wrapper to store data into database. + +The reason we leave **value_type** as the raw type is that we want the existing algorithms in the STL library to work with dbstl because we have seen that without doing so, a few tests will fail. + +You need to use the same type as the return type of the data element retrieval functions to hold a value in order to properly manipulate the data element. For example, when calling + +``` c +db_vector::operator[] +``` + +check that the return type for this function is + +``` c +db_vector::datatype_wrap +``` + +Then, hold the return value using an object of the same type: + +``` c +db_vector::datatype_wrap refelem = vctr[3]; +``` diff --git a/docs-src/guides/programmer_reference/stl_mt_usage.md b/docs-src/guides/programmer_reference/stl_mt_usage.md new file mode 100644 index 000000000..06a473899 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_mt_usage.md @@ -0,0 +1,47 @@ +--- +title: "Using dbstl in multithreaded applications" +api-name: "Using dbstl in multithreaded applications" +source: docs/programmer_reference/stl_mt_usage.html +--- +## Using dbstl in multithreaded applications + +Multithreaded use of dbstl must obey the following guidelines: + +1. For a few non-standard platforms, you must first configure dbstl for that platform, but usually the configure script will detect the applicable thread local storage (TLS) modifier to use, and then use it. If no appropriate TLS is found, the pthread TLS API is used. + +2. Perform all initializations in a single thread. `dbstl::dbstl_startup()` should be called mutually exclusive in a single thread before using dbstl. If dbstl is used in only a single thread, this function does not need to be called. + + If necessary, callback functions for a complex type T must be registered to the singleton of DbstlElemTraits\ before any container related to T (for example, `db_vector`), is used, and certain isolation may be required among multiple threads. The best way to do this is to register all callback function pointers into the singleton in a single thread before making use of the containers. + + All container cursor open flags and auto commit transaction begin/commit flags must be set in a single thread before storing objects into or reading objects from the container. + +3. Environment and database handles can optionally be shared across threads. If handles are shared, they must be registered in each thread that is using the handle (either directly, or indirectly using the containers that own the handles). You do this using the `dbstl::register_db()` and `dbstl::register_db_env()` functions. Note that these functions are not necessary if the current thread called `dbstl::open_db()` or `dbstl::open_env()` for the handle that is being shared. This is because the open functions automatically register the handle for you. + + Note that the get/set functions that provide access to container data members are not mutex-protected because these data members are supposed to be set only once at container object initialization. Applications wishing to modify them after initialization must supply their own protection. + +4. While container objects can be shared between multiple threads, iterators and transactions can not be shared. + +5. Set the **directdb_get** parameter of the container `begin()` method to `true` in order to guarantee that referenced key/data pairs are always obtained from the database and not from an iterator's cached value. (This is the default behavior.) You should do this because otherwise a rare situation may occur. Given db_vector_iterator i1 and i2 used in the same iteration, setting \*i1 = new_value will not update i2, and \*i2 will return the original value. + +6. If using a CDS database, only const iterators or read-only non-const iterators should be used for read only iterations. Otherwise, when multiple threads try to open read-write iterators at the same time, performance is greatly degraded because CDS only supports one write cursor open at any moment. The use of read-only iterators is good practice in general because dbstl contains internal optimizations for read-only iterators. + + To create a read-only iterator, do one of the following: + + - Use a `const` reference to the container object, then call the container's `begin()` method using the const reference, and then store the return value from the `begin()` method in a `db_vector::const_iterator`. + + - If you are using a non-const container object, then simply pass `true` to the **readonly** parameter of the non-const `begin()` method. + +7. When using DS, CDS or TDS, enable the locking subsystem by passing the DB_INIT_LOCK flag to `DbEnv::open()`. + +8. Perform portable thread synchronization within a process by calling the following functions. These are all global functions in the "dbstl" name space: + + | | + |---------------------------------| + | `db_mutex_t alloc_mutex();` | + | `int lock_mutex(db_mutex_t);` | + | `int unlock_mutex(db_mutex_t);` | + | `void free_mutex(db_mutex_t);` | + + These functions use an internal dbstl environment's mutex functionality to synchronize. As a result, the synchronization is portable across all platforms supported by Berkeley DB. + +The `WorkerThread` class provides example code demonstrating the use of dbstl in multi-threaded applications. You can find this class implemented in the dbstl test suite. diff --git a/docs-src/guides/programmer_reference/stl_persistence.md b/docs-src/guides/programmer_reference/stl_persistence.md new file mode 100644 index 000000000..c7a52c57b --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_persistence.md @@ -0,0 +1,161 @@ +--- +title: "Dbstl persistence" +api-name: "Dbstl persistence" +source: docs/programmer_reference/stl_persistence.html +--- +## Dbstl persistence + + [Direct database get](stl_persistence.md#directdbget) + + [Change persistence](stl_persistence.md#chg_persistence) + + [Object life time and persistence](stl_persistence.md#obj_life_persistence) + +The following sections provide information on how to achieve persistence using dbstl. + +### Direct database get + +Each container has a **begin()** method which produces an iterator. These **begin** methods take a boolean parameter, **directdb_get**, which controls the caching behavior of the iterator. The default value of this parameter is `true`. + +If **directdb_get** is `true`, then the persistent object is fetched anew from the database each time the iterator is dereferenced as a pointer by use of the star-operator (**\*iterator**) or by use of the arrow-operator (**iterator-\>member**). If **directdb_get** is `false`, then the first dereferencing of the iterator fetches the object from the database, but later dereferences can return cached data. + +With **directdb_get** set to `true`, if you call: + +``` c +(*iterator).datamember1=new-value1; +(*iterator).datamember2=new-value2; +``` + +then the assignment to `datamember1` will be lost, because the second dereferencing of the iterator would cause the cached copy of the object to be overwritten by the object's persistent data from the database. + +You also can use the arrow operator like this: + +``` c +iterator->datamember1=new-value1; +iterator->datamember2=new-value2; +``` + +This works exactly the same way as **iterator::operator\***. For this reason, the same caching rules apply to arrow operators as they do for star operators. + +One way to avoid this problem is to create a reference to the object, and use it to access the object: + +``` c +container::value_type &ref = *iterator; +ref.datamember1=new-value1; +ref.datamember2=new-value2; +...// more member function calls and datamember assignments +ref._DB_STL_StoreElement(); +``` + +The above code will not lose the newly assigned value of `ref.datamember1` in the way that the previous example did. + +In order to avoid these complications, you can assign to the object referenced by an iterator with another object of the same type like this: + +``` c +container::value_type obj2; +obj2.datamember1 = new-value1; +obj2.datamember2 = new-value2; +*itr = obj2; +``` + +This code snippet causes the new values in `obj2` to be stored into the underlying database. + +If you have two iterators going through the same container like this: + +``` c +for (iterator1 = v.begin(), iterator2 = v.begin(); + iterator1 != v.end(); + ++iterator1, ++iterator2) { + *iterator1 = new_value; + print(*iterator2); +} +``` + +then the printed value will depend on the value of **directdb_get** with which the iterator had been created. If **directdb_get** is `false`, then the original, persistent value is printed; otherwise the newly assigned value is returned from the cache when `iterator2` is dereferenced. This happens because each iterator has its own cached copy of the persistent object, and the dereferencing of `iterator2` refreshes `iterator2`'s copy from the database, retrieving the value stored by the assignment to `*iterator1`. + +Alternatively, you can set **directdb_get** to `false` and call `iterator2->refresh()` immediately before the dereferencing of `iterator2`, so that `iterator2`'s cached value is refreshed. + +If **directdb_get** is `false`, a few of the tests in dbstl's test kit will fail. This is because the above contrived case appears in several of C++ STL tests. Consequently, the default value of the **directdb_get** parameter in the `container::begin()` methods is `true`. If your use cases avoid such bizarre usage of iterators, you can set it to `false`, which makes the iterator read operation faster. + +### Change persistence + +If you modify the object to which an iterator refers by using one of the following: + +``` c +(*iterator).member_function_call() +``` + +or + +``` c +(*iterator).data_member = new_value +``` + +then you should call `iterator->_DB_STL_StoreElement()` to store the change. Otherwise the change is lost after the iterator moves on to other elements. + +If you are storing a sequence, and you modified some part of it, you should also call `iterator->_DB_STL_StoreElement()` before moving the iterator. + +And in both cases, if **directdb_get** is `true` (this is the default value), you should call `_DB_STL_StoreElement()` after the change and before the next iterator movement OR the next dereferencing of the iterator by the star or arrow operators (`iterator::operator*` or `iterator::operator->`). Otherwise, you will lose the change. + +If you update the element by assigning to a dereferenced iterator like this: + +``` c +*iterator = new_element; +``` + +then you never have to call `_DB_STL_StoreElement()` because the change is stored in the database automatically. + +### Object life time and persistence + +Dbstl is an interface to Berkeley DB, so it is used to store data persistently. This is really a different purpose from that of regular C++ STL. This difference in their goals has implications on expected object lifetime: In standard STL, when you store an object A of type ID into C++ stl vector V using V.push_back(A), if a proper copy constructor is provided in A's class type, then the copy of A (call it B) and everything in B, such as another object C pointed to by B's data member B.c_ptr, will be stored in V and will live as long as B is still in V and V is alive. B will be destroyed when V is destroyed or B is erased from V. + +This is not true for dbstl, which will copy A's data and store it in the underlying database. The copy is by default a shallow copy, but users can register their object marshalling and unmarshalling functions using the `DbstlElemTraits` class template. So if A is passed to a `db_vector` container, `dv`, by using `dv.push_back(A)`, then dbstl copies A's data using the registered functions, and stores data into the underlying database. Consequently, A will be valid, even if the container is destroyed, because it is stored into the database. + +If the copy is simply a shallow copy, and A is later destroyed, then the pointer stored in the database will become invalid. The next time we use the retrieved object, we will be using an invalid pointer, which probably will result in errors. To avoid this, store the referred object C rather than the pointer member A.c_ptr itself, by registering the right marshalling/unmarshalling function with `DbstlElemTraits`. + +For example, consider the following example class declaration: + +``` c +class ID +{ +public: + string Name; + int Score; +}; +``` + +Here, the class ID has a data member **Name**, which refers to a memory address of the actual characters in the string. If we simply shallow copy an object, `id`, of class ID to store it, then the stored data, `idd`, is invalid when `id` is destroyed. This is because `idd` and `id` refer to a common memory address which is the base address of the memory space storing all characters in the string, and this memory space is released when `id` is destroyed. So `idd` will be referring to an invalid address. The next time we retrieve `idd` and use it, there will probably be memory corruption. + +The way to store `id` is to write a marshal/unmarshal function pair like this: + +``` c +void copy_id(void *dest, const ID&elem) +{ + memcpy(dest, &elem.Score, sizeof(elem.Score)); + char *p = ((char *)dest) + sizeof(elem.Score); + strcpy(p, elem.Name.c_str()); +} + +void restore_id(ID& dest, const void *srcdata) +{ + memcpy(&dest.Score, srcdata, sizeof(dest.Score)); + const char *p = ((char *)srcdata) + sizeof(dest.Score); + dest.Name = p; +} + +size_t size_id(const ID& elem) +{ + return sizeof(elem.Score) + elem.Name.size() + + 1;// store the '\0' char. +} +``` + +Then register the above functions before storing any instance of `ID`: + +``` c +DbstlElemTraits::instance()->set_copy_function(copy_id); +DbstlElemTraits::instance()->set_size_function(size_id); +DbstlElemTraits::instance()->set_restore_function(restore_id); +``` + +This way, the actual data of instances of ID are stored, and so the data will persist even if the container itself is destroyed. diff --git a/docs-src/guides/programmer_reference/stl_primitive_rw.md b/docs-src/guides/programmer_reference/stl_primitive_rw.md new file mode 100644 index 000000000..57719582b --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_primitive_rw.md @@ -0,0 +1,80 @@ +--- +title: "Working with primitive types" +api-name: "Working with primitive types" +source: docs/programmer_reference/stl_primitive_rw.html +--- +## Working with primitive types + + [Storing strings](stl_primitive_rw.md#idp51467888) + +To store simple primitive types such as `int`, `long`, `double`, and so forth, an additional type parameter for the container class templates is needed. For example, to store an `int` in a `db_vector`, use this container class: + +``` c +db_vector >; +``` + +To map integers to doubles, use this: + +``` c +db_map >; +``` + +To store a `char*` string with `long` keys, use this: + +``` c +db_map >; +``` + +Use this for `const char*` strings: + +``` c +db_map >; +``` + +To map one const string to another, use this type: + +``` c +db_map >; +``` + +The `StlAdvancedFeaturesExample::primitive()` method demonstrates more of these examples. + +### Storing strings + +For `char*` and `wchar_t*` strings, `_DB_STL_StoreElement()` must be called following partial or total modifications before iterator movement, `container::operator[]` or `iterator::operator*/->` calls. Without the `_DB_STL_StoreElement()` call, the modified change will be lost. If storing an new value like this: + +``` c +*iterator = new_char_star_string; +``` + +the call to `_DB_STL_StoreElement()` is not needed. + +Note that passing a NULL pointer to a container of `char*` type or passing a `std::string` with no contents at all will insert an empty string of zero length into the database. + +The string returned from a container will not live beyond the next iterator movement call, `container::operator[]` or `iterator::operator*/->` call. + +A **db_map::value_type::second_type** or **db_map::datatype_wrap** should be used to hold a reference to a `container::operator[]` return value. Then the reference should be used for repeated references to that value. The \*iterator is of type `ElementHolder`, which can be automatically converted to a `char *` pointer using its type conversion operator. Wherever an auto conversion is done by the compiler, the conversion operator of `ElementHolder` is called. This avoids almost all explicit conversions, except for two use cases: + +1. The \*iterator is used as a "..." parameter like this: + + ``` c + printf("this is the special case %s", *iterator); + ``` + + This compiles but causes errors. Instead, an explicit cast should be used: + + ``` c + printf("this is the special case %s", (char *)*iterator); + ``` + +2. For some old compilers, such as gcc3.4.6, the \*iterator cannot be used with the ternary `?` operator, like this: + + ``` c + expr ? *iterator : var + ``` + + Even when **var** is the same type as the iterator's `value_type`, the compiler fails to perform an auto conversion. + +When using `std::string` or `std::wstring` as the data type for dbstl containers — that is, `db_vector`, and `db_map` — the string's content rather than the string object itself is stored in order to maintain persistence. + +You can find example code demonstrating string storage in the `StlAdvancedFeaturesExample::char_star_string_storage()` and `StlAdvancedFeaturesExample::storing_std_strings()` methods. diff --git a/docs-src/guides/programmer_reference/stl_txn_usage.md b/docs-src/guides/programmer_reference/stl_txn_usage.md new file mode 100644 index 000000000..e3ec67a91 --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_txn_usage.md @@ -0,0 +1,20 @@ +--- +title: "Using transactions in dbstl" +api-name: "Using transactions in dbstl" +source: docs/programmer_reference/stl_txn_usage.html +--- +## Using transactions in dbstl + +When using transactions with dbstl, you must call the dbstl transaction functions instead of the corresponding methods from the Berkeley DB C or C++ transaction API. That is, you must use `dbstl::begin_txn()`, `dbstl::commit_txn()` and `dbstl::abort_txn()` in order to begin/commit/abort transactions. + +A container can be configured to use auto commit by setting the DB_AUTO_COMMIT flag when the environment or database handle is opened. In this case, any container method that supports auto commit will automatically form an independent transaction if the method is not in an external transactional context; Otherwise, the operation will become part of that transaction. + +You can configure the flags used internally by dbstl when it is creating and committing these independent transactions required by auto commit. To do so, use the `db_container::set_txn_begin_flags()` and/or `db_container::set_commit_flags()` methods. + +When a transaction is committed or aborted, dbstl will automatically close any cursors opened for use by the transaction. For this reason, any iterators opened within the transaction context should not be used after the transaction commits or aborts. + +You can use nested transactions explicitly and externally, by calling `dbstl::begin_txn()` in a context already operating under the protection of a transaction. But you can not designate which transaction is the parent transaction. The parent transaction is automatically the most recently created and unresolved transaction in current thread. + +It is also acceptable to use explicit transactions in a container configured for auto commit. The operation performed by the method will become part of the provided external transaction. + +Finally, transactions and iterators cannot be shared among multiple threads. That is, they are not free-threaded, or thread-safe. diff --git a/docs-src/guides/programmer_reference/stl_usecase.md b/docs-src/guides/programmer_reference/stl_usecase.md new file mode 100644 index 000000000..e1e4764dc --- /dev/null +++ b/docs-src/guides/programmer_reference/stl_usecase.md @@ -0,0 +1,18 @@ +--- +title: "Dbstl typical use cases" +api-name: "Dbstl typical use cases" +source: docs/programmer_reference/stl_usecase.html +--- +## Dbstl typical use cases + +Among others, the following are some typical use cases where dbstl would be prefered over C++ STL: + +- Working with a large amount of data, more than can reside in memory. Using C++ STL would force a number of page swaps, which will degrade performance. When using dbstl, data is stored in a database and Berkeley DB ensures the needed data is in memory, so that the overall performance of the machine is not slowed down. + +- Familiar Interface. dbstl provides a familiar interface to Berkeley DB, hiding the marshalling and unmashalling details and automatically managing Berkeley DB structures and objects. + +- Transaction semantics. dbstl provides the ACID properties (or a subset of the ACID properties) in addition to supporting all of the STL functionality. + +- Concurrent access. Few (if any) existing C++ STL implementations support reading/writing to the same container concurrently, dbstl does. + +- Object persistence. dbstl allows your application to store objects in a database, and use the objects across different runs of your application. dbstl is capable of storing complicated objects which are not located in a contiguous chunk of memory, with some user configurations. diff --git a/docs-src/guides/programmer_reference/tcl.md b/docs-src/guides/programmer_reference/tcl.md new file mode 100644 index 000000000..436f1cac0 --- /dev/null +++ b/docs-src/guides/programmer_reference/tcl.md @@ -0,0 +1,75 @@ +--- +title: "Chapter 21.  Berkeley DB Extensions: Tcl" +api-name: "Chapter 21.  Berkeley DB Extensions: Tcl" +source: docs/programmer_reference/tcl.html +--- +## Chapter 21.  Berkeley DB Extensions: Tcl + +**Table of Contents** + + [Loading Berkeley DB with Tcl](tcl.md#tcl_intro) + + [Installing as a Tcl Package](tcl.md#idp53366464) + + [Loading Berkeley DB with Tcl](tcl.md#idp53356912) + + [Using Berkeley DB with Tcl](tcl_using.md) + + [Tcl API programming notes](tcl_program.md) + + [Tcl error handling](tcl_error.md) + + [Tcl FAQ](tcl_faq.md) + +## Loading Berkeley DB with Tcl + + [Installing as a Tcl Package](tcl.md#idp53366464) + + [Loading Berkeley DB with Tcl](tcl.md#idp53356912) + +Berkeley DB includes a dynamically loadable Tcl API, which requires that Tcl/Tk 8.5 or later already be installed on your system. You can download a copy of Tcl from the Tcl Developer Xchange Web site. + +This document assumes that you already configured Berkeley DB for Tcl support, and you have built and installed everything where you want it to be. If you have not done so, see Configuring Berkeley DB or Building the Tcl API in the Berkeley DB Installation and Build Guide for more information. + +### Installing as a Tcl Package + +Once enabled, the Berkeley DB shared library for Tcl is automatically installed as part of the standard installation process. However, if you want to be able to dynamically load it as a Tcl package into your script, there are several steps that must be performed: + +1. Run the Tcl shell in the install directory. +2. Append this directory to your auto_path variable. +3. Run the pkg_mkIndex proc, giving the name of the Berkeley DB Tcl library. + +For example: + +``` c +# tclsh8.5 +% lappend auto_path /usr/local/BerkeleyDB.5.2/lib +% pkg_mkIndex /usr/local/BerkeleyDB.5.2/lib libdb_tcl-5.2.so +``` + +Note that your Tcl and Berkeley DB version numbers may differ from the example, and so your tclsh and library names may be different. + +### Loading Berkeley DB with Tcl + +The Berkeley DB package may be loaded into the user's interactive Tcl script (or wish session) via the **load** command. For example: + +``` c +load /usr/local/BerkeleyDB.5.2/lib/libdb_tcl-5.2.so +``` + +Note that your Berkeley DB version numbers may differ from the example, and so the library name may be different. + +If you installed your library to run as a Tcl package, Tcl application scripts should use the **package** command to indicate to the Tcl interpreter that it needs the Berkeley DB package and where to find it. For example: + +``` c +lappend auto_path "/usr/local/BerkeleyDB.5.2/lib" +package require Db_tcl +``` + +No matter which way the library gets loaded, it creates a command named **berkdb**. All the Berkeley DB functionality is accessed via this command and additional commands it creates on behalf of the application. A simple test to determine whether everything is loaded and ready is to display the library version, as follows: + +``` c +berkdb version -string +``` + +This should return you the Berkeley DB version in a string format. diff --git a/docs-src/guides/programmer_reference/tcl_error.md b/docs-src/guides/programmer_reference/tcl_error.md new file mode 100644 index 000000000..e1b850809 --- /dev/null +++ b/docs-src/guides/programmer_reference/tcl_error.md @@ -0,0 +1,42 @@ +--- +title: "Tcl error handling" +api-name: "Tcl error handling" +source: docs/programmer_reference/tcl_error.html +--- +## Tcl error handling + +The Tcl interfaces to Berkeley DB generally return TCL_OK on success and throw a Tcl error on failure, using the appropriate Tcl interfaces to provide the user with an informative error message. There are some "expected" failures, however, for which no Tcl error will be thrown and for which Tcl commands will return TCL_OK. These failures include times when a searched-for key is not found, a requested key/data pair was previously deleted, or a key/data pair cannot be written because the key already exists. + +These failures can be detected by searching the Berkeley DB error message that is returned. For example, use the following to detect that an attempt to put a record into the database failed because the key already existed: + +``` c +% berkdb open -create -btree a.db +db0 +% db0 put dog cat +0 +% set ret [db0 put -nooverwrite dog newcat] +DB_KEYEXIST: Key/data pair already exists +% if { [string first DB_KEYEXIST $ret] != -1 } { + puts "This was an error; the key existed" +} +This was an error; the key existed +% db0 close +0 +% exit +``` + +To simplify parsing, it is recommended that the initial Berkeley DB error name be checked; for example, DB_MULTIPLE in the previous example. To ensure that Tcl scripts are not broken by upgrading to new releases of Berkeley DB, these values will not change in future releases of Berkeley DB. There are currently only three such "expected" error returns: + +``` c +DB_NOTFOUND: No matching key/data pair found +DB_KEYEMPTY: Nonexistent key/data pair +DB_KEYEXIST: Key/data pair already exists +``` + +Finally, sometimes Berkeley DB will output additional error information when a Berkeley DB error occurs. By default, all Berkeley DB error messages will be prefixed with the created command in whose context the error occurred (for example, "env0", "db2", and so on). There are several ways to capture and access this information. + +First, if Berkeley DB invokes the error callback function, the additional information will be placed in the error result returned from the command and in the errorInfo backtrace variable in Tcl. + +Also, the two calls to open an environment and open a database take an option, **-errfile filename**, which sets an output file to which these additional error messages should be written. + +Additionally, the two calls to open an environment and open a database take an option, **-errpfx string**, which sets the error prefix to the given string. This option may be useful in circumstances where a more descriptive prefix is desired or where a constant prefix indicating an error is desired. diff --git a/docs-src/guides/programmer_reference/tcl_faq.md b/docs-src/guides/programmer_reference/tcl_faq.md new file mode 100644 index 000000000..e4511a0a9 --- /dev/null +++ b/docs-src/guides/programmer_reference/tcl_faq.md @@ -0,0 +1,24 @@ +--- +title: "Tcl FAQ" +api-name: "Tcl FAQ" +source: docs/programmer_reference/tcl_faq.html +--- +## Tcl FAQ + +1. **I have several versions of Tcl installed. How do I configure Berkeley DB to use a particular version?** + + To compile the Tcl interface with a particular version of Tcl, use the --with-tcl option to specify the Tcl installation directory that contains the tclConfig.sh file. See the Changing compile or load options section in the Berkeley DB Installation and Build Guide for more information. + +2. **Berkeley DB was configured using --enable-tcl or --with-tcl and fails to build.** + + The Berkeley DB Tcl interface requires Tcl version 8.5 or greater. + +3. **Berkeley DB was configured using --enable-tcl or --with-tcl and fails to build.** + + If the Tcl installation was moved after it was configured and installed, try reconfiguring and reinstalling Tcl. + + Also, some systems do not search for shared libraries by default, or do not search for shared libraries named the way the Tcl installation names them, or are searching for a different kind of library than those in your Tcl installation. For example, Linux systems often require linking "libtcl.a" to "libtcl#.#.a", whereas AIX systems often require adding the "-brtl" flag to the linker. A simpler solution that almost always works on all systems is to create a link from "libtcl.#.#.a" or "libtcl.so" (or whatever you happen to have) to "libtcl.a" and reconfigure. + +4. **Loading the Berkeley DB library into Tcl on AIX causes a core dump.** + + In some versions of Tcl, the "tclConfig.sh" autoconfiguration script created by the Tcl installation does not work properly under AIX, and you may have to modify values in the tclConfig.sh file to in order to load the Berkeley DB library into Tcl. Specifically, the TCL_LIB_SPEC variable should contain sufficient linker flags to find and link against the installed libtcl library. In some circumstances, the tclConfig.sh file built by Tcl does not. diff --git a/docs-src/guides/programmer_reference/tcl_program.md b/docs-src/guides/programmer_reference/tcl_program.md new file mode 100644 index 000000000..10d02bc19 --- /dev/null +++ b/docs-src/guides/programmer_reference/tcl_program.md @@ -0,0 +1,14 @@ +--- +title: "Tcl API programming notes" +api-name: "Tcl API programming notes" +source: docs/programmer_reference/tcl_program.html +--- +## Tcl API programming notes + +The Berkeley DB Tcl API does not attempt to avoid evaluating input as Tcl commands. For this reason, it may be dangerous to pass unreviewed user input through the Berkeley DB Tcl API, as the input may subsequently be evaluated as a Tcl command. Additionally, the Berkeley DB Tcl API initialization routine resets process' effective user and group IDs to the real user and group IDs, to minimize the effectiveness of a Tcl injection attack. + +The Tcl API closely parallels the Berkeley DB programmatic interfaces. If you are already familiar with one of those interfaces, there will not be many surprises in the Tcl API. + +The Tcl API currently does not support multithreading although it could be made to do so. The Tcl shell itself is not multithreaded and the Berkeley DB extensions use global data unprotected from multiple threads. + +Several pieces of Berkeley DB functionality are not available in the Tcl API. Any of the functions that require a user-provided function are not supported via the Tcl API. For example, there is no equivalent to the DB->set_dup_compare() or DB_ENV->set_errcall() methods. Additionally, the heap access method is not available. diff --git a/docs-src/guides/programmer_reference/tcl_using.md b/docs-src/guides/programmer_reference/tcl_using.md new file mode 100644 index 000000000..93c749a10 --- /dev/null +++ b/docs-src/guides/programmer_reference/tcl_using.md @@ -0,0 +1,35 @@ +--- +title: "Using Berkeley DB with Tcl" +api-name: "Using Berkeley DB with Tcl" +source: docs/programmer_reference/tcl_using.html +--- +## Using Berkeley DB with Tcl + +All commands in the Berkeley DB Tcl interface are in the following form: + +``` c +command_handle operation options +``` + +The *command handle* is **berkdb** or one of the additional commands that may be created. The *operation* is what you want to do to that handle, and the *options* apply to the operation. Commands that get created on behalf of the application have their own sets of operations. Generally, any calls in DB that result in new object handles will translate into a new command handle in Tcl. Then, the user can access the operations of the handle via the new Tcl command handle. + +Newly created commands are named with an abbreviated form of their objects, followed by a number. Some created commands are subcommands of other created commands and will be the first command, followed by a period (.), and then followed by the new subcommand. For example, suppose that you have a database already existing called my_data.db. The following example shows the commands created when you open the database and when you open a cursor: + +``` c +# First open the database and get a database command handle +% berkdb open my_data.db +db0 +#Get some data from that database +% db0 get my_key +{{my_key my_data0}{my_key my_data1}} +#Open a cursor in this database, get a new cursor handle +% db0 cursor +db0.c0 +#Get the first data from the cursor +% db0.c0 get -first +{{first_key first_data}} +``` + +All commands in the library support a special option **-?** that will list the correct operations for a command or the correct options. + +A list of commands and operations can be found in the Tcl API documentation. diff --git a/docs-src/guides/programmer_reference/transapp.md b/docs-src/guides/programmer_reference/transapp.md new file mode 100644 index 000000000..30f8c6267 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp.md @@ -0,0 +1,68 @@ +--- +title: "Chapter 11.  Berkeley DB Transactional Data Store Applications" +api-name: "Chapter 11.  Berkeley DB Transactional Data Store Applications" +source: docs/programmer_reference/transapp.html +--- +## Chapter 11.  Berkeley DB Transactional Data Store Applications + +**Table of Contents** + + [Transactional Data Store introduction](transapp.md#transapp_intro) + + [Why transactions?](transapp_why.md) + + [Terminology](transapp_term.md) + + [Handling failure in Transactional Data Store applications](transapp_fail.md) + + [Architecting Transactional Data Store applications](transapp_app.md) + + [Opening the environment](transapp_env_open.md) + + [Opening the databases](transapp_data_open.md) + + [Recoverability and deadlock handling](transapp_put.md) + + [Atomicity](transapp_atomicity.md) + + [Isolation](transapp_inc.md) + + [Degrees of isolation](transapp_read.md) + + [Snapshot Isolation](transapp_read.md#snapshot_isolation) + + [Transactional cursors](transapp_cursor.md) + + [Nested transactions](transapp_nested.md) + + [Environment infrastructure](transapp_admin.md) + + [Deadlock detection](transapp_deadlock.md) + + [Checkpoints](transapp_checkpoint.md) + + [Database and log file archival](transapp_archival.md) + + [Log file removal](transapp_logfile.md) + + [Recovery procedures](transapp_recovery.md) + + [Hot failover](transapp_hotfail.md) + + [Using Recovery on Journaling Filesystems](transapp_journal.md) + + [Recovery and filesystem operations](transapp_filesys.md) + + [Berkeley DB recoverability](transapp_reclimit.md) + + [Transaction tuning](transapp_tune.md) + + [Transaction throughput](transapp_throughput.md) + + [Transaction FAQ](transapp_faq.md) + +## Transactional Data Store introduction + +It is difficult to write a useful transactional tutorial and still keep within reasonable bounds of documentation; that is, without writing a book on transactional programming. We have two goals in this section: to familiarize readers with the transactional interfaces of Berkeley DB and to provide code building blocks that will be useful for creating applications. + +We have not attempted to present this information using a real-world application. First, transactional applications are often complex and time-consuming to explain. Also, one of our goals is to give you an understanding of the wide variety of tools Berkeley DB makes available to you, and no single application would use most of the interfaces included in the Berkeley DB library. For these reasons, we have chosen to simply present the Berkeley DB data structures and programming solutions, using examples that differ from page to page. All the examples are included in a standalone program you can examine, modify, and run; and from which you will be able to extract code blocks for your own applications. Fragments of the program will be presented throughout this chapter, and the complete text of the example program for IEEE/ANSI Std 1003.1 (POSIX) standard systems is included in the Berkeley DB distribution. diff --git a/docs-src/guides/programmer_reference/transapp_admin.md b/docs-src/guides/programmer_reference/transapp_admin.md new file mode 100644 index 000000000..1c75c1e26 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_admin.md @@ -0,0 +1,16 @@ +--- +title: "Environment infrastructure" +api-name: "Environment infrastructure" +source: docs/programmer_reference/transapp_admin.html +--- +## Environment infrastructure + +When building transactional applications, it is usually necessary to build an administrative infrastructure around the database environment. There are five components to this infrastructure, and each is supported by the Berkeley DB package in two different ways: a standalone utility and one or more library interfaces. + +- Deadlock detection: db_deadlock utility, DB_ENV->lock_detect(), DB_ENV->set_lk_detect() +- Checkpoints: the db_checkpoint utility, DB_ENV->txn_checkpoint() +- Database and log file archival: the db_archive utility, DB_ENV->log_archive() +- Log file removal: db_archive utility, DB_ENV->log_archive() +- Recovery procedures: db_recover utility, DB_ENV->open() + +When writing multithreaded server applications and/or applications intended for download from the Web, it is usually simpler to create local threads that are responsible for administration of the database environment as scheduling is often simpler in a single-process model, and only a single binary need be installed and run. However, the supplied utilities can be generally useful tools even when the application is responsible for doing its own administration because applications rarely offer external interfaces to database administration. The utilities are required when programming to a Berkeley DB scripting interface because the scripting APIs do not always offer interfaces to the administrative functionality. diff --git a/docs-src/guides/programmer_reference/transapp_app.md b/docs-src/guides/programmer_reference/transapp_app.md new file mode 100644 index 000000000..7edcd3192 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_app.md @@ -0,0 +1,68 @@ +--- +title: "Architecting Transactional Data Store applications" +api-name: "Architecting Transactional Data Store applications" +source: docs/programmer_reference/transapp_app.html +--- +## Architecting Transactional Data Store applications + +When building Transactional Data Store applications, the architecture decisions involve application startup (running recovery) and handling system or application failure. For details on performing recovery, see the Recovery procedures. + +Recovery in a database environment is a single-threaded procedure, that is, one thread of control or process must complete database environment recovery before any other thread of control or process operates in the Berkeley DB environment. + +Performing recovery first marks any existing database environment as "failed" and then removes it, causing threads of control running in the database environment to fail and return to the application. This feature allows applications to recover environments without concern for threads of control that might still be running in the removed environment. The subsequent re-creation of the database environment is serialized, so multiple threads of control attempting to create a database environment will serialize behind a single creating thread. + +One consideration in removing (as part of recovering) a database environment which may be in use by another thread, is the type of mutex being used by the Berkeley DB library. In the case of database environment failure when using test-and-set mutexes, threads of control waiting on a mutex when the environment is marked "failed" will quickly notice the failure and will return an error from the Berkeley DB API. In the case of environment failure when using blocking mutexes, where the underlying system mutex implementation does not unblock mutex waiters after the thread of control holding the mutex dies, threads waiting on a mutex when an environment is recovered might hang forever. Applications blocked on events (for example, an application blocked on a network socket, or a GUI event) may also fail to notice environment recovery within a reasonable amount of time. Systems with such mutex implementations are rare, but do exist; applications on such systems should use an application architecture where the thread recovering the database environment can explicitly terminate any process using the failed environment, or configure Berkeley DB for test-and-set mutexes, or incorporate some form of long-running timer or watchdog process to wake or kill blocked processes should they block for too long. + +Regardless, it makes little sense for multiple threads of control to simultaneously attempt recovery of a database environment, since the last one to run will remove all database environments created by the threads of control that ran before it. However, for some applications, it may make sense for applications to have a single thread of control that performs recovery and then removes the database environment, after which the application launches a number of processes, any of which will create the database environment and continue forward. + +There are three common ways to architect Berkeley DB Transactional Data Store applications. The one chosen is usually based on whether or not the application is comprised of a single process or group of processes descended from a single process (for example, a server started when the system first boots), or if the application is comprised of unrelated processes (for example, processes started by web connections or users logged into the system). + +1. The first way to architect Transactional Data Store applications is as a single process (the process may or may not be multithreaded.) + + When this process starts, it runs recovery on the database environment and then opens its databases. The application can subsequently create new threads as it chooses. Those threads can either share already open Berkeley DB DB_ENV and DB handles, or create their own. In this architecture, databases are rarely opened or closed when more than a single thread of control is running; that is, they are opened when only a single thread is running, and closed after all threads but one have exited. The last thread of control to exit closes the databases and the database environment. + + This architecture is simplest to implement because thread serialization is easy and failure detection does not require monitoring multiple processes. + + If the application's thread model allows processes to continue after thread failure, the DB_ENV->failchk() method can be used to determine if the database environment is usable after thread failure. If the application does not call DB_ENV->failchk(), or DB_ENV->failchk() returns DB_RUNRECOVERY, the application must behave as if there has been a system failure, performing recovery and re-creating the database environment. Once these actions have been taken, other threads of control can continue (as long as all existing Berkeley DB handles are first discarded). + +2. The second way to architect Transactional Data Store applications is as a group of related processes (the processes may or may not be multithreaded). + + This architecture requires the order in which threads of control are created be controlled to serialize database environment recovery. + + In addition, this architecture requires that threads of control be monitored. If any thread of control exits with open Berkeley DB handles, the application may call the DB_ENV->failchk() method to detect lost mutexes and locks and determine if the application can continue. If the application does not call DB_ENV->failchk(), or DB_ENV->failchk() returns that the database environment can no longer be used, the application must behave as if there has been a system failure, performing recovery and creating a new database environment. Once these actions have been taken, other threads of control can be continued (as long as all existing Berkeley DB handles are first discarded), or + + The easiest way to structure groups of related processes is to first create a single "watcher" process (often a script) that starts when the system first boots, runs recovery on the database environment and then creates the processes or threads that will actually perform work. The initial thread has no further responsibilities other than to wait on the threads of control it has started, to ensure none of them unexpectedly exit. If a thread of control exits, the watcher process optionally calls the DB_ENV->failchk() method. If the application does not call DB_ENV->failchk() or if DB_ENV->failchk() returns that the environment can no longer be used, the watcher kills all of the threads of control using the failed environment, runs recovery, and starts new threads of control to perform work. + +3. The third way to architect Transactional Data Store applications is as a group of unrelated processes (the processes may or may not be multithreaded). This is the most difficult architecture to implement because of the level of difficulty in some systems of finding and monitoring unrelated processes. There are several possible techniques to implement this architecture. + + One solution is to log a thread of control ID when a new Berkeley DB handle is opened. For example, an initial "watcher" process could run recovery on the database environment and then create a sentinel file. Any "worker" process wanting to use the environment would check for the sentinel file. If the sentinel file does not exist, the worker would fail or wait for the sentinel file to be created. Once the sentinel file exists, the worker would register its process ID with the watcher (via shared memory, IPC or some other registry mechanism), and then the worker would open its DB_ENV handles and proceed. When the worker finishes using the environment, it would unregister its process ID with the watcher. The watcher periodically checks to ensure that no worker has failed while using the environment. If a worker fails while using the environment, the watcher removes the sentinel file, kills all of the workers currently using the environment, runs recovery on the environment, and finally creates a new sentinel file. + + The weakness of this approach is that, on some systems, it is difficult to determine if an unrelated process is still running. For example, POSIX systems generally disallow sending signals to unrelated processes. The trick to monitoring unrelated processes is to find a system resource held by the process that will be modified if the process dies. On POSIX systems, flock- or fcntl-style locking will work, as will LockFile on Windows systems. Other systems may have to use other process-related information such as file reference counts or modification times. In the worst case, threads of control can be required to periodically re-register with the watcher process: if the watcher has not heard from a thread of control in a specified period of time, the watcher will take action, recovering the environment. + + The Berkeley DB library includes one built-in implementation of this approach, the DB_ENV->open() method's DB_REGISTER flag: + + If the DB_REGISTER flag is set, each process opening the database environment first checks to see if recovery needs to be performed. If recovery needs to be performed for any reason (including the initial creation of the database environment), and DB_RECOVER is also specified, recovery will be performed and then the open will proceed normally. If recovery needs to be performed and DB_RECOVER is not specified, DB_RUNRECOVERY will be returned. If recovery does not need to be performed, DB_RECOVER will be ignored. + + Prior to the actual recovery beginning, the DB_EVENT_REG_PANIC event is set for the environment. Processes in the application using the DB_ENV->set_event_notify() method will be notified when they do their next operations in the environment. Processes receiving this event should exit the environment. Also, the DB_EVENT_REG_ALIVE event will be triggered if there are other processes currently attached to the environment. Only the process doing the recovery will receive this event notification. It will receive this notification once for each process still attached to the environment. The parameter of the DB_ENV->set_event_notify() callback will contain the process identifier of the process still attached. The process doing the recovery can then signal the attached process or perform some other operation prior to recovery (i.e. kill the attached process). + + The DB_ENV->set_timeout() method's DB_SET_REG_TIMEOUT flag can be set to establish a wait period before starting recovery. This creates a window of time for other processes to receive the DB_EVENT_REG_PANIC event and exit the environment. + + There are three additional requirements for the DB_REGISTER architecture to work: + + - First, all applications using the database environment must specify the DB_REGISTER flag when opening the environment. However, there is no additional requirement if the application chooses a single process to recover the environment, as the first process to open the database environment will know to perform recovery. + + - Second, there can only be a single DB_ENV handle per database environment in each process. As the DB_REGISTER locking is per-process, not per-thread, multiple DB_ENV handles in a single environment could race with each other, potentially causing data corruption. + + - Third, the DB_REGISTER implementation does not explicitly terminate processes using the database environment which is being recovered. Instead, it relies on the processes themselves noticing the database environment has been discarded from underneath them. For this reason, the DB_REGISTER flag should be used with a mutex implementation that does not block in the operating system, as that risks a thread of control blocking forever on a mutex which will never be granted. Using any test-and-set mutex implementation ensures this cannot happen, and for that reason the DB_REGISTER flag is generally used with a test-and-set mutex implementation. + + A second solution for groups of unrelated processes is also based on a "watcher process". This solution is intended for systems where it is not practical to monitor the processes sharing a database environment, but it is possible to monitor the environment to detect if a thread of control has failed holding open Berkeley DB handles. This would be done by having a "watcher" process periodically call the DB_ENV->failchk() method. If DB_ENV->failchk() returns that the environment can no longer be used, the watcher would then take action, recovering the environment. + + The weakness of this approach is that all threads of control using the environment must specify an "ID" function and an "is-alive" function using the DB_ENV->set_thread_id() method. (In other words, the Berkeley DB library must be able to assign a unique ID to each thread of control, and additionally determine if the thread of control is still running. It can be difficult to portably provide that information in applications using a variety of different programming languages and running on a variety of different platforms.) + + A third solution for groups of unrelated processes is a hybrid of the two above. Along with implementing the built-in sentinel approach with the the DB_ENV->open() methods DB_REGISTER flag, the DB_FAILCHK flag can be specified. When using both flags, each process opening the database environment first checks to see if recocvery needs to be performed. If recovery needs to be performed for any reason, it will first determine if a thread of control exited while holding database read locks, and release those. Then it will abort any unresolved transactions. If these steps are successful, the process opening the environment will continue without the need for any additional recocvery. If these steps are unsuccessful, then additional recovery will be performed if DB_RECOVER is specified and if DB_RECOVER is not specified, DB_RUNRECOVERYwill be returned. + + Since this solution is hybrid of the first two, all of the requirements of both of them must be implemented (will need "ID" function, "is-alive" function, single DB_ENV handle per database, etc.) + + The described approaches are different, and should not be combined. Applications might use either the DB_REGISTER approach, the DB_ENV->failchk() or the hybrid approach, but not together in the same application. For example, a POSIX application written as a library underneath a wide variety of interfaces and differing APIs might choose the DB_REGISTER approach for a few reasons: first, it does not require making periodic calls to the DB_ENV->failchk() method; second, when implementing in a variety of languages, is may be more difficult to specify unique IDs for each thread of control; third, it may be more difficult determine if a thread of control is still running, as any particular thread of control is likely to lack sufficient permissions to signal other processes. Alternatively, an application with a dedicated watcher process, running with appropriate permissions, might choose the DB_ENV->failchk() approach as supporting higher overall throughput and reliability, as that approach allows the application to abort unresolved transactions and continue forward without having to recover the database environment. The hybrid approach is useful in situations where running a dedicated watcher process is not practical but getting the equivalent of DB_ENV->failchk() on the DB_ENV->open() is important. + +Obviously, when implementing a process to monitor other threads of control, it is important the watcher process' code be as simple and well-tested as possible, because the application may hang if it fails. diff --git a/docs-src/guides/programmer_reference/transapp_archival.md b/docs-src/guides/programmer_reference/transapp_archival.md new file mode 100644 index 000000000..75953fbe8 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_archival.md @@ -0,0 +1,91 @@ +--- +title: "Database and log file archival" +api-name: "Database and log file archival" +source: docs/programmer_reference/transapp_archival.html +--- +## Database and log file archival + +The third component of the administrative infrastructure, archival for catastrophic recovery, concerns the recoverability of the database in the face of catastrophic failure. Recovery after catastrophic failure is intended to minimize data loss when physical hardware has been destroyed — for example, loss of a disk that contains databases or log files. Although the application may still experience data loss in this case, it is possible to minimize it. + +First, you may want to periodically create snapshots (that is, backups) of your databases to make it possible to recover from catastrophic failure. These snapshots are either a standard backup, which creates a consistent picture of the databases as of a single instant in time; or an on-line backup (also known as a *hot* backup), which creates a consistent picture of the databases as of an unspecified instant during the period of time when the snapshot was made. The advantage of a hot backup is that applications may continue to read and write the databases while the snapshot is being taken. The disadvantage of a hot backup is that more information must be archived, and recovery based on a hot backup is to an unspecified time between the start of the backup and when the backup is completed. + +Second, after taking a snapshot, you should periodically archive the log files being created in the environment. It is often helpful to think of database archival in terms of full and incremental filesystem backups. A snapshot is a full backup, whereas the periodic archival of the current log files is an incremental backup. For example, it might be reasonable to take a full snapshot of a database environment weekly or monthly, and archive additional log files daily. Using both the snapshot and the log files, a catastrophic crash at any time can be recovered to the time of the most recent log archival; a time long after the original snapshot. + +When incremental backups are implemented using this procedure, it is important to know that a database copy taken prior to a bulk loading event (that is, a transaction started with the DB_TXN_BULK flag) can no longer be used as the target of an incremental backup. This is true because bulk loading omits logging of some record insertions, so these insertions cannot be rolled forward by recovery. It is recommended that a full backup be scheduled following a bulk loading event. + +To create a standard backup of your database that can be used to recover from catastrophic failure, take the following steps: + +1. Commit or abort all ongoing transactions. + +2. Stop writing your databases until the backup has completed. Read-only operations are permitted, but no write operations and no filesystem operations may be performed (for example, the DB_ENV->remove() and DB->open() methods may not be called). + +3. Force an environment checkpoint (see the db_checkpoint utility for more information). + +4. Run the db_archive utility with option **-s** to identify all the database data files, and copy them to a backup device such as CD-ROM, alternate disk, or tape. + + If the database files are stored in a separate directory from the other Berkeley DB files, it may be simpler to archive the directory itself instead of the individual files (see DB_ENV->set_data_dir() for additional information). + + ### Note + + If any of the database files did not have an open DB handle during the lifetime of the current log files, the db_archive utility will not list them in its output. This is another reason it may be simpler to use a separate database file directory and archive the entire directory instead of archiving only the files listed by the db_archive utility. + +5. Run the db_archive utility with option **-l** to identify all the log files, and copy the last one (that is, the one with the highest number) to a backup device such as CD-ROM, alternate disk, or tape. + +To create a *hot* backup of your database that can be used to recover from catastrophic failure, take the following steps: + +1. Set the DB_HOTBACKUP_IN_PROGRESS flag in the environment. This affects the behavior of transactions started with the DB_TXN_BULK flag. + +2. Archive your databases, as described in the previous step \#4. You do not have to halt ongoing transactions or force a checkpoint. As this is a hot backup, and the databases may be modified during the copy, it is critical that database pages be read atomically as described by Berkeley DB recoverability. + + Note that only UNIX based systems are known to support the atomicity of reads. These systems include: Solaris, Mac OSX, HPUX and various BSD based systems. Linux and Windows based systems do not support atomic filesystem reads directly. The XFS file system supports atomic reads despite the lack of it in Linux. On systems that do not support atomic file system reads, the db_hotbackup utility should be used or a tool can be constructed using the DB_ENV->backup() method. Alternatively, you can construct a tool using the the db_copy() method. You can also perform a hot backup of just a single database in your environment using the DB_ENV->dbbackup() method. + +3. Archive **all** of the log files. The order of these two operations is required, and the database files must be archived **before** the log files. This means that if the database files and log files are in the same directory, you cannot simply archive the directory; you must make sure that the correct order of archival is maintained. + + To archive your log files, run the db_archive utility using the **-l** option to identify all the database log files, and copy them to your backup media. If the database log files are stored in a separate directory from the other database files, it may be simpler to archive the directory itself instead of the individual files (see the DB_ENV->set_lg_dir() method for more information). + +4. Reset the DB_HOTBACKUP_IN_PROGRESS flag. + +To minimize the archival space needed for log files when doing a hot backup, run db_archive to identify those log files which are not in use. Log files which are not in use do not need to be included when creating a hot backup, and you can discard them or move them aside for use with previous backups (whichever is appropriate), before beginning the hot backup. + +After completing one of these two sets of steps, the database environment can be recovered from catastrophic failure (see Recovery procedures for more information). + +To update either a hot or cold backup so that recovery from catastrophic failure is possible to a new point in time, repeat step \#2 under the hot backup instructions and archive **all** of the log files in the database environment. Each time both the database and log files are copied to backup media, you may discard all previous database snapshots and saved log files. Archiving additional log files does not allow you to discard either previous database snapshots or log files. Generally, updating a backup must be integrated with the application's log file removal procedures. + +The time to restore from catastrophic failure is a function of the number of log records that have been written since the snapshot was originally created. Perhaps more importantly, the more separate pieces of backup media you use, the more likely it is that you will have a problem reading from one of them. For these reasons, it is often best to make snapshots on a regular basis. + +**Obviously, the reliability of your archive media will affect the safety of your data. For archival safety, ensure that you have multiple copies of your database backups, verify that your archival media is error-free and readable, and that copies of your backups are stored offsite!** + +The functionality provided by the db_archive utility is also available directly from the Berkeley DB library. The following code fragment prints out a list of log and database files that need to be archived: + +``` c +void +log_archlist(DB_ENV *dbenv) +{ + int ret; + char **begin, **list; + + /* Get the list of database files. */ + if ((ret = dbenv->log_archive(dbenv, + &list, DB_ARCH_ABS | DB_ARCH_DATA)) != 0) { + dbenv->err(dbenv, ret, "DB_ENV->log_archive: DB_ARCH_DATA"); + exit (1); + } + if (list != NULL) { + for (begin = list; *list != NULL; ++list) + printf("database file: %s\n", *list); + free (begin); + } + + /* Get the list of log files. */ + if ((ret = dbenv->log_archive(dbenv, + &list, DB_ARCH_ABS | DB_ARCH_LOG)) != 0) { + dbenv->err(dbenv, ret, "DB_ENV->log_archive: DB_ARCH_LOG"); + exit (1); + } + if (list != NULL) { + for (begin = list; *list != NULL; ++list) + printf("log file: %s\n", *list); + free (begin); + } +} +``` diff --git a/docs-src/guides/programmer_reference/transapp_atomicity.md b/docs-src/guides/programmer_reference/transapp_atomicity.md new file mode 100644 index 000000000..f7af7ba48 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_atomicity.md @@ -0,0 +1,14 @@ +--- +title: "Atomicity" +api-name: "Atomicity" +source: docs/programmer_reference/transapp_atomicity.html +--- +## Atomicity + +The second reason listed for using transactions was *atomicity*. Atomicity means that multiple operations can be grouped into a single logical entity, that is, other threads of control accessing the database will either see all of the changes or none of the changes. Atomicity is important for applications wanting to update two related databases (for example, a primary database and secondary index) in a single logical action. Or, for an application wanting to update multiple records in one database in a single logical action. + +Any number of operations on any number of databases can be included in a single transaction to ensure the atomicity of the operations. There is, however, a trade-off between the number of operations included in a single transaction and both throughput and the possibility of deadlock. The reason for this is because transactions acquire locks throughout their lifetime and do not release the locks until commit or abort time. So, the more operations included in a transaction, the more likely it is that a transaction will block other operations and that deadlock will occur. However, each transaction commit requires a synchronous disk I/O, so grouping multiple operations into a transaction can increase overall throughput. (There is one exception to this: the DB_TXN_WRITE_NOSYNC and DB_TXN_NOSYNC flags cause transactions to exhibit the ACI (atomicity, consistency and isolation) properties, but not D (durability); avoiding the write and/or synchronous disk I/O on transaction commit greatly increases transaction throughput for some applications.) + +When applications do create complex transactions, they often avoid having more than one complex transaction at a time because simple operations like a single DB->put() are unlikely to deadlock with each other or the complex transaction; while multiple complex transactions are likely to deadlock with each other because they will both acquire many locks over their lifetime. Alternatively, complex transactions can be broken up into smaller sets of operations, and each of those sets may be encapsulated in a nested transaction. Because nested transactions may be individually aborted and retried without causing the entire transaction to be aborted, this allows complex transactions to proceed even in the face of heavy contention, repeatedly trying the suboperations until they succeed. + +It is also helpful to order operations within a transaction; that is, access the databases and items within the databases in the same order, to the extent possible, in all transactions. Accessing databases and items in different orders greatly increases the likelihood of operations being blocked and failing due to deadlocks. diff --git a/docs-src/guides/programmer_reference/transapp_checkpoint.md b/docs-src/guides/programmer_reference/transapp_checkpoint.md new file mode 100644 index 000000000..dfcab0070 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_checkpoint.md @@ -0,0 +1,95 @@ +--- +title: "Checkpoints" +api-name: "Checkpoints" +source: docs/programmer_reference/transapp_checkpoint.html +--- +## Checkpoints + +The second component of the infrastructure is performing checkpoints of the log files. Performing checkpoints is necessary for two reasons. + +First, you may be able to remove Berkeley DB log files from your database environment after a checkpoint. Change records are written into the log files when databases are modified, but the actual changes to the database are not necessarily written to disk. When a checkpoint is performed, changes to the database are written into the backing database file. Once the database pages are written, log files can be archived and removed from the database environment because they will never be needed for anything other than catastrophic failure. (Log files which are involved in active transactions may not be removed, and there must always be at least one log file in the database environment.) + +The second reason to perform checkpoints is because checkpoint frequency is inversely proportional to the amount of time it takes to run database recovery after a system or application failure. This is because recovery after failure has to redo or undo changes only since the last checkpoint, as changes before the checkpoint have all been flushed to the databases. + +Berkeley DB provides the db_checkpoint utility, which can be used to perform checkpoints. Alternatively, applications can write their own checkpoint thread using the underlying DB_ENV->txn_checkpoint() function. The following code fragment checkpoints the database environment every 60 seconds: + +``` c +int +main(int argc, char *argv) +{ + extern int optind; + DB *db_cats, *db_color, *db_fruit; + DB_ENV *dbenv; + pthread_t ptid; + int ch; + + while ((ch = getopt(argc, argv, "")) != EOF) + switch (ch) { + case '?': + default: + usage(); + } + argc -= optind; + argv += optind; + + env_dir_create(); + env_open(&dbenv); + + /* Start a checkpoint thread. */ + if ((errno = pthread_create( + &ptid, NULL, checkpoint_thread, (void *)dbenv)) != 0) { + fprintf(stderr, + "txnapp: failed spawning checkpoint thread: %s\n", + strerror(errno)); + exit (1); + } + + /* Open database: Key is fruit class; Data is specific type. */ + db_open(dbenv, &db_fruit, "fruit", 0); + + /* Open database: Key is a color; Data is an integer. */ + db_open(dbenv, &db_color, "color", 0); + + /* + * Open database: + * Key is a name; Data is: company name, cat breeds. + */ + db_open(dbenv, &db_cats, "cats", 1); + + add_fruit(dbenv, db_fruit, "apple", "yellow delicious"); + + add_color(dbenv, db_color, "blue", 0); + add_color(dbenv, db_color, "blue", 3); + + add_cat(dbenv, db_cats, + "Amy Adams", + "Oracle", + "abyssinian", + "bengal", + "chartreaux", + NULL); + + return (0); +} + +void * +checkpoint_thread(void *arg) +{ + DB_ENV *dbenv; + int ret; + + dbenv = arg; + dbenv->errx(dbenv, "Checkpoint thread: %lu", (u_long)pthread_self()); + + /* Checkpoint once a minute. */ + for (;; sleep(60)) + if ((ret = dbenv->txn_checkpoint(dbenv, 0, 0, 0)) != 0) { + dbenv->err(dbenv, ret, "checkpoint thread"); + exit (1); + } + + /* NOTREACHED */ +} +``` + +Because checkpoints can be quite expensive, choosing how often to perform a checkpoint is a common tuning parameter for Berkeley DB applications. diff --git a/docs-src/guides/programmer_reference/transapp_cursor.md b/docs-src/guides/programmer_reference/transapp_cursor.md new file mode 100644 index 000000000..dfac77c68 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_cursor.md @@ -0,0 +1,148 @@ +--- +title: "Transactional cursors" +api-name: "Transactional cursors" +source: docs/programmer_reference/transapp_cursor.html +--- +## Transactional cursors + +Berkeley DB cursors may be used inside a transaction, exactly as any other DB method. The enclosing transaction ID must be specified when the cursor is created, but it does not then need to be further specified on operations performed using the cursor. One important point to remember is that a cursor **must be closed** before the enclosing transaction is committed or aborted. + +The following code fragment uses a cursor to store a new key in the cats database with four associated data items. The key is a name. The data items are a company name and a list of the breeds of cat owned. Each of the data entries is stored as a duplicate data item. In this example, transactions are necessary to ensure that either all or none of the data items appear in case of system or application failure. + +``` c +int +main(int argc, char *argv) +{ + extern int optind; + DB *db_cats, *db_color, *db_fruit; + DB_ENV *dbenv; + int ch; + + while ((ch = getopt(argc, argv, "")) != EOF) + switch (ch) { + case '?': + default: + usage(); + } + argc -= optind; + argv += optind; + + env_dir_create(); + env_open(&dbenv); + + /* Open database: Key is fruit class; Data is specific type. */ + db_open(dbenv, &db_fruit, "fruit", 0); + + /* Open database: Key is a color; Data is an integer. */ + db_open(dbenv, &db_color, "color", 0); + + /* + * Open database: + * Key is a name; Data is: company name, cat breeds. + */ + db_open(dbenv, &db_cats, "cats", 1); + + add_fruit(dbenv, db_fruit, "apple", "yellow delicious"); + + add_color(dbenv, db_color, "blue", 0); + add_color(dbenv, db_color, "blue", 3); + + add_cat(dbenv, db_cats, + "Amy Adams", + "Oracle", + "abyssinian", + "bengal", + "chartreaux", + NULL); + + return (0); +} + +int +add_cat(DB_ENV *dbenv, DB *db, char *name, ...) +{ + va_list ap; + DBC *dbc; + DBT key, data; + DB_TXN *tid; + int fail, ret, t_ret; + char *s; + + /* Initialization. */ + fail = 0; + + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + key.data = name; + key.size = strlen(name); + +retry: /* Begin the transaction. */ + if ((ret = dbenv->txn_begin(dbenv, NULL, &tid, 0)) != 0) { + dbenv->err(dbenv, ret, "DB_ENV->txn_begin"); + exit (1); + } + + /* Delete any previously existing item. */ + switch (ret = db->del(db, tid, &key, 0)) { + case 0: + case DB_NOTFOUND: + break; + case DB_LOCK_DEADLOCK: + default: + /* Retry the operation. */ + if ((t_ret = tid->abort(tid)) != 0) { + dbenv->err(dbenv, t_ret, "DB_TXN->abort"); + exit (1); + } + if (fail++ == MAXIMUM_RETRY) + return (ret); + goto retry; + } + + /* Create a cursor. */ + if ((ret = db->cursor(db, tid, &dbc, 0)) != 0) { + dbenv->err(dbenv, ret, "db->cursor"); + exit (1); + } + + /* Append the items, in order. */ + va_start(ap, name); + while ((s = va_arg(ap, char *)) != NULL) { + data.data = s; + data.size = strlen(s); + switch (ret = dbc->put(dbc, &key, &data, DB_KEYLAST)) { + case 0: + break; + case DB_LOCK_DEADLOCK: + default: + va_end(ap); + + /* Retry the operation. */ + if ((t_ret = dbc->close(dbc)) != 0) { + dbenv->err( + dbenv, t_ret, "dbc->close"); + exit (1); + } + if ((t_ret = tid->abort(tid)) != 0) { + dbenv->err(dbenv, t_ret, "DB_TXN->abort"); + exit (1); + } + if (fail++ == MAXIMUM_RETRY) + return (ret); + goto retry; + } + } + va_end(ap); + + /* Success: commit the change. */ + if ((ret = dbc->close(dbc)) != 0) { + dbenv->err(dbenv, ret, "dbc->close"); + exit (1); + } + if ((ret = tid->commit(tid, 0)) != 0) { + dbenv->err(dbenv, ret, "DB_TXN->commit"); + exit (1); + } + return (0); +} +``` diff --git a/docs-src/guides/programmer_reference/transapp_data_open.md b/docs-src/guides/programmer_reference/transapp_data_open.md new file mode 100644 index 000000000..726a5b74a --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_data_open.md @@ -0,0 +1,116 @@ +--- +title: "Opening the databases" +api-name: "Opening the databases" +source: docs/programmer_reference/transapp_data_open.html +--- +## Opening the databases + +Next, we open three databases ("color" and "fruit" and "cats"), in the database environment. Again, our DB database handles are declared to be free-threaded using the DB_THREAD flag, and so may be used by any number of threads we subsequently create. + +``` c +int +main(int argc, char *argv[]) +{ + extern int optind; + DB *db_cats, *db_color, *db_fruit; + DB_ENV *dbenv; + int ch; + + while ((ch = getopt(argc, argv, "")) != EOF) + switch (ch) { + case '?': + default: + usage(); + } + argc -= optind; + argv += optind; + + env_dir_create(); + env_open(&dbenv); + ... + + /* Open database: Key is fruit class; Data is specific type. */ + if (db_open(dbenv, &db_fruit, "fruit", 0)) + return (1); + + /* Open database: Key is a color; Data is an integer. */ + if (db_open(dbenv, &db_color, "color", 0)) + return (1); + + /* + * Open database: + * Key is a name; Data is: company name, cat breeds. + */ + if (db_open(dbenv, &db_cats, "cats", 1)) + return (1); + + ... + + return (0); +} + +int +db_open(DB_ENV *dbenv, DB **dbp, char *name, int dups) +{ + DB *db; + int ret; + + /* Create the database handle. */ + if ((ret = db_create(&db, dbenv, 0)) != 0) { + dbenv->err(dbenv, ret, "db_create"); + return (1); + } + + /* Optionally, turn on duplicate data items. */ + if (dups && (ret = db->set_flags(db, DB_DUP)) != 0) { + (void)db->close(db, 0); + dbenv->err(dbenv, ret, "db->set_flags: DB_DUP"); + return (1); + } + + /* + * Open a database in the environment: + * create if it doesn't exist + * free-threaded handle + * read/write owner only + */ + if ((ret = db->open(db, NULL, name, NULL, DB_BTREE, + DB_AUTO_COMMIT | DB_CREATE | DB_THREAD, S_IRUSR | S_IWUSR)) != 0) { + (void)db->close(db, 0); + dbenv->err(dbenv, ret, "db->open: %s", name); + return (1); + } + + *dbp = db; + return (0); +} +``` + +After opening the database, we can use the db_stat utility to display information about a database we have created: + +``` c +prompt> db_stat -h TXNAPP -d color +53162 Btree magic number. +8 Btree version number. +Flags: +2 Minimum keys per-page. +8192 Underlying database page size. +1 Number of levels in the tree. +0 Number of unique keys in the tree. +0 Number of data items in the tree. +0 Number of tree internal pages. +0 Number of bytes free in tree internal pages (0% ff). +1 Number of tree leaf pages. +8166 Number of bytes free in tree leaf pages (0.% ff). +0 Number of tree duplicate pages. +0 Number of bytes free in tree duplicate pages (0% ff). +0 Number of tree overflow pages. +0 Number of bytes free in tree overflow pages (0% ff). +0 Number of pages on the free list. +``` + +The database open must be enclosed within a transaction in order to be recoverable. The transaction will ensure that created files are re-created in recovered environments (or do not appear at all). Additional database operations or operations on other databases can be included in the same transaction, of course. In the simple case, where the open is the only operation in the transaction, an application can set the DB_AUTO_COMMIT flag instead of creating and managing its own transaction handle. The DB_AUTO_COMMIT flag will internally wrap the operation in a transaction, simplifying application code. + +The previous example is the simplest case of transaction protection for database open. Obviously, additional database operations can be done in the scope of the same transaction. For example, an application maintaining a list of the databases in a database environment in a well-known file might include an update of the list in the same transaction in which the database is created. Or, an application might create both a primary and secondary database in a single transaction. + +DB handles that will later be used for transactionally protected database operations must be opened within a transaction. Specifying a transaction handle to database operations using DB handles not opened within a transaction will return an error. Similarly, not specifying a transaction handle to database operations that will modify the database, using handles that were opened within a transaction, will also return an error. diff --git a/docs-src/guides/programmer_reference/transapp_deadlock.md b/docs-src/guides/programmer_reference/transapp_deadlock.md new file mode 100644 index 000000000..866bd0bbc --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_deadlock.md @@ -0,0 +1,57 @@ +--- +title: "Deadlock detection" +api-name: "Deadlock detection" +source: docs/programmer_reference/transapp_deadlock.html +--- +## Deadlock detection + +The first component of the infrastructure, *deadlock detection*, is not so much a requirement specific to transaction-protected applications, but instead is necessary for almost all applications in which more than a single thread of control will be accessing the database at one time. Even when Berkeley DB automatically handles database locking, it is normally possible for deadlock to occur. Because the underlying database access methods may update multiple pages during a single Berkeley DB API call, deadlock is possible even when threads of control are making only single update calls into the database. The exception to this rule is when all the threads of control accessing the database are read-only or when the Berkeley DB Concurrent Data Store product is used; the Berkeley DB Concurrent Data Store product guarantees deadlock-free operation at the expense of reduced concurrency. + +When the deadlock occurs, two (or more) threads of control each request additional locks that can never be granted because one of the threads of control waiting holds the requested resource. For example, consider two processes: A and B. Let's say that A obtains a write lock on item X, and B obtains a write lock on item Y. Then, A requests a lock on Y, and B requests a lock on X. A will wait until resource Y becomes available and B will wait until resource X becomes available. Unfortunately, because both A and B are waiting, neither will release the locks they hold and neither will ever obtain the resource on which it is waiting. For another example, consider two transactions, A and B, each of which may want to modify item X. Assume that transaction A obtains a read lock on X and confirms that a modification is needed. Then it is descheduled and the thread containing transaction B runs. At that time, transaction B obtains a read lock on X and confirms that it also wants to make a modification. Both transactions A and B will block when they attempt to upgrade their read locks to write locks because the other already holds a read lock. This is a deadlock. Transaction A cannot make forward progress until Transaction B releases its read lock on X, but Transaction B cannot make forward progress until Transaction A releases its read lock on X. + +In order to detect that deadlock has happened, a separate process or thread must review the locks currently held in the database. If deadlock has occurred, a victim must be selected, and that victim will then return the error DB_LOCK_DEADLOCK from whatever Berkeley DB call it was making. Berkeley DB provides the db_deadlock utility that can be used to perform this deadlock detection. Alternatively, applications can create their own deadlock utility or thread using the underlying DB_ENV->lock_detect() function, or specify that Berkeley DB run the deadlock detector internally whenever there is a conflict over a lock (see DB_ENV->set_lk_detect() for more information). The following code fragment does the latter: + +``` c +void +env_open(DB_ENV **dbenvp) +{ + DB_ENV *dbenv; + int ret; + + /* Create the environment handle. */ + if ((ret = db_env_create(&dbenv, 0)) != 0) { + fprintf(stderr, + "txnapp: db_env_create: %s\n", db_strerror(ret)); + exit (1); + } + + /* Set up error handling. */ + dbenv->set_errpfx(dbenv, "txnapp"); + dbenv->set_errfile(dbenv, stderr); + + /* Do deadlock detection internally. */ + if ((ret = dbenv->set_lk_detect(dbenv, DB_LOCK_DEFAULT)) != 0) { + dbenv->err(dbenv, ret, "set_lk_detect: DB_LOCK_DEFAULT"); + exit (1); + } + + /* + * Open a transactional environment: + * create if it doesn't exist + * free-threaded handle + * run recovery + * read/write owner only + */ + if ((ret = dbenv->open(dbenv, ENV_DIRECTORY, + DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | + DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER | DB_THREAD, + S_IRUSR | S_IWUSR)) != 0) { + dbenv->err(dbenv, ret, "dbenv->open: %s", ENV_DIRECTORY); + exit (1); + } + + *dbenvp = dbenv; +} +``` + +Deciding how often to run the deadlock detector and which of the deadlocked transactions will be forced to abort when the deadlock is detected is a common tuning parameter for Berkeley DB applications. diff --git a/docs-src/guides/programmer_reference/transapp_env_open.md b/docs-src/guides/programmer_reference/transapp_env_open.md new file mode 100644 index 000000000..ba3c1e4d7 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_env_open.md @@ -0,0 +1,152 @@ +--- +title: "Opening the environment" +api-name: "Opening the environment" +source: docs/programmer_reference/transapp_env_open.html +--- +## Opening the environment + +Creating transaction-protected applications using the Berkeley DB library is quite easy. Applications first use DB_ENV->open() to initialize the database environment. Transaction-protected applications normally require all four Berkeley DB subsystems, so the DB_INIT_MPOOL, DB_INIT_LOCK, DB_INIT_LOG, and DB_INIT_TXN flags should be specified. + +Once the application has called DB_ENV->open(), it opens its databases within the environment. Once the databases are opened, the application makes changes to the databases inside of transactions. Each set of changes that entails a unit of work should be surrounded by the appropriate DB_ENV->txn_begin(), DB_TXN->commit() and DB_TXN->abort() calls. The Berkeley DB access methods will make the appropriate calls into the Lock, Log and Memory Pool subsystems in order to guarantee transaction semantics. When the application is ready to exit, all outstanding transactions should have been committed or aborted. + +Databases accessed by a transaction must not be closed during the transaction. Once all outstanding transactions are finished, all open Berkeley DB files should be closed. When the Berkeley DB database files have been closed, the environment should be closed by calling DB_ENV->close(). + +The following code fragment creates the database environment directory then opens the environment, running recovery. Our DB_ENV database environment handle is declared to be free-threaded using the DB_THREAD flag, and so may be used by any number of threads that we may subsequently create. + +``` c +#include +#include + +#include +#include +#include +#include +#include + +#include + +#define ENV_DIRECTORY "TXNAPP" + +void env_dir_create(void); +void env_open(DB_ENV **); +... + +int +main(int argc, char *argv[]) +{ + extern int optind; + DB_ENV *dbenv; + int ch; + + while ((ch = getopt(argc, argv, "")) != EOF) + switch (ch) { + case '?': + default: + usage(); + } + argc -= optind; + argv += optind; + + env_dir_create(); + env_open(&dbenv); + ... + + return (0); +} + +... + +void +env_dir_create() +{ + struct stat sb; + + /* + * If the directory exists, we're done. We do not further check + * the type of the file, DB will fail appropriately if it's the + * wrong type. + */ + if (stat(ENV_DIRECTORY, &sb) == 0) + return; + + /* Create the directory, read/write/access owner only. */ + if (mkdir(ENV_DIRECTORY, S_IRWXU) != 0) { + fprintf(stderr, + "txnapp: mkdir: %s: %s\n", ENV_DIRECTORY, strerror(errno)); + exit (1); + } +} + +void +env_open(DB_ENV **dbenvp) +{ + DB_ENV *dbenv; + int ret; + + /* Create the environment handle. */ + if ((ret = db_env_create(&dbenv, 0)) != 0) { + fprintf(stderr, + "txnapp: db_env_create: %s\n", db_strerror(ret)); + exit (1); + } + + /* Set up error handling. */ + dbenv->set_errpfx(dbenv, "txnapp"); + dbenv->set_errfile(dbenv, stderr); + + /* + * Open a transactional environment: + * create if it doesn't exist + * free-threaded handle + * run recovery + * read/write owner only + */ + if ((ret = dbenv->open(dbenv, ENV_DIRECTORY, + DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | + DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER | DB_THREAD, + S_IRUSR | S_IWUSR)) != 0) { + (void)dbenv->close(dbenv, 0); + fprintf(stderr, "dbenv->open: %s: %s\n", + ENV_DIRECTORY, db_strerror(ret)); + exit (1); + } + + *dbenvp = dbenv; +} +``` + +After running this initial program, we can use the db_stat utility to display the contents of the environment directory: + +``` c +prompt> db_stat -e -h TXNAPP +3.2.1 Environment version. +120897 Magic number. +0 Panic value. +1 References. +6 Locks granted without waiting. +0 Locks granted after waiting. +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= +Mpool Region: 4. +264KB Size (270336 bytes). +-1 Segment ID. +1 Locks granted without waiting. +0 Locks granted after waiting. +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= +Log Region: 3. +96KB Size (98304 bytes). +-1 Segment ID. +3 Locks granted without waiting. +0 Locks granted after waiting. +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= +Lock Region: 2. +240KB Size (245760 bytes). +-1 Segment ID. +1 Locks granted without waiting. +0 Locks granted after waiting. +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= +Txn Region: 5. +8KB Size (8192 bytes). +-1 Segment ID. +1 Locks granted without waiting. +0 Locks granted after waiting. +``` diff --git a/docs-src/guides/programmer_reference/transapp_fail.md b/docs-src/guides/programmer_reference/transapp_fail.md new file mode 100644 index 000000000..2c116b6b2 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_fail.md @@ -0,0 +1,24 @@ +--- +title: "Handling failure in Transactional Data Store applications" +api-name: "Handling failure in Transactional Data Store applications" +source: docs/programmer_reference/transapp_fail.html +--- +## Handling failure in Transactional Data Store applications + +When building Transactional Data Store applications, there are design issues to consider whenever a thread of control with open Berkeley DB handles fails for any reason (where a thread of control may be either a true thread or a process). + +The first case is handling system failure: if the system fails, the database environment and the databases may be left in a corrupted state. In this case, recovery must be performed on the database environment before any further action is taken, in order to: + +- recover the database environment resources, +- release any locks or mutexes that may have been held to avoid starvation as the remaining threads of control convoy behind the held locks, and +- resolve any partially completed operations that may have left a database in an inconsistent or corrupted state. + +For details on performing recovery, see the Recovery procedures. + +The second case is handling the failure of a thread of control. There are resources maintained in database environments that may be left locked or corrupted if a thread of control exits unexpectedly. These resources include data structure mutexes, logical database locks and unresolved transactions (that is, transactions which were never aborted or committed). While Transactional Data Store applications can treat the failure of a thread of control in the same way as they do a system failure, they have an alternative choice, the DB_ENV->failchk() method. + +The DB_ENV->failchk() will return DB_RUNRECOVERY if the database environment is unusable as a result of the thread of control failure. (If a data structure mutex or a database write lock is left held by thread of control failure, the application should not continue to use the database environment, as subsequent use of the environment is likely to result in threads of control convoying behind the held locks.) The DB_ENV->failchk() call will release any database read locks that have been left held by the exit of a thread of control, and abort any unresolved transactions. In this case, the application can continue to use the database environment. + +A Transactional Data Store application recovering from a thread of control failure should call DB_ENV->failchk(), and, if it returns success, the application can continue. If DB_ENV->failchk() returns DB_RUNRECOVERY, the application should proceed as described for the case of system failure. + +It greatly simplifies matters that recovery may be performed regardless of whether recovery needs to be performed; that is, it is not an error to recover a database environment for which recovery is not strictly necessary. For this reason, applications should not try to determine if the database environment was active when the application or system failed. Instead, applications should run recovery any time the DB_ENV->failchk() method returns DB_RUNRECOVERY, or, if the application is not calling the DB_ENV->failchk() method, any time any thread of control accessing the database environment fails, as well as any time the system reboots. diff --git a/docs-src/guides/programmer_reference/transapp_faq.md b/docs-src/guides/programmer_reference/transapp_faq.md new file mode 100644 index 000000000..08a6643bc --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_faq.md @@ -0,0 +1,34 @@ +--- +title: "Transaction FAQ" +api-name: "Transaction FAQ" +source: docs/programmer_reference/transapp_faq.html +--- +## Transaction FAQ + +1. **What should a transactional program do when an error occurs?** + + Any time an error occurs, such that a transactionally protected set of operations cannot complete successfully, the transaction must be aborted. While deadlock is by far the most common of these errors, there are other possibilities; for example, running out of disk space for the filesystem. In Berkeley DB transactional applications, there are three classes of error returns: "expected" errors, "unexpected but recoverable" errors, and a single "unrecoverable" error. Expected errors are errors like DB_NOTFOUND, which indicates that a searched-for key item is not present in the database. Applications may want to explicitly test for and handle this error, or, in the case where the absence of a key implies the enclosing transaction should fail, simply call DB_TXN->abort(). Unexpected but recoverable errors are errors like DB_LOCK_DEADLOCK, which indicates that an operation has been selected to resolve a deadlock, or a system error such as EIO, which likely indicates that the filesystem has no available disk space. Applications must immediately call DB_TXN->abort() when these returns occur, as it is not possible to proceed otherwise. The only unrecoverable error is DB_RUNRECOVERY, which indicates that the system must stop and recovery must be run. + +2. **How can hot backups work? Can't you get an inconsistent picture of the database when you copy it?** + + First, Berkeley DB is based on the technique of "write-ahead logging", which means that before any change is made to a database, a log record is written that describes the change. Further, Berkeley DB guarantees that the log record that describes the change will always be written to stable storage (that is, disk) before the database page where the change was made is written to stable storage. Because of this guarantee, we know that any change made to a database will appear either in just a log file, or both the database and a log file, but never in just the database. + + Second, you can always create a consistent and correct database based on the log files and the databases from a database environment. So, during a hot backup, we first make a copy of the databases and then a copy of the log files. The tricky part is that there may be pages in the database that are related for which we won't get a consistent picture during this copy. For example, let's say that we copy pages 1-4 of the database, and then are swapped out. For whatever reason (perhaps because we needed to flush pages from the cache, or because of a checkpoint), the database pages 1 and 5 are written. Then, the hot backup process is re-scheduled, and it copies page 5. Obviously, we have an inconsistent database snapshot, because we have a copy of page 1 from before it was written by the other thread of control, and a copy of page 5 after it was written by the other thread. What makes this work is the order of operations in a hot backup. Because of the write-ahead logging guarantees, we know that any page written to the database will first be referenced in the log. If we copy the database first, then we can also know that any inconsistency in the database will be described in the log files, and so we know that we can fix everything up during recovery. + +3. **My application has DB_LOCK_DEADLOCK errors. Is the normal, and what should I do?** + + It is quite rare for a transactional application to be deadlock free. All applications should be prepared to handle deadlock returns, because even if the application is deadlock free when deployed, future changes to the application or the Berkeley DB implementation might introduce deadlocks. + + Practices which reduce the chance of deadlock include: + + - Not using cursors which move backwards through the database (DB_PREV), as backward scanning cursors can deadlock with page splits; + - Configuring DB_REVSPLITOFF to turn off reverse splits in applications which repeatedly delete and re-insert the same keys, to minimize the number of page splits as keys are re-inserted; + - Not configuring DB_READ_UNCOMMITTED as that flag requires write transactions upgrade their locks when aborted, which can lead to deadlock. Generally, DB_READ_COMMITTED or non-transactional read operations are less prone to deadlock than DB_READ_UNCOMMITTED. + +4. **How can I move a database from one transactional environment into another?** + + Because database pages contain references to log records, databases cannot be simply moved into different database environments. To move a database into a different environment, dump and reload the database before moving it. If the database is too large to dump and reload, the database may be prepared in place using the DB_ENV->lsn_reset() method or the **-r** argument to the db_load utility. + +5. **I'm seeing the error "log_flush: LSN past current end-of-log", what does that mean?** + + The most common cause of this error is that a system administrator has removed all of the log files from a database environment. You should shut down your database environment as gracefully as possible, first flushing the database environment cache to disk, if that's possible. Then, dump and reload your databases. If the database is too large to dump and reload, the database may be reset in place using the DB_ENV->lsn_reset() method or the **-r** argument to the db_load utility. However, if you reset the database in place, you should verify your databases before using them again. (It is possible for the databases to be corrupted by running after all of the log files have been removed, and the longer the application runs, the worse it can get.) diff --git a/docs-src/guides/programmer_reference/transapp_filesys.md b/docs-src/guides/programmer_reference/transapp_filesys.md new file mode 100644 index 000000000..0fd94dc2a --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_filesys.md @@ -0,0 +1,26 @@ +--- +title: "Recovery and filesystem operations" +api-name: "Recovery and filesystem operations" +source: docs/programmer_reference/transapp_filesys.html +--- +## Recovery and filesystem operations + +The Berkeley DB API supports creating, removing and renaming files. Creating files is supported by the DB->open() method. Removing files is supported by the DB_ENV->dbremove() and DB->remove() methods. Renaming files is supported by the DB_ENV->dbrename() and DB->rename() methods. (There are two methods for removing and renaming files because one of the methods is transactionally protected and one is not.) + +Berkeley DB does not permit specifying the DB_TRUNCATE flag when opening a file in a transaction-protected environment. This is an implicit file deletion, but one that does not always require the same operating system file permissions as deleting and creating a file do. + +If you have changed the name of a file or deleted it outside of the Berkeley DB library (for example, you explicitly removed a file using your normal operating system utilities), then it is possible that recovery will not be able to find a database to which the log refers. In this case, the db_recover utility will produce a warning message, saying it was unable to locate a file it expected to find. This message is only a warning because the file may have been subsequently deleted as part of normal database operations before the failure occurred, so is not necessarily a problem. + +Generally, any filesystem operations that are performed outside the Berkeley DB interface should be performed at the same time as making a snapshot of the database. To perform filesystem operations correctly, do the following: + +1. Cleanly shut down database operations. + + To shut down database operations cleanly, all applications accessing the database environment must be shut down and a transaction checkpoint must be taken. If the applications are not implemented so they can be shut down gracefully (that is, closing all references to the database environment), recovery must be performed after all applications have been killed to ensure that the underlying databases are consistent on disk. + +2. Perform the filesystem operations; for example, remove or rename one or more files. + +3. Make an archival snapshot of the database. + + Although this step is not strictly necessary, it is strongly recommended. If this step is not performed, recovery from catastrophic failure will require that recovery first be performed up to the time of the filesystem operations, the filesystem operations be redone, and then recovery be performed from the filesystem operations forward. + +4. Restart the database applications. diff --git a/docs-src/guides/programmer_reference/transapp_hotfail.md b/docs-src/guides/programmer_reference/transapp_hotfail.md new file mode 100644 index 000000000..1868f44c1 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_hotfail.md @@ -0,0 +1,42 @@ +--- +title: "Hot failover" +api-name: "Hot failover" +source: docs/programmer_reference/transapp_hotfail.html +--- +## Hot failover + +For some applications, it may be useful to periodically snapshot the database environment for use as a hot failover should the primary system fail. The following steps can be taken to keep a backup environment in close synchrony with an active environment. The active environment is entirely unaffected by these procedures, and both read and write operations are allowed during all steps described here. + +The procedure described here is not compatible with the concurrent use of the transactional bulk insert optimization (transactions started with the DB_TXN_BULK flag). After the bulk optimization is used, the archive must be created again from scratch starting with step 1. + +The db_hotbackup utility is the preferred way to automate generating a hot failover system. The first step is to run db_hotbackup utility without the **-u** flag. This will create hot backup copy of the databases in your environment. After that point periodically running the db_hotbackup utility with the **-u** flag will copy the new log files and run recovery on the backup copy to bring it current with the primary environment. + +Note that you can also create your own hot backup solution using the DB_ENV->backup() or DB_ENV->dbbackup() methods. + +To implement your own hot fail over system, the steps below can be followed. However, care should be taken on non-UNIX based systems when copying the database files to be sure that they are either quiescent, or that either the DB_ENV->backup() or db_copy() routine is used to ensure atomic reads of the database pages. + +1. Run the db_archive utility with the **-s** option in the active environment to identify all of the active environment's database files, and copy them to the backup directory. + + If the database files are stored in a separate directory from the other Berkeley DB files, it will be simpler (and much faster!) to copy the directory itself instead of the individual files (see DB_ENV->add_data_dir() for additional information). + + ### Note + + If any of the database files did not have an open DB handle during the lifetime of the current log files, the db_archive utility will not list them in its output. This is another reason it may be simpler to use a separate database file directory and copy the entire directory instead of archiving only the files listed by the db_archive utility. + +2. Remove all existing log files from the backup directory. + +3. Run the db_archive utility with the **-l** option in the active environment to identify all of the active environment's log files, and copy them to the backup directory. + +4. Run the db_recover utility with the **-c** option in the backup directory to catastrophically recover the copied environment. + +Steps 2, 3 and 4 may be repeated as often as you like. If Step 1 (the initial copy of the database files) is repeated, then Steps 2, 3 and 4 **must** be performed at least once in order to ensure a consistent database environment snapshot. + +These procedures must be integrated with your other archival procedures, of course. If you are periodically removing log files from your active environment, you must be sure to copy them to the backup directory before removing them from the active directory. Not copying a log file to the backup directory and subsequently running recovery with it present may leave the backup snapshot of the environment corrupted. A simple way to ensure this never happens is to archive the log files in Step 2 as you remove them from the backup directory, and move inactive log files from your active environment into your backup directory (rather than copying them), in Step 3. The following steps describe this procedure in more detail: + +1. Run the db_archive utility with the **-s** option in the active environment to identify all of the active environment's database files, and copy them to the backup directory. +2. Archive all existing log files from the backup directory, moving them to a backup device such as CD-ROM, alternate disk, or tape. +3. Run the db_archive utility (without any option) in the active environment to identify all of the log files in the active environment that are no longer in use, and **move** them to the backup directory. +4. Run the db_archive utility with the **-l** option in the active environment to identify all of the remaining log files in the active environment, and **copy** the log files to the backup directory. +5. Run the db_recover utility with the **-c** option in the backup directory to catastrophically recover the copied environment. + +As before, steps 2, 3, 4 and 5 may be repeated as often as you like. If Step 1 (the initial copy of the database files) is repeated, then Steps 2 through 5 **must** be performed at least once in order to ensure a consistent database environment snapshot. diff --git a/docs-src/guides/programmer_reference/transapp_inc.md b/docs-src/guides/programmer_reference/transapp_inc.md new file mode 100644 index 000000000..4c080e5fa --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_inc.md @@ -0,0 +1,140 @@ +--- +title: "Isolation" +api-name: "Isolation" +source: docs/programmer_reference/transapp_inc.html +--- +## Isolation + +The third reason listed for using transactions was *isolation*. Consider an application suite in which multiple threads of control (multiple processes or threads in one or more processes) are changing the values associated with a key in one or more databases. Specifically, they are taking the current value, incrementing it, and then storing it back into the database. + +Such an application requires isolation. Because we want to change a value in the database, we must make sure that after we read it, no other thread of control modifies it. For example, assume that both thread \#1 and thread \#2 are doing similar operations in the database, where thread \#1 is incrementing records by 3, and thread \#2 is incrementing records by 5. We want to increment the record by a total of 8. If the operations interleave in the right (well, wrong) order, that is not what will happen: + +``` c +thread #1 read record: the value is 2 +thread #2 read record: the value is 2 +thread #2 write record + 5 back into the database (new value 7) +thread #1 write record + 3 back into the database (new value 5) +``` + +As you can see, instead of incrementing the record by a total of 8, we've incremented it only by 3 because thread \#1 overwrote thread \#2's change. By wrapping the operations in transactions, we ensure that this cannot happen. In a transaction, when the first thread reads the record, locks are acquired that will not be released until the transaction finishes, guaranteeing that all writers will block, waiting for the first thread's transaction to complete (or to be aborted). + +Here is an example function that does transaction-protected increments on database records to ensure isolation: + +``` c +int +main(int argc, char *argv) +{ + extern int optind; + DB *db_cats, *db_color, *db_fruit; + DB_ENV *dbenv; + int ch; + + while ((ch = getopt(argc, argv, "")) != EOF) + switch (ch) { + case '?': + default: + usage(); + } + argc -= optind; + argv += optind; + + env_dir_create(); + env_open(&dbenv); + + /* Open database: Key is fruit class; Data is specific type. */ + db_open(dbenv, &db_fruit, "fruit", 0); + + /* Open database: Key is a color; Data is an integer. */ + db_open(dbenv, &db_color, "color", 0); + + /* + * Open database: + * Key is a name; Data is: company name, cat breeds. + */ + db_open(dbenv, &db_cats, "cats", 1); + + add_fruit(dbenv, db_fruit, "apple", "yellow delicious"); + + add_color(dbenv, db_color, "blue", 0); + add_color(dbenv, db_color, "blue", 3); + + return (0); +} + +int +add_color(DB_ENV *dbenv, DB *dbp, char *color, int increment) +{ + DBT key, data; + DB_TXN *tid; + int fail, original, ret, t_ret; + char buf64; + + /* Initialization. */ + memset(&key, 0, sizeof(key)); + key.data = color; + key.size = strlen(color); + memset(&data, 0, sizeof(data)); + data.flags = DB_DBT_MALLOC; + + for (fail = 0;;) { + /* Begin the transaction. */ + if ((ret = dbenv->txn_begin(dbenv, NULL, &tid, 0)) != 0) { + dbenv->err(dbenv, ret, "DB_ENV->txn_begin"); + exit (1); + } + + /* + * Get the key. If it exists, we increment the value. If it + * doesn't exist, we create it. + */ + switch (ret = dbp->get(dbp, tid, &key, &data, DB_RMW)) { + case 0: + original = atoi(data.data); + break; + case DB_LOCK_DEADLOCK: + default: + /* Retry the operation. */ + if ((t_ret = tid->abort(tid)) != 0) { + dbenv->err(dbenv, t_ret, "DB_TXN->abort"); + exit (1); + } + if (fail++ == MAXIMUM_RETRY) + return (ret); + continue; + case DB_NOTFOUND: + original = 0; + break; + } + if (data.data != NULL) + free(data.data); + + /* Create the new data item. */ + (void)snprintf(buf, sizeof(buf), "%d", original + increment); + data.data = buf; + data.size = strlen(buf) + 1; + + /* Store the new value. */ + switch (ret = dbp->put(dbp, tid, &key, &data, 0)) { + case 0: + /* Success: commit the change. */ + if ((ret = tid->commit(tid, 0)) != 0) { + dbenv->err(dbenv, ret, "DB_TXN->commit"); + exit (1); + } + return (0); + case DB_LOCK_DEADLOCK: + default: + /* Retry the operation. */ + if ((t_ret = tid->abort(tid)) != 0) { + dbenv->err(dbenv, t_ret, "DB_TXN->abort"); + exit (1); + } + if (fail++ == MAXIMUM_RETRY) + return (ret); + break; + } + } +} +``` + +The DB_RMW flag in the DB->get() call specifies a write lock should be acquired on the key/data pair, instead of the more obvious read lock. We do this because the application expects to write the key/data pair in a subsequent operation, and the transaction is much more likely to deadlock if we first obtain a read lock and subsequently a write lock, than if we obtain the write lock initially. diff --git a/docs-src/guides/programmer_reference/transapp_journal.md b/docs-src/guides/programmer_reference/transapp_journal.md new file mode 100644 index 000000000..110818a8a --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_journal.md @@ -0,0 +1,34 @@ +--- +title: "Using Recovery on Journaling Filesystems" +api-name: "Using Recovery on Journaling Filesystems" +source: docs/programmer_reference/transapp_journal.html +--- +## Using Recovery on Journaling Filesystems + +In some cases, the use of meta-data only journaling file systems can lead to log file corruption. The window of vulnerability is quite small, but if the operating system experiences an unclean shutdown while Berkeley DB is creating a new log file, it is possible that upon file system recovery, the system will be in a state where the log file has been created, but its own meta-data has not. + +When a log file is corrupted to this degree, normal recovery can fail and your application may be unable to open your environment. Instead, an error something like this is issued when you attempt to run normal recovery on environment open: + +``` c + Ignoring log file: /var/dblog/log.0000000074: magic number + 6c73732f, not 40988 + Invalid log file: log.0000000074: Invalid argument + PANIC: Invalid argument + process-private: unable to find environment + txn_checkpoint interface requires an environment configured for + the transaction subsystem +``` + +In this case, it may be possible to successfully recover the environment by ignoring the log file that was being created — to do this, rename the log file with the highest number to a temporary name: + +``` c + mv DBHOME/log.000000XXX my-temporary-log-file +``` + +and try running normal environment recovery again. If recovery is successful, and your application is able to open the environment, then you can delete the log file that you renamed. + +If recovery is not successful, then you must perform a catastrophic recovery from a previous backup. + +This situation has been shown to occur when using ext3 in writeback mode, but other journaling filesystems could exhibit similar behavior. + +To be absolutely certain of your application's ability to recover your environment in the event of a system crash, either use non-journaling filesystems, or use a journaling filesystem in a safe (albeit slower) configuration, such as ext3 in ordered mode. diff --git a/docs-src/guides/programmer_reference/transapp_logfile.md b/docs-src/guides/programmer_reference/transapp_logfile.md new file mode 100644 index 000000000..f5677e2fa --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_logfile.md @@ -0,0 +1,24 @@ +--- +title: "Log file removal" +api-name: "Log file removal" +source: docs/programmer_reference/transapp_logfile.html +--- +## Log file removal + +The fourth component of the infrastructure, log file removal, concerns the ongoing disk consumption of the database log files. Depending on the rate at which the application writes to the databases and the available disk space, the number of log files may increase quickly enough so that disk space will be a resource problem. For this reason, you will periodically want to remove log files in order to conserve disk space. This procedure is distinct from database and log file archival for catastrophic recovery, and you cannot remove the current log files simply because you have created a database snapshot or copied log files to archival media. + +Log files may be removed at any time, as long as: + +- the log file is not involved in an active transaction. +- a checkpoint has been written subsequent to the log file's creation. +- the log file is not the only log file in the environment. + +Additionally, when Replication Manager is running the log file is older than the most out of date active site in the replication group. + +If you are preparing for catastrophic failure, you will want to copy the log files to archival media before you remove them as described in Database and log file archival. + +If you are not preparing for catastrophic failure, any one of the following methods can be used to remove log files: + +1. Run the standalone db_archive utility with the **-d** option, to remove any log files that are no longer needed at the time the command is executed. +2. Call the DB_ENV->log_archive() method from the application, with the DB_ARCH_REMOVE flag, to remove any log files that are no longer needed at the time the call is made. +3. Call the DB_ENV->log_set_config() method from the application, with the DB_LOG_AUTO_REMOVE flag, to remove any log files that are no longer needed on an ongoing basis. With this configuration, Berkeley DB will automatically remove log files, and the application will not have an opportunity to copy the log files to backup media. diff --git a/docs-src/guides/programmer_reference/transapp_nested.md b/docs-src/guides/programmer_reference/transapp_nested.md new file mode 100644 index 000000000..e9a98ed37 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_nested.md @@ -0,0 +1,18 @@ +--- +title: "Nested transactions" +api-name: "Nested transactions" +source: docs/programmer_reference/transapp_nested.html +--- +## Nested transactions + +Berkeley DB provides support for nested transactions. Nested transactions allow an application to decompose a large or long-running transaction into smaller units that may be independently aborted. + +Normally, when beginning a transaction, the application will pass a NULL value for the parent argument to DB_ENV->txn_begin(). If, however, the parent argument is a TXN handle, the newly created transaction will be treated as a nested transaction within the parent. Transactions may nest arbitrarily deeply. For the purposes of this discussion, transactions created with a parent identifier will be called *child transactions*. + +Once a transaction becomes a parent, as long as any of its child transactions are unresolved (that is, they have neither committed nor aborted), the parent may not issue any Berkeley DB calls except to begin more child transactions, or to commit or abort. For example, it may not issue any access method or cursor calls. After all of a parent's children have committed or aborted, the parent may again request operations on its own behalf. + +The semantics of nested transactions are as follows. When a child transaction is begun, it inherits all the locks of its parent. This means that the child will never block waiting on a lock held by its parent. Further, locks held by two children of the same parent will also conflict. To make this concrete, consider the following set of transactions and lock acquisitions. + +Transaction T1 is the parent transaction. It acquires a write lock on item A and then begins two child transactions: C1 and C2. C1 also wants to acquire a write lock on A; this succeeds. If C2 attempts to acquire a write lock on A, it will block until C1 releases the lock, at which point it will succeed. Now, let's say that C1 acquires a write lock on B. If C2 now attempts to obtain a lock on B, it will block. However, let's now assume that C1 commits. Its locks are anti-inherited, which means they are given to T1, so T1 will now hold a lock on B. At this point, C2 would be unblocked and would then acquire a lock on B. + +Child transactions are entirely subservient to their parent transaction. They may abort, undoing their operations regardless of the eventual fate of the parent. However, even if a child transaction commits, if its parent transaction is eventually aborted, the child's changes are undone and the child's transaction is effectively aborted. Any child transactions that are not yet resolved when the parent commits or aborts are resolved based on the parent's resolution -- committing if the parent commits and aborting if the parent aborts. Any child transactions that are not yet resolved when the parent prepares are also prepared. diff --git a/docs-src/guides/programmer_reference/transapp_put.md b/docs-src/guides/programmer_reference/transapp_put.md new file mode 100644 index 000000000..13b4278e5 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_put.md @@ -0,0 +1,119 @@ +--- +title: "Recoverability and deadlock handling" +api-name: "Recoverability and deadlock handling" +source: docs/programmer_reference/transapp_put.html +--- +## Recoverability and deadlock handling + +The first reason listed for using transactions was recoverability. Any logical change to a database may require multiple changes to underlying data structures. For example, modifying a record in a Btree may require leaf and internal pages to split, so a single DB->put() method call can potentially require that multiple physical database pages be written. If only some of those pages are written and then the system or application fails, the database is left inconsistent and cannot be used until it has been recovered; that is, until the partially completed changes have been undone. + +*Write-ahead-logging* is the term that describes the underlying implementation that Berkeley DB uses to ensure recoverability. What it means is that before any change is made to a database, information about the change is written to a database log. During recovery, the log is read, and databases are checked to ensure that changes described in the log for committed transactions appear in the database. Changes that appear in the database but are related to aborted or unfinished transactions in the log are undone from the database. + +For recoverability after application or system failure, operations that modify the database must be protected by transactions. More specifically, operations are not recoverable unless a transaction is begun and each operation is associated with the transaction via the Berkeley DB interfaces, and then the transaction successfully committed. This is true even if logging is turned on in the database environment. + +Here is an example function that updates a record in a database in a transactionally protected manner. The function takes a key and data items as arguments and then attempts to store them into the database. + +``` c +int +main(int argc, char *argv) +{ + extern int optind; + DB *db_cats, *db_color, *db_fruit; + DB_ENV *dbenv; + int ch; + + while ((ch = getopt(argc, argv, "")) != EOF) + switch (ch) { + case '?': + default: + usage(); + } + argc -= optind; + argv += optind; + + env_dir_create(); + env_open(&dbenv); + + /* Open database: Key is fruit class; Data is specific type. */ + db_open(dbenv, &db_fruit, "fruit", 0); + + /* Open database: Key is a color; Data is an integer. */ + db_open(dbenv, &db_color, "color", 0); + + /* + * Open database: + * Key is a name; Data is: company name, cat breeds. + */ + db_open(dbenv, &db_cats, "cats", 1); + + add_fruit(dbenv, db_fruit, "apple", "yellow delicious"); + + return (0); +} + +int +add_fruit(DB_ENV *dbenv, DB *db, char *fruit, char *name) +{ + DBT key, data; + DB_TXN *tid; + int fail, ret, t_ret; + + /* Initialization. */ + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + key.data = fruit; + key.size = strlen(fruit); + data.data = name; + data.size = strlen(name); + + for (fail = 0;;) { + /* Begin the transaction. */ + if ((ret = dbenv->txn_begin(dbenv, NULL, &tid, 0)) != 0) { + dbenv->err(dbenv, ret, "DB_ENV->txn_begin"); + exit (1); + } + + /* Store the value. */ + switch (ret = db->put(db, tid, &key, &data, 0)) { + case 0: + /* Success: commit the change. */ + if ((ret = tid->commit(tid, 0)) != 0) { + dbenv->err(dbenv, ret, "DB_TXN->commit"); + exit (1); + } + return (0); + case DB_LOCK_DEADLOCK: + default: + /* Retry the operation. */ + if ((t_ret = tid->abort(tid)) != 0) { + dbenv->err(dbenv, t_ret, "DB_TXN->abort"); + exit (1); + } + if (fail++ == MAXIMUM_RETRY) + return (ret); + break; + } + } +} +``` + +Berkeley DB also uses transactions to recover from deadlock. Database operations (that is, any call to a function underlying the handles returned by DB->open() and DB->cursor()) are usually performed on behalf of a unique locker. Transactions can be used to perform multiple calls on behalf of the same locker within a single thread of control. For example, consider the case in which an application uses a cursor scan to locate a record and then the application accesses another other item in the database, based on the key returned by the cursor, without first closing the cursor. If these operations are done using default locker IDs, they may conflict. If the locks are obtained on behalf of a transaction, using the transaction's locker ID instead of the database handle's locker ID, the operations will not conflict. + +There is a new error return in this function that you may not have seen before. In transactional (not Concurrent Data Store) applications supporting both readers and writers, or just multiple writers, Berkeley DB functions have an additional possible error return: DB_LOCK_DEADLOCK. This means two threads of control deadlocked, and the thread receiving the `DB_LOCK_DEADLOCK` error return has been selected to discard its locks in order to resolve the problem. When an application receives a `DB_LOCK_DEADLOCK` return, the correct action is to close any cursors involved in the operation and abort any enclosing transaction. In the sample code, any time the DB->put() method returns `DB_LOCK_DEADLOCK`, DB_TXN->abort() is called (which releases the transaction's Berkeley DB resources and undoes any partial changes to the databases), and then the transaction is retried from the beginning. + +There is no requirement that the transaction be attempted again, but that is a common course of action for applications. Applications may want to set an upper bound on the number of times an operation will be retried because some operations on some data sets may simply be unable to succeed. For example, updating all of the pages on a large Web site during prime business hours may simply be impossible because of the high access rate to the database. + +The DB_TXN->abort() method is called in error cases other than deadlock. Any time an error occurs, such that a transactionally protected set of operations cannot complete successfully, the transaction must be aborted. While deadlock is by far the most common of these errors, there are other possibilities; for example, running out of disk space for the filesystem. In Berkeley DB transactional applications, there are three classes of error returns: "expected" errors, "unexpected but recoverable" errors, and a single "unrecoverable" error. Expected errors are errors like DB_NOTFOUND, which indicates that a searched-for key item is not present in the database. Applications may want to explicitly test for and handle this error, or, in the case where the absence of a key implies the enclosing transaction should fail, simply call DB_TXN->abort(). Unexpected but recoverable errors are errors like DB_LOCK_DEADLOCK, which indicates that an operation has been selected to resolve a deadlock, or a system error such as EIO, which likely indicates that the filesystem has no available disk space. Applications must immediately call DB_TXN->abort() when these returns occur, as it is not possible to proceed otherwise. The only unrecoverable error is DB_RUNRECOVERY, which indicates that the system must stop and recovery must be run. + +The above code can be simplified in the case of a transaction comprised entirely of a single database put or delete operation, as operations occurring in transactional databases are implicitly transaction protected. For example, in a transactional database, the above code could be more simply written as: + +``` c + for (fail = 0; fail++ <= MAXIMUM_RETRY && + (ret = db->put(db, NULL, &key, &data, 0)) == DB_LOCK_DEADLOCK;) + continue; + return (ret == 0 ? 0 : 1); +``` + +and the underlying transaction would be automatically handled by Berkeley DB. + +Programmers should not attempt to enumerate all possible error returns in their software. Instead, they should explicitly handle expected returns and default to aborting the transaction for the rest. It is entirely the choice of the programmer whether to check for DB_RUNRECOVERY explicitly or not — attempting new Berkeley DB operations after DB_RUNRECOVERY is returned does not worsen the situation. Alternatively, using the DB_ENV->set_event_notify() method to handle an unrecoverable error and simply doing some number of abort-and-retry cycles for any unexpected Berkeley DB or system error in the mainline code often results in the simplest and cleanest application code. diff --git a/docs-src/guides/programmer_reference/transapp_read.md b/docs-src/guides/programmer_reference/transapp_read.md new file mode 100644 index 000000000..7d7f00d4b --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_read.md @@ -0,0 +1,38 @@ +--- +title: "Degrees of isolation" +api-name: "Degrees of isolation" +source: docs/programmer_reference/transapp_read.html +--- +## Degrees of isolation + + [Snapshot Isolation](transapp_read.md#snapshot_isolation) + +Transactions can be isolated from each other to different degrees. *Serializable* provides the most isolation, and means that, for the life of the transaction, every time a thread of control reads a data item, it will be unchanged from its previous value (assuming, of course, the thread of control does not itself modify the item). By default, Berkeley DB enforces serializability whenever database reads are wrapped in transactions. This is also known as *degree 3 isolation*. + +Most applications do not need to enclose all reads in transactions, and when possible, transactionally protected reads at serializable isolation should be avoided as they can cause performance problems. For example, a serializable cursor sequentially reading each key/data pair in a database, will acquire a read lock on most of the pages in the database and so will gradually block all write operations on the databases until the transaction commits or aborts. Note, however, that if there are update transactions present in the application, the read operations must still use locking, and must be prepared to repeat any operation (possibly closing and reopening a cursor) that fails with a return value of DB_LOCK_DEADLOCK. Applications that need repeatable reads are ones that require the ability to repeatedly access a data item knowing that it will not have changed (for example, an operation modifying a data item based on its existing value). + +*Snapshot isolation* also guarantees repeatable reads, but avoids read locks by using multiversion concurrency control (MVCC). This makes update operations more expensive, because they have to allocate space for new versions of pages in cache and make copies, but avoiding read locks can significantly increase throughput for many applications. Snapshot isolation is discussed in detail below. + +A transaction may only require *cursor stability*, that is only be guaranteed that cursors see committed data that does not change so long as it is addressed by the cursor, but may change before the reading transaction completes. This is also called *degree 2 isolation*. Berkeley DB provides this level of isolation when a transaction is started with the DB_READ_COMMITTED flag. This flag may also be specified when opening a cursor within a fully isolated transaction. + +Berkeley DB optionally supports reading uncommitted data; that is, read operations may request data which has been modified but not yet committed by another transaction. This is also called *degree 1 isolation*. This is done by first specifying the DB_READ_UNCOMMITTED flag when opening the underlying database, and then specifying the DB_READ_UNCOMMITTED flag when beginning a transaction, opening a cursor, or performing a read operation. The advantage of using DB_READ_UNCOMMITTED is that read operations will not block when another transaction holds a write lock on the requested data; the disadvantage is that read operations may return data that will disappear should the transaction holding the write lock abort. + +### Snapshot Isolation + +To make use of snapshot isolation, databases must first be configured for multiversion access by calling DB->open() with the DB_MULTIVERSION flag. Then transactions or cursors must be configured with the DB_TXN_SNAPSHOT flag. + +When configuring an environment for snapshot isolation, it is important to realize that having multiple versions of pages in cache means that the working set will take up more of the cache. As a result, snapshot isolation is best suited for use with larger cache sizes. + +If the cache becomes full of page copies before the old copies can be discarded, additional I/O will occur as pages are written to temporary "freezer" files. This can substantially reduce throughput, and should be avoided if possible by configuring a large cache and keeping snapshot isolation transactions short. The amount of cache required to avoid freezing buffers can be estimated by taking a checkpoint followed by a call to DB_ENV->log_archive(). The amount of cache required is approximately double the size of logs that remains. + +The environment should also be configured for sufficient transactions using DB_ENV->set_tx_max(). The maximum number of transactions needs to include all transactions executed concurrently by the application plus all cursors configured for snapshot isolation. Further, the transactions are retained until the last page they created is evicted from cache, so in the extreme case, an additional transaction may be needed for each page in the cache. Note that cache sizes under 500MB are increased by 25%, so the calculation of number of pages needs to take this into account. + +So when *should* applications use snapshot isolation? + +- There is a large cache relative to size of updates performed by concurrent transactions; and +- Read/write contention is limiting the throughput of the application; or +- The application is all or mostly read-only, and contention for the lock manager mutex is limiting throughput. + +The simplest way to take advantage of snapshot isolation is for queries: keep update transactions using full read/write locking and set DB_TXN_SNAPSHOT on read-only transactions or cursors. This should minimize blocking of snapshot isolation transactions and will avoid introducing new DB_LOCK_DEADLOCK errors. + +If the application has update transactions which read many items and only update a small set (for example, scanning until a desired record is found, then modifying it), throughput may be improved by running some updates at snapshot isolation as well. diff --git a/docs-src/guides/programmer_reference/transapp_reclimit.md b/docs-src/guides/programmer_reference/transapp_reclimit.md new file mode 100644 index 000000000..edecc770f --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_reclimit.md @@ -0,0 +1,39 @@ +--- +title: "Berkeley DB recoverability" +api-name: "Berkeley DB recoverability" +source: docs/programmer_reference/transapp_reclimit.html +--- +## Berkeley DB recoverability + +Berkeley DB recovery is based on write-ahead logging. This means that when a change is made to a database page, a description of the change is written into a log file. This description in the log file is guaranteed to be written to stable storage before the database pages that were changed are written to stable storage. This is the fundamental feature of the logging system that makes durability and rollback work. + +If the application or system crashes, the log is reviewed during recovery. Any database changes described in the log that were part of committed transactions and that were never written to the actual database itself are written to the database as part of recovery. Any database changes described in the log that were never committed and that were written to the actual database itself are backed-out of the database as part of recovery. This design allows the database to be written lazily, and only blocks from the log file have to be forced to disk as part of transaction commit. + +There are two interfaces that are a concern when considering Berkeley DB recoverability: + +1. The interface between Berkeley DB and the operating system/filesystem. +2. The interface between the operating system/filesystem and the underlying stable storage hardware. + +Berkeley DB uses the operating system interfaces and its underlying filesystem when writing its files. This means that Berkeley DB can fail if the underlying filesystem fails in some unrecoverable way. Otherwise, the interface requirements here are simple: The system call that Berkeley DB uses to flush data to disk (normally fsync or fdatasync), must guarantee that all the information necessary for a file's recoverability has been written to stable storage before it returns to Berkeley DB, and that no possible application or system crash can cause that file to be unrecoverable. + +In addition, Berkeley DB implicitly uses the interface between the operating system and the underlying hardware. The interface requirements here are not as simple. + +First, it is necessary to consider the underlying page size of the Berkeley DB databases. The Berkeley DB library performs all database writes using the page size specified by the application, and Berkeley DB assumes pages are written atomically. This means that if the operating system performs filesystem I/O in blocks of different sizes than the database page size, it may increase the possibility for database corruption. For example, assume that Berkeley DB is writing 32KB pages for a database, and the operating system does filesystem I/O in 16KB blocks. If the operating system writes the first 16KB of the database page successfully, but crashes before being able to write the second 16KB of the database, the database has been corrupted and this corruption may or may not be detected during recovery. For this reason, it may be important to select database page sizes that will be written as single block transfers by the underlying operating system. If you do not select a page size that the underlying operating system will write as a single block, you may want to configure the database to use checksums (see the DB->set_flags() flag for more information). By configuring checksums, you guarantee this kind of corruption will be detected at the expense of the CPU required to generate the checksums. When such an error is detected, the only course of recovery is to perform catastrophic recovery to restore the database. + +Second, if you are copying database files (either as part of doing a hot backup or creation of a hot failover area), there is an additional question related to the page size of the Berkeley DB databases. You must copy databases atomically, in units of the database page size. In other words, the reads made by the copy program must not be interleaved with writes by other threads of control, and the copy program must read the databases in multiples of the underlying database page size. On Unix systems, this is not a problem, as these operating systems already make this guarantee and system utilities normally read in power-of-2 sized chunks, which are larger than the largest possible Berkeley DB database page size. Other operating systems, particularly those based on Linux and Windows, do not provide this guarantee and hot backups may not be performed on these systems by reading data from the file system. The db_hotbackup utility should be used on these systems. + +An additional problem we have seen in this area was in some releases of Solaris where the cp utility was implemented using the mmap system call rather than the read system call. Because the Solaris' mmap system call did not make the same guarantee of read atomicity as the read system call, using the cp utility could create corrupted copies of the databases. Another problem we have seen is implementations of the tar utility doing 10KB block reads by default, and even when an output block size was specified to that utility, not reading from the underlying databases in multiples of the block size. Using the dd utility instead of the cp or tar utilities (and specifying an appropriate block size), fixes these problems. If you plan to use a system utility to copy database files, you may want to use a system call trace utility (for example, ktrace or truss) to check for an I/O size smaller than or not a multiple of the database page size and system calls other than read. + +Third, it is necessary to consider the behavior of the system's underlying stable storage hardware. For example, consider a SCSI controller that has been configured to cache data and return to the operating system that the data has been written to stable storage, when, in fact, it has only been written into the controller RAM cache. If power is lost before the controller is able to flush its cache to disk, and the controller cache is not stable (that is, the writes will not be flushed to disk when power returns), the writes will be lost. If the writes include database blocks, there is no loss because recovery will correctly update the database. If the writes include log file blocks, it is possible that transactions that were already committed may not appear in the recovered database, although the recovered database will be coherent after a crash. + +If the underlying hardware can fail in any way so that only part of the block was written, the failure conditions are the same as those described previously for an operating system failure that writes only part of a logical database block. In such cases, configuring the database for checksums will ensure the corruption is detected. + +For these reasons, it may be important to select hardware that does not do partial writes and does not cache data writes (or does not return that the data has been written to stable storage until it has either been written to stable storage or the actual writing of all of the data is guaranteed, barring catastrophic hardware failure — that is, your disk drive exploding). + +If the disk drive on which you are storing your databases explodes, you can perform normal Berkeley DB catastrophic recovery, because it requires only a snapshot of your databases plus the log files you have archived since those snapshots were taken. In this case, you should lose no database changes at all. + +If the disk drive on which you are storing your log files explodes, you can also perform catastrophic recovery, but you will lose any database changes made as part of transactions committed since your last archival of the log files. Alternatively, if your database environment and databases are still available after you lose the log file disk, you should be able to dump your databases. However, you may see an inconsistent snapshot of your data after doing the dump, because changes that were part of transactions that were not yet committed may appear in the database dump. Depending on the value of the data, a reasonable alternative may be to perform both the database dump and the catastrophic recovery and then compare the databases created by the two methods. + +Regardless, for these reasons, storing your databases and log files on different disks should be considered a safety measure as well as a performance enhancement. + +Finally, you should be aware that Berkeley DB does not protect against all cases of stable storage hardware failure, nor does it protect against simple hardware misbehavior (for example, a disk controller writing incorrect data to the disk). However, configuring the database for checksums will ensure that any such corruption is detected. diff --git a/docs-src/guides/programmer_reference/transapp_recovery.md b/docs-src/guides/programmer_reference/transapp_recovery.md new file mode 100644 index 000000000..96ece88b5 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_recovery.md @@ -0,0 +1,32 @@ +--- +title: "Recovery procedures" +api-name: "Recovery procedures" +source: docs/programmer_reference/transapp_recovery.html +--- +## Recovery procedures + +The fifth component of the infrastructure, recovery procedures, concerns the recoverability of the database. After any application or system failure, there are two possible approaches to database recovery: + +1. There is no need for recoverability, and all databases can be re-created from scratch. Although these applications may still need transaction protection for other reasons, recovery usually consists of removing the Berkeley DB environment home directory and all files it contains, and then restarting the application. Such an application may use the DB_TXN_NOT_DURABLE flag to avoid writing log records. + +2. It is necessary to recover information after system or application failure. In this case, recovery processing must be performed on any database environments that were active at the time of the failure. Recovery processing involves running the db_recover utility or calling the DB_ENV->open() method with the DB_RECOVER or DB_RECOVER_FATAL flags. + + During recovery processing, all database changes made by aborted or unfinished transactions are undone, and all database changes made by committed transactions are redone, as necessary. Database applications must not be restarted until recovery completes. After recovery finishes, the environment is properly initialized so that applications may be restarted. + +If performing recovery, there are two types of recovery processing: *normal* and *catastrophic*. Which you choose depends on the source for the database and log files you are using to recover. + +If up-to-the-minute database and log files are accessible on a stable filesystem, normal recovery is sufficient. Run the db_recover utility or call the DB_ENV->open() method specifying the DB_RECOVER flag. However, the normal recovery case **never** includes recovery using hot backups of the database environment. For example, you cannot perform a hot backup of databases and log files, restore the backup and then run normal recovery — you must always run catastrophic recovery when using hot backups. + +If the database or log files have been destroyed or corrupted, or normal recovery fails, catastrophic recovery is required. For example, catastrophic failure includes the case where the disk drive on which the database or log files are stored has been physically destroyed, or when the underlying filesystem is corrupted and the operating system's normal filesystem checking procedures cannot bring that filesystem to a consistent state. This is often difficult to detect, and a common sign of the need for catastrophic recovery is when normal Berkeley DB recovery procedures fail, or when checksum errors are displayed during normal database procedures. When catastrophic recovery is necessary, take the following steps: + +1. Restore the most recent snapshots of the database and log files from the backup media into the directory where recovery will be performed. + +2. If any log files were archived since the last snapshot was made, they should be restored into the directory where recovery will be performed. + + If any log files are available from the database environment that failed (for example, the disk holding the database files crashed, but the disk holding the log files is fine), those log files should be copied into the directory where recovery will be performed. + + Be sure to restore all log files in the order they were written. The order is important because it's possible the same log file appears on multiple backups, and you want to run recovery using the most recent version of each log file. + +3. Run the db_recover utility, specifying its **-c** option; or call the DB_ENV->open() method, specifying the DB_RECOVER_FATAL flag. The catastrophic recovery process will review the logs and database files to bring the environment databases to a consistent state as of the time of the last uncorrupted log file that is found. It is important to realize that only transactions committed before that date will appear in the databases. + + It is possible to re-create the database in a location different from the original by specifying appropriate pathnames to the **-h** option of the db_recover utility. In order for this to work properly, it is important that your application refer to files by names relative to the database home directory or the pathname(s) specified in calls to DB_ENV->set_data_dir(), instead of using full pathnames. diff --git a/docs-src/guides/programmer_reference/transapp_term.md b/docs-src/guides/programmer_reference/transapp_term.md new file mode 100644 index 000000000..ad98e5206 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_term.md @@ -0,0 +1,29 @@ +--- +title: "Terminology" +api-name: "Terminology" +source: docs/programmer_reference/transapp_term.html +--- +## Terminology + +Here are some definitions that will be helpful in understanding transactions: + +Thread of control +Berkeley DB is indifferent to the type or style of threads being used by the application; or, for that matter, if threads are being used at all — because Berkeley DB supports multiprocess access. In the Berkeley DB documentation, any time we refer to a *thread of control*, it can be read as a true thread (one of many in an application's address space) or a process. + +Free-threaded +A Berkeley DB handle that can be used by multiple threads simultaneously without any application-level synchronization is called *free-threaded*. + +Transaction +A *transaction* is a one or more operations on one or more databases that should be treated as a single unit of work. For example, changes to a set of databases, in which either all of the changes must be applied to the database(s) or none of them should. Applications specify when each transaction starts, what database operations are included in it, and when it ends. + +Transaction abort/commit +Every transaction ends by *committing* or *aborting*. If a transaction commits, Berkeley DB guarantees that any database changes included in the transaction will never be lost, even after system or application failure. If a transaction aborts, or is uncommitted when the system or application fails, then the changes involved will never appear in the database. + +System or application failure +*System or application failure* is the phrase we use to describe something bad happening near your data. It can be an application dumping core, being interrupted by a signal, the disk filling up, or the entire system crashing. In any case, for whatever reason, the application can no longer make forward progress, and its databases are left in an unknown state. + +Recovery +*Recovery* is what makes the database consistent after a system or application failure. The recovery process includes review of log files and databases to ensure that the changes from each committed transaction appear in the database, and that no changes from an unfinished (or aborted) transaction do. Whenever system or application failure occurs, applications must usually run recovery. + +Deadlock +*Deadlock*, in its simplest form, happens when one thread of control owns resource A, but needs resource B; while another thread of control owns resource B, but needs resource A. Neither thread of control can make progress, and so one has to give up and release all its resources, at which time the remaining thread of control can make forward progress. diff --git a/docs-src/guides/programmer_reference/transapp_throughput.md b/docs-src/guides/programmer_reference/transapp_throughput.md new file mode 100644 index 000000000..e77a8072c --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_throughput.md @@ -0,0 +1,55 @@ +--- +title: "Transaction throughput" +api-name: "Transaction throughput" +source: docs/programmer_reference/transapp_throughput.html +--- +## Transaction throughput + +Generally, the speed of a database system is measured by the *transaction throughput*, expressed as a number of transactions per second. The two gating factors for Berkeley DB performance in a transactional system are usually the underlying database files and the log file. Both are factors because they require disk I/O, which is slow relative to other system resources such as CPU. + +In the worst-case scenario: + +- Database access is truly random and the database is too large for any significant percentage of it to fit into the cache, resulting in a single I/O per requested key/data pair. +- Both the database and the log are on a single disk. + +This means that for each transaction, Berkeley DB is potentially performing several filesystem operations: + +- Disk seek to database file +- Database file read +- Disk seek to log file +- Log file write +- Flush log file information to disk +- Disk seek to update log file metadata (for example, inode information) +- Log metadata write +- Flush log file metadata to disk + +There are a number of ways to increase transactional throughput, all of which attempt to decrease the number of filesystem operations per transaction. First, the Berkeley DB software includes support for *group commit*. Group commit simply means that when the information about one transaction is flushed to disk, the information for any other waiting transactions will be flushed to disk at the same time, potentially amortizing a single log write over a large number of transactions. There are additional tuning parameters which may be useful to application writers: + +- Tune the size of the database cache. If the Berkeley DB key/data pairs used during the transaction are found in the database cache, the seek and read from the database are no longer necessary, resulting in two fewer filesystem operations per transaction. To determine whether your cache size is too small, see Selecting a cache size. +- Put the database and the log files on different disks. This allows reads and writes to the log files and the database files to be performed concurrently. +- Set the filesystem configuration so that file access and modification times are not updated. Note that although the file access and modification times are not used by Berkeley DB, this may affect other programs -- so be careful. +- Upgrade your hardware. When considering the hardware on which to run your application, however, it is important to consider the entire system. The controller and bus can have as much to do with the disk performance as the disk itself. It is also important to remember that throughput is rarely the limiting factor, and that disk seek times are normally the true performance issue for Berkeley DB. +- Turn on the DB_TXN_NOSYNC or DB_TXN_WRITE_NOSYNC flags. This changes the Berkeley DB behavior so that the log files are not written and/or flushed when transactions are committed. Although this change will greatly increase your transaction throughput, it means that transactions will exhibit the ACI (atomicity, consistency, and isolation) properties, but not D (durability). Database integrity will be maintained, but it is possible that some number of the most recently committed transactions may be undone during recovery instead of being redone. + +If you are bottlenecked on logging, the following test will help you confirm that the number of transactions per second that your application does is reasonable for the hardware on which you are running. Your test program should repeatedly perform the following operations: + +- Seek to the beginning of a file +- Write to the file +- Flush the file write to disk + +The number of times that you can perform these three operations per second is a rough measure of the minimum number of transactions per second of which the hardware is capable. This test simulates the operations applied to the log file. (As a simplifying assumption in this experiment, we assume that the database files are either on a separate disk; or that they fit, with some few exceptions, into the database cache.) We do not have to directly simulate updating the log file directory information because it will normally be updated and flushed to disk as a result of flushing the log file write to disk. + +Running this test program, in which we write 256 bytes for 1000 operations on reasonably standard commodity hardware (Pentium II CPU, SCSI disk), returned the following results: + +``` c +% testfile -b256 -o1000 +running: 1000 ops +Elapsed time: 16.641934 seconds +1000 ops: 60.09 ops per second +``` + +Note that the number of bytes being written to the log as part of each transaction can dramatically affect the transaction throughput. The test run used 256, which is a reasonable size log write. Your log writes may be different. To determine your average log write size, use the db_stat utility to display your log statistics. + +As a quick sanity check, the average seek time is 9.4 msec for this particular disk, and the average latency is 4.17 msec. That results in a minimum requirement for a data transfer to the disk of 13.57 msec, or a maximum of 74 transfers per second. This is close enough to the previous 60 operations per second (which was not done on a quiescent disk) that the number is believable. + +An implementation of the previous example test program for IEEE/ANSI Std 1003.1 (POSIX) standard systems is included in the Berkeley DB distribution. diff --git a/docs-src/guides/programmer_reference/transapp_tune.md b/docs-src/guides/programmer_reference/transapp_tune.md new file mode 100644 index 000000000..1d6b6d783 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_tune.md @@ -0,0 +1,59 @@ +--- +title: "Transaction tuning" +api-name: "Transaction tuning" +source: docs/programmer_reference/transapp_tune.html +--- +## Transaction tuning + +There are a few different issues to consider when tuning the performance of Berkeley DB transactional applications. First, you should review Access method tuning, as the tuning issues for access method applications are applicable to transactional applications as well. The following are additional tuning issues for Berkeley DB transactional applications: + +access method +Highly concurrent applications should use the Queue access method, where possible, as it provides finer-granularity of locking than the other access methods. Otherwise, applications usually see better concurrency when using the Btree access method than when using either the Hash or Recno access methods. + +record numbers +Using record numbers outside of the Queue access method will often slow down concurrent applications as they limit the degree of concurrency available in the database. Using the Recno access method, or the Btree access method with retrieval by record number configured can slow applications down. + +Btree database size +When using the Btree access method, applications supporting concurrent access may see excessive numbers of deadlocks in small databases. There are two different approaches to resolving this problem. First, as the Btree access method uses page-level locking, decreasing the database page size can result in fewer lock conflicts. Second, in the case of databases that are cyclically growing and shrinking, turning off reverse splits (with DB_REVSPLITOFF) can leave the database with enough pages that there will be fewer lock conflicts. + +read locks +Performing all read operations outside of transactions or at Degrees of isolation can often significantly increase application throughput. In addition, limiting the lifetime of non-transactional cursors will reduce the length of times locks are held, thereby improving concurrency. + +DB_DIRECT_DB, DB_LOG_DIRECT +On some systems, avoiding caching in the operating system can improve write throughput and allow the creation of larger Berkeley DB caches. + +DB_READ_UNCOMMITTED, DB_READ_COMMITTED +Consider decreasing the level of isolation of transaction using the DB_READ_UNCOMMITTED, or DB_READ_COMMITTED flags for transactions or cursors or the DB_READ_UNCOMMITTED flag on individual read operations. The DB_READ_COMMITTED flag will release read locks on cursors as soon as the data page is nolonger referenced. This is also called *degree 2 isolation*. This will tend to block write operations for shorter periods for applications that do not need to have repeatable reads for cursor operations. + +The DB_READ_UNCOMMITTED flag will allow read operations to potentially return data which has been modified but not yet committed, and can significantly increase application throughput in applications that do not require data be guaranteed to be permanent in the database. This is also called *degree 1 isolation*, or *dirty reads*. + + DB_RMW +If there are many deadlocks, consider using the DB_RMW flag to immediately acquire write locks when reading data items that will subsequently be modified. Although this flag may increase contention (because write locks are held longer than they would otherwise be), it may decrease the number of deadlocks that occur. + +DB_TXN_WRITE_NOSYNC, DB_TXN_NOSYNC +By default, transactional commit in Berkeley DB implies durability, that is, all committed operations will be present in the database after recovery from any application or system failure. For applications not requiring that level of certainty, specifying the DB_TXN_NOSYNC flag will often provide a significant performance improvement. In this case, the database will still be fully recoverable, but some number of committed transactions might be lost after application or system failure. + +access databases in order +When modifying multiple databases in a single transaction, always access physical files and databases within physical files, in the same order where possible. In addition, avoid returning to a physical file or database, that is, avoid accessing a database, moving on to another database and then returning to the first database. This can significantly reduce the chance of deadlock between threads of control. + +large key/data items +Transactional protections in Berkeley DB are guaranteed by before and after physical image logging. This means applications modifying large key/data items also write large log records, and, in the case of the default transaction commit, threads of control must wait until those log records have been flushed to disk. Applications supporting concurrent access should try and keep key/data items small wherever possible. + +mutex selection +During configuration, Berkeley DB selects a mutex implementation for the architecture. Berkeley DB normally prefers blocking-mutex implementations over non-blocking ones. For example, Berkeley DB will select POSIX pthread mutex interfaces rather than assembly-code test-and-set spin mutexes because pthread mutexes are usually more efficient and less likely to waste CPU cycles spinning without getting any work accomplished. + +For some applications and systems (generally highly concurrent applications on large multiprocessor systems), Berkeley DB makes the wrong choice. In some cases, better performance can be achieved by configuring with the --with-mutex argument and selecting a different mutex implementation than the one selected by Berkeley DB. When a test-and-set spin mutex implementation is selected, it may be useful to tune the number of spins made before yielding the processor and sleeping. For more information, see the DB_ENV->mutex_set_tas_spins() method. + +Finally, Berkeley DB may put multiple mutexes on individual cache lines. When tuning Berkeley DB for large multiprocessor systems, it may be useful to tune mutex alignment using the DB_ENV->mutex_set_align() method. + + --enable-posix-mutexes +By default, the Berkeley DB library will only select the POSIX pthread mutex implementation if it supports mutexes shared between multiple processes. If your application does not share its database environment between processes and your system's POSIX mutex support was not selected because it did not support inter-process mutexes, you may be able to increase performance and transactional throughput by configuring with the --enable-posix-mutexes argument. + +log buffer size +Berkeley DB internally maintains a buffer of log writes. The buffer is written to disk at transaction commit, by default, or, whenever it is filled. If it is consistently being filled before transaction commit, it will be written multiple times per transaction, costing application performance. In these cases, increasing the size of the log buffer can increase application throughput. + +log file location +If the database environment's log files are on the same disk as the databases, the disk arms will have to seek back-and-forth between the two. Placing the log files and the databases on different disk arms can often increase application throughput. + +trickle write +In some applications, the cache is sufficiently active and dirty that readers frequently need to write a dirty page in order to have space in which to read a new page from the backing database file. You can use the db_stat utility (or the statistics returned by the DB_ENV->memp_stat() method) to see how often this is happening in your application's cache. In this case, using a separate thread of control and the DB_ENV->memp_trickle() method to trickle-write pages can often increase the overall throughput of the application. diff --git a/docs-src/guides/programmer_reference/transapp_why.md b/docs-src/guides/programmer_reference/transapp_why.md new file mode 100644 index 000000000..afbd907d9 --- /dev/null +++ b/docs-src/guides/programmer_reference/transapp_why.md @@ -0,0 +1,17 @@ +--- +title: "Why transactions?" +api-name: "Why transactions?" +source: docs/programmer_reference/transapp_why.html +--- +## Why transactions? + +Perhaps the first question to answer is "Why transactions?" There are a number of reasons to include transactional support in your applications. The most common ones are the following: + +Recoverability +Applications often need to ensure that no matter how the system or application fails, previously saved data is available the next time the application runs. This is often called Durability. + +Atomicity +Applications may need to make multiple changes to one or more databases, but ensure that either all of the changes happen, or none of them happens. Transactions guarantee that a group of changes are atomic; that is, if the application or system fails, either all of the changes to the databases will appear when the application next runs, or none of them. + +Isolation +Applications may need to make changes in isolation, that is, ensure that only a single thread of control is modifying a key/data pair at a time. Transactions ensure each thread of control sees all records as if all other transactions either completed before or after its transaction. diff --git a/docs-src/guides/programmer_reference/txn.md b/docs-src/guides/programmer_reference/txn.md new file mode 100644 index 000000000..850f9ce15 --- /dev/null +++ b/docs-src/guides/programmer_reference/txn.md @@ -0,0 +1,46 @@ +--- +title: "Chapter 19.  The Transaction Subsystem" +api-name: "Chapter 19.  The Transaction Subsystem" +source: docs/programmer_reference/txn.html +--- +## Chapter 19.  The Transaction Subsystem + +**Table of Contents** + + [Introduction to the transaction subsystem](txn.md#txn_intro) + + [Configuring transactions](txn_config.md) + + [Transaction limits](txn_limits.md) + + [Transaction IDs](txn_limits.md#idp53352320) + + [Cursors](txn_limits.md#idp53275624) + + [Multiple Threads of Control](txn_limits.md#idp53223368) + +## Introduction to the transaction subsystem + +The Transaction subsystem makes operations atomic, consistent, isolated, and durable in the face of system and application failures. The subsystem requires that the data be properly logged and locked in order to attain these properties. Berkeley DB contains all the components necessary to transaction-protect the Berkeley DB access methods, and other forms of data may be protected if they are logged and locked appropriately. + +The Transaction subsystem is created, initialized, and opened by calls to DB_ENV->open() with the DB_INIT_TXN flag specified. Note that enabling transactions automatically enables logging, but does not enable locking because a single thread of control that needed atomicity and recoverability would not require it. + +The DB_ENV->txn_begin() function starts a transaction, returning an opaque handle to a transaction. If the parent parameter to DB_ENV->txn_begin() is non-NULL, the new transaction is a child of the designated parent transaction. + +The DB_TXN->abort() function ends the designated transaction and causes all updates performed by the transaction to be undone. The end result is that the database is left in a state identical to the state that existed prior to the DB_ENV->txn_begin(). If the aborting transaction has any child transactions associated with it (even ones that have already been committed), they are also aborted. Any transactions that are unresolved (neither committed nor aborted) when the application or system fails are aborted during recovery. + +The DB_TXN->commit() function ends the designated transaction and makes all the updates performed by the transaction permanent, even in the face of application or system failure. If this is a parent transaction committing, all child transactions that individually committed or had not been resolved are also committed. + +Transactions are identified by 32-bit unsigned integers. The ID associated with any transaction can be obtained using the DB_TXN->id() function. If an application is maintaining information outside of Berkeley DB it wants to transaction-protect, it should use this transaction ID as the locking ID. + +The DB_ENV->txn_checkpoint() function causes a transaction checkpoint. A checkpoint is performed using to a specific log sequence number (LSN), referred to as the checkpoint LSN. When a checkpoint completes successfully, it means that all data buffers whose updates are described by LSNs less than the checkpoint LSN have been written to disk. This, in turn, means that the log records less than the checkpoint LSN are no longer necessary for normal recovery (although they would be required for catastrophic recovery if the database files were lost), and all log files containing only records prior to the checkpoint LSN may be safely archived and removed. + +The time required to run normal recovery is proportional to the amount of work done between checkpoints. If a large number of modifications happen between checkpoints, many updates recorded in the log may not have been written to disk when failure occurred, and recovery may take longer to run. Generally, if the interval between checkpoints is short, data may be being written to disk more frequently, but the recovery time will be shorter. Often, the checkpoint interval is tuned for each specific application. + +The DB_TXN->stat() method returns information about the status of the transaction subsystem. It is the programmatic interface used by the db_stat utility. + +The transaction system is closed by a call to DB_ENV->close(). + +Finally, the entire transaction system may be removed using the DB_ENV->remove() method. + +For more information on the transaction subsystem methods, see the Transaction Subsystem and Related Methods section in the *Berkeley DB C API Reference Guide.* diff --git a/docs-src/guides/programmer_reference/txn_config.md b/docs-src/guides/programmer_reference/txn_config.md new file mode 100644 index 000000000..a96e7fafc --- /dev/null +++ b/docs-src/guides/programmer_reference/txn_config.md @@ -0,0 +1,14 @@ +--- +title: "Configuring transactions" +api-name: "Configuring transactions" +source: docs/programmer_reference/txn_config.html +--- +## Configuring transactions + +The application may change the number of simultaneous outstanding transactions supported by the Berkeley DB environment by calling the DB_ENV->set_tx_max() method. This will also set the size of the underlying transaction subsystem's region. When the number of outstanding transactions is reached, additional calls to DB_ENV->txn_begin() will fail until some active transactions complete. + +The application can limit how long a transaction runs or blocks on contested resources. The DB_ENV->set_timeout() method specifies the length of the timeout. This value is checked whenever deadlock detection is performed or when the transaction is about to block on a lock that cannot be immediately granted. Because timeouts are only checked at these times, the accuracy of the timeout depends on how often deadlock detection is performed or how frequently the transaction blocks. + +There is an additional parameter used in configuring transactions; the DB_TXN_NOSYNC. Setting the DB_TXN_NOSYNC flag to DB_ENV->set_flags() when opening a transaction region changes the behavior of transactions to not write or synchronously flush the log during transaction commit. + +This change may significantly increase application transactional throughput. However, it means that although transactions will continue to exhibit the ACI (atomicity, consistency, and isolation) properties, they will not have D (durability). Database integrity will be maintained, but it is possible that some number of the most recently committed transactions may be undone during recovery instead of being redone. diff --git a/docs-src/guides/programmer_reference/txn_limits.md b/docs-src/guides/programmer_reference/txn_limits.md new file mode 100644 index 000000000..757c48a97 --- /dev/null +++ b/docs-src/guides/programmer_reference/txn_limits.md @@ -0,0 +1,24 @@ +--- +title: "Transaction limits" +api-name: "Transaction limits" +source: docs/programmer_reference/txn_limits.html +--- +## Transaction limits + + [Transaction IDs](txn_limits.md#idp53352320) + + [Cursors](txn_limits.md#idp53275624) + + [Multiple Threads of Control](txn_limits.md#idp53223368) + +### Transaction IDs + +Transactions are identified by 31-bit unsigned integers, which means there are just over two billion unique transaction IDs. When a database environment is initially created or recovery is run, the transaction ID name space is reset, and new transactions are numbered starting from 0x80000000 (2,147,483,648). The IDs will wrap if the maximum transaction ID is reached, starting again from 0x80000000. The most recently allocated transaction ID is the **st_last_txnid** value in the transaction statistics information, and can be displayed by the db_stat utility. + +### Cursors + +When using transactions, cursors are localized to a single transaction. That is, a cursor may not span transactions, and must be opened and closed within a single transaction. In addition, intermingling transaction-protected cursor operations and non-transaction-protected cursor operations on the same database in a single thread of control is practically guaranteed to deadlock because the locks obtained for transactions and non-transactions can conflict. + +### Multiple Threads of Control + +Because transactions must hold all their locks until commit, a single transaction may accumulate a large number of long-term locks during its lifetime. As a result, when two concurrently running transactions access the same database, there is strong potential for conflict. Although Berkeley DB allows an application to have multiple outstanding transactions active within a single thread of control, great care must be taken to ensure that the transactions do not block each other (for example, attempt to obtain conflicting locks on the same data). If two concurrently active transactions in the same thread of control do encounter a lock conflict, the thread of control will deadlock so that the deadlock detector cannot detect the problem. In this case, there is no true deadlock, but because the transaction on which a transaction is waiting is in the same thread of control, no forward progress can be made. diff --git a/docs-src/guides/programmer_reference/witold.md b/docs-src/guides/programmer_reference/witold.md new file mode 100644 index 000000000..6f1d337df --- /dev/null +++ b/docs-src/guides/programmer_reference/witold.md @@ -0,0 +1,8 @@ +--- +title: "Berkeley DB Reference Guide: Witold Litwin" +api-name: "Berkeley DB Reference Guide: Witold Litwin" +source: docs/programmer_reference/witold.html +--- +**Witold Litwin** + +Witold is a hell of a guy to take you on a late-night high-speed car chase up the mountains of Austria in search of very green wine. diff --git a/docs-src/guides/programmer_reference/xa.md b/docs-src/guides/programmer_reference/xa.md new file mode 100644 index 000000000..38f36b76b --- /dev/null +++ b/docs-src/guides/programmer_reference/xa.md @@ -0,0 +1,46 @@ +--- +title: "Chapter 13.  Distributed Transactions" +api-name: "Chapter 13.  Distributed Transactions" +source: docs/programmer_reference/xa.html +--- +## Chapter 13.  Distributed Transactions + +**Table of Contents** + + [Introduction](xa.md#xa_intro) + + [Berkeley DB XA Implementation](ch13s02.md) + + [Building a Global Transaction Manager](xa_build.md) + + [Communicating with multiple Berkeley DB environments](xa_build.md#idp52778488) + + [Recovering from GTM failure](xa_build.md#idp52779432) + + [Managing the Global Transaction ID (GID) name space](xa_build.md#idp52703176) + + [Maintaining state for each distributed transaction.](xa_build.md#idp52758336) + + [Recovering from the failure of a single environment](xa_build.md#idp52777008) + + [Recovering from GTM failure](xa_build.md#idp52779896) + + [XA Introduction](xa_xa_intro.md) + + [Configuring Berkeley DB with the Tuxedo System](xa_xa_config.md) + + [Update the Resource Manager File in Tuxedo](xa_xa_config.md#idp52786896) + + [Build the Transaction Manager Server](xa_xa_config.md#idp52812512) + + [Update the UBBCONFIG File](xa_xa_config.md#idp52759288) + + [Restrictions on XA Transactions](xa_xa_restrict.md) + + [XA: Frequently Asked Questions](xa_faq.md) + +## Introduction + +An application must use distributed transactions whenever it wants transactional semantics either across operations in multiple Berkeley DB environments (even if they are on the same machine) or across operations in Berkeley DB and some other database systems (for example, Oracle server). Berkeley DB provides support for distributed transactions using a two-phase commit protocol. In order to use the two-phase commit feature of Berkeley DB, an application must either implement its own global transaction manager or use an XA-compliant transaction manager such as Oracle Tuxedo (as Berkeley DB can act as an XA-compliant resource manager). + +This chapter explains Berkeley DB's XA-compliant resource manager, which can be used in any X/Open distributed transaction processing system, and explains how to configure Oracle Tuxedo to use the Berkeley DB resource manager. diff --git a/docs-src/guides/programmer_reference/xa_build.md b/docs-src/guides/programmer_reference/xa_build.md new file mode 100644 index 000000000..61c049dc8 --- /dev/null +++ b/docs-src/guides/programmer_reference/xa_build.md @@ -0,0 +1,93 @@ +--- +title: "Building a Global Transaction Manager" +api-name: "Building a Global Transaction Manager" +source: docs/programmer_reference/xa_build.html +--- +## Building a Global Transaction Manager + + [Communicating with multiple Berkeley DB environments](xa_build.md#idp52778488) + + [Recovering from GTM failure](xa_build.md#idp52779432) + + [Managing the Global Transaction ID (GID) name space](xa_build.md#idp52703176) + + [Maintaining state for each distributed transaction.](xa_build.md#idp52758336) + + [Recovering from the failure of a single environment](xa_build.md#idp52777008) + + [Recovering from GTM failure](xa_build.md#idp52779896) + +Managing distributed transactions and using the two-phase commit protocol of Berkeley DB from an application requires the application to provide the functionality of a global transaction manager (GTM). The GTM is responsible for the following: + +- Communicating with the multiple environments (potentially on separate systems). +- Managing the global transaction ID name space. +- Maintaining state information about each distributed transaction. +- Recovering from failures of individual environments. +- Recovering the global transaction state after failure of the global transaction manager. + +### Communicating with multiple Berkeley DB environments + +Two-phase commit is required if a transaction spans operations in multiple Berkeley DB environments or across operations in Berkeley DB and some other database systems. If the environments reside on the same machine, the application can communicate with each environment through its own address space with no additional complexity. If the environments reside on separate machines, the application may use its own messaging capability, translating messages on the remote machine into calls into the Berkeley DB library (including the recovery calls). + +### Recovering from GTM failure + +If the GTM fails, it must first recover its local state. Assuming the GTM uses Berkeley DB tables to maintain state, it should run the db_recover utility (or the DB_RECOVER option to DB_ENV->open()) upon startup. Once the GTM is back up and running, it needs to review all its outstanding global transactions, that is, all transactions that are recorded but not yet completed. + +The global transactions that have not yet reached the prepare phase should be aborted. For each global transaction that has not yet prepared, the GTM should send a message to each participant telling it to abort its transaction. + +Next the GTM should review its log to identify all participating environments that have transactions in the preparing, aborting, or committing states. For each such participant, the GTM should issue a DB_ENV->txn_recover() call. Upon receiving responses from each participant, the GTM must decide the fate of each transaction and issue appropriate calls. The correct behavior is defined as follows: + +preparing +if all participating environments return the transaction in the list of prepared but not yet completed transactions, then the GTM should commit the transaction. If any participating environment fails to return the transaction in this list, then the GTM must issue an abort to all environments participating in that global transaction. + +committing +the GTM should send a commit to any environment that returned this transaction in its list of prepared but not yet completed transactions. + +aborting +the GTM should send an abort to any environment that returned this transaction in its list of prepared but not yet completed transactions. + +### Managing the Global Transaction ID (GID) name space + +A global transaction is a transaction that spans multiple environments. Each global transaction must have a unique transaction ID. This unique ID is the global transaction ID (GID). In Berkeley DB, global transaction IDs must be represented by the character array, `DB_GID_SIZE` (currently 128 bytes). It is the responsibility of the global transaction manager to assign GIDs, guarantee their uniqueness, and manage the mapping of local transactions to GID. That is, for each GID, the GTM should know which local transaction managers participated. The Berkeley DB logging system or a Berkeley DB table could be used to record this information. + +### Maintaining state for each distributed transaction. + +In addition to knowing which local environments participate in each global transaction, the GTM must also know the state of each active global transaction. As soon as a transaction becomes distributed (that is, a second environment participates), the GTM must record the existence of the global transaction and all participants (whether this must reside on stable storage or not depends on the exact configuration of the system). As new environments participate, the GTM must keep this information up to date. + +When the GTM is ready to begin commit processing, it should issue DB_TXN->prepare() calls to each participating environment, indicating the GID of the global transaction. Once all the participants have successfully prepared, then the GTM must record that the global transaction will be committed. This record should go to stable storage. Once written to stable storage, the GTM can send DB_TXN->commit() requests to each participating environment. Once all environments have successfully completed the commit, the GTM can either record the successful commit or can somehow "forget" the global transaction. + +If an application uses nested transactions (that is, the parent parameter is non-NULL in a call to DB_ENV->txn_begin()) then, only the parent transaction should call DB_TXN->prepare(), not any of the child transactions. + +Should any participant fail to prepare, then the GTM must abort the global transaction. The fact that the transaction is going to be aborted should be written to stable storage. Once written, the GTM can then issue DB_TXN->abort() requests to each environment. When all aborts have returned successfully, the GTM can either record the successful abort or "forget" the global transaction. + +In summary, for each transaction, the GTM must maintain the following: + +- A list of participating environments +- The current state of each transaction (pre-prepare, preparing, committing, aborting, done) + +### Recovering from the failure of a single environment + +If a single environment fails, there is no need to bring down or recover other environments (the only exception to this is if all environments are managed in the same application address space and there is a risk that the failure of the environment corrupted other environments). Instead, once the failing environment comes back up, it should be recovered (that is, conventional recovery, via the db_recover utility or by specifying the DB_RECOVER flag to DB_ENV->open() should be run). If the db_recover utility is used, then the -e option must be specified. In this case, the application will almost certainly want to specify environmental parameters via a DB_CONFIG configuration file in the environment's home directory, so that the db_recover utility can create an appropriately configured environment. If the db_recover utility is not used, then the GTM should call DB_ENV->open() specifying the DB_RECOVER flag. It should then call DB_ENV->txn_recover(), which will return an array of DB_TXN handles for the set of prepared, but not yet completed transactions. For each transaction, the GTM should combine this knowledge with its transaction state table and call either DB_TXN->commit() or DB_TXN->abort(). After that process is complete, the environment is ready to participate in new transactions. + +If the GTM is running in a system with multiple GTMs, it is possible that some of the transactions returned via DB_ENV->txn_recover() do not belong to the current environment. The GTM should detect this and call DB_TXN->discard() on each such transaction handle. Furthermore, it is important to note the environment does not retain information about which GTM has issued DB_ENV->txn_recover() operations. Therefore, each GTM should issue all its DB_ENV->txn_recover() calls before another GTM issues its calls. If the calls are interleaved, each GTM may not get a complete and consistent set of transactions. The simplest way to enforce this is for each GTM to make sure it can receive all its outstanding transactions in a single DB_ENV->txn_recover() call. The maximum number of possible outstanding transactions is roughly the maximum number of active transactions in the environment (whose value can be obtained using the db_stat utility). To simplify this procedure, the caller should allocate an array large enough to hold the list of transactions (for example, allocate an array able to hold three times the maximum number of transactions). If that is not possible, callers should check that the array was not completely filled in when DB_ENV->txn_recover() returns. If the array was completely filled in, each transaction should be explicitly discarded, and the call repeated with a larger array. + +The newly recovered environment will forbid any new transactions from being started until the prepared but not yet completed transactions have been resolved. In the multiple GTM case, this means that all GTMs must recover before any GTM can begin issuing new transactions. + +The GTM must determine how long it needs to retain global transaction commit and abort records. If the participating environments are following a DB_TXN_SYNC policy, that is, they are forcing commit and abort records to disk before replying to the GTM, then once the GTM has heard from all participants, it need not retain its persistent log records. However, if participating environments are running at weaker durability levels, such as DB_TXN_WRITE_NOSYNC or DB_TXN_NOSYNC, then the GTM must retain all commit and abort records until all participants have completed a checkpoint following the completion of a transaction. + +### Recovering from GTM failure + +If the GTM fails, it must first recover its local state. Assuming the GTM uses Berkeley DB tables to maintain state, it should run the db_recover utility (or the DB_RECOVER option to DB_ENV->open()) upon startup. Once the GTM is back up and running, it needs to review all its outstanding global transactions, that is, all transactions that are recorded but not yet completed. + +The global transactions that have not yet reached the prepare phase should be aborted. For each global transaction that has not yet prepared, the GTM should send a message to each participant telling it to abort its transaction. + +Next the GTM should review its log to identify all participating environments that have transactions in the preparing, aborting, or committing states. For each such participant, the GTM should issue a DB_ENV->txn_recover() call. Upon receiving responses from each participant, the GTM must decide the fate of each transaction and issue appropriate calls. The correct behavior is defined as follows: + +preparing +if all participating environments return the transaction in the list of prepared but not yet completed transactions, then the GTM should commit the transaction. If any participating environment fails to return the transaction in this list, then the GTM must issue an abort to all environments participating in that global transaction. + +committing +the GTM should send a commit to any environment that returned this transaction in its list of prepared but not yet completed transactions. + +aborting +the GTM should send an abort to any environment that returned this transaction in its list of prepared but not yet completed transactions. diff --git a/docs-src/guides/programmer_reference/xa_faq.md b/docs-src/guides/programmer_reference/xa_faq.md new file mode 100644 index 000000000..a2526e68c --- /dev/null +++ b/docs-src/guides/programmer_reference/xa_faq.md @@ -0,0 +1,30 @@ +--- +title: "XA: Frequently Asked Questions" +api-name: "XA: Frequently Asked Questions" +source: docs/programmer_reference/xa_faq.html +--- +## XA: Frequently Asked Questions + +1. **Is it possible to mix XA and non-XA transactions?** + + Yes. It is also possible for XA and non-XA transactions to coexist in the same Berkeley DB environment. To do this, specify the same environment to the non-XA DB_ENV->open() calls as was specified in the Tuxedo configuration file. + +2. **Does converting an application to run within XA change any of the already existing C/C++ API calls it does?** + + When converting an application to run under XA, the application's Berkeley DB calls are unchanged, with three exceptions: + + 1. The application must specify the DB_XA_CREATE flag to the db_create() function. + 2. Unless the application is performing an operation for a non-XA transaction, the application should never explicitly call DB_TXN->commit(), DB_TXN->abort(), and DB_ENV->txn_begin(), and those calls should be replaced by calls into the Tuxedo transaction manager. + 3. Unless the application is performing an operation for a non-XA transaction, the application should specify a transaction argument of NULL to Berkeley DB methods taking transaction arguments (for example, DB->put() or DB->cursor()). + + Otherwise, the application should be unchanged. + +3. **How does Berkeley DB recovery interact with recovery by the Tuxedo transaction manager?** + + Recovery is completed in two steps. First, each resource manager should recover its environment(s). This can be done via a program that calls DB_ENV->open() or by calling the db_recover utility. If using the db_recover utility, then the option should be specified so that the regions that are recovered persist after the utility exits. Any transactions that were prepared, but neither completed nor aborted, are restored to their prepared state so that they may be aborted or committed via the Tuxedo recovery mechanisms. After each resource manager has recovered, then Tuxedo recovery may begin. Tuxedo will interact with each resource manager via the \_\_db_xa_recover function which returns the list of prepared, but not yet completed transactions. It should issue a commit or abort for each one, and only after having completed each transaction will normal processing resume. + + Finally, standard log file archival and catastrophic recovery procedures should occur independently of XA operation. + +4. **Does Berkeley DB provide multi-threaded support for XA transactions?** + + Yes. For information on how to build multi-threaded servers for XA transactions, see http://download.oracle.com/docs/cd/E13161_01/tuxedo/docs10gr3/pgc/pgthr.html. All databases used by servers should be opened with handles created with the DB_XA_CREATE flag in the db_create() method and must be opened in the tpsvrinit routine. Note that the environment parameter of the db_create() method must be assigned NULL. For more information on the tpsvrinit routine, see http://download.oracle.com/docs/cd/E13161_01/tuxedo/docs10gr3/pgc/pgthr.html. diff --git a/docs-src/guides/programmer_reference/xa_xa_config.md b/docs-src/guides/programmer_reference/xa_xa_config.md new file mode 100644 index 000000000..db5f6b91a --- /dev/null +++ b/docs-src/guides/programmer_reference/xa_xa_config.md @@ -0,0 +1,84 @@ +--- +title: "Configuring Berkeley DB with the Tuxedo System" +api-name: "Configuring Berkeley DB with the Tuxedo System" +source: docs/programmer_reference/xa_xa_config.html +--- +## Configuring Berkeley DB with the Tuxedo System + + [Update the Resource Manager File in Tuxedo](xa_xa_config.md#idp52786896) + + [Build the Transaction Manager Server](xa_xa_config.md#idp52812512) + + [Update the UBBCONFIG File](xa_xa_config.md#idp52759288) + +To configure the Tuxedo system to use Berkeley DB resource managers, do the following: + +### Update the Resource Manager File in Tuxedo + +For the purposes of this discussion, assume that the Tuxedo home directory is in + +``` c +/home/tuxedo +``` + +In that case, the resource manager file will be located in + +``` c +/home/tuxedo/udataobj/RM +``` + +Edit the resource manager file to identify the Berkeley DB resource manager, the name of the resource manager switch, and the name of the library for the resource manager. + +For example, on a RedHat Linux Enterprise (64-bit) installation of Oracle Tuxedo 11gR1, you can update the resource manager file by adding the following line: + +``` c +BERKELEY-DB:db_xa_switch:-L${DB_INSTALL}/lib -ldb +``` + +where `${DB_INSTALL}` is the directory into which you installed the Berkeley DB library. + +Note that the load options may differ depending on the platform of your system. + +### Build the Transaction Manager Server + +To do this, use the Tuxedo **buildtms(1)** utility. The **buildtms** command will create the `Berkeley-DB` resource manager in the directory from which it was run. The parameters to the **buildtms** command should be: + +``` c +buildtms -v -o DBRM -r BERKELEY-DB +``` + +This will create an executable transaction manager server, `DBRM`, which is called by Tuxedo to process begins, commits, and aborts. + +### Update the UBBCONFIG File + +You must make sure that your TUXCONFIG environment variable identifies an UBBCONFIG file that properly identifies your resource managers. In the GROUPS section of the UBBCONFIG file, you should identify the group's LMID and GRPNO, as well as the transaction manager server name "TMSNAME=DBRM." You must also specify the OPENINFO parameter, setting it equal to the string + +``` c +rm_name:dir +``` + +where rm_name is the resource name specified in the RM file (that is, BERKELEY-DB) and dir is the directory for the Berkeley DB home environment (see DB_ENV->open() for a discussion of Berkeley DB environments). + +Because Tuxedo resource manager startup accepts only a single string for configuration, any environment customization that might have been done via the config parameter to DB_ENV->open() must instead be done by placing a DB_CONFIG configuration file in the Berkeley DB environment directory. See File naming for further information. + +Consider the following configuration. We have built a transaction manager server, as described previously. We want the Berkeley DB environment to be `/home/dbhome`, our database files to be maintained in `/home/datafiles`, our log files to be maintained in `/home/log`, and we want a duplexed server. + +The GROUPS section of the ubb file might look like the following: + +``` c +group_tm LMID=myname GRPNO=1 TMSNAME=DBRM TMSCOUNT=2 \ + OPENINFO="BERKELEY-DB:/home/dbhome" +``` + +There would be a DB_CONFIG configuration file in the directory `/home/dbhome` that contained the following two lines: + +``` c +set_data_dir /home/datafiles +set_lg_dir /home/log +``` + +Finally, the UBBCONFIG file must be translated into a binary version using Tuxedo's **tmloadcf**(1) utility, and then the pathname of that binary file must be specified as your TUXCONFIG environment variable. + +At this point, your system is properly initialized to use the Berkeley DB resource manager. + +See DB class for further information on accessing data files using XA. diff --git a/docs-src/guides/programmer_reference/xa_xa_intro.md b/docs-src/guides/programmer_reference/xa_xa_intro.md new file mode 100644 index 000000000..978a3bccb --- /dev/null +++ b/docs-src/guides/programmer_reference/xa_xa_intro.md @@ -0,0 +1,28 @@ +--- +title: "XA Introduction" +api-name: "XA Introduction" +source: docs/programmer_reference/xa_xa_intro.html +--- +## XA Introduction + +Berkeley DB can be used as an XA-compliant resource manager. The XA implementation is known to work with the Tuxedo transaction manager. + +The XA support is encapsulated in the resource manager switch db_xa_switch, which defines the following functions: + +- *\_\_db_xa_close.* Close the resource manager. +- *\_\_db_xa_commit.* Commit the specified transaction. +- *\_\_db_xa_complete.* Wait for asynchronous operations to complete. +- *\_\_db_xa_end.* Disassociate the application from a transaction. +- *\_\_db_xa_forget.* Forget about a transaction that was heuristically completed. (Berkeley DB does not support heuristic completion.) +- *\_\_db_xa_open.* Open the resource manager. +- *\_\_db_xa_prepare.* Prepare the specified transaction. +- *\_\_db_xa_recover.* Return a list of prepared, but not yet committed transactions. +- *\_\_db_xa_rollback.* Abort the specified transaction. +- *\_\_db_xa_start.* Associate the application with a transaction. + +The Berkeley DB resource manager does not support the following optional XA features: + +- Asynchronous operations +- Transaction migration + +The Tuxedo System is available from Oracle BEA Systems. diff --git a/docs-src/guides/programmer_reference/xa_xa_restrict.md b/docs-src/guides/programmer_reference/xa_xa_restrict.md new file mode 100644 index 000000000..6507b31e5 --- /dev/null +++ b/docs-src/guides/programmer_reference/xa_xa_restrict.md @@ -0,0 +1,39 @@ +--- +title: "Restrictions on XA Transactions" +api-name: "Restrictions on XA Transactions" +source: docs/programmer_reference/xa_xa_restrict.html +--- +## Restrictions on XA Transactions + +When you are using Berkeley DB for XA transactions, there are a few restrictions you should be aware of: + +- Configure environment using the DB_CONFIG file + + For most options, you must configure your environment via the DB_CONFIG file because an XA application or server cannot control the environment creation. + +- Snapshot isolation must be configured for the entire environment. + + Transactions managed by the Berkeley DB X/open compliant XA resource manager can be configured for transaction snapshots using either database open flags or the DB_CONFIG file file. To configure using database open flags, open the XA managed database with the flag DB_MULTIVERSION. When using DB_CONFIG, include both of the following lines: + + ``` c + set_flags DB_MULTIVERSION + set_flags DB_TXN_SNAPSHOT + ``` + + Note that both methods will results in all transactions using transaction snapshots, there is no way to enable transaction snapshots in just a subset of XA managed transactions. + +- No in-memory logging + + Upon return from xa_open, Berkeley DB checks to ensure there is no in-memory logging. If in-memory logging is detected, a FAILURE message is returned to the application. + +- No application-level child transactions + + Berkeley DB verifies in the xa_start and xa_end calls that no XA transaction has a parent. If application-level child transactions are detected, a FAILURE message is returned to the application. + +- All database-level operations, such as create, rename, and remove, must be performed in local BDB transactions, not distributed XA transactions + + Berkeley DB checks that there is no XA transaction currently active during these operations, and if detected, a FAILURE message is returned to the application. + +- Close cursors before a service invocation returns + + Berkeley DB checks in the `xa_end` call that the `DB_TXN` has no active cursors open and and if detected, a FAILURE message is returned to the application. diff --git a/docs-src/guides/upgrading/_meta.toml b/docs-src/guides/upgrading/_meta.toml new file mode 100644 index 000000000..973db5254 --- /dev/null +++ b/docs-src/guides/upgrading/_meta.toml @@ -0,0 +1,187 @@ +# Nav/index metadata for the upgrading guide (auto-derived from the +# source index.html TOC chain). `order` pins the reading order for nav +# and later PDF assembly; `landing` is the tree's index page. + +title = "Berkeley DB Upgrade Guide" +landing = "index.md" +order = [ + "preface", + "moreinfo", + "introduction", + "upgrade_process", + "upgrade_4_7_toc", + "upgrade_4_7_rtc", + "upgrade_4_7_repapi", + "upgrade_4_7_tcl", + "upgrade_4_7_interdir", + "upgrade_4_7_log", + "upgrade_4_7_disk", + "changelog_4_7", + "upgrade_4_6_toc", + "upgrade_4_6_cursor", + "upgrade_4_6_memp_fput", + "upgrade_4_6_memp_fset", + "upgrade_4_6_event", + "upgrade_4_6_full_election", + "upgrade_4_6_verbose", + "upgrade_4_6_verb", + "upgrade_4_6_win", + "upgrade_4_6_disk", + "changelog_4_6", + "upgrade_4_5_toc", + "upgrade_4_5_deprecate", + "upgrade_4_5_alive", + "upgrade_4_5_elect", + "upgrade_4_5_rep_set", + "upgrade_4_5_rep_event", + "upgrade_4_5_memp", + "upgrade_4_5_paniccall", + "upgrade_4_5_pagesize", + "upgrade_4_5_collect", + "upgrade_4_5_config", + "upgrade_4_5_source", + "upgrade_4_5_applog", + "upgrade_4_5_disk", + "changelog_4_5_20", + "upgrade_4_4_toc", + "upgrade_4_4_autocommit", + "upgrade_4_4_isolation", + "upgrade_4_4_joinenv", + "upgrade_4_4_mutex", + "upgrade_4_4_clear", + "upgrade_4_4_lockstat", + "upgrade_4_4_disk", + "changelog_4_4_16", + "changelog_4_4_20", + "upgrade_4_3_toc", + "upgrade_4_3_java", + "upgrade_4_3_err", + "upgrade_4_3_cput", + "upgrade_4_3_stat", + "upgrade_4_3_verb", + "upgrade_4_3_log", + "upgrade_4_3_fileopen", + "upgrade_4_3_enomem", + "upgrade_4_3_repl", + "upgrade_4_3_rtc", + "upgrade_4_3_disk", + "changelog_4_3_29", + "upgrade_4_2_toc", + "upgrade_4_2_java", + "upgrade_4_2_queue", + "upgrade_4_2_cksum", + "upgrade_4_2_client", + "upgrade_4_2_del", + "upgrade_4_2_priority", + "upgrade_4_2_verify", + "upgrade_4_2_lockng", + "upgrade_4_2_repinit", + "upgrade_4_2_nosync", + "upgrade_4_2_tcl", + "upgrade_4_2_disk", + "changelog_4_2_52", + "upgrade_4_1_toc", + "upgrade_4_1_excl", + "upgrade_4_1_fop", + "upgrade_4_1_log_register", + "upgrade_4_1_log_stat", + "upgrade_4_1_checkpoint", + "upgrade_4_1_incomplete", + "upgrade_4_1_memp_sync", + "upgrade_4_1_hash_nelem", + "upgrade_4_1_java", + "upgrade_4_1_cxx", + "upgrade_4_1_app_dispatch", + "upgrade_4_1_disk", + "changelog_4_1_24", + "changelog_4_1_25", + "upgrade_4_0_toc", + "upgrade_4_0_deadlock", + "upgrade_4_0_lock", + "upgrade_4_0_log", + "upgrade_4_0_mp", + "upgrade_4_0_txn", + "upgrade_4_0_env", + "upgrade_4_0_rpc", + "upgrade_4_0_set_lk_max", + "upgrade_4_0_lock_id_free", + "upgrade_4_0_java", + "upgrade_4_0_cxx", + "upgrade_4_0_asr", + "upgrade_4_0_disk", + "changelog_4_0_14", + "upgrade_3_3_toc", + "upgrade_3_3_rpc", + "upgrade_3_3_gettype", + "upgrade_3_3_getswap", + "upgrade_3_3_alloc", + "upgrade_3_3_conflict", + "upgrade_3_3_memp_fget", + "upgrade_3_3_txn_prepare", + "upgrade_3_3_shared", + "upgrade_3_3_bigfile", + "upgrade_3_3_disk", + "upgrade_3_2_toc", + "upgrade_3_2_set_flags", + "upgrade_3_2_callback", + "upgrade_3_2_renumber", + "upgrade_3_2_incomplete", + "upgrade_3_2_tx_recover", + "upgrade_3_2_mutexlock", + "upgrade_3_2_handle", + "upgrade_3_2_notfound", + "upgrade_3_2_db_dump", + "upgrade_3_2_disk", + "upgrade_3_1_toc", + "upgrade_3_1_config", + "upgrade_3_1_set_tx_recover", + "upgrade_3_1_set_feedback", + "upgrade_3_1_set_paniccall", + "upgrade_3_1_put", + "upgrade_3_1_dup", + "upgrade_3_1_btstat", + "upgrade_3_1_sysmem", + "upgrade_3_1_log_register", + "upgrade_3_1_memp_register", + "upgrade_3_1_txn_check", + "upgrade_3_1_env", + "upgrade_3_1_tcl", + "upgrade_3_1_tmp", + "upgrade_3_1_logalloc", + "upgrade_3_1_disk", + "upgrade_3_0_toc", + "upgrade_3_0_envopen", + "upgrade_3_0_func", + "upgrade_3_0_dbenv", + "upgrade_3_0_open", + "upgrade_3_0_xa", + "upgrade_3_0_db", + "upgrade_3_0_dbinfo", + "upgrade_3_0_join", + "upgrade_3_0_stat", + "upgrade_3_0_close", + "upgrade_3_0_lock_put", + "upgrade_3_0_lock_detect", + "upgrade_3_0_lock_stat", + "upgrade_3_0_log_register", + "upgrade_3_0_log_stat", + "upgrade_3_0_memp_stat", + "upgrade_3_0_txn_begin", + "upgrade_3_0_txn_commit", + "upgrade_3_0_txn_stat", + "upgrade_3_0_rmw", + "upgrade_3_0_lock_notheld", + "upgrade_3_0_eagain", + "upgrade_3_0_eacces", + "upgrade_3_0_jump_set", + "upgrade_3_0_value_set", + "upgrade_3_0_dbenv_cxx", + "upgrade_3_0_db_cxx", + "upgrade_3_0_cxx", + "upgrade_3_0_java", + "upgrade_3_0_disk", + "upgrade_2_0_toc", + "upgrade_2_0_system", + "upgrade_2_0_convert", + "upgrade_2_0_disk", +] diff --git a/docs-src/guides/upgrading/changelog_4_0_14.md b/docs-src/guides/upgrading/changelog_4_0_14.md new file mode 100644 index 000000000..6de3a1021 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_0_14.md @@ -0,0 +1,175 @@ +--- +title: "4.0.14 Change Log" +api-name: "4.0.14 Change Log" +source: docs/upgrading/changelog_4_0_14.html +--- +## 4.0.14 Change Log + + [Major New Features:](changelog_4_0_14.md#idp51113768) + + [General Environment Changes:](changelog_4_0_14.md#idp51101344) + + [General Access Method Changes:](changelog_4_0_14.md#idp51103296) + + [Btree Access Method Changes:](changelog_4_0_14.md#idp51105152) + + [Hash Access Method Changes:](changelog_4_0_14.md#idp51109416) + + [Queue Access Method Changes:](changelog_4_0_14.md#idp51112664) + + [Recno Access Method Changes:](changelog_4_0_14.md#idp51113832) + + [C++ API Changes:](changelog_4_0_14.md#idp51115760) + + [Java API Changes:](changelog_4_0_14.md#idp51126328) + + [Tcl API Changes:](changelog_4_0_14.md#idp51116840) + + [RPC Client/Server Changes:](changelog_4_0_14.md#idp51117920) + + [XA Resource Manager Changes:](changelog_4_0_14.md#idp51118608) + + [Locking Subsystem Changes:](changelog_4_0_14.md#idp51118928) + + [Logging Subsystem Changes:](changelog_4_0_14.md#idp51103680) + + [Memory Pool Subsystem Changes:](changelog_4_0_14.md#idp51122816) + + [Transaction Subsystem Changes:](changelog_4_0_14.md#idp51109800) + + [Utility Changes:](changelog_4_0_14.md#idp51113048) + + [Database or Log File On-Disk Format Changes:](changelog_4_0_14.md#idp51125248) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_0_14.md#idp51126712) + +### Major New Features: + +1. Group commit. \[#42\] +2. Single-master replication. \[#44\] +3. Support for VxWorks AE; Vxworks support certified by WindRiver Systems Inc. \[#4401\] + +### General Environment Changes: + +1. The db_env_set_pageyield interface has been replaced by a new flag (DB_YIELDCPU) for the DB_ENV-\>set_flags interface. +2. The db_env_set_panicstate interface has been replaced by a new flag (DB_PANIC_STATE) for the DB_ENV-\>set_flags interface. +3. The db_env_set_region_init interface has been replaced by a new flag (DB_REGION_INIT) for the DB_ENV-\>set_flags interface. +4. The db_env_set_tas_spins interface has been replaced by the DB_ENV-\>set_tas_spins method. +5. The DB_ENV-\>set_mutexlocks interface has been replaced by a new flag (DB_NOLOCKING) for the DB_ENV-\>set_flags interface. +6. Fix a bug where input values from the DB_CONFIG file could overflow. +7. The C API lock, log, memory pool and transaction interfaces have been converted to method based interfaces; see the Upgrade documentation for specific details. \[#920\] +8. Fix a bug in which some DB_ENV configuration information could be lost by a failed DB_ENV-\>open command. \[#4608\] +9. Fix a bug where Berkeley DB could fail if the application attempted to allocate new database pages while the system was unable to write new log file buffers. \[#4928\] + +### General Access Method Changes: + +1. Add a new flag (DB_GET_BOTH_RANGE) that adds support for range searches within sorted duplicate data sets. \[#3378\] +2. Fix a bug in which the DB-\>get or DB-\>pget methods, when used with secondary indices, could incorrectly leave an internally-created database cursor open. \[#4465\] +3. The DB-\>set_alloc method can no longer be called when the database is part of a database environment. \[#4599\] + +### Btree Access Method Changes: + +1. Fix a bug where a lock could be leaked when a thread calling DB-\>stat on a Btree database was selected to resolve a deadlock. \[#4509\] + +### Hash Access Method Changes: + +1. Fix a bug where bulk return using the MULTIPLE_KEY flag on a Hash database would only return entries from a single bucket. \[#4313\] + +### Queue Access Method Changes: + +1. Delete extent files whenever the leading record is deleted, instead of only when a DB_CONSUME operation was performed. \[#4307\] + +### Recno Access Method Changes: + +1. Fix a bug where the delete of a record in a Recno database could leak a lock in non-transactional applications. \[#4351\] +2. Fix a bug where the DB_THREAD flag combined with a backing source file could cause an infinite loop. \[#4581\] + +### C++ API Changes: + +> None. + +### Java API Changes: + +1. Added implementation of DbEnv.lock_vec for Java. \[#4094\] Added some minimal protection so that the same Java Dbt cannot be used twice in the same API call, this will often catch multithreading programming errors with Dbts. \[#4094\] +2. Fix a bug in which a Db.put call with the Db.DB_APPEND would fail to correctly return the newly put record's record number. \[#4527\] +3. Fixed problems occurring in multithreaded java apps that use callbacks. \[#4467\] + +### Tcl API Changes: + +1. Fix a bug in which large integers could be handled incorrectly by the Tcl interface on 64-bit machines. \[#4371\] + +### RPC Client/Server Changes: + +1. The DB_ENV-\>set_server interface has been removed. + +### XA Resource Manager Changes: + +> None. + +### Locking Subsystem Changes: + +1. The C++ (Java) API DbLock::put (DbLock.put) method has been changed to be a method off the DbEnv handle rather than the DbLock handle. +2. Locker IDs may now wrap-around. \[#864\] +3. Explicitly allocated locker IDs must now be freed. \[#864\] +4. Add per-environment, per-lock and per-transaction interfaces to support timeout based lock requests and "deadlock" detection. \[#1855\] +5. Add support for interrupting a waiting locker. \[#1976\] +6. Implemented DbEnv.lock_vec for Java. \[#4094\] + +### Logging Subsystem Changes: + +1. Fix a bug where the size of a log file could not be set to the default value. \[#4567\] +2. Fix a bug where specifying a non-default log file size could cause other processes to be unable to join the environment and read its log files. \[#4567\] +3. Fix a bug where Berkeley DB could keep open file descriptors to log files returned by the DB_ENV-\>log_archive method (or the db_archive utility), making it impossible to move or remove them on Windows systems. \[#3969\] +4. Replace the log_get interface with a cursor into the log file. \[#0043\] + +### Memory Pool Subsystem Changes: + +1. Add the DB_ODDFILESIZE flag to the DB_MPOOLFILE-\>open method supporting files not a multiple of the underlying page size in length. +2. Convert memp_XXX functional interfaces to a set of methods, either base methods off the DB_ENV handle or methods off of a DB_MPOOLFILE handle. \[#920\] +3. Add the DB_ODDFILESIZE flag to the DB_MPOOLFILE-\>open method supporting files not a multiple of the underlying page size in length. +4. Fix a bug where threads of control could deadlock opening a database environment with multiple memory pool caches. \[#4696\] +5. Fix a bug where the space needed for per-file memory pool statistics was incorrectly calculated. \[#4772\] + +### Transaction Subsystem Changes: + +1. Transaction IDs may now wrap-around. \[#864\] +2. Release read locks before performing logging operations at commit. \[#4219\] + +### Utility Changes: + +1. Fix a bug in which the db_dump utility would incorrectly attach to transaction, locking, or logging regions when salvaging, and thus could not be used to salvage databases in environments where these regions were present. \[#4305\] +2. Fix a bug in which the DB salvager could produce incorrectly formatted output for certain classes of corrupt database. \[#4305\] +3. Fix a bug in which the DB salvager could incorrectly salvage files containing multiple databases. \[#4305\] +4. Fix a bug where unprintable characters in subdatabase names could cause a dump of a database that could not then be loaded. \[#4688\] +5. Increase the size of the cache created by the db_stat and db_verify utilities to avoid failure on large databases. \[#4688\] \[#4787\] +6. Fix a bug in which a database verification performed with the DB_ORDERCHKONLY flag could fail incorrectly. \[#4757\] +7. Fix a bug which caused db_stat to display incorrect information about GB size caches. \[#4812\] + +### Database or Log File On-Disk Format Changes: + +1. The on-disk log format changed. + +### Configuration, Documentation, Portability and Build Changes: + +1. Fix a bug where Win9X systems region names could collide. +2. Fix a bug where configuring Berkeley DB to build the C++ API without also configuring for a shared library build would fail to build the C++ library. \[#4343\] +3. Change Berkeley DB installation to not strip binaries if --enable-debug was specified as a configuration option. \[#4318\] +4. Add the -pthread flag to AIX, FreeBSD and OSF/1 library loads. \[#4350\] +5. Fix a bug where the Berkeley DB 1.85 compatibility API failed to load in the 3.3.11 release. \[#4368\] +6. Port the Berkeley DB utility programs to the VxWorks environment. \[#4378\] +7. Made change to configuration so that dynamic libraries link correctly when C++ is used on AIX. \[#4381\] +8. Fix a variety of problems that prevented the Berkeley DB source tree from building on systems without ANSI C compiler support (for example, SunOS 4.X). \[#4398\] +9. Added missing DbMultiple\*Iterator Java files to Makefile.in. \[#4404\] +10. Fix a bug that could prevent the db_dump185 utility from dumping Berkeley DB version 1.86 hash databases. \[#4418\] +11. Reduce the number of calls setting the errno value, to improve performance on Windows/NT in MT environments. \[#4432\] +12. Fix for Darwin (and probably some other) OS's that were getting 'yes' or other garbage in generated makefiles in place of a shared library name. \[#4453\] +13. C++: Remove inlining for constructor of tmpString internal class. This fixes warnings on Solaris profiling builds. \[#4473\] +14. DB now restarts system calls that are interrupted by signals. \[#4480\] +15. Fixed warnings for compiling Java native code on Solaris and OSF/1. \[#4571\] +16. Added better configuration for Java on Tru64 (OSF/1), Solaris, +17. Java files are now built as jar files. Berkeley DB classes are put into db.jar (which is an installed file on UNIX) and examples are put into dbexamples.jar. The classes directory is now a subdirectory of the build directory, rather than in java/classes. \[#4575\] +18. Support Cygwin installation process. \[#4611\] +19. Correct the Java secondary_key_create method signature. \[#4777\] +20. Export additional Berkeley DB interfaces on Windows to support application-specific logging and recovery. \[#4827\] +21. Always complain when using version 2.96 of the gcc compiler. \[#4878\] +22. Add compile and load-time flags to configure for threads on UnixWare and OpenUNIX. \[#4552\] \[#4950\] diff --git a/docs-src/guides/upgrading/changelog_4_1_24.md b/docs-src/guides/upgrading/changelog_4_1_24.md new file mode 100644 index 000000000..af2fbf74c --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_1_24.md @@ -0,0 +1,242 @@ +--- +title: "Berkeley DB 4.1.24 and 4.1.25 Change Log" +api-name: "Berkeley DB 4.1.24 and 4.1.25 Change Log" +source: docs/upgrading/changelog_4_1_24.html +--- +## Berkeley DB 4.1.24 and 4.1.25 Change Log + + [Database or Log File On-Disk Format Changes:](changelog_4_1_24.md#idp50963888) + + [Major New Features:](changelog_4_1_24.md#idp50959088) + + [General Environment Changes:](changelog_4_1_24.md#idp50962280) + + [General Access Method Changes:](changelog_4_1_24.md#idp50961984) + + [Btree Access Method Changes:](changelog_4_1_24.md#idp50964272) + + [Hash Access Method Changes:](changelog_4_1_24.md#idp50967400) + + [Queue Access Method Changes:](changelog_4_1_24.md#idp50969240) + + [Recno Access Method Changes:](changelog_4_1_24.md#idp50972088) + + [C++-specific API Changes:](changelog_4_1_24.md#idp50973928) + + [Java-specific API Changes:](changelog_4_1_24.md#idp50975768) + + [Tcl-specific API Changes:](changelog_4_1_24.md#idp50950328) + + [RPC-specific Client/Server Changes:](changelog_4_1_24.md#idp50958680) + + [Replication Changes:](changelog_4_1_24.md#idp50977144) + + [XA Resource Manager Changes:](changelog_4_1_24.md#idp50964336) + + [Locking Subsystem Changes:](changelog_4_1_24.md#idp50987264) + + [Logging Subsystem Changes:](changelog_4_1_24.md#idp50989192) + + [Memory Pool Subsystem Changes:](changelog_4_1_24.md#idp50992072) + + [Transaction Subsystem Changes:](changelog_4_1_24.md#idp50993160) + + [Utility Changes:](changelog_4_1_24.md#idp50994744) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_1_24.md#idp50997648) + +### Database or Log File On-Disk Format Changes: + +1. All of the access method database formats changed in the Berkeley DB 4.1 release (Btree/Recno: version 8 to version 9, Hash: version 7 to version 8, and Queue: version 3 to version 4). *The format changes are entirely backward-compatible, and no database upgrades are needed.* + +### Major New Features: + +1. Berkeley DB now includes support for database encryption using the AES encryption standard. \[#1797\] +2. Berkeley DB now includes support for database page checksums to allow detection of database corruption during I/O. \[#1797\] +3. The shared memory buffer pool code base was substantially reworked in the 4.1 release to improve concurrent throughput. \[#4655\] + +### General Environment Changes: + +1. Allow applications to specify transaction handles to the DB-\>open method call, so database creation can be grouped with other Berkeley DB calls in a single transaction. \[#4257\] +2. Add the DB_ENV-\>remove and DB_ENV-\>rename method calls that support transactional protection of database removal and renaming. \[#4257\] +3. Add the DB_ENV-\>set_flags flags DB_DIRECT_DB and DB_DIRECT_LOG, which disable the system's buffer cache where possible. \[#4526\] +4. Unlock the pthread mutex if pthread_cond_wait() returns an error. \[#4872\] +5. Fix a memory leak caused by running recovery. \[#4913\] +6. Fix a bug in which closing an environment with open database handles could result in application crashes. \[#4991\] +7. Fix a bug where DB_CONFIG files were ignored if the database environment defaulted to the application's current working directory. \[#5265\] +8. Fix a bug where transaction abort or commit could fail to destroy the handle. \[#5633\] +9. Fix a set of bugs where the Berkeley DB API could return DB_RUNRECOVERY without panic-ing the database environment itself or calling the application's panic-callback function. \[#5743\] +10. Fix a bug in where DB=\>rename and DB-\>remove method calls could leak a transaction and its locks. \[#5824\] +11. Fix a bug where recovery feedback could return values greater than 100. \[#6193\] +12. Fix a bug where a page allocated by a transaction, eventually aborted because of application or system failure, could appear twice in the free list, if catastrophic recovery was performed. \[#6222\] +13. Add a new flag, DB_AUTO_COMMIT, that wraps all database modification operations inside a transaction, to the DB_ENV-\>set_flags method. \[#6395\] +14. Fix a bug where recovery could fail when upgrading between releases. \[#6372\] +15. Fix a recovery bug where pages that were repeatedly freed and allocated could be lost. \[#6479\] \[#6501\] +16. Change DB_CONFIG reading to handle non-\ terminated last line. \[#6490\] + +### General Access Method Changes: + +1. Allow applications to specify transaction handles to the DB-\>associate method call, so secondary index creation can be grouped with other Berkeley DB calls in a single transaction. \[#4185\] +2. Add a new flag, DB_AUTO_COMMIT, that wraps single database operations inside a transaction. This flag is supported by the DB-\>del, DB-\>open, DB-\>put, DB-\>truncate,DB_ENV-\>remove, and DB_ENV-\>rename methods. \[#4257\] +3. The DB_EXCL DB-\>open method flag has been enhanced to work on subdatabases. \[#4257\] +4. Fix a bug in which a DB-\>put(DB_APPEND) could result in leaked memory or a corruption in the returned record number. \[#5002\] +5. Fix a bug in the database salvage code that could leave pages pinned in the cache. \[#5037\] +6. Add a flag to the DB-\>verify method to output salvaged key/data pairs in printable characters. \[#5037\] +7. Fix a bug in which DB-\>verify() might continue and report extraneous database corruption after a fatal error. \[#5131\] +8. Fix a bug where calling the DB-\>stat method before the DB-\>open method could drop core. \[#5190\] +9. Fix a bug in which a DB-\>get, DBcursor-\>c_get, or DBcursor-\>c_pget on a secondary index, in the Concurrent Data Store product, could result in a deadlock. \[#5192\] +10. Fix a bug in which DB-\>verify() could correctly report errors but still return success. \[#5297\] +11. Add support for the DB-\>set_cache_priority interface, that allows applications to set the underlying cache priority for their database files. \[#5375\] +12. Fix a bug where calling DBcursor-\>c_pget with a database that is not a secondary index would drop core. \[#5391\] +13. Fix a bug where a bug in the DB-\>truncate method could cause recovery to fail. \[#5679\] +14. Fix a bug where DB_GET_RECNO would fail if specified to a secondary index. \[#5811\] +15. Fix a bug where building a secondary index for an existing primary database could fail in Concurrent Data Store environments. \[#5811\] +16. Fix a bug where the DB-\>rename method could fail, causing a problem during recovery. \[#5893\] +17. Fix a bug in which a DB-\>get or DB-\>pget call on a secondary index could fail when done with a handle shared among multiple threads. \[#5899\] +18. Fix a bug in which a DB-\>put operation on a database with off-page duplicates could leak a duplicate cursor, thereby preventing transactions being able to commit. \[#5936\] +19. Fix a bug where overflow page reference counts were not properly maintained when databases were truncated. \[#6168\] +20. Fix a bug where the bulk get APIs could allocate large amounts of heap memory. \[#6439\] \[#6520\] + +### Btree Access Method Changes: + +1. Fix a bug that prevented loads of sorted data, with duplicates at the end of the tree, from creating compact trees. \[#4926\] +2. No longer return a copy of the key if the DB_GET_BOTH or DB_GET_BOTH_RANGE flags are specified. \[#4470\] +3. Fix a bug where the fast-search code could hold an unlocked reference to a page, which could lead to recovery failure. \[#5518\] +4. Fix a bug where some cursor operations on a database, for which the bt_minkey size had been specified, could fail to use the correct overflow key/data item size. \[#6183\] +5. Fix a bug where the recovery of an aborted transaction that did a reverse Btree split might leave a page in an inconsistent state. \[#6393\] + +### Hash Access Method Changes: + +1. Fix bugs that could cause hash recovery to drop core. \[#4978\] +2. Use access method flags instead of interface flags to check for read-only access to a hash database with an application-specified hash function. \[#5121\] +3. Fix a bug where a hash database allocation of a new set of buckets may be improperly recovered by catastrophic recovery if the transaction is split across log files and the beginning segment of the transaction is not included in the set of logs to be recovered. \[#5942\] +4. Fix a bug where aborting particular hash allocations could lead to a database on which the verifier would loop infinitely. \[#5966\] +5. Fix a bug where a memory allocation failure could result in a system hang. \[#5988\] +6. Remove nelem from the Hash access method statistics (the value was incorrect once items had been added or removed from the database). \[#6101\] +7. Fix a bug where a page allocated by an aborted transaction might not be placed on the free list by recovery, if the file holding the page was created as part of recovery, and a later page was part of a hash bucket allocation. \[#6184\] +8. Fix a bug where allocated pages could be improperly recovered on systems that require explicit zero-ing of filesystem pages. \[#6534\] + +### Queue Access Method Changes: + +1. No longer return a copy of the key if the DB_SET_RANGE flag is specified. \[#4470\] +2. Fix a bug where DBcursor-\>c_get (with DB_MULTIPLE or DB_MULTIPLE_KEY specified) could fail on a Queue database if the record numbers had wrapped. \[#6397\] + +### Recno Access Method Changes: + +1. No longer return a copy of the key if the DB_GET_BOTH or DB_GET_BOTH_RANGE flags are specified. \[#4470\] +2. Fix a bug where non-transactional locking applications could leak locks when modifying Recno databases. \[#5766\] +3. Fix a bug where DBcursor-\>c_get with the DB_GET_RECNO flag would panic the environment if the cursor was uninitialized. \[#5935\] +4. Fix a bug where deleting pages from a three-level Recno tree could cause the database environment to panic. \[#6232\] + +### C++-specific API Changes: + +1. C++ DbLock::put is replaced by DbEnv::lock_put to match the C and Java API change in Release 4.0. \[#5170\] +2. Declared destructors and methods within Db and DbEnv classes to be virtual, making subclassing safer. \[#5264\] +3. Fixed a bug where Dbt objects with no flags set would not be filled with data by some operations. \[#5706\] +4. Added DbDeadlockException, DbRunRecoveryException, and DbLockNotGrantedException classes to C++, and throw them accordingly. \[#6134\] +5. Added C++ methods to support remaining conversions between C++ classes and C structs where appropriate. In particular, DbTxn/DB_TXN conversions and DbMpoolFile/DB_MPOOLFILE were added. \[#6278\] +6. Fix a bug in DbEnv::~DbEnv() that could cause memory corruption if a DbEnv was deleted without being closed. \[#6342\] +7. Reordered C++ class declarations to avoid a GCC g++ warning about function inlining. \[#6406\] +8. Fix a bug in the DbEnv destructor that could cause memory corruption when an environment was destroyed without closing first. \[#6342\] +9. Change DbEnv and Db destructor behavior to close the handle if it was not already closed. \[#6342\] + +### Java-specific API Changes: + +1. Added check for system property "sleepycat.Berkeley DB.libfile" that can be used to specify a complete pathname for the JNI shared library. This is needed as a workaround on Mac OS X, where libtool cannot currently create a library with a .jnilib extension which is what the current JDK expects by default. \[#5664\] +2. Fixed handling of JVM out of memory conditions, when some JNI methods return NULL. When the JVM runs out of memory, calls should consistently fail with OutOfMemoryErrors. \[#5995\] +3. Added Dbt.get_object and Dbt.set_object convenience routines to the Java API to make using serialization easier. \[#6113\] +4. Fixed a bug that prevented Java's Db.set_feedback from working, fixed document for Java's Db.set_feedback, some callback methods were misnamed. \[#6137\] +5. Fix a NullPointerException in Db.finalize() if the database had been closed. \[#6504\] +6. Marked DbEnv constructor with "throws DbException". \[#6342\] + +### Tcl-specific API Changes: + +None. + +### RPC-specific Client/Server Changes: + +1. Fix a bug where Db and DbEnv handles were not thread-safe. \[#6102\] + +### Replication Changes: + +1. A large number of replication bugs were fixed in this release. The replication support is now believed to be production quality. +2. Add the DB_ENV-\>set_rep_limit interface, allowing applications to limit the data sent in response to a single DB_ENV-\>rep_process_message call. \[#5999\] +3. Add the DB_ENV-\>set_rep_stat interface, returning information from the replication subsystem \[#5919\] + +### XA Resource Manager Changes: + +1. Added support for multithreaded XA. Environments can now have multiple XA transactions active. db_env_xa_attach() can be used to get a DB_TXN that corresponds to the XA transaction in the current thread. \[#5049\] +2. Added a com.sleepycat.Berkeley DB.xa package that implements J2EE support for XA. This includes new DbXAResource, DbXid classes that implement the XAResource and Xid interfaces. \[#5049\] +3. Fix a bug where aborting a prepared transaction after recovery may fail. \[#6383\] +4. Fix a bug where recovery might fail if a prepared transaction had previously extended the size of a file and then was aborted. \[#6387\] +5. Fix a bug where if the commit of a prepared transaction fails the transaction would be aborted. \[#6389\] + +### Locking Subsystem Changes: + +1. Fix a bug where lock counts were incorrect if a lock request returned DB_LOCK_NOTGRANTED or an error occurred. \[#4923\] +2. Fix a bug where lock downgrades were counted as releases, so the lock release statistics could be wrong. \[#5762\] +3. Fix a bug where the lock and transaction timeout values could not be reset by threads of control joining Berkeley DB database environments. \[#5996\] +4. Fix a bug where applications using lock and/or transaction timeouts could hit a race condition that would lead to a segmentation fault. \[#6061\] + +### Logging Subsystem Changes: + +1. DB_ENV-\>log_register and DB_ENV-\>log_unregister have been removed from the interface. \[#0046\] +2. Fix a bug where creating a database environment with a nonexistent logging directory could drop core. \[#5833\] +3. Add support allowing applications to change the log file size in existing database environments. \[#4875\] +4. Fix a bug where a write error on a log record spanning a buffer could cause transaction abort to fail and the database environment to panic. \[#5830\] + +### Memory Pool Subsystem Changes: + +1. The DB_INCOMPLETE error has been removed, as cache flushing can no longer return without completing. \[#4655\] +2. Fix a bug where Berkeley DB might refuse to open a file if the open was attempted while another thread was writing a large buffer. \[#4885\] +3. Prefer clean buffers to dirty buffers when selecting a buffer for eviction. \[#4934\] +4. Fix a bug where transaction checkpoint might miss flushing a buffer to disk. \[#5033\] +5. Fix a bug where Berkeley DB applications could run out of file descriptors. \[#5535\] +6. Fix bugs where Berkeley DB could self-deadlock on systems requiring mutex resource reclamation after application failure. \[#5722\] \[#6523\] + +### Transaction Subsystem Changes: + +1. Go back only one checkpoint, not two, when performing normal recovery. \[#4284\] +2. Fix a bug where an abort of a transaction could fail if there was no disk space for the log. \[#5740\] +3. Fix a bug where the checkpoint log-sequence-number could reference a nonexistent log record. \[#5789\] +4. Fix a bug where subtransactions which allocated pages from the filesystem and subsequently aborted could cause other pages allocated by sibling transactions to not be freed if the parent transaction then aborted. \[#5903\] +5. Fix a bug where transactions doing multiple updates to a queue database which spanned a checkpoint could be improperly handled by recovery. \[#5898\] + +### Utility Changes: + +1. Fix a bug where the -p option could not be specified with the -R or -r options. \[#5037\] +2. The utilities were modified to correctly size their private caches in order to handle databases with large page sizes. \[#5055\] +3. Fix a bug in which utilities run with the -N option would fail to ignore the environment's panic flag. \[#5082\] +4. Fix a bug where invalid log records could cause db_printlog to drop core. \[#5173\] +5. Add a new option to the db_verify utility to support verification of files that include databases having non-standard sorting or hash functions. \[#5237\] + +### Configuration, Documentation, Portability and Build Changes: + +1. Replace test-and-set mutexes on Windows with a new mutex implementation that signals an event to wake blocked threads. \[#4413\] +2. Support configuration of POSIX pthread mutexes on systems where the pthread mutexes do not support inter-process locks. \[#4942\] +3. Add mutex support for the ARM architecture using the gcc compiler. \[#5018\] +4. On Windows NT/2000/XP, switched to atomic seek-and-read/write operations to improve performance of concurrent reads \[#0654\]. +5. Support cross-compilation using the GNU compiler tool chain. \[#4558\] +6. Fix a bug where libraries were always installed read-only. \[#5096\] +7. Fix a bug where temporary files on VxWorks could fail. \[#5160\] +8. Fix a bug where Berkeley DB did not install correctly if the system cp utility did not support the -f option. \[#5111\] +9. Correct the documentation for the Queue access method statistics field qs_cur_recno to be the "Next available record number". \[#5190\] +10. Fix a bug where file rename could fail on Windows/9X. \[#5223\] +11. Removed support for Microsoft Visual Studio 5.0 \[#5231\] +12. Switched to using HANDLEs for all I/O operations on Windows to overcome a hard limit of 2048 open file descriptors in Microsoft's C runtime library. \[#5249\] +13. Fix a bug where Berkeley DB error message routines could drop core on the PowerPC and UltraSPARC architectures. \[#5331\] +14. Rename OSTREAMCLASS to \_\_DB_OSTREAMCLASS in db_cxx.h to avoid stepping on application name space. \[#5402\] +15. Support Linux on the S/390 architecture. \[#5608\] +16. Work around a bug in Solaris where the pthread_cond_wait call could return because a signal was delivered to the application. \[#5640\] +17. Fix build line for loadable libraries to include -module to support Mac OS X. \[#5664\] +18. Fix a bug in the PPC mutex support for the Mac OS X system. \[#5781\] +19. Added support for Java on Mac OS X. A workaround on the Java command line is currently necessary; it is documented. \[#5664\] +20. Added support for Tcl on Mac OS X. \[#5664\] +21. Update Windows build instructions to cover Visual C++ .NET. \[#5684\] +22. AIX configuration changes for building on AIX 4.3.3 and 5 with both standard and Visual Age compilers. \[#5779\] +23. Add a new UNIX configuration argument, --with-mutex=MUTEX, to allow applications to select a mutex implementation. \[#6040\] +24. Changed libtool and configure so we can now correctly build and install Tcl and Java loadable shared libraries that work on Mac OS X. \[#6117\] +25. Fix mutex alignment problems on historic HP-UX releases that could make multiprocess applications fail. \[#6250\] +26. Installed static .a archives on Mac OS X need to be built with the ranlib -c option so linked applications will not see undefined \_\_db_jump errors. \[#6215\] +27. Upgrade pthread and mmap support in the uClibc library to support Berkeley DB. \[#6268\] +28. Fixed error in determining include directories during configuration for --enable-java. The error can cause compilation errors on certain systems with newer versions of gcc. \[#6445\] diff --git a/docs-src/guides/upgrading/changelog_4_1_25.md b/docs-src/guides/upgrading/changelog_4_1_25.md new file mode 100644 index 000000000..31f7064b8 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_1_25.md @@ -0,0 +1,8 @@ +--- +title: "Berkeley DB 4.1.25 Change Log" +api-name: "Berkeley DB 4.1.25 Change Log" +source: docs/upgrading/changelog_4_1_25.html +--- +## Berkeley DB 4.1.25 Change Log + +Berkeley DB version 4.1.25 is version 4.1.24 with all public patches applied. There were no public interface changes or new features. diff --git a/docs-src/guides/upgrading/changelog_4_2_52.md b/docs-src/guides/upgrading/changelog_4_2_52.md new file mode 100644 index 000000000..f17e76b6d --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_2_52.md @@ -0,0 +1,427 @@ +--- +title: "Berkeley DB 4.2.52 Change Log" +api-name: "Berkeley DB 4.2.52 Change Log" +source: docs/upgrading/changelog_4_2_52.html +--- +## Berkeley DB 4.2.52 Change Log + + [Database or Log File On-Disk Format Changes:](changelog_4_2_52.md#idp50822856) + + [New Features:](changelog_4_2_52.md#idp50784344) + + [Database Environment Changes:](changelog_4_2_52.md#idp50809288) + + [Concurrent Data Store Changes:](changelog_4_2_52.md#idp50822104) + + [General Access Method Changes:](changelog_4_2_52.md#idp50824288) + + [Btree Access Method Changes:](changelog_4_2_52.md#idp50825368) + + [Hash Access Method Changes:](changelog_4_2_52.md#idp50844704) + + [Queue Access Method Changes:](changelog_4_2_52.md#idp50828568) + + [Recno Access Method Changes:](changelog_4_2_52.md#idp50858440) + + [C++-specific API Changes:](changelog_4_2_52.md#idp50832248) + + [Java-specific API Changes:](changelog_4_2_52.md#idp50815840) + + [Tcl-specific API Changes:](changelog_4_2_52.md#idp50867864) + + [RPC-specific Client/Server Changes:](changelog_4_2_52.md#idp50852544) + + [Replication Changes:](changelog_4_2_52.md#idp50858528) + + [XA Resource Manager Changes:](changelog_4_2_52.md#idp50877816) + + [Locking Subsystem Changes:](changelog_4_2_52.md#idp50865088) + + [Logging Subsystem Changes:](changelog_4_2_52.md#idp50868008) + + [Memory Pool Subsystem Changes:](changelog_4_2_52.md#idp50865504) + + [Transaction Subsystem Changes:](changelog_4_2_52.md#idp50845064) + + [Utility Changes:](changelog_4_2_52.md#idp50858944) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_2_52.md#idp50892568) + +### Database or Log File On-Disk Format Changes: + +1. Queue databases that use encryption or data checksum features with extent files will need to be dumped and reloaded prior to using with release 4.2. For more details, see Queue access method. \[#8671\] +2. The on-disk log format changed. + +### New Features: + +1. Add support for a reduced memory footprint build of the Berkeley DB library. \[#1967\] +2. Add the DB_MPOOLFILE-\>set_flags interface which disallows the creation of backing filesystem files for in-memory databases. \[#4224\] +3. Add cache interfaces to limit the number of buffers written sequentially to allow applications to bound the time they will monopolize the disk. \[#4935\] +4. Support auto-deletion of log files. \[#0040\] \[#6252\] +5. The new Java DBX API for Berkeley DB allows Java programmers to use a familiar Java Collections style API, including Map, while interacting with the transactional Berkeley DB core engine. \[#6260\] +6. Support auto-commit with the DB-\>get method's consume operations. \[#6954\] +7. Add "get" methods to retrieve most settings. \[#7061\] +8. Add Javadoc documentation to the Berkeley DB release. \[#7110\] +9. Add support to Concurrent Data Store to allow duplication of write cursors. \[#7167\] +10. Add C++ utility classes for iterating over multiple key and data items returned from a cursor when using the DB_MULTIPLE or DB_MULTIPLE_KEY flags. \[#7351\] +11. Add CamelCased methods to the Java API. \[#7396\] +12. Add the DB_MPOOLFILE-\>set_maxsize interface to enforce a maximum database size. \[#7582\] +13. Add a toString() method for all Java \*Stat classes (DbBtreeStat, DbHashStat, DbMpoolStat, etc.). This method creates a listing of values of all of the class member variables. \[#7712\] + +### Database Environment Changes: + +1. Add cache interfaces to limit the number of buffers written sequentially to allow applications to bound the time they will monopolize the disk. \[#4935\] +2. Fix a bug which could cause database environment open to hang, in database environments supporting cryptography. \[#6621\] +3. Fix a bug where a database environment panic might result from an out-of-disk-space error while rolling back a page allocation. \[#6694\] +4. Fix a bug where a database page write failure, in a database environment configured for encryption or byte-swapping, could cause page corruption. \[#6791\] +5. Fix a bug where DB-\>truncate could drop core if there were active cursors in the database. \[#6846\] +6. Fix a bug where for databases sharing a physical file required a file descriptor per database. \[#6981\] +7. Fix a bug where the panic callback routine was only being called in the first thread of control to detect the error when returning DB_RUNRECOVERY. \[#7019\] +8. Fix a bug where a transaction which contained a remove of a subdatabase and an allocation to another subdatabase in the same file might not properly be aborted. \[#7356\] +9. Fix a bug to now disallow DB_TRUNCATE on opens in locking environments, since we cannot prevent race conditions. In the absence of locking and transactions, DB_TRUNCATE will truncate ANY file for which the user has appropriate permissions. \[#7345\] +10. Fix several bugs around concurrent creation of databases. \[#7363\] +11. Change methods in DbEnv that provide access to statistics information so that they now return instances of the proper classes. \[#7395\] +12. Replace the DB-\>set_cache_priority API with the DB_MPOOLFILE-\>set_priority API. \[#7545\] +13. Fix a bug where a failure during a creation of a subdatabase could then fail in the dbremove cleanup, causing a crash. \[#7579\] +14. Allow creating into a file that was renamed within the same transaction. \[#7581\] +15. Fix a bug where DB_ENV-\>txn_stat could drop core if there are more-than-expected active transactions. \[#7638\] +16. Change Berkeley DB to ignore user-specified byte orders when creating a database in an already existing physical file. \[#7640\] +17. Fix a bug where a database rename that is aborted would leak some memory. \[#7789\] +18. Fix a bug where files could not be renamed or removed if they were not writable. \[#7819\] +19. Fix a bug where an error during a database open may leak memory in the mpool region. \[#7834\] +20. Fix a bug where the DB_ENV-\>trickle_sync method could flush all of the dirty buffers in the cache rather than a subset. \[#7863\] +21. Fix a bug where an attempt to rename or remove an open file in the same transaction could succeed, even though this is not allowed and will not work on Windows. \[#7917\] +22. Fix a bug where if a recovery interval in the log contained only database opens then a recovery might report "Improper file close". \[#7886\] +23. Add a flag, DB_INIT_REP to DB_ENV-\>open to initialize Replication subsystem. \[#8299\] +24. Fix a bug where file remove and rename operations would not block each other if they were in different transactions. \[#8340\] +25. Change Berkeley DB to not propagate error returns from the application's rep_send function out of the Berkeley DB API. \[#8496\] \[#8522\] +26. Remove restriction that DB_TRUNCATE is not allowed on files with subdatabases. This restriction was introduced in 4.1.25. \[#8852\] + +### Concurrent Data Store Changes: + +1. Fix a bug where opens with other threads/processes actively acquiring locks on database handles could deadlock. \[#6286\] +2. Add support to Concurrent Data Store to allow duplication of write cursors. \[#7167\] + +### General Access Method Changes: + +1. Fix a bug where the truncate of a database with associated secondary databases did not truncate the secondaries. \[#6585\] +2. Fix a bug in which an out-of-disk condition during a transactional database create, remove, or rename could cause a crash. \[#6695\] +3. Fix a bug where system errors unknown to the C library could cause Berkeley DB utilities to drop core on Solaris. \[#6728\] +4. Fix a bug where Berkeley DB could overwrite incorrectly formatted files rather than returning an error to the application during open. \[#6769\] +5. Fix a bug DB handle reference counts were incorrect, leading to spurious warning about open DB handles. \[#6818\] +6. Fix a bug where cursor adjustments across multiple DB handles could fail. \[#6820\] +7. Fix a bug where a failure during open could result in a hang. \[#6902\] +8. Fix a bug where repeated failures during certain stages of opens could cause error messages to appear during recovery. \[#7008\] +9. Fix a bug in secondary indices with multiple threads calling DBC-\>put that resulted in DB_NOTFOUND being returned. \[#7124\] +10. Fix a bug where database verification might reference memory which was previously freed after reporting an error. \[#7137\] +11. Rename the DB_CHKSUM_SHA1 to DB_CHKSUM as Berkeley DB only uses SHA1 for encrypted pages, not for clear text pages. \[#7095\] +12. Fix a bug where DB-\>rename could fail silently if the underlying system rename call failed. \[#7322\] +13. Fix a bug where Berkeley DB failed to open a file with FCNTL locking and 0-length files. \[#7345\] +14. Prohibit the use of the DB_RMW flag on get operations for DB handles opened in transactional mode. \[#7407\] +15. Standardize when Berkeley DB will return DB_LOCK_NOTGRANTED, or throw DbLockNotGrantedException, versus returning DB_LOCK_DEADLOCK or throwing DbDeadlockException. Fix bugs in the C++ and Java APIs where DbException was thrown, encapsulating DB_LOCK_NOTGRANTED, rather than throwing DbLockNotGrantedException. \[#7549\] +16. Fix a bug where Berkeley DB could hang on a race condition if a checkpoint was running at the same time another thread was closing a database for the last time. \[#7604\] +17. Fix several bugs that made multiple filesystem level operations inside a single transaction break. \[#7728\] +18. Fix a memory leak in the abort path of a sub-database create. \[#7790\] +19. Fix a race condition with file close that could cause NULL pointer deference under load. \[#8235\] +20. Fix a bug to correct the calculation of the amount of space needed to return off page duplicates using the DB_MULTIPLE interface. \[#8437\] +21. Fix a bug where the duplicate data item count could be incorrect if a cursor was used to first overwrite and then delete a duplicate which was part of a set of duplicates large enough to have been stored outside the standard access method pages. \[#8445\] +22. Fix a bug where The DB_MULTIPLE interface might fail to return the proper duplicates in some edge cases. \[#8485\] +23. Fix a bug where DB-\>get(...DB_MULTIPLE) would not return a reasonable estimate of the buffer size required to return a set of duplicates. \[#8513\] +24. Fix a bug where the DbCursor.count method could return the wrong count in the case of small (on-page) duplicate sets, where a still-open cursor has been used to delete one of the duplicate data items. \[#8851\] +25. Fix a bug where a non-transactional cursor using DB_MULTIPLE_KEY could briefly be left pointing at an unlocked page. This could lead to a race condition with another thread deleting records resulting in the wrong record being deleted. \[#8926\] +26. Fix a bug where a key/data item could be lost if a cursor is used to do a delete, and then immediately used to do an insert which causes a set of duplicates to be shifted to an off-page Btree. \[#9085\] + +### Btree Access Method Changes: + +1. Fix a bug where a deleted item could be left on a database page causing database verification to fail. \[#6059\] +2. Fix a bug where a page may be left pinned in the cache if a deadlock occurs during a DB-\>put operation. \[#6875\] +3. Fix a bug where a deleted record may not be removed from a Btree page if the page is split while another cursor is trying to delete a record on the page. \[#6059\] +4. Fix a bug where records marked for deletion were incorrectly counted when retrieving in a Btree by record number. \[#7133\] +5. Fix a bug where a page and lock were left pinned if an application requested a record number past the end of the file when retrieving in a Btree by record number. \[#7133\] +6. Fix a bug where deleted keys were included in the key count for the DB-\>stat call. \[#7133\] +7. Fix a bug where specifying MULTIPLE_KEY and NEXT_DUP to the bulk get interfaces might return the wrong data if all the duplicates could not fit in a single buffer. \[#7192\] +8. Remove assertions that triggered failures that were correct executions. \[#8032\] +9. Fix a bug where duplicate data items were moved onto overflow pages before it was necessary. \[#8082\] +10. Fix a bug where the DB-\>verify method might incorrectly complain about a tree's overflow page reference count. \[#8061\] +11. Fix a bug that could cause DB_MULTIPLE on a Btree database to return an incorrect data field at the end of buffer. \[#8442\] +12. Fix a bug where DBC-\>c_count was returning an incorrect count if the cursor was positioned on an item that had just been deleted. \[#8851\] +13. Remove the test for bt_maxkey in the Btree put code. If it is set to 1 it can cause an infinite loop. \[#8904\] + +### Hash Access Method Changes: + +1. Fix a bug where Hash databases could be corrupted on filesystems that do not zero-fill implicitly created blocks. \[#6588\] +2. Fix a bug where creating a Hash database with an initial size larger than 4GB would fail. \[#6805\] +3. Fix a bug where a page in an unused hash bucket might not be empty if there was a disk error while writing the log record for the bucket split. \[#7035\] +4. Fix a bug where two threads opening a hash database at the same time might deadlock. \[#7159\] +5. Fix a bug where a hash cursor was not updated properly when doing a put with DB_NODUPDATA specified. \[#7361\] +6. Fix a bug that could cause DB_MULTIPLE_KEY on Hash databases to return improper results when moving from a key with duplicates to a key without duplicates. \[#8442\] + +### Queue Access Method Changes: + +1. Fix a bug where opening an in-memory Queue database with extent size specified will dump core. \[#6795\] +2. Support auto-commit with the DB-\>get method's consume operations. \[#6954\] +3. Fix a bug where calling the sync method on a queue database with extents may hang if there are active consumers. \[#7022\] +4. Fix a bug where a get(...MULTIPLE...) might lead to an infinite loop or return the wrong record number(s) if there was a deleted record at the beginning of a page or the buffer was filled exactly at the end of a page. \[#7064\] +5. Fix a bug where a database environment checkpoint might hang if a thread was blocked waiting for a record while doing a DB_CONSUME_WAIT on a Queue database. \[#7086\] +6. Fix a bug where queue extent files would not be removed if a queue with extents was removed and its record numbers wrapped around the maximum record number. \[#7191\] +7. Fix a bug where a DB-\>remove of an extent based Queue with a small number of pages per extent would generate a segmentation fault. \[#7249\] +8. Fix a bug where verify and salvage on queues with extent files did not consider the extent files. \[#7294\] +9. Fix a bug when transaction timeouts are set in the environment they would get applied to some non-transactional operations and could cause a failure during the abort of a queue operation. \[#7641\] +10. Fix a bug when the record numbers in a queue database wrap around at 232, a cursor positioned on a record near the head of the queue that is then deleted, may return DB_NOTFOUND when get is specified with DB_NEXT rather than the next non-deleted record. \[#7979\] +11. Fix a bug where a record lock will not be removed when the first record in the queue is deleted without a transaction (not using DB_CONSUME). \[#8434\] +12. Fix a bug where byte swapping was not handled properly in queue extent files. \[#8358\] +13. Fix a bug where Queue extent file pages were not properly typed, causing the extent files not to use encryption or checksums, even if those options had been specified. This fix requires a database upgrade for any affected Queue databases. \[#8671\] +14. Fix a bug where truncating a queue with extents may fail to remove the last extent file. \[#8716\] +15. Fix a bug where a rename or remove of a QUEUE database with extents might leave empty extent files behind. \[#8729\] +16. Fix a bug where on Windows operating systems a "Permission denied" error may be raised if a Queue extent is reopened while it is in the process of being unlinked. \[#8710\] + +### Recno Access Method Changes: + +1. Fix a bug where the DB-\>truncate method may return the wrong record count if there are deleted records in the database. \[#6788\] +2. Fix a bug where internal nodes of Recno trees could get wrong record count if a log write failed and the log was later applied during recovery. \[#6841\] +3. Fix a bug where a cursor next operation could infinitely loop after deleting a record, when the deleted record was immediately followed by implicitly created records. \[#8133\] + +### C++-specific API Changes: + +1. Document the DB-\>del method can return DB_KEYEMPTY for Queue or Recno databases. The C++ and Java APIs now return this value rather than throwing an exception. \[#7030\] +2. Add "get" methods to retrieve most settings. \[#7061\] +3. Fix a bug where applications calling DB-\>verify from the C++ or Java APIs could drop core. Change the DB-\>verify method API to act as a DB handle destructor. \[#7418\] +4. Add utility classes for iterating over multiple key and data items returned from a cursor when using the DB_MULTIPLE or DB_MULTIPLE_KEY flags. These classes, DbMultipleDataIterator, DbMultipleKeyDataIterator, and DbMultipleRecnoDataIterator, mirror the DB Java API and are provided as replacements for the C macros, DB_MULTIPLE_INIT, DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE. \[#7351\] +5. Fix a bug DbException was thrown, encapsulating DB_LOCK_NOTGRANTED, rather than throwing DbLockNotGrantedException. \[#7549\] +6. Add the DbEnv handle to exceptions thrown by the C++ and Java APIs, where possible. \[#7303\] +7. Fix a bug in the C++ DbEnv::set_rep_transport signature so that the envid parameter is signed. \[#8303\] +8. Make the fields of DB_LSN public in the DbLsn class. \[#8422\] + +### Java-specific API Changes: + +1. Db.put(), Dbc.get() and Dbc.put() preserve key size +2. Dbc.get() returns DB_KEYEMPTY rather than throwing an exception +3. The return type of Db.close() is now void. \[#7002\] + + + +1. New Java API (com.sleepycat.dbx.\*) for the transactional storage of data using the Java Collections design pattern. \[#6569\] +2. Fix a bug in the Java Dbt.get_recno_key_data() method when used inside callbacks. \[#6668\] +3. Fix Java DbMpoolStat class to match the DB_MPOOL_STAT struct. \[#6821\] +4. Fix a bug where Dbc.put expected key data even if the key was unused. \[#6932\] +5. Fix a bug in the Java API secondary_key_create callback where memory was freed incorrectly, causing JVM crashes. \[#6970\] +6. Re-implement the Java API to improve performance and maintenance. Fix several inconsistencies in the Java API: +7. Document the DB-\>del method can return DB_KEYEMPTY for Queue or Recno databases. The C++ and Java APIs now return this value rather than throwing an exception. \[#7030\] +8. Add "get" methods to retrieve most settings. \[#7061\] +9. Add Javadoc documentation to the Berkeley DB release. \[#7110\] +10. Fix a bug that caused potential memory corruption when using the Java API and specifying the DB_DBT_REALLOC flag. \[#7215\] +11. Add the DbEnv handle to exceptions thrown by the C++ and Java APIs, where possible. \[#7303\] +12. Map existing c-style API to a more Java camel case API with Java style naming. Retained deprecated older API for the 4.2 release for backwards support in all cases except callback interfaces. Also overloaded methods such as get/pget() into multiple different get() calls to clean up call structure. \[#7378\] +13. Add CamelCased methods to the Java API. \[#7396\] +14. Fix a bug where applications calling DB-\>verify from the C++ or Java APIs could drop core. Change the DB-\>verify method API to act as a DB handle destructor. \[#7418\] +15. Fix a bug DbException was thrown, encapsulating DB_LOCK_NOTGRANTED, rather than throwing DbLockNotGrantedException. \[#7549\] +16. Add a toString() method for all Java \*Stat classes (DbBtreeStat, DbHashStat, DbMpoolStat, etc.). This method creates a listing of values of all of the class member variables. \[#7712\] +17. Remove Db.fd() method from Java API as it has no value to a Java programmer. \[#7716\] +18. Add an accessible timeout field in the DbLockRequest class, needed for the DB_LOCK_GET_TIMEOUT operation of DbEnv.lockVector. \[#8043\] +19. Fix replication method calls from Java API. \[#8467\] +20. Fix a bug where exception returns were inconsistent. \[#8622\] +21. Change the Java API so that it throws an IllegalArgumentException rather than a DbException with the platform-specific EINVAL. \[#8978\] + +### Tcl-specific API Changes: + +1. Add "get" methods to retrieve most settings. \[#7061\] +2. Brought Tcl's \$env set_flags command up to date with available flags. \[#7385\] +3. Update Berkeley DB to compile cleanly against the Tcl/Tk 8.4 release. \[#7612\] +4. Made txn_checkpoint publicly available. \[#8594\] + +### RPC-specific Client/Server Changes: + +1. Fix two bugs in the RPC server where incorrect handling of illegal environment home directories caused server crashes. \[#7075\] +2. Fix a bug where the DB_ENV-\>close method would fail in RPC clients if the DB_ENV-\>open method was never called. \[#8200\] + +### Replication Changes: + +1. Write prepare records synchronously on replication clients so that prepare operations are always honored in the case of failure. \[#6416\] +2. Change replication elections so that the client with the biggest LSN wins, and priority is a secondary factor. \[#6568\] +3. Fix a bug where replicas could not remove log files because the checkpoint lsn was not being updated properly. \[#6620\] +4. Force prepare records out to disk regardless of the setting of the DB_TXN_NOSYNC flag. \[#6614\] +5. Add a new flag, DB_REP_NOBUFFER, which gets passed to the rep_send function specified in DBENV-\>rep_set_transport, to indicate that the message should not be buffered on the master, but should be immediately transmitted to the client(s). \[#6680\] +6. Fix a replication election bug where Berkeley DB could fail to elect a master even if a master already existed. \[#6702\] +7. Allow environment wide setting of DB_AUTO_COMMIT on replication clients. \[#6732\] +8. Fix a replication bug where a client coming up in the midst of an election might not participate in the election. \[#6826\] +9. Add log_flushes when sites become replication masters. If log_flush fails, panic the environment since the clients already have the commits. \[#6873\] +10. Fix a replication bug where a brand new client syncing up could generate an error on the master. \[#6927\] +11. Fix a bug where clients synchronize with the master when they come up with the same master after a client-side disconnect or failures. \[#6986\] +12. Fix several bugs in replication elections turned up by test rep005. \[#6990\] +13. Fix a bug where aborted hash group allocations were not properly applied on replicas. \[#7039\] +14. Fix race conditions between running client recovery and other threads calling replication and other Berkeley DB functions. \[#7402\] \[#8035\] +15. Use shared memory region for all replication flags. \[#7573\] +16. Fix a bug where log archive on clients could prematurely remove log files. \[#7659\] +17. Return an error if a non-replication dbenv handle attempts to write log records to a replication environment. \[#7752\] +18. Fix a race condition when clients applied log records, where we would store a log record locally and then never notice we have it, and need to re-request it from the master, causing the client to get far behind the master. \[#7765\] +19. Fix inconsistencies between the documentation and actual code regarding when replication methods can be called. \[#7775\] +20. Fix a bug where Berkeley DB would wait forever if a NEWMASTER message got dropped. \[#7897\] +21. Fix a bug where the master environment ID did not get set when you called DBENV-\>rep_start as a master. \[#7899\] +22. Fix a bug where operations on a queue database will not get replicated if the transactions that include the operations are committed out of order with the operations. \[#7904\] +23. Fix bugs in log_c_get where an invalid LSN could access invalid addresses. Fix bug in elections where a client upgrading to master didn't write a txn_recycle record. \[#7964\] +24. Fix a bug where REP_VERIFY_FAIL during client recovery wasn't being handled. \[#8040\] +25. Return an error if the application calls rep_process_message before calling rep_start when starting. \[#8057\] +26. Fix a bug to ensure that replication generation numbers always increase and are never reset to 1. \[#8136\] +27. Modify log message retransmission protocol to efficiently handle the case where a large number of contiguous messages were dropped at once. \[#8182\] \[#8169\] \[#8188\] +28. Fix a bug where using the wrong mutex in replication which under certain conditions could cause replication to hang. Also fix a bug where incorrectly setting the checkpoint LSN could cause recovery to take a very long time. \[#8183\] +29. Fix bug where a message could get sent to an invalid master. \[#8184\] +30. Fix a bug where a local variable in log_archive was not initialized. \[#8230\] +31. Fix a bug where elections could hang. \[#8254\] +32. Fix a bug to ensure that we can always remove/re-create the temporary replication database after a failure. \[#8266\] +33. Add a flag, DB_INIT_REP to DB_ENV-\>open to initialize Replication subsystem. \[#8299\] +34. Add new ret_lsnp argument to rep_process_message so that LSNs can be returned to clients on permanent records. Add new lsnp arg to the send callback function so that the master can know the LSNs of records as well. \[#8308\] +35. Narrow the window where we block due to client recovery. \[#8316\] +36. Fix a bug in log_c_incursor where we would not detect that a record was already in the buffer. \[#8330\] +37. Fix a bug that would allow elections to be managed incorrectly. \[#8360\] +38. Fix a bug where replicas were not maintaining meta-\>last_pgno correctly. \[#8378\] +39. Fix a bug in truncating log after recovery to a timestamp or replication-based recovery. \[#8387\] +40. Fix a bug where a checkpoint record written as the first record in a log could cause recovery to fail. \[#8391\] +41. Fix a bug where a client would return DB_NOTFOUND instead of DB_REP_OUTDATED when it was unable to synchronize with the master because it ran out of log records. \[#8399\] +42. Fix a bug where log file changes were not handled properly in replication. \[#8400\] \[#8420\] +43. Fix a bug where checking for invalid log header data could fail incorrectly. \[#8460\] +44. Fix a bug where DB_REP_PERMANENT was not being set when log records were re-transmitted. \[#8473\] +45. Modify elections so that all participants elect in the same election generation. \[#8590\] +46. Fix bug where rep_apply was masking an error return. Also return DB_RUNRECOVERY if the replication client cannot commit or checkpoint. \[#8636\] +47. Fix a bug to update the last_pgno on the meta page on free as well as alloc. \[#8637\] +48. Fix a bug to roll back the LSN on a queue database metapage if we're going to truncate the log. Fix a bug in MASTER_CHECK so we don't apply log messages from an unknown master. Fix a bug to perform a sync on rep_close. \[#8601\] +49. Fix a bug so that we reset the LSN when putting pages on the free list. \[#8685\] +50. Fix a bug where replication was not properly calling db_shalloc. \[#8811\] +51. Fix a bug where replication flags were getting set in multiple steps which could cause an Assertion Failure in log_compare. \[#8889\] +52. Fix a bug where open database handles could cause problems on clients. \[#8936\] +53. Fix a bug where in dbreg code where an fnp with an invalid fileid could be found on the lp-\>fq list. \[#8963\] +54. Fix a bug where a reader on a replication client could see partial updates when replicating databases with off page duplicates or overflow records. \[#9041\] +55. Fix a bug that could result in a self deadlock in dbreg under replication. \[#9138\] +56. Fix a memory leak in replication. \[#9255\] + +### XA Resource Manager Changes: + +1. Fix a bug where a failed write during XA transaction prepare could result in a checksum error in the log. \[#6760\] +2. Fix a bug where we were not properly handling DB_AUTO_COMMIT in XA transactions and where we were not honoring the XA transaction during an XA-protected open. \[#6851\] +3. Add infrastructure support for multithreaded XA. \[#6865\] +4. Display XA status and ID as part of db_stat -t output. \[#6413\] + +### Locking Subsystem Changes: + +1. failure to remove dirty read locks prior to aborting a transaction, +2. calling upgrade on other than WWRITE locks, +3. failure to remove expired locks from the locker queue, +4. clearing the lock timeout before looking at it. \[#7267\] + + + +1. Fix a bug where locks were not cleared in an off-page duplicate cursor. \[#6950\] +2. Fix a bug where a deadlock may not be detected if dirty reads are enabled and the deadlock involves an aborting transaction. \[#7143\] +3. Fix a bug where a transaction doing updates while using dirty read locking might fail while aborting the transaction with a deadlock. Several other locking issues were also fixed: +4. Fix a bug when dirty reads are enabled a writer might be blocked on a lock that it had previously obtained. Dirty readers would also wait behind regular readers when they could have safely read a page. \[#7502\] +5. Fix a bug where a DB-\>put using CDB gets a lock timeout then the error "Closing already closed cursor". \[#7597\] +6. Modify the maximum test-and-set mutex sleep for logical page locks at 10ms, everything else at 25ms. \[#7675\] +7. Fix a bug where the DB_LOCK_TIMEOUT mode of env-\>lock_vec could hang. \[#7682\] +8. Fix a bug where running with only transaction timeouts for deadlock detection might deadlock without being detected if more than one transaction times out while trying to avoid searching a Btree on repeated inserts. \[#7787\] +9. Fix a bug that could cause detection to not run when there was a lock that should be timed out. \[#8588\] +10. Fix a bug with using dirty reads with subtransactions. If a writing subtransaction aborts and then is blocked, the deadlock may not be detected. \[#9193\] +11. Fix a bug where handle locks were not being correctly updated when releasing read locks during transaction prepare. \[#9275\] + +### Logging Subsystem Changes: + +1. Fix a bug where if a write error occurred while committing a transaction with DB_WRITE_NOSYNC enabled the transaction may appear to be committed in the log while it was really aborted. \[#7034\] +2. Fix a bug where multiprocess applications could violate write-ahead logging requirements if one process wrote a log record but didn't flush it, the current log file then changed, and another process wrote a database page before the log record was written to disk. \[#6999\] +3. Fix a bug where fatal recovery could fail with a "Transaction already committed" error if recovery had been run and there are no active transactions in the part of the log following the last checkpoint. \[#7234\] +4. Fix a bug where recovery would fail to put freed pages onto the free list, when both committed and aborted subtransactions that allocated new pages were present. This only affected prepared transactions. \[#7403\] +5. Fix a bug where open errors during recovery get propagated unless they are reporting missing files, which might correctly have been removed. \[#7578\] +6. Fix a bug so that we now validate a log file before writing to it. \[#7580\] +7. Fix a bug where Berkeley DB could display the unnecessary error message "DB_LOGC-\>get: short read" during recovery. \[#7700\] +8. Fix a bug where recovery may fail if it tries to reallocate a page to a file that is out of space. \[#7780\] +9. Change Berkeley DB so that operations on databases opened in a non-transactional mode do not write records into the database logs. \[#7843\] +10. Fix a bug where Berkeley DB could timeout waiting for locks (on Queue databases) during recovery. \[#7927\] +11. Fix a bug in truncating log after recovery to a timestamp or replication-based recovery. \[#8387\] +12. Fix a bug where recovery can be slow if the log contains many opens of files which contain multiple databases. \[#8423\] +13. Fix a bug where a file id could be used before its open was logged. \[#8496\] +14. Fix a bug where recovery would partially undo a database create if the transaction which created it spanned log files and not all of the log files were present during recovery. \[#9039\] + +### Memory Pool Subsystem Changes: + +1. Fix a bug where checksummed files could not be read on different endian systems. \[#6429\] +2. Fix a bug where read-only databases were not mapped into memory but were instead read through the Berkeley DB buffer cache. \[#6671\] +3. Fix a bug where Berkeley DB could loop infinitely if the cache was sized so small that all of its pages were simultaneously pinned by the application. \[#6681\] +4. Fix a bug where DbEnv.sync could fail to write a page if another thread unpinned the page at the same time and there were no other pages in that hash bucket. \[#6793\] +5. Fix a bug where threads of control may hang if multiple threads of control are opening and closing a database at the same time. \[#6953\] +6. Fix a bug where a database created without checksums but later opened with checksums would result in a checksum error. \[#6959\] +7. Fix a bug where a multiprocess application suite could see incorrect data if one process opened a non-checksummed database +8. Change to avoid database open and flush when handles are discarded, if the handle was never used to write anything. \[#7232\] +9. Fix a bug where applications dirtying the entire cache in a single database operation would see large performance degradation. \[#7273\] +10. Fix a bug where contention in the buffer pool could cause the buffer allocation algorithm to unnecessarily sleep waiting for buffers to be freed. \[#7572\] + +### Transaction Subsystem Changes: + +1. Fix a bug where disk write errors in encrypted database environments, causing transaction abort, could corrupt the log. \[#6768\] +2. Fix a bug where catastrophic recovery may fail on a log which has a prepared transaction which aborted the allocation of a new page and was rolled forward previously by another recovery session. \[#6790\] +3. Fix a bug where a transaction that contains a database truncate followed by page allocations, may not properly undo the truncate if aborted. \[#6862\] +4. Fix a bug which causes Berkeley DB to checkpoint quiescent database environments. \[#6933\] +5. Fix a bug where if a transaction prepare fails while writing the prepare log record, and it contains a subtransaction which did an allocation later, recovery of the database may fail with a log sequence error. \[#6874\] +6. Do not abort prepared but not yet completed transactions when closing an environment. \[#6993\] +7. Fix a bug where operations on the source of a rename in the same transaction would fail. \[#7537\] +8. Fix a bug where a parent transaction which aborts when it tries to write its commit record could fail with a log sequence error, if the parent transaction has an aborted child transaction which allocated a new page from the operating system. \[#7251\] +9. Fix a bug where Berkeley DB could try to abort a partial transaction because it contained a partial subtransaction. \[#7922\] +10. Fix a bug where Berkeley DB could drop core when transactions were configured without locking support. \[#9255\] + +### Utility Changes: + +1. Fix a bug where db_load could core dump or corrupt record numbers by walking off the end of a string. \[#6985\] +2. Fix a bug where db_load could run out of locks when loading large numbers of records. \[#7173\] +3. Fix a bug where db_dump could drop core when salvaging unaligned entries on a Btree page. \[#7247\] +4. Fix a bug where hash statistics did not include overflow items in the count of database data elements. \[#7473\] +5. Fix a bug where an corruption in an overflow page list could cause DB-\>verify to infinitely loop. \[#7663\] +6. Fix a bug where verify could display extraneous error messages when verifying a Btree with corrupt or missing pages. \[#7750\] +7. Fix a bug that could cause the db_stat utility to display values larger than 100 for various percentages. \[#7779\] +8. Fix a memory overflow bug in db_load. \[#8124\] +9. Fix a minor leak when verifying queue databases. \[#8620\] + +### Configuration, Documentation, Portability and Build Changes: + +1. Add support for a reduced memory footprint build of the Berkeley DB library. \[#1967\] +2. Change DB_SYSTEM_MEM on Windows to fail immediately when opening an environment whose regions were deleted on last close. \[#4882\] +3. Update queue.h to current FreeBSD version. \[#5494\] +4. Support for and certification under Tornado 2.2/VxWorks 5.5. \[#5522\] +5. Add support for IBM OS/390 using the IBM C compiler. \[#6486\] +6. Specify -pthread as a compile flag for Tru64 systems, not just as a linker flag. \[#6637\] +7. Remove automatic aggregate initialization for non-ANSI compilers. \[#6664\] +8. Fix a link error ("GetLongPathNameA could not be located in the dynamic link library KERNEL32.dll") that prevented Berkeley DB from loading on Windows NT. \[#6665\] +9. Remove use of U suffix in crypto build to denote unsigned integers for non-ANSI compilers. \[#6663\] +10. Fix Java API documentation problems where API return values were int and should have been void, or vice versa. \[#6675\] +11. Add an include of \ for old Solaris systems with the directio call. \[#6707\] +12. Fix Java API documentation problem where the Db.associate call was missing a DbTxn handle. \[#6714\] +13. Clean up source based on gcc's -Wmissing-prototypes option. \[#6759\] +14. Ignore pread/pwrite interfaces on NCR's System V R 4.3 system. \[#6766\] +15. Fix an interface compatibility with Sendmail and Postfix releases. \[#6769\] +16. Fix warnings when the Tcl API was built without TEST_CONFIG defined. \[#6789\] +17. Change Win32 mutexes to use the shared code for all mutexes to fix handle leak. \[#6822\] \[#6853\] +18. Fix the Windows/Tcl API export list for Berkeley DB XML. \[#6931\] +19. Add the --enable-mingw configuration option to build Berkeley DB for MinGW. \[#6973\] +20. Add a CPU pause to the mutex spinlock code to improve performance on newer Pentium CPUs. \[#6975\] +21. Upgrade read-only file descriptors to read-write during checkpoint, it's an error to call FlushFileBuffers on a read-only Windows file handle. \[#7051\] +22. Fix configure so that Java applications on HP/UX can access RPC environments. \[#7066\] +23. Update Berkeley DB to use libtool 1.5 to allow building of shared libraries on various platforms. This should not be visible except for changes to the Makefile and internal build procedures. \[#7080\] +24. Fix a bug where the configure script displayed incorrect default installation directory information. \[#7081\] +25. Fix a signed/unsigned warning with some Windows compilers. \[#7100\] +26. Fix macro redefinition conflicts between queue.h and Vc7\PlatformSDK\Include\WinNT.h when building with Visual Studio.NET 7.0. \[#7103\] +27. Add a loop to retry system calls that return EBUSY. Also limit retries on EINTR to 100 times. \[#7118\] +28. Fix a bug in our use of GetDiskFreeSpace that caused access violations on some versions of Windows with DB_DIRECT_DB. \[#7122\] +29. Fix a bug where regions in system memory on Windows were incorrectly reinitialized because the magic number was overwritten. \[#7127\] +30. Change version provided to Tcl's package system to reflect Berkeley DB's major and minor number. \[#7174\] +31. Support for the Berkeley DB Embedix port has been removed. \[#7209\] +32. Merge all public C++ headers into db_cxx.h, which fixes name clashes between Berkeley DB headers and system headers (specifically mutex.h). \[#7221\] +33. Fix a bug where the configured Makefile could try and build objects for which there were no existing rules. \[#7227\] +34. Port the ex_repquote example to Windows. \[#7328\] +35. Fix a race in the ARM/gcc mutex code which could cause almost anything bad you can imagine. \[#7468\] +36. Fix a bug where shared region removal could hang. \[#7613\] +37. Fix a bug so that when using Java in Debug mode on Windows, automatically pick the Debug DLL. \[#7722\] +38. Fix configure --disable-shared so that it now creates a Makefile that installs static libraries that look the same as a regular shared build. This flag will create a libdb\.\.a and make a libdb.a that is a symlink to it. \[#7755\] +39. Add support for OS/390 2.10 and all versions of z/OS. \[#7972\] +40. Support Java builds on Windows with spaces in the project path. \[#8141\] +41. Fix a bug where Berkeley DB mutex locking code for OS X was not multiprocessor safe. \[#8255\] +42. Add an error to DB_ENV-\>set_flags if the OS does not support Direct +43. Enable verbose error logging from the test suite on Windows. \[#8634\] +44. Fix a bug with DLL linking on Cygwin under Windows. \[#8628\] +45. Add support for JDK on HP/UX. \[#8813\] +46. Fix a bug where pathnames longer than 2KB could cause processes to core dump. \[#8886\] +47. Fix a bug in VxWorks when yielding the CPU, so that we delay at least one tick. \[#9061\] diff --git a/docs-src/guides/upgrading/changelog_4_3_29.md b/docs-src/guides/upgrading/changelog_4_3_29.md new file mode 100644 index 000000000..c65d737e4 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_3_29.md @@ -0,0 +1,300 @@ +--- +title: "Berkeley DB 4.3.29 Change Log" +api-name: "Berkeley DB 4.3.29 Change Log" +source: docs/upgrading/changelog_4_3_29.html +--- +## Berkeley DB 4.3.29 Change Log + + [Database or Log File On-Disk Format Changes:](changelog_4_3_29.md#idp50694880) + + [New Features:](changelog_4_3_29.md#idp50670248) + + [Database Environment Changes:](changelog_4_3_29.md#idp50673968) + + [Concurrent Data Store Changes:](changelog_4_3_29.md#idp50703424) + + [General Access Method Changes:](changelog_4_3_29.md#idp50690848) + + [Btree Access Method Changes:](changelog_4_3_29.md#idp50691272) + + [Hash Access Method Changes:](changelog_4_3_29.md#idp50694944) + + [Queue Access Method Changes:](changelog_4_3_29.md#idp50697104) + + [Recno Access Method Changes](changelog_4_3_29.md#idp50720352) + + [C++-specific API Changes:](changelog_4_3_29.md#idp50700784) + + [Java-specific API Changes:](changelog_4_3_29.md#idp50670632) + + [Tcl-specific API Changes:](changelog_4_3_29.md#idp50702384) + + [RPC-specific Client/Server Changes:](changelog_4_3_29.md#idp50703784) + + [Replication Changes:](changelog_4_3_29.md#idp50685776) + + [XA Resource Manager Changes:](changelog_4_3_29.md#idp50733112) + + [Locking Subsystem Changes:](changelog_4_3_29.md#idp50712384) + + [Logging Subsystem Changes:](changelog_4_3_29.md#idp50740760) + + [Memory Pool Subsystem Changes:](changelog_4_3_29.md#idp50695328) + + [Transaction Subsystem Changes:](changelog_4_3_29.md#idp50720440) + + [Utility Changes:](changelog_4_3_29.md#idp50724480) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_3_29.md#idp50724864) + +### Database or Log File On-Disk Format Changes: + +1. The on-disk log format has changed. + +### New Features: + +1. Add support for light weight, transactionally protected Sequence Number generation. \[#5739\] +2. Add support for Degree 2 isolation. \[#8689\] +3. Add election generation information to replication to support Paxos compliance. \[#9068\] +4. Add support for 64-bit and ANSI C implementations of the RPCGEN utility. \[#9548\] + +### Database Environment Changes: + +1. Fix a bug where the permissions on system shared memory segments did not match the mode specified in the DB_ENV-\>open() method. \[#8921\] +2. Add a new return error from the DB_ENV-\>open() method call, DB_VERSION_MISMATCH, which is returned in the case of an application compiled under one version of Berkeley DB attempting to open an environment created under a different version. \[#9077\] +3. Add support for importing databases from a transactional database environment into a different environment. \[#9324\] +4. Fix a bug where a core dump could occur if a zero-length database environment name was specified. \[#9233\] +5. Increase the number of environment regions to 100. \[#9297\] +6. Remove the DB_ENV-\>set_verbose() method flag DB_VERB_CHKPOINT. \[#9405\] +7. Fix bugs where database environment getters could return incorrect information after the database environment was opened, if a different thread of control changed the database environment values. Fix bugs where database environment getter/setter functions could race with other threads of control. \[#9724\] +8. Change the DbEnv.set_lk_detect method to match the DbEnv.open semantics. That is, DbEnv.set_lk_detect may be called after the database environment is opened, allowing applications to configure automatic deadlock detection if it has not yet been configured. \[#9724\] +9. Fix cursor locks for environments opened without DB_THREAD so that they use the same locker ID. This eliminates many common cases of application self-deadlock, particularly in CDS. \[#9742\] +10. Fix a bug in DB-\>get_env() in the C API where it could return an error when it should only return the DB_ENV handle. C++ and Java are unchanged. \[#9828\] +11. Fix a bug where we only need to initialize the cryptographic memory region when MPOOL, Log or Transactions have been configured. \[#9872\] +12. Change private database environments at process startup to only allocate the heap memory required at any particular time, rather than always allocating the maximum amount of heap memory configured for the environment. \[#9889\] +13. Add a method to create nonexistent intermediate directories when opening database files. \[#9898\] +14. Add support for in-memory logging within database environments. \[#9927\] +15. Change Berkeley DB so configuring a database environment for automatic log file removal affects all threads in the environment, not just the DbEnv handle in which the configuration call is made. \[#9947\] +16. Change the signature of the error callback passed to the DB_ENV-\>set_errcall and DB-\>set_errcall methods to add a DB_ENV handle, to provide database environment context for the callback function. \[#10025\] +17. Fix a race condition between DB-\>close and DB-\>{remove,rename} on filesystems that don't allow file operations on open files (such as Windows). \[#10180\] +18. Add a DB_DSYNC_LOG flag to the DbEnv::set_flags method, which configures O_DSYNC on POSIX systems and FILE_FLAG_WRITE_THROUGH on Win32 systems. This offers significantly better performance for some applications on certain Solaris/filesystem combinations. \[#10205\] +19. Fix a bug where calling the DB or DBEnv database remove or rename methods could cause a transaction checkpoint or cache flush to fail. \[#10286\] +20. Change file operations not to flush a file if it hasn't been written. \[#10537\] +21. Remove 4GB restriction on region sizes on 64 bit machines. \[#10668\] +22. Simplify the signature of substitute system calls for ftruncate and seek. \[#10668\] +23. Change Berkeley DB so that opening an environment without specifying a home directory will cause the DB_CONFIG file in the current directory to be read, if it exists. \[#11424\] +24. Fix a bug that caused a core dump if DB handles without associated database environments were used for database verification. \[#11649\] +25. Fix Windows mutexes shared between processes run as different users. \[#11985\] +26. Fix Windows mutexes for some SMP machines. \[#12417\] + +### Concurrent Data Store Changes: + +1. Fix cursor locks for environments opened without DB_THREAD so that they use the same locker ID. This eliminates many common cases of application self-deadlock, particularly in CDS. \[#9742\] + +### General Access Method Changes: + +1. Fix a bug where Berkeley DB log cursors would close and reopen the underlying log file each time the log file was read. \[#8934\] +2. Improve performance of DB-\>open() for existing subdatabases maintained within the same database file. \[#9156\] +3. Add a new error, DB_BUFFER_SMALL, to differentiate from ENOMEM. The new error indicates that the supplied DBT is too small. ENOMEM is now always fatal. \[#9314\] +4. Fix a bug when an update through a secondary index is deadlocked it is possible for the deadlock to be ignored, resulting in a partial update to the data. \[#9492\] +5. Fix a bug where a record could get inserted into the wrong database when a page was deallocated from one subdatabase and reallocated to another subdatabase maintained within the same database file. \[#9510\] +6. Enhance file allocation so that if the operating system supports decreasing the size of a file and the last page of the file is freed, it will be returned to the operating system. \[#9620\] +7. Fix a bug where DB_RUNRECOVERY could be returned if there was no more disk space while aborting the allocation of a new page in a database. \[#9643\] +8. Fix a bug where the cryptographic code could memcpy too many bytes. \[#9648\] +9. Fix a bug with DB-\>join() cursors that resulted in a memory leak and incomplete results. \[#9763\] +10. Disallow cursor delete, followed a cursor put of the current item across all access methods. \[#9900\] +11. Fix a bug where recovery of operations on unnamed databases that were never closed, could fail. \[#10118\] +12. Fix a bug where DB-\>truncate of a database with overflow records that spanned more than one page would loop. \[#10151\] +13. Improve performance in the database open/close path. \[#10266\] +14. Fix a bug that restricted the number of temporary files that could be created to 127. \[#10415\] +15. Fix a bug which could cause a Too many files error when trying to create temporary files. Limit the number of temporary file creation retries. \[#10760\] \[#10773\] +16. Fix a memory leak bug with Sequence Numbers. \[#11589\] +17. Fix a bug on Windows platforms that prevents database files from growing to over 2GB. \[#11839\] +18. Fix a platform independence bug with sequence numbers. Existing sequence numbers will be automatically upgraded upon next access. \[#12202\] +19. Fix a race between truncate and read/write operations on Windows platforms that could cause corrupt database files. \[#12598\] + +### Btree Access Method Changes: + +1. Fix a bug where a record could get placed on the wrong page when two threads are simultaneously trying to split a four level (or greater) Btree. \[#9542\] +2. Fix a bug where calling DB-\>truncate() on a Btree which has duplicate keys that overflow the leaf page would not properly free the overflow pages and possibly loop. \[#10666\] + +### Hash Access Method Changes: + +1. Fix a bug where a delete to a HASH database with off page duplicates could fail to have the proper lock when deleting an off page duplicate tree. \[#9585\] +2. Fix a bug where a dirty reader using a HASH database would leave a lock on the meta page. \[#10105\] +3. Fix a bug where a DB-\>del() on a HASH database supporting dirty reads could fail to upgrade a WWRITE lock to a WRITE lock when deleting an off page duplicate. \[#10649\] + +### Queue Access Method Changes: + +1. Fix a bug where DB_CONSUME_WAIT may loop rather than wait for a new record to enter the queue, if the queue gets into a state where there are only deleted records between the head and the end of queue. \[#9215\] +2. Fix a bug where a Queue extent file could be closed when it was empty, even if a thread was still accessing a page from that file. \[#9291\] +3. Fix a bug where DBC-\>c_put(key, data, DB_CURRENT) where inserting a new record after the current record had been deleted was returning DB_KEYEMPTY. \[#9314\] +4. Fix a bug where a Queue extent file could be reported as not found if a race condition was encountered between removing the file and writing out a stale buffer. \[#9462\] +5. Fix a bug where the Queue access method might fail to release a record lock when running without transactions. \[#9487\] +6. Add DB_INORDER flag for Queue databases to guarantee FIFO (First In, First Out) ordering when using DB_CONSUME or DB_CONSUME_WAIT. \[#9689\] +7. Fix a bug where remove and rename calls could fail with a "Permission denied" error. \[#9775\] +8. Fix a bug where aborting a transaction that opened and renamed a queue database using extents could leave some of the extent files with the wrong name on Windows. \[#9781\] +9. Fix a bug where a db_dump of a queue database could return an error at the end of the queue if the head or tail of the queue is the first record on a page. \[#10215\] +10. Fix a race condition which would leave a Queue extent file open until the database handle was closed, preventing it from being removed. \[#10591\] +11. Fix a bug where a deadlock of a put on a database handle with dirty readers could generate a lock downgrade error. \[#10678\] +12. Fix a bug which caused DB_SET_RANGE and DB_GET_BOTH_RANGE to not return the next record when an exact match was not found. \[#10860\] + +### Recno Access Method Changes + +1. Fix a bug where the key/data counts returned by the Db-\>stat method for Recno databases did not match the documentation. \[#8639\] +2. Fix a bug where DBC-\>c_put(key, data, DB_CURRENT) where inserting a new record after the current record had been deleted was returning DB_KEYEMPTY. \[#9314\]. + +### C++-specific API Changes: + +1. Change DbException to extend std::exception, making it possible for applications to catch all exceptions in one place. \[#10022\] +2. Fix a bug where errors during transaction destructors (commit, abort) could cause an invalid memory access. \[#10302\] +3. Fix a bug that could lead to a read through an uninitialized pointer when a DbLockNotGrantedException is thrown. \[#10470\] +4. Fix a bug in the C++ DbEnv::rep_elect method API where the arguments were swapped, leading to an "Invalid Argument" return when that method is called. \[#11906\] + +### Java-specific API Changes: + +1. Fix a bug where the Java API did not respect non-zero return values from secondaryKeyCreate, including DB_DONOTINDEX. \[#9474\] +2. Fix a bug where a self-deadlock occurred with a non-transactional class catalog database used in a transactional environment. The bug only occurred when the collections API was not used for starting transactions. \[#9521\] +3. Improve Javadoc for the Java API. \[#9614\] +4. Improve memory management and performance when large byte arrays are being passed to DB methods. \[#9801\] +5. Improve performance of accessing statistics information from the Java API. \[#9835\] +6. Allow Java application to run without DB_THREAD so they can be used as RPC clients. \[#10097\] +7. Fix a bug where an uninitialized pointer is dereferenced for logArchive(Db.DB_ARCH_REMOVE). \[#10225\] +8. Fix a bug in the Collections API where a deadlock exception could leave a cursor open. \[#10516\] +9. Fix the replication callback in the Java API so that the parameter names match the C API. \[#10550\] +10. Add get methods to the Java statistics classes. \[#10807\] +11. Fix bugs in the Java API handling of null home directories and environments opened without a memory pool. \[#11424\] +12. Fix the Java API in the non-crypto package. \[#11752\] +13. Fix a bug that would cause corruption of error prefix strings. \[#11967\] +14. Fix handling of LSNs in the Java API. \[#12223\] +15. Dont throw a NullPointerException if the list of files returned by log_archive is empty. \[#12383\] + +### Tcl-specific API Changes: + +None. + +### RPC-specific Client/Server Changes: + +1. Add support for 64-bit and ANSI C implementations of the RPCGEN utility. \[#9548\] +2. Fix a small memory leak in RPC clients. \[#9595\] +3. Fix a bug in the RPC server to avoid self-deadlock by always setting DB_TXN_NOWAIT. \[#10181\] +4. Fix a bug in the RPC server so that if it times out an environment, it first closes all the Berkeley DB handles in that environment. \[#10623\] + +### Replication Changes: + +1. Add an Environment ID to distinguish between clients from different replication groups. \[#7786\] +2. Add number of votes required and flags parameters to DB_ENV-\>rep_elect() method. \[#7812\] +3. Fix a bug where a client's env_openfiles pass could start with the wrong LSN. This could result in very long initial sync-up times for clients joining a replication group. \[#8635\] +4. Add election generation information to replication to support Paxos compliance. \[#9068\] +5. Add rep019 to test running normal recovery on clients to make sure we synch to the correct LSNs. \[#9151\] +6. Remove support for logs-only replication clients. Use of the DB_REP_LOGSONLY flag to the DB_ENV-\>rep_start() method should be replaced with the DB_REP_CLIENT flag. \[#9331\] +7. Fix a bug where replication clients fail to lock all the necessary pages when applying updates when there are more than one database in the transaction. \[#9569\] +8. Fix a bug in replication elections where when elections are called by multiple threads the wrong master could get elected. \[#9770\] +9. Fix a bug where the master could get a DB_REP_OUTDATED error. Instead send an OUTDATED message to the client. \[#9881\] +10. Add support for automatic initialization of replication clients. \[#9927\] +11. Modify replication timestamp so that non-replication client applications can get a DB_REP_HANDLE_DEAD. \[#9986\] +12. Add a new DB_REP_STARTUPDONE return value for rep_process_message() and st_startup_done to rep_stat() to indicate when a client has finished syncing up to a master and is processing live messages. \[#10310\] +13. Add \_pp to secondary handles, add RPRINT, fix a deadlock. \[#10429\] +14. Fix a bug where an old client (and no master) that dropped the ALIVE message would never update to the current generation. \[#10469\] +15. Fix a bug where a message could get sent to a new client before NEWSITE has been returned to the application. Broadcast instead. \[#10508\] +16. Fix a crash when verbose replication messages are configured and a NULL DB_LSN pointer is passed to rep_process_message. \[#10508\] +17. Add code to respect set_rep_limit in LOG_REQ processing. \[#10716\] +18. Fix a synchronization problem between replication recovery and database open. \[#10731\] +19. Change elections to adjust timeout if egen changes while we are waiting. \[#10686\] +20. Client perm messages now return ISPERM/NOTPERM instead of 0. \[#10855\] \[#10905\] +21. Fix a race condition during rep_start when a role change occurs. Fix memory leaks. \[#11030\] +22. Fix problems with duplicate records. A failure will no longer occur if the records are old records (LOG_MORE) and archived. \[#11090\] +23. Fix a bug where the replication temporary database would grow during automatic client initialization. \[#11090\] +24. Add throttling to PAGE_REQ. \[#11130\] +25. Remove optimization-causing problems with racing threads in rep_verify_match. \[#11208\] +26. Fix memory leaks. \[#11239\] +27. Fix an initialization bug when High Availability configurations are combined with private database environments, which can cause intermittent failures. \[#11795\] +28. Fix a bug in the C++ DbEnv::rep_elect method API where the arguments were swapped, leading to an "Invalid Argument" return when that method is called. \[#11906\] + +### XA Resource Manager Changes: + +None. + +### Locking Subsystem Changes: + +1. Fix a bug where a deadlock of an upgrade from a dirty read to a write lock during an aborted transaction, may not be detected. \[#7143\] +2. Add support for Degree 2 isolation. \[#8689\] +3. Change the system to return DB_LOCK_DEADLOCK if a transaction attempts to get new locks after it has been selected as the deadlock victim. \[#9111\] +4. Fix a bug where when configured to support dirty reads, a writer may not downgrade a write lock as soon as possible, potentially blocking dirty readers. \[#9197\] +5. Change the test-and-set mutex implementation to avoid interlocked instructions when we know the instruction is unlikely to succeed. \[#9204\] +6. Fix a bug where a thread supporting dirty readers can get blocked while trying to get a write lock. It will allocate a new lock rather than using an existing WAS_WRITE lock when it becomes unblocked, causing the application to hang. \[#10093\] +7. The deadlock detector will now note that a parent transaction should be considered in abort if one of its children is. \[#10394\] +8. Remove a deadlock where database closes could deadlock with page acquisition. \[#10726\] +9. Fix a bug where a dirty reader could read an overflow page that was about to be deleted. \[#10979\] +10. Fix a bug that failed to downgrade existing write locks during a btree page split when supporting dirty reads. \[#10983\] +11. Fix a bug that would fail to upgrade a write lock when moving a cursor off a previously deleted record. \[#11042\] + +### Logging Subsystem Changes: + +1. Fix a bug where recovery could leave too many files open. \[#9452\] +2. Fix a bug where aborting a transaction with a file open in it could result in an unrecoverable log file. \[#9636\] +3. Fix a bug where recovery would not return a fatal error if the transaction log was corrupted. \[#9841\] +4. Fix a bug in recovery so that the final checkpoint no longer tries to flush the log. This will permit recovery to complete even if there is no disk space to grow the log file. \[#10204\] +5. Improve performance of log flushes by pre-allocating log files and using fdatasync() in preference to fsync(). \[#10228\] +6. Fix a bug where recovery of a page split after a non-transactional update to the next page would fail to update the back pointer. \[#10421\] +7. Fix a bug in log_archive() where \_\_env_rep_enter() was called twice. \[#10577\] +8. Fix a bug with in-memory logs that could cause a memory leak in the log region. \[#11505\] + +### Memory Pool Subsystem Changes: + +1. Fix a bug in the MPOOLFILE file_written flag value so that checkpoint doesn't repeatedly open, flush and sync files in the cache for which there are no active application handles. \[#9529\] + +### Transaction Subsystem Changes: + +1. Fix a bug where the same transaction ID could get allocated twice if you wrapped the transaction ID space twice and then had a very old transaction. \[#9036\] +2. Fix a bug where a transaction abort that contained a page allocation could loop if the filesystem was full. \[#9461\] +3. Fix implementation of DB-\>get_transactional() to match documentation: there is no possibility of error return, only 0 or 1. \[#9526\] +4. Fix a bug where re-setting any of the DB_TXN_NOSYNC, DB_TXN_NOT_DURABLE and DB_TXN_WRITE_NOSYNC flags could fail to clear previous state, potentially leading to incorrect transactional behavior in the application. \[#9947\] +5. Add a feature to configure the maximum number of files that a checkpoint will hold open. \[#10026\] +6. An aborting transaction will no longer generate an undetected deadlock. \[#10394\] +7. Fix a bug that prevented a child transaction from accessing a database handle that was opened by its parent transaction. \[#10783\] +8. Fix a bug where a checkpoint or a delayed write in another process could raise an EINVAL error if the database had been opened with the DB_TXN_NOT_DURABLE flag. \[#10824\] +9. Fix private transactional environments on 64-bit systems. \[#11983\] + +### Utility Changes: + +1. Add debugging and performance tuning information to db_stat. Add new Berkeley DB handle methods to output debugging and performance tuning information to a C library FILE handle (C and C++ APIs only). \[#9204\] +2. Fix a bug where db_stat could drop core if DB-\>open fails and no subdatabase was specified. \[#9273\] +3. Add command-line arguments to the db_printlog utility to restrict the range of log file records that are displayed. \[#9307\] +4. Fix a bug in the locking statistics where current locks included failed lock requests. \[#9314\] +5. Fix a bug where db_archive would remove all log files when --enable-diagnostic and DB_NOTDURABLE were both specified. \[#9459\] +6. Fix a bug where db_dump with the -r flag would output extra copies of the subdatabase information. \[#9808\] +7. Fix a bug in db_archive that would cause log file corruption if the application had configured the environment with DB_PRIVATE. \[#9841\] +8. Add support in db_load for resetting database LSNs and file IDs without having to reload the database. \[#9916\] +9. Change the DB-\>stat() method to take a transaction handle as an argument, allowing DB-\>stat() to be called from within a transaction. \[#9920\] +10. Fix a bug in db_printlog where only the first filewould be displayed for in-memory logs. \[#11505\] +11. Fix a bug that prevented database salvage from working in Berkeley DB 4.3.21. \[#11649\] +12. Fix a bug in db_load which made it impossible to specify more than a single option on the command line. \[#11676\] + +### Configuration, Documentation, Portability and Build Changes: + +1. Add pread and pwrite to the list of system calls applications can replace at run-time. \[#8954\] +2. Add support for UTF-8 encoding of filenames on Windows. \[#9122\] +3. Remove C++ dependency on snprintf. Compilers on HPUX 10.20 are missing header file support for snprintf(). \[#9284\] +4. Change Berkeley DB to not use the open system call flag O_DIRECT, unless DB configured using --enable-o_direct. \[#9298\] +5. Fix several problems with mutex alignment on HP/UX 10.20. \[#9404\] +6. Fix a memory leak when HAVE_MUTEX_SYSTEM_RESOURCES is enabled. \[#9546\] +7. Fix a bug in the sec002.tcl test for binary data. \[#9626\] +8. Fix a bug where filesystem blocks were not being zeroed out in the On-Time embedded Windows OS. \[#9640\] +9. Fix build problems with the Java API in Visual Studio .NET 2003. \[#9701\] +10. Add support for the gcc compiler on the Opteron platform. \[#9725\] +11. Add support for the small_footprint build option for VxWorks. \[#9820\] +12. Add support for linking of DLLs with MinGW. \[#9957\] +13. Remove the make target which builds the RPM package from the Berkeley DB distribution. \[#10233\] +14. Add a C++/XML example for ex_repquote. \[#10380\] +15. Fix a bug to link with lrt only if detected by configure (Mac OS X issue). \[#10418\] +16. Fix a bug and link Java and Tcl shared libraries with lpthread if required, for mutexes. \[#10418\] +17. Add support for building Berkeley DB on the HP NonStop OSS (Tandem) platform. \[10483\] +18. Change Berkeley DB to ignore EAGAIN from system calls. This fixed problems on NFS mounted databases. \[#10531\] +19. Remove a line with bt_compare and bt_prefix from the db_dump recovery test suite, which can cause failures on OpenBSD. \[#10567\] +20. Fix a conflict with the lock_init for building Berkeley DB on Cygwin. \[#10582\] +21. Add Unicode support for the Berkeley DB Windows API. \[#10598\] +22. Add support for 64-bit builds on Windows. \[#10664\] +23. Libtool version is now 1.5.8. \[#10950\] +24. Remove mt compilation flag for HP-UX 11.0. \[#11427\] +25. Fix a bug to link with lrt on Solaris to support fdatasync. \[#11437\] diff --git a/docs-src/guides/upgrading/changelog_4_4_16.md b/docs-src/guides/upgrading/changelog_4_4_16.md new file mode 100644 index 000000000..d195e04d6 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_4_16.md @@ -0,0 +1,266 @@ +--- +title: "Berkeley DB 4.4.16 Change Log" +api-name: "Berkeley DB 4.4.16 Change Log" +source: docs/upgrading/changelog_4_4_16.html +--- +## Berkeley DB 4.4.16 Change Log + + [Database or Log File On-Disk Format Changes:](changelog_4_4_16.md#idp50595920) + + [New Features:](changelog_4_4_16.md#idp50583264) + + [Database Environment Changes:](changelog_4_4_16.md#idp50583648) + + [Concurrent Data Store Changes:](changelog_4_4_16.md#idp50567656) + + [General Access Method Changes:](changelog_4_4_16.md#idp50591960) + + [Btree Access Method Changes:](changelog_4_4_16.md#idp50592384) + + [Hash Access Method Changes:](changelog_4_4_16.md#idp50595984) + + [Queue Access Method Changes:](changelog_4_4_16.md#idp50597856) + + [Recno Access Method Changes](changelog_4_4_16.md#idp50598936) + + [C++-specific API Changes:](changelog_4_4_16.md#idp50598072) + + [Java-specific API Changes:](changelog_4_4_16.md#idp50600424) + + [Java collections and bind API Changes:](changelog_4_4_16.md#idp50621112) + + [Tcl-specific API Changes:](changelog_4_4_16.md#idp50604672) + + [RPC-specific Client/Server Changes:](changelog_4_4_16.md#idp50589536) + + [Replication Changes:](changelog_4_4_16.md#idp50610200) + + [XA Resource Manager Changes:](changelog_4_4_16.md#idp50594920) + + [Locking Subsystem Changes:](changelog_4_4_16.md#idp50614600) + + [Logging Subsystem Changes:](changelog_4_4_16.md#idp50614888) + + [Memory Pool Subsystem Changes:](changelog_4_4_16.md#idp50635800) + + [Transaction Subsystem Changes:](changelog_4_4_16.md#idp50617400) + + [Utility Changes:](changelog_4_4_16.md#idp50617824) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_4_16.md#idp50621200) + +### Database or Log File On-Disk Format Changes: + +1. The on-disk log format has changed. + +### New Features: + +1. Add support to compact an existing Btree database. \[#6750\] +2. Add support for named in-memory databases. \[#9927\] +3. Add support for database environment recovery serialization. This simplifies multiprocess application architectures. Add DB_REGISTER flag to DB_ENV-\>open(). \[#11511\] +4. Add utility for performing hot backups of a database environment. \[#11536\] +5. Add replication configuration API. \[#12110\] +6. Add replication support to return error instead of waiting for client sync to complete. \[#12110\] +7. Add replication support for delayed client synchronization. \[#12110\] +8. Add replication support for client-to-client synchronization. \[#12110\] +9. Add replication support for bulk transfer. \[#12110\] +10. Add new flags DB_DSYNC_DB and DB_DSYNC_LOG \[12941\] +11. Add DbEnv.log_printf, a new DbEnv method which logs printf style formatted strings into the Berkeley DB database environment log. \[#13241\] + +### Database Environment Changes: + +1. Add a feature to support arbitrary alignment of mutexes in order to minimize cache line collisions. \[#9580\] +2. Change cache regions on 64-bit machines to allow regions larger than 4GB. \[#10668\] +3. Fix a bug where a loop could occur if the application or system failed during modification of the linked list of shared regions. \[#11532\] +4. Fix mutex alignment on Linux/PA-RISC, add test-and-set mutexes for MIPS and x86_64. \[#11575\] +5. Fix a bug where private database environments (DB_PRIVATE) on 64-bit machines would core dump because of 64-bit address truncation. \[#11983\] +6. Fix a bug where freed memory is accessed when DB_PRIVATE environments are closed. This can happen on systems where the operating system holds mutex resources that must be freed when the mutex is destroyed. \[#12591\] +7. Fix a bug where the DbEnv.stat_print method could self-deadlock and hang. The DbEnv.stat_print method no longer displays statistics for any of the database environments databases. \[#12039\] +8. Fix a bug where Berkeley DB could create fragmented filesystem-backed shared region files. \[#12125\] +9. Fix a bug where Berkeley DB stat calls could report a cache size of 0 after the statistics were cleared. \[#12307\] +10. Threads of control joining database environments are now configured for all of the subsystems (lock, log, cache, or transaction) for which the environment was originally configured, it is now an error to attempt configuration of additional subsystems after an environment is created. \[#12422\] +11. Fix a bug where negative percentages could be displayed in statistics output. \[#12673\] +12. Fix a bug that could cause a panic if the cache is filled with non-logging updated pages. \[#12763\] +13. Fix a bug that could cause an unreported deadlock if the application was using the DB_DIRTY_READ flag and the record was an off page duplicate record. \[#12893\] +14. Fix a bug where a handle lock could be incorrectly retained during a delete or rename operation. \[#12906\] + +### Concurrent Data Store Changes: + +1. Lock upgrades and downgrades are now accounted for separately from lock requests and releases. \[#11155\] +2. Fix a bug where a second process joining a Concurrent Data Store environment, with the DB_CDB_ALLDB flag set, would fail. This would happen if the first thread were not entirely finished with initialization. \[#12277\] + +### General Access Method Changes: + +1. Fix a bug where filesystem operations are improperly synchronized. \[#10564\] +2. Add support for database files larger than 2GB on Windows. \[#11839\] +3. Rename DB_DEGREE_2 (and all related flags) to DB_READ_COMMITTED. Rename DB_DIRTY (and all related flags) to DB_READ_UNCOMMITTED. \[#11776\] +4. Fix a bug where wrapping of sequences was incorrect when the cache size is smaller than the range of the maximum value minus the minimum value. \[#11997\] +5. Fix a bug that could result in a hot backup having a page missing from a database file if a file truncation was in progress during the backup but was then aborted. \[#12017\] +6. Fix a bug where a long filename could cause one too few bytes to be allocated when opening a file. \[#12085\] +7. Fix a bug in secondary cursor code if a write lock is not granted. \[#12109\] +8. Fix a bug in secondary cursors where the current record would change on error. \[#12141\] +9. Fix a bug in Db-\>truncate where the method was not checking to see if the handle was opened read-only. \[#12179\] +10. Fix a bug in sequences so that they are now platform independent, taking into account little-endian and big-endian architectures. They will be automatically upgraded in 4.4. \[#12202\] +11. Fix a bug with non-wrapping sequences when initial value was INT64_MIN. \[#12390\] +12. Add a retry for operating system operations that return EIO (IO Error) to better support NFS mounted filesystems. \[#12426\] +13. Fix sequence wrapping at INT64 limits. \[#12520\] +14. Fix a bug where errors during DB-\>associate could leave secondaries half associated. \[#13173\] +15. Fix a bug so that we no longer will update in CDS and DS if the file size limit will be exceeded. \[#13222\] + +### Btree Access Method Changes: + +1. Remove maxkey configuration. \[#8904\] +2. Fix a memory leak in operations on large Btrees. \[#12000\] + +### Hash Access Method Changes: + +1. Fix a bug where access to HASH or encrypted database pages might be blocked during a checkpoint. \[#11031\] +2. Fix a bug where recovery would fail when a database has a hash page on the free list and that hash page was freed without using transactions and later allocated and aborted within a transaction. \[#11214\] +3. Fix a bug in hash duplicates where if the caller left garbage in the partial length field, we were using it. Fix a bug where a replacement of a hash item that should have gone on an overflow page, did not. \[#11966\] +4. Fix a bug where free space was miscalculated when adding the first duplicate to an existing item and the existing item plus the new item does not fit on a page. \[#12270\] +5. Fix a bug where allocations of hash buckets are not recovered correctly. \[#12846\] + +### Queue Access Method Changes: + +1. Improve performance of deletes from a QUEUE database that does not have a secondary index. \[#11538\] +2. Fix a bug where updates that do not use transactions, but do enable locking, failed to release locks. \[#11669\] +3. Fix a bug where a transaction might not be rolled forward if the site was performing hot backups and an application aborted a prepared but not committed transaction. \[#12181\] +4. Fix a bug with queue extents not being reclaimed. \[#12249\] +5. Fix a bug where a record being inserted before the head of the queue could appear missing if DB_CONSUME is not specified. \[#12919\] +6. Fix a bug that might cause recovery to move the head or tail of the queue to exclude a record that was deleted but whose transaction did not commit. \[#13256\] +7. Fix a bug that could cause recovery to move the head or tail pointer beyond a record that was aborted but was rolled backward by recovery. \[#13318\] + +### Recno Access Method Changes + +None. + +### C++-specific API Changes: + +1. Fix a bug so that a DbMemoryException will be raised during a DB_BUFFER_SMALL error. \[#13273\] + +### Java-specific API Changes: + +1. Add VersionMismatchException to map the DB_VERSION_MISMATCH error. \[#11429\] +2. Fix a bug in Environment.getConfiguration() method in non-crypto builds. \[#11752\] +3. Fix a bug that caused a NullPointerException when using the MultipleDataEntry default constructor. \[#11753\] +4. Fix handling of replication errors. \[#11822\] +5. Remove EnvironmentConfig.setReadOnly() method. \[#11882\] +6. Fix a bug where prefix strings in the error handler may be corrupted. \[#11967\] +7. Fix a bug so that nested exceptions will appear in stack traces. \[#11992\] +8. Fix a bug on LogSequenceNumber objects in the Java API. \[#12223\] +9. Fix a bug when no files are returned from a call to DB_ENV-\>log_archive. \[#12383\] +10. Fix a bug when multiple verbose flags are set. \[#12383\] +11. Fix a bug so that an OutOfMemoryError is thrown when allocation fails in the JNI layer. \[#13434\] + +### Java collections and bind API Changes: + +1. Binding performance has been improved by using System.arraycopy in the FastOutputStream and FastInputStream utility classes. \[#12002\] +2. The objectToEntry method is now implemented in all TupleBinding subclasses (IntegerBinding, etc) so that tuple bindings are fully nestable. An example of this usage is a custom binding that dynamically discovers the data types of each of the properties of a Java bean class. For each property, it calls TupleBinding.getPrimitiveBinding using the property's type (class). When the custom binding's objectToEntry method is called, it in turn calls the objectToEntry method of the nested bindings for each property. \[#12124\] +3. The getCause method for IOExceptionWrapper and RuntimeExceptionWrapper is now defined so that nested exceptions appear in stack traces for exceptions thrown by the collections API. \[#11992\] +4. TupleBinding.getPrimitiveBinding can now be passed a primitive type class as well as a primitive wrapper class. The return value for Integer.TYPE and Integer.class, for example, will be the same binding. \[#12035\] +5. Improvements have been made to prevent the buffer used in serial and tuple bindings from growing inefficiently, and to provide more alternatives for the application to specify the desired size. For details see com.sleepycat.bind.serial.SerialBase and com.sleepycat.bind.tuple.TupleBase. \[#12398\] +6. Add StoredContainer.getCursorConfig, deprecate isDirtyRead. Deprecate StoredCollections.dirtyReadMap (dirtyReadSet, etc) which is replaced by configuredMap (configuredSet, etc). Deprecated StoredContainer.isDirtyReadAllowed with no replacement (please use DatabaseConfig.getDirtyRead). Also note that StoredCollections.configuredMap (configuredSet, etc) can be used to configure read committed and write lock containers, as well as read uncommitted containers, since all CursorConfig properties are supported. \[#11776\] +7. Add the protected method SerialBinding.getClassLoader so that subclasses may return a specific or dynamically determined class loader. Useful for applications which use multiple class loaders, including applications that serialize Groovy-defined classes. \[#12764\] \[#12749\] + +### Tcl-specific API Changes: + +1. Fix a bug that could cause a memory leak in the replication test code. \[#13436\] + +### RPC-specific Client/Server Changes: + +1. Fix double-free in RPC server when handling an out-of-memory error. \[#11852\] + +### Replication Changes: + +1. Fix race condition (introduced in 4.3) in rep_start function. \[#11030\] +2. Changed internal initialization to no longer store records. \[#11090\] +3. Add support for replication bulk transfer. \[#11099\] +4. Berkeley DB now calls check_doreq function for MASTER_REQ messages. \[#11207\] +5. Fix a bug where transactions could be counted incorrectly during txn_recover. \[#11257\] +6. Add DB_REP_IGNORE flag so that old messages (especially PERM messages) can be ignored by applications. \[#11585\] +7. Fix a bug where op_timestamp was not initialized. \[#11795\] +8. Fix a bug in db_refresh where a client would write a log record on closing a file. \[#11892\] +9. Fix backward arguments in C++ rep_elect API. \[#11906\] +10. Fix a bug where a race condition could happen between downgrading a master and a database update operation. \[#11955\] +11. Fix a bug on VERIFY_REQ. We now honor wait recs/rcvd. \[#12097\] +12. Fix a bug in rebroadcast of verify_req by initializing lp-\>wait_recs when finding a new master. \[#12097\] +13. Fix a bug by adding lockout checking to \_\_env_rep_enter since rename/remove now call it. \[#12192\] +14. Fix a bug so that we now skip \_\_db_chk_meta if we are a rep client. \[#12316\] +15. Fix a replication failure on Windows. \[#12331\] +16. Remove master discovery phase from rep_elect as a performance improvement to speed up elections. \[#12551\] +17. Fix a bug to avoid multiple data streams when issuing al ALL_REQ. \[#12595\] +18. Fix a bug to request the remaining gap again if the gap record is dropped after we receive the singleton. \[#12974\] +19. Fix a bug in internal initialization when master changes in the middle of initializing. \[#13074\] +20. Fix a bug in replication/archiving with internal init. \[#13110\] +21. Fix pp handling of db_truncate. \[#13115\] +22. Fix a bug where rep_timestamp could be updated when it should not be updated. \[#13331\] +23. Fix a bug with bulk transfer when toggling during updates. \[#13339\] +24. Change EINVAL error return to DB_REP_JOIN_FAILURE. \[#12110\] +25. Add C++ exception for DB_REP_HANDLE_DEAD. \[#13361\] +26. Fix a bug where starting an election concurrently with processing a NEWMASTER message could cause the send function to be called with an invalid eid. \[#13403\] + +### XA Resource Manager Changes: + +None. + +### Locking Subsystem Changes: + +None. + +### Logging Subsystem Changes: + +1. Add set_log_filemode for applications that need to set an absolute file mode on log files. \[#8747\] +2. Fix a bug that caused Not Found to be returned if a log file exists but is not readable. \[#11185\] +3. Removed checksum of records with an in-memory log buffer. \[#11280\] +4. Fix a bug so that the DB_LOG_INMEMORY flag can no longer be set after calling DB_ENV-\>open. \[#11436\] +5. Fix a bug introduced after release 4.0 where two simultaneous checkpoints could cause ckp_lsn values to be out of order. \[#12094\] +6. Fix a bug when in debug mode and using the DEBUG_ROP which will now log read operations in \_\_dbc_logging. \[#12303\] +7. Fix a bug where failing to write a log record on a file close would result in a core dump later. \[#12460\] +8. Fix a bug where automatic log file removal, or the return of log files using an absolute path, could fail silently if the applications current working directory could not be reached using the systems getcwd library call. \[#12505\] +9. Avoid locking the log region if we are not going to flush the log. This can improve performance for some write-intensive application workloads. \[#13090\] +10. Fix a bug with a possible segment fault when memp_stat_print is called on a temporary database. \[#13315\] +11. Fix a bug where log_stat_print could deadlock with threads during a checkpoint. \[#13315\] + +### Memory Pool Subsystem Changes: + +1. Fix a bug where modified database pages might not be flushed if recovery were run and all pages from a database were found in the system cache and up to date, followed by a system crash. \[#11654\] + +### Transaction Subsystem Changes: + +1. Add new DbTxn class methods allowing applications to set/get a descriptive name associated with a transaction. The descriptive name is also displayed by the db_stat utility. \[#0382\] +2. Fix a bug where aborting a transaction with a large number of nested transactions could take a long time. \[#10972\] +3. Add support to allow the TXN_WRITE_NOSYNC flag to be specified on the transaction handle. \[#11151\] +4. Fix a bug that could cause a page to be on the free list twice if it was originally put on the free list by a non-transactional update and then reallocated in a transaction that aborts. \[#11159\] +5. Remove the requirement for the DB_AUTO_COMMIT flag to make database operations transactional. Specifying the database environment as transactional or opening the database handle transactionally is sufficient. \[#11302\] +6. Fix a bug so that environments created from errant programs that called dbp-\>close while transactions were still active can now be recovered. \[#11384\] +7. Fix a bug that caused free pages at the end of a file to be truncated during recovery rather than placed on the free page list. \[#11643\] +8. Fix a bug that caused a page to have the wrong type if the truncate of a BREE or RECNO database needed to be rolled forward. \[#11670\] +9. Fix a bug when manually undoing a subdb create, dont try to free a root page that has not been allocated. \[#11925\] +10. Add a check on database open to see if log files were incorrectly removed by system administration mistakes. \[#12178\] +11. Fix a bug when calling DB-\>pget and then specifying the DB_READ_COMMITTED (DB_DEGREE_2) on a cursor. If followed by a DBC-\>c_pget, the primary database would incorrectly remain locked. \[#12410\] +12. Fix a bug where the abort of a transaction in which a sub database was opened with the DB_TXN_NOT_DURABLE flag could fail. \[#12420\] +13. Fix a bug that could cause an abort transaction that allocated new pages to a file that were not flushed to disk prior to the abort transaction to report out of disk space. \[#12743\] +14. Fix a bug that could prevent multiple creates and destroys of the same file to be recovered correctly. \[#13026\] +15. Fix a bug when recovery previously handled a section of the log that did not contain any transactions. \[#13139\] +16. Fix a bug that could result in the loss of durability in Transactional Environments on Mac OS X. \[#13149\] +17. Fix a bug that could cause the improper reuse of a transaction id when recovery restores prepared transactions. \[#13256\] + +### Utility Changes: + +1. Add utility for performing hot backups of a database environment. \[#11536\] +2. Change the Verify utility to now identify any nodes that have incorrect record counts. \[#11934\] +3. Fix a bug in the 1.85 compatibility code supporting per-application Btree comparison and prefix compression functions. The functions would not work on big-endian 64-bit hardware. \[#13316\] + +### Configuration, Documentation, Portability and Build Changes: + +1. Change the ex_tpcb sample application to no longer displays intermediate results. It displays results at the end of the run. \[#11259\] +2. Change the Visual Studio projects on Windows so that each is in an intermediate directory. \[#11441\] +3. Fix errors in test subdb011. \[#11799\] +4. Fix a bug that could cause applications using gcc on Power PC platforms to hang. \[#12233\] +5. Fix a bug where installation will fail if a true program cannot be found. \[#12278\] +6. Fix a bug that prevented C++ applications from configuring XA \[#12300\]. +7. Fix a race condition in the Windows mutex implementation found on 8-way Itanium systems. \[#12417\] +8. Add pthread mutex support for IBM OS/390 platform (z/OS or MVS). \[#12639\] +9. Fix a bug where the Tcl API did not configure on OS X 10.4. \[#12699\] +10. Fix portability issues with queue or recno primary databases. \[#12872\] +11. Fix a bug where utility attempted to send replication message. \[#13446\] diff --git a/docs-src/guides/upgrading/changelog_4_4_20.md b/docs-src/guides/upgrading/changelog_4_4_20.md new file mode 100644 index 000000000..a5274c898 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_4_20.md @@ -0,0 +1,24 @@ +--- +title: "Berkeley DB 4.4.20 Change Log" +api-name: "Berkeley DB 4.4.20 Change Log" +source: docs/upgrading/changelog_4_4_20.html +--- +## Berkeley DB 4.4.20 Change Log + + [Changes since Berkeley DB 4.4.16:](changelog_4_4_20.md#idp50624312) + +### Changes since Berkeley DB 4.4.16: + +1. Add support for Visual Studio 2005. \[#13521\] +2. Fix a bug with in-memory transaction logs when files wrapped around the buffer. \[#13589\] +3. Fix a bug where we needed to close replications open files during replication initialization. \[#13623\] +4. Fix a bug which could leave locks in the environment if database compaction was run in a transactional environment on a non-transactional database. This might have also have triggered deadlocks if the database was opened transactionally. \[#13680\] +5. Fix a bug where setting the DB_REGISTER flag could result in unnecessarily running recovery, or corruption of the registry file on Windows systems. \[#13789\] +6. Fix a bug in Database.compact that could cause JVM crashes or NullPointerException. \[#13791\] +7. Fix a bug that would cause a trap if an environment was opened specifying DB_REGISTER and the environment directory could not be found. \[#13793\] +8. Fix a buffer overflow bug when displaying process and thread IDs in the Berkeley DB statistics output. \[#13796\] +9. Fix a bug where if there is insufficient memory for a database key in a DBT configured to return a key value into user-specified memory, the cursor is moved forward to the next entry in the database, which can cause applications to skip key/data pairs. \[#13815\] +10. Fix a bug that could cause the loss of an update to a QUEUE database in a hot backup. \[#13823\] +11. Fix a bug where retrieval from a secondary index could result in a core dump. \[#13843\] +12. Fix a bug that could cause part of the free list to become unlinked if a btree compaction was rolled back due to a transaction abort. \[#13891\] +13. Fix a bug with in-memory logging that could cause a race condition to corrupt the logs. \[#13919\] diff --git a/docs-src/guides/upgrading/changelog_4_5_20.md b/docs-src/guides/upgrading/changelog_4_5_20.md new file mode 100644 index 000000000..42bfcfe98 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_5_20.md @@ -0,0 +1,210 @@ +--- +title: "Berkeley DB 4.5.20 Change Log" +api-name: "Berkeley DB 4.5.20 Change Log" +source: docs/upgrading/changelog_4_5_20.html +--- +## Berkeley DB 4.5.20 Change Log + + [Database or Log File On-Disk Format Changes:](changelog_4_5_20.md#idp50532016) + + [New Features:](changelog_4_5_20.md#idp50511048) + + [Database Environment Changes:](changelog_4_5_20.md#idp50513672) + + [Concurrent Data Store Changes:](changelog_4_5_20.md#idp50516704) + + [General Access Method Changes:](changelog_4_5_20.md#idp50520456) + + [Btree Access Method Changes:](changelog_4_5_20.md#idp50542544) + + [Hash Access Method Changes:](changelog_4_5_20.md#idp50536608) + + [Queue Access Method Changes:](changelog_4_5_20.md#idp50533200) + + [Recno Access Method Changes:](changelog_4_5_20.md#idp50539504) + + [C++-specific API Changes:](changelog_4_5_20.md#idp50539760) + + [Java-specific API Changes:](changelog_4_5_20.md#idp50541672) + + [Java collections and bind API Changes:](changelog_4_5_20.md#idp50542632) + + [Tcl-specific API Changes:](changelog_4_5_20.md#idp50546176) + + [RPC-specific Client/Server Changes:](changelog_4_5_20.md#idp50548752) + + [Replication Changes:](changelog_4_5_20.md#idp50547824) + + [XA Resource Manager Changes:](changelog_4_5_20.md#idp50557816) + + [Locking Subsystem Changes:](changelog_4_5_20.md#idp50534496) + + [Logging Subsystem Changes:](changelog_4_5_20.md#idp50532216) + + [Memory Pool Subsystem Changes:](changelog_4_5_20.md#idp50542056) + + [Transaction Subsystem Changes:](changelog_4_5_20.md#idp50543016) + + [Utility Changes:](changelog_4_5_20.md#idp50556608) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_5_20.md#idp50557880) + +### Database or Log File On-Disk Format Changes: + +1. The on-disk log format has changed. + +### New Features: + +1. Multi-Version Concurrency Control for the Btree/Recno access methods. +2. A new replication framework with a default TCP/IP setup. +3. Online replication upgrades for high availability replicated 24/7 systems. +4. A new event-style notification. +5. Several enhancements to the Java Collections API including the implementation of the size() method. + +### Database Environment Changes: + +1. Update the DB_ENV-\>failchk method to garbage collect per-process mutexes stranded after unexpected process failure. \[#13964\] +2. Fix a bug that could cause memory used to track threads for DB_ENV-\>failchk to not be reused when a thread no longer exists. \[#14425\] +3. Add set_event_notify behavior as part of new event notification in Berkeley DB. \[#14534\] +4. Fix a bug so that we no longer panic on DB_ENV-\>close() if a previous environment close failed to log. This condition will now return an error. \[#14693\] +5. Created os_getenv, removed clib/getenv, implemented Windows specific behavior. \[#14942\] +6. Fix a bug where it was possible to corrupt the DB_REGISTER information file, making it impossible for Berkeley DB applications to join database environments. \[#14998\] + +### Concurrent Data Store Changes: + +1. Fix a bug where renaming a subdatabase in a Concurrent Data Store environment could fail. \[#14185\] + +### General Access Method Changes: + +1. Fix a bug that could leave extra unallocated pages at the end of a database file. \[#14031\] +2. Optimize secondary updates when overwriting primary records. \[#14075\] +3. Fix a bug to prevent a trap when creating a named in-memory database and there are already temporary files open. \[#14133\] +4. Fix a bug which caused a trap if the key parameter to DBC-\>c_get was omitted with DB_CURRENT. \[#14143\] +5. Fix a bug with secondary cursors when the secondary has off-page duplicates. This bug resulted in incorrect primary data being returned. \[#14240\] +6. Improve performance when removing a subdatabase by not locking every page. \[#14366\] +7. Fix a bug that would not properly upgrade database files from releases 3.2.9 (and earlier) to releases 4.0 (and greater). \[#14461\] +8. Fix a bug that could cause a DB_READ_UNCOMMITTED get through a secondary index to return DB_SECONDARY_CORRUPT. \[#14487\] +9. Fix a bug so that non-transactional cursor updates of a transactional database will generate an error. \[#14519\] +10. Add a message when the system panics due to a page in the wrong state at its time of allocation. \[#14527\] +11. Fix a remove failure when attempting to remove a file that is open in another thread of control. \[#14780\] +12. Fix a bug where the key was not ignored when doing a cursor put with the DB_CURRENT flag. \[#14988\] + +### Btree Access Method Changes: + +1. When deleting a page don't check the next key in the parent if we are going to delete the parent too. +2. Need to check that the tree has not collapsed between dropping a read lock and getting the write lock. If it has collapsed we will fetch the root of the tree. +3. Fix a case where we fail to lock the next page before reading it. + + + +1. Changed the implementation of internal nodes in btrees so that they no longer share references to overflow pages with leaf nodes. \[#10717\] +2. Fix a bug that could cause a diagnostic assertion by setting the deleted bit on a record in an internal node. \[#13944\] +3. Fix three problems in BTREE compaction: \[#14238\] +4. Fix a bug that could cause the compaction of a Btree with sorted duplicates to fail when attempting to compact an off page duplicate tree if a key could not fit in an internal node. \[#14771\] +5. Fix a bug that causes a loop if an empty Btree was compacted. \[#14493\] + +### Hash Access Method Changes: + +1. Fix a bug to allow creation of hash pages during truncate recovery. \[#14247\] + +### Queue Access Method Changes: + +1. Fix a bug where reads of data items outside the range of the queue were not kept locked to the end of the transaction, breaking serializability. \[#13719\] +2. Fix a bug that could cause corruption in queue extent files if multiple processes tried to open the same extent at the same time. \[#14438\] +3. Improve concurrency for in-place updates in queue databases. \[#14918\] + +### Recno Access Method Changes: + +None. + +### C++-specific API Changes: + +1. C++ applications that check the error code in exceptions should note that DbMemoryException has been changed to have the error code DB_BUFFER_SMALL rather than ENOMEM, to match the error returned by the C API. DbMemoryException will be thrown when a Dbt is too small to contain data returned by Berkeley DB. When a call to malloc fails, or some other resource is exhausted, a plain DbException will be thrown with error code set to ENOMEM. \[#13939\] + +### Java-specific API Changes: + +1. Database.verify may now be called. This method is now static and takes a DatabaseConfig parameter. \[#13971\] +2. Add DB_ENV-\>{fileid_reset, lsn_reset} to the public API. \[#14076\] + +### Java collections and bind API Changes: + +1. The com.sleepycat.collections package is now fully compatible with the Java Collections framework. \[#14732\] + +### Tcl-specific API Changes: + +1. Fix a conflicting variable, sysscript.tcl. \[#15051\] + +### RPC-specific Client/Server Changes: + +None. + +### Replication Changes: + +1. Fix a bug when running with DEBUG_ROP or DEBUG_WOP. \[#13394\] +2. Add live replication upgrade support \[#13670\] +3. Fix a bug so that client databases are removed at the start of internal initialization. \[#14147\] +4. Fix a bug in replication internal initialization so that data_dir will be handled correctly. Make internal initialization resilient to multiple data_dir calls with the same directory. \[#14489\] +5. Fix a bug in the 4.2 sync-up algorithm that could result in no open files. \[#14552\] +6. Fix a bug when clients decide to re-request. \[#14642\] +7. Fix a bug where a PERM bulk buffer could have a zero LSN passed to the application callback. \[#14675\] +8. Change names of some existing replication API methods as described in Replication method naming. \[#14723\] +9. Fix a bug which could cause an election to succeed only after waiting for the timeout to expire, even when all sites responded in a timely manner. The bug was most easily visible in an election between 2 sites. \[#14752\] +10. Fix a bug where a process could have an old file handle to a log file. \[#14797\] +11. Fix a bug where a "log_more" message could be on a log file boundary. \[#15034\] +12. Fix a bug that could cause log corruption if a database open operation were attempted during a call to rep_start in another thread. \[#15035\] +13. Fix a bug during elections where a vote2 arrives before its vote1. \[#15055\] +14. Fix a bug to make sure we are a client if sending a REP_REREQUEST. \[#15066\] + +### XA Resource Manager Changes: + +None. + +### Locking Subsystem Changes: + +1. Fix a bug that could cause a write to hang if DB_READ_UNCOMMITTED is enabled and it tries to reacquire a write lock. \[#14919\] + +### Logging Subsystem Changes: + +1. Fix a bug so that log headers are now included in the checksum. This avoids a possible race in doing hot backups. \[#11636\]. +2. Add a check so that some log sequence errors are diagnosed at run time rather than during recovery. \[#13231\] +3. Fix a bug where recovery fails if there is no disk space for the forced checkpoint that occurs at the end of processing the log. \[#13986\] +4. Fix a bug which could cause a page to be missing from the end of a database file if the page at the end of the file was freed while it contained data and the system was restarted before the log record for that free was flushed to disk. \[#14090\] +5. Fix a bug that could cause log files to be incorrectly removed by log_archive if it was run immediately after recovery. \[#14874\] + +### Memory Pool Subsystem Changes: + +1. Fix a bug that could cause corruption to the buffer pool cache if a race condition was hit while using DB-\>compact. \[#14360\] +2. Fix a bug where cache pages could be leaked in applications creating temporary files for which the DB_MPOOL_NOFILE flag was set. \[#14544\] + +### Transaction Subsystem Changes: + +1. Fix a bug that could cause extra empty pages to appear in a database file after recovery. \[#11118\] +2. Fix a bug triggered when running recovery with a feedback function that could cause a NULL pointer dereference. \[#13834\] +3. Fix a bug where running recovery could create duplicate entries in the data directory list. \[#13884\] +4. Fix a bug to not trade locks if a write lock is already owned. \[#13917\] +5. Fix a bug that could cause traps or hangs if the DB_TXN-\>set_name function is used in a multithreaded application. \[#14033\] +6. Fix a bug so that a transaction can no longer be committed after it had deadlocked. \[#14037\] +7. Fix a bug that could cause a trap during recovery if multiple operations that could remove the same extent are recovered. \[#14061\] +8. Fix a bug that could cause an extent file to be deleted after the last record in the extent was consumed but the consuming transaction was aborted. \[#14179\] +9. Fix a bug where the parent database would not use DB_READ_UNCOMMITTED in certain cases when calling DBC-\>c_pget. \[#14361\] +10. Fix a bug so that it is no longer possible to do a non-transactional cursor update on a database that is opened transactionally. \[#14519\] +11. Fix a bug that causes a sequence to ignore the DB_AUTO_COMMIT settings. \[#14582\] +12. Fix a bug, change txn_recover so that multiple processes will recover prepared transactions without requiring that the first process stay active. \[#14707\] +13. Fix a bug that could cause the wrong record to be deleted if a transaction had a cursor on a record with a pending delete and then replaced a record that contained overflow data or replaced a record with overflow data and that replace failed. \[#14834\] + +### Utility Changes: + +1. Fix a bug that caused db_verify to not check the order on leaf pages which were the leftmost children of an internal node. \[#13004\] +2. Fix a bug that caused db_hotbackup to not backup queue extent files. \[#13848\] +3. Fix a bug so that db_verify no longer reports that an unused hash page is not fully zeroed. \[#14030\] +4. Fix a bug where db_stat ignored the -f option to return "fast statistics". \[#14283\] +5. Fix a bug that prevented the db_stat utility from opening database files with write permission so that meta data statistics would be updated. \[#14755\] +6. Fix a bug in db_hotbackup related to windows. Sub-directories are now ignored. \[#14757\] + +### Configuration, Documentation, Portability and Build Changes: + +1. The Berkeley DB 4.3 and 4.4 releases disallowed using the --with-uniquename configuration option with the C++, Java, or RPC --enable-XXX options. The 4.5 release returns to the 4.2 release behavior, allowing those combinations of configuration options. \[#14067\] +2. Fix build issues when CONFIG_TEST is not enabled for Tcl. \[#14507\] +3. There are updated build instructions for Berkeley DB PHP module on Linux. \[#14249\] +4. Use libtool's "standard" environment variable names so that you can set "AR" to "ar -X64" for example, and modify both libtool and the Makefile commands. Remove the install-strip target from the Makefile, it is no longer used. \[#14726\] +5. Fix a bug where, when a database is opened with the DB_THREAD flag (the default in Java), and an operation in one thread causes the database to be truncated (typically when the last page in the database is freed) concurrently with a read or write in another thread, there can be arbitrary data loss, as Windows zeros out pages from the read/write location to the end of the file. \[#15063\] diff --git a/docs-src/guides/upgrading/changelog_4_6.md b/docs-src/guides/upgrading/changelog_4_6.md new file mode 100644 index 000000000..7b60c55b9 --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_6.md @@ -0,0 +1,244 @@ +--- +title: "Berkeley DB 4.6.21 Change Log" +api-name: "Berkeley DB 4.6.21 Change Log" +source: docs/upgrading/changelog_4_6.html +--- +## Berkeley DB 4.6.21 Change Log + + [4.6.21 Patches:](changelog_4_6.md#idp50449856) + + [4.6.19 Patches](changelog_4_6.md#idp50370888) + + [Database or Log File On-Disk Format Changes:](changelog_4_6.md#idp50361912) + + [New Features:](changelog_4_6.md#idp50454856) + + [Database Environment Changes:](changelog_4_6.md#idp50457960) + + [Concurrent Data Store Changes:](changelog_4_6.md#idp50459800) + + [General Access Method Changes:](changelog_4_6.md#idp50458344) + + [Btree Access Method Changes:](changelog_4_6.md#idp50475672) + + [Hash Access Method Changes:](changelog_4_6.md#idp50460536) + + [Queue Access Method Changes:](changelog_4_6.md#idp50444272) + + [Recno Access Method Changes:](changelog_4_6.md#idp50463616) + + [C++-specific API Changes:](changelog_4_6.md#idp50463872) + + [Java-specific API Changes:](changelog_4_6.md#idp50481800) + + [Java collections and bind API Changes:](changelog_4_6.md#idp50464456) + + [Tcl-specific API Changes:](changelog_4_6.md#idp50464944) + + [RPC-specific Client/Server Changes:](changelog_4_6.md#idp50465232) + + [Replication Changes:](changelog_4_6.md#idp50486584) + + [XA Resource Manager Changes:](changelog_4_6.md#idp50466136) + + [Locking Subsystem Changes:](changelog_4_6.md#idp50465496) + + [Logging Subsystem Changes:](changelog_4_6.md#idp50451848) + + [Memory Pool Subsystem Changes:](changelog_4_6.md#idp50452712) + + [Transaction Subsystem Changes:](changelog_4_6.md#idp50468064) + + [Utility Changes:](changelog_4_6.md#idp50475736) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_6.md#idp50479800) + +### 4.6.21 Patches: + +1. Fix a bug where mutex contention in database environments configured for hybrid mutex support could result in performance degradation. \[#15646\] +2. Fix a bug where closing a database handle after aborting a transaction which included a failed open of that database handle could result in application failure. \[#15650\] +3. Fix multiple MVCC bugs including a race which *could result in incorrect data being returned* to the application. \[#15653\] +4. Fix a bug where a database store into a Hash database could self-deadlock in a database environment configured for the Berkeley DB Concurrent Data Store product and with a free-threaded DB_ENV or DB handle. \[#15718\] +5. Fix an installation bug where Berkeley DB's PHP header file was not installed in the correct place. + +### 4.6.19 Patches + +1. Fix a bug where a client in a two-site replication group could become master, after failure of the existing master, even if the client had priority 0. \[#15388\] +2. Fix a bug where 32-bit builds on 64-bit machines could immediately core dump because of a misaligned access. \[#15643\] +3. Fix a bug where attempts to configure a database for MVCC in the Java API were silently ignored. \[#15644\] +4. Fix a bug where database environments configured for replication and verbose output could drop core. \[#15651\] + +### Database or Log File On-Disk Format Changes: + +1. The on-disk log format has changed. +2. The format of Hash database pages was changed in the Berkeley DB 4.6 release, and items are now stored in sorted order. *The format changes are entirely backward-compatible, and no database upgrades are needed.* However, upgrading existing databases can offer significant performance improvements. Note that databases created using the 4.6 release may not be usable with earlier Berkeley DB releases. + +### New Features: + +1. Add support for a cursor DB_PREV_DUP flag, which moves the cursor to the previous key/data pair if it's a duplicate of the current key/data pair. \[#4801\] +2. Add the ability to set cache page priority on a database or cursor handle. \[#11886\] +3. Add verbose output tracing for filesystem operations. \[#13760\] +4. Port Berkeley DB to Qualcomm's Binary Runtime Environment for Wireless (BREW). \[#14562\] +5. Port Berkeley DB to WinCE. \[#15312\] +6. Port Berkeley DB to S60. \[#15371\] +7. Add a key_exists method to the DB handle. \[#15374\] +8. Applications may now begin processing new transactions while previously prepared, but unresolved, transactions are still pending. \[#14754\] +9. Significant performance improvements in the Hash access method. \[#15017\] + +### Database Environment Changes: + +1. Add support to close open file handles in the case of catastrophic database environment failure so applications that do not exit and restart on failure won't leak file handles. \[#6538\] +2. Replace the Berkeley DB shared memory allocator with a new implementation, intended to decrease the performance drop-off seen in database environments having working sets that are larger than the cache, especially database environments with multiple cache page sizes. \[#13122\] +3. Fix a bug that would incorrectly cause a thread to appear to be in the Berkeley DB API after a call to db_create. \[#14562\] +4. Allow database close prior to resolving all transactions updating the database. \[#14785\] +5. Fix a bug where the db_stat utility -Z flag and the statistics method's DB_STAT_CLEAR flag could clear mutex statistics too quickly, leading to incorrect values being displayed. \[#15032\] +6. Fix a bug where removal of a file after and open/close pair spanning the most recent checkpoint log-sequence-numbers made recovery fail. \[#15092\] +7. Fix a bug that could leave an environment unrecoverable if FTRUNCATE was not set and a roll-forward to a timestamp was interrupted between the truncation of the log and the recording of aborted allocations. \[#15108\] +8. Fix a bug where recovery of a rename operation could fail if the rename occurred in a directory that no longer existed. \[#15119\] +9. Fix a bug that could cause recovery to report a "File exists" error if a committed create was partially recovered by a previously failed recovery operation. \[#15151\] +10. Fix a bug where the DbEnv.get_thread_count method implementation was missing from the Berkeley DB 4.5 release. \[#15201\] +11. Fix a bug where replication operations were not reported properly when the DbEnv.failchk method was called. \[#15094\] +12. Fixed a bug that caused SEQ-\>remove not to use a transaction if the sequence was opened on a transactional database handle but no transaction was specified on the call. \[#15235\] +13. Fix a bug where accesses to the database environment reference count could race, causing the DB_ENV-\>remove method to incorrectly remove or not remove a database environment. \[#15240\] +14. Fix a bug that could cause a recovery failure if a partial record was written near the end of a log file before a crash and then never overwritten after recovery runs and before a log file switch occurs. \[#15302\] +15. Fix a bug that could fire a diagnostic assertion if an error occurred during a database environment open. \[#15309\] +16. Fix a bug where memp_trickle attempts to flush an infinite number of buffers. \[#15342\] +17. Cause application updates of the DB_ENV-\>set_mp_max_write values to affect already running cache flush operations. \[#15342\] +18. Fix a bug which could cause system hang if a checkpoint happened at the same time as a database file create or rename. \[#15346\] +19. Fix a bug which could cause application failure if the open of a subdatabase failed while other database opens were happening. \[#15346\] +20. Fix a bug that could cause recovery to not process a transaction properly if the transaction was started before the transaction IDs were reset but did not put its first record into the log until after the txn_recycle record. \[#15400\] +21. Fix a bug that could cause a thread in cache allocation to loop infinitely. \[#15406\] +22. Fix a bug that could cause recovery to report a Log Sequence Error on systems without the ftruncate system call where a page allocation occurred and the database metadata page was forced out of cache without being marked dirty and then had to be recovered. \[#15441\] +23. Fix a bug on systems lacking the ftruncate system call, where a page may be improperly linked into the free list if archive recovery was done in multiple steps, that is, applying additional logs to the same databases. \[#15557\] + +### Concurrent Data Store Changes: + +None. + +### General Access Method Changes: + +1. Add a feature where applications can specify a custom comparison function for the Hash access method \[#4109\] +2. Open, create, close and removal of non-transactional databases is are longer logged in transactional database environments unless debug logging is enabled. \[#8037\] +3. Add the ability to set cache page priority on a database or cursor handle. \[#11886\] +4. fix a bug where the DB_ENV-\>fileid_reset method failed when called on on encrypted or check-summed databases. \[#13990\] +5. Fix a bug where the DB-\>fd method failed when called on in-memory databases. \[#14157\] +6. Fix a bug where an attempt to open a Recno database with a backing file that does not exist could report an error because it couldn't remove a temporary file. \[#14160\] +7. Reverse a change found in previous releases which disallowed setting "partial" flags on key DBTs for DB and DbCursor put method calls. \[#14520\] +8. Fix a bug where transactional file operations, such as remove or rename, could leak file handles. \[#15222\] +9. Fix a bug that could cause the in-memory sorted freelist used by the DB-\>compact method not to be freed if transaction or lock timeouts were set in the environment. \[#15292\] +10. Add the DB-\>get_multiple method, which returns if the DB handle references a "master" database in the physical file. \[#15352\] +11. Fix a bug that could cause an DB_INORDER, DB-\>get method DB_CONSUME operation to loop if the Queue database was missing a record due to a rollback by a writer or a non-queue insert in the queue. \[#15452\] +12. Fix a bug preventing database removal after application or system failure in a database environment configured for in-memory logging. \[#15459\] + +### Btree Access Method Changes: + +None. + +### Hash Access Method Changes: + +1. Change the internal format of Hash database pages, storing items in sorted order. There are no externally visible changes, and hash databases using historic on-page formats do not require an explicit upgrade. (However, upgrading existing databases can offer significant performance improvements.) \[#15017\] +2. Fix a bug preventing LSNs from being reset on hash databases when the databases were configured with a non-standard hash function. \[#15567\] + +### Queue Access Method Changes: + +1. Fix a bug which could cause a Queue extent file to be incorrectly removed if an empty extent file was being closed by one thread and being updated by another thread (which was using random access operations). \[#9101\] + +### Recno Access Method Changes: + +None. + +### C++-specific API Changes: + +None. + +### Java-specific API Changes: + +1. Add a feature where an exception is thrown by the Java API, the Berkeley DB error message is now included in the exception object. \[#11870\] +2. Fix a bug which can cause a JVM crash when doing a partial get operation. \[#15143\] +3. Fix a bug which prevented the use of Berkeley DB sequences from Java. \[#15220\] +4. Fix multiple bugs where DBTs were not being copied correctly in the Java replication APIs. \[#15223\] +5. Add transaction.commitWriteNoSync to the Java API. \[#15376\] + +### Java collections and bind API Changes: + +1. Change SerialBinding to use the current thread's context class loader when loading application classes. This allows the JE jar file to be deployed in application servers and other containers as a shared library rather than as an application jar. \[#15447\] +2. Tuple bindings now support the java.math.BigInteger type. Like other tuple binding values, BigInteger values are sorted in natural integer order by default, without using a custom comparator. For details please see the Javadoc for: com.sleepycat.bind.tuple.TupleInput.readBigInteger com.sleepycat.bind.tuple.TupleOutput.writeBigInteger com.sleepycat.bind.tuple.BigIntegerBinding \[#15244\] +3. Add tuple binding methods for reading and writing packed int and long values. Packed integer values take less space, but take slightly more processing time to read and write. See: TupleInput.readPackedInt TupleInput.getPackedIntByteLength TupleInput.readPackedLong TupleInput.getPackedLongByteLength TupleOutput.writePackedInt TupleOutput.writePackedLong PackedInteger \[#15422\] +4. The Collections API has been enhanced so that auto-commit works for the standard Java Iterator.remove(), set() and add() methods. Previously it was necessary to explicitly begin and commit a transaction in order to call these methods, when the underlying Database was transactional. Note that starting a transaction is still necessary when calling these methods if the StoredCollection.storedIterator method is used. \[#15401\] +5. Fix a bug that causes a memory leak for applications where both of the following are true: many Environment objects are opened and closed, and the CurrentTransaction or TransactionRunner class is used. \[#15444\] + +### Tcl-specific API Changes: + +None. + +### RPC-specific Client/Server Changes: + +None. + +### Replication Changes: + +1. Fix a bug where transactions could be rolled-back if an existing replication group master was partitioned and unable to participate in an election. \[#14752\] +2. Add a new event when a replication manager framework master fails to send and confirm receipt by clients of a "permanent" message. \[#14775\] +3. Fix a race where multiple threads might attempt to process a LOGREADY condition. \[#14902\] +4. Change the DB_VERB_REPLICATION flag to no longer require the Berkeley DB library be built with the --enable-diagnostic configuration option to output additional replication logging information. \[#14991\] +5. Fix a bug with elections occurring during internal init of a replication client site. \[#15057\] +6. Fix lockout code to lockout message threads and API separately. Send indication that log requests is for internal init. \[#15067\] +7. Replication manager changed to retry host-name look-up failures, since they could be caused by transient name server outage. \[#15081\] +8. Fix a bug which led to memory corruption when the sending of a bulk buffer resulted in an error. \[#15100\] +9. A throttling limit of 10 megabytes is now set by default in a newly created database environment (see the DbEnv.rep_set_limit method). \[#15115\] +10. Fix a bug in ALL_REQ handling where master could get a DB_NOTFOUND. \[#15116\] +11. Fix a bug which could lead to client sites repeatedly but unproductively calling for an election, when a master site already exists. \[#15128\] +12. Modify gap processing algorithms so XXX_MORE messages ask for data beyond what it just processed, not an earlier gap that might exist. \[#15136\] +13. Fixed a bug in the ex_rep example application which could cause the last few transactions to disappear when shutting down the sites of the replication group gracefully. \[#15162\] +14. Fix a bug where if a client crashed during internal init, its database environment would be left in a confused state, making it impossible to synchronize again with the master. \[#15177\] +15. Fix a bug where election flags are not cleared atomically with the setting of the new master ID. \[#15186\] +16. Fix a bug which would cause Berkeley DB to crash if an internal init happened when there were no database files at the master. \[#15227\] +17. It is now guaranteed that the DB_EVENT_REP_STARTUPDONE event will be presented to the application after the corresponding DB_EVENT_REP_NEWMASTER event, even in the face of extreme scheduling anomalies. \[#15265\] +18. Fix minor memory leaks in the replication manager. \[#15239\] \[#15256\] +19. Fix a bug which caused the replication manager to lose track of a failed connection, resulting in the inability to accept a replacement connection. \[#15311\] +20. Fix a bug where a client starting an election when the rest of the replication group already had an established master could confuse replication management at the other client sites, leading to failure to properly acknowledge PERM transactions from the master. \[#15428\] +21. Add support for reporting Replication Manager statistics. \[#15430\] +22. Fix a bug where a send failure during processing of a request message from a client could erroneously appear to the application as an EPERM system error. \[#15436\] +23. Client now sets STARTUPDONE at the end of the synchronization phase when it has caught up to the end of the master's transaction log, without requiring ongoing transactions at the master. \[#15542\] +24. Fix a bug in sleep-time calculation which could cause a Replication Manager failure. \[#15552\] + +### XA Resource Manager Changes: + +None. + +### Locking Subsystem Changes: + +1. Change the DB_ENV-\>lock_detect method to return the number of transactions timed out in addition to those were rejected due to deadlock. \[#15281\] + +### Logging Subsystem Changes: + +None. + +### Memory Pool Subsystem Changes: + +1. Fix a bug that could cause a checkpoint to hang if a database was closed while the checkpoint was forcing that file to disk and all the pages for that database were replaced in the cache. \[#15135\] +2. Fix a bug where a system error in closing a file could result in a core dump. \[#15137\] +3. Fix MVCC statistics counts for private database environments. \[#15218\] + +### Transaction Subsystem Changes: + +1. Fix a bug where creating a database with the DB_TXN_NOTDURABLE flag set would still write a log record. \[#15386\] +2. Change transaction checkpoint to wait only for pages being updated during the checkpoint. \[#14710\] + +### Utility Changes: + +1. Fix a bug that prevented db_load from handling subdatabase names that were of zero length. \[#8204\] +2. Fix a bug where the db_hotbackup utility did not clean out and record the log file numbers in the backup directory when both the -u and -D flags were specified. \[#15395\] + +### Configuration, Documentation, Portability and Build Changes: + +1. Berkeley DB no longer supports process-shared database environments on Windows 9X platforms; the DB_PRIVATE flag must always be specified to the DB_ENV-\>open method. \[#13766\] +2. Port Berkeley DB to Qualcomm's Binary Runtime Environment for Wireless (BREW). \[#14562\] +3. Compile SWIG-generated code with the -fno-strict-aliasing flag when using the GNU gcc compiler. \[#14953\] +4. Changed include files so ENOENT is resolved on Windows. \[#15078\] +5. Port Berkeley DB to WinCE. \[#15312\] +6. Port Berkeley DB to S60. \[#15371\] +7. Add the db_hotbackup executable to the Windows MSI installer. \[#15372\] +8. Change the db_hotbackup utility to use the Berkeley DB library portability layer. \[#15415\] +9. Re-write the GNU gcc mutex implementation on the x86 platform to avoid compiler errors. \[#15461\] +10. Fix a bug with non-HFS filesystems under OS X which could affect data durability. \[#15501\] diff --git a/docs-src/guides/upgrading/changelog_4_7.md b/docs-src/guides/upgrading/changelog_4_7.md new file mode 100644 index 000000000..6739ef52d --- /dev/null +++ b/docs-src/guides/upgrading/changelog_4_7.md @@ -0,0 +1,224 @@ +--- +title: "Berkeley DB 4.7.25 Change Log" +api-name: "Berkeley DB 4.7.25 Change Log" +source: docs/upgrading/changelog_4_7.html +--- +## Berkeley DB 4.7.25 Change Log + + [Database or Log File On-Disk Format Changes:](changelog_4_7.md#idp50357648) + + [New Features:](changelog_4_7.md#idp50378912) + + [Database Environment Changes:](changelog_4_7.md#idp50380752) + + [Concurrent Data Store Changes:](changelog_4_7.md#idp50382592) + + [General Access Method Changes:](changelog_4_7.md#idp50381120) + + [Btree Access Method Changes:](changelog_4_7.md#idp50391248) + + [Hash Access Method Changes:](changelog_4_7.md#idp50365280) + + [Queue Access Method Changes:](changelog_4_7.md#idp50355136) + + [Recno Access Method Changes:](changelog_4_7.md#idp50346288) + + [C-specific API Changes:](changelog_4_7.md#idp50346816) + + [Java-specific API Changes:](changelog_4_7.md#idp50347072) + + [Direct Persistence Layer (DPL), Bindings and Collections API:](changelog_4_7.md#idp50347296) + + [Tcl-specific API Changes:](changelog_4_7.md#idp50386200) + + [RPC-specific Client/Server Changes:](changelog_4_7.md#idp50395072) + + [Replication Changes:](changelog_4_7.md#idp50395328) + + [XA Resource Manager Changes:](changelog_4_7.md#idp50391504) + + [Locking Subsystem Changes:](changelog_4_7.md#idp50357904) + + [Logging Subsystem Changes:](changelog_4_7.md#idp50397528) + + [Memory Pool Subsystem Changes:](changelog_4_7.md#idp50385512) + + [Mutex Subsystem Changes:](changelog_4_7.md#idp50386616) + + [Transaction Subsystem Changes:](changelog_4_7.md#idp50391632) + + [Utility Changes:](changelog_4_7.md#idp50396200) + + [Configuration, Documentation, Sample Application, Portability and Build Changes:](changelog_4_7.md#idp50412288) + +### Database or Log File On-Disk Format Changes: + +1. The log file format changed in 4.7. + +### New Features: + +1. The lock manager may now be fully partitioned, improving performance on some multi-CPU systems. \[#15880\] +2. Replication groups are now architecture-neutral, supporting connections between differing architectures (big-endian or little-endian, independent of structure padding). \[#15787\] \[#15840\] +3. Java: A new Direct Persistence Layer adds a built-in Plain Old Java Object (POJO)-based persistent object model, which provides support for complex object models without compromises in performance. For an introduction to the Direct Persistence Layer API, see Getting Started with Data Storage. \[#15936\] +4. Add the DB_ENV-\>set_intermediate_dir_mode method to support the creation of intermediate directories needed during recovery. \[#15097\] +5. The DB_ENV-\>failchk method can now abort transactions for threads, which have failed while blocked on a concurrency lock. This significantly decreases the need for database environment recovery after thread of control failure. \[#15626\] +6. Replication Manager clients now can be configured to monitor the connection to the master using heartbeat messages, in order to promptly discover connection failures. \[#15714\] +7. The logging system may now be configured to pre-zero log files when they are created, improving performance on some systems. \[#15758\] + +### Database Environment Changes: + +1. Restructure aborted page allocation handling on systems without an ftruncate system call. This enables the Berkeley DB High Availability product on systems, which do not support ftruncate. \[#15602\] +2. Fix a bug where closing a database handle after aborting a transaction which included a failed open of that handle could result in application failure. \[#15650\] +3. Fix minor memory leaks when closing a private database environment. \[#15663\] +4. Fix a bug leading to a panic of "unpinned page returned" if a cursor was used for a delete multiple times and deadlocked during one of the deletes. \[#15944\] +5. Optionally signal processes still running in the environment before running recovery. \[#15984\] + +### Concurrent Data Store Changes: + +None. + +### General Access Method Changes: + +1. Fix a bug where closing a database handle after aborting a transaction which included a failed open of that database handle could result in application failure. \[#15650\] +2. Fix a bug that could cause panic in a database environment configured with POSIX-style thread locking, if a database open failed. \[#15662\] +3. Fix bug in the DB-\>compact method which could cause a panic if a thread was about to release a page while another thread was truncating the database file. \[#15671\] +4. Fix an obscure case of interaction between a cursor scan and delete that was prematurely returning DB_NOTFOUND. \[#15785\] +5. Fix a bug in the DB-\>compact method where if read-uncommitted was configured, a reader reading uncommitted data my see an inconsistent entry between when the compact method detects an error and when it aborts the enclosing transaction. \[#15856\] +6. Fix a bug in the DB-\>compact method where a thread of control mail fail if two threads are compacting the same section of a Recno database. \[#15856\] +7. Fix a bug in DB-\>compact method, avoid an assertion failure when zero pages can be freed. \[#15965\] +8. Fix a bug return a non-zero error when DB-\>truncate is called with open cursors. \[#15973\] +9. Fix a bug add HANDLE_DEAD checking for DB cursors. \[#15990\] +10. Fix a bug to now generate errors when DB_SEQUENCE-\>stat is called without first opening the sequence. \[#15995\] +11. Fix a bug to no longer dereference a pointer into a hash structure, when hash functionality is disabled.  \[#16095\] + +### Btree Access Method Changes: + +None. + +### Hash Access Method Changes: + +1. Fix a bug where a database store into a Hash database could self-deadlock in a database environment configured for the Berkeley DB Concurrent Data Store product, and with a free-threaded DB_ENV or DB handle. \[#15718\] + +### Queue Access Method Changes: + +1. Fix a bug that could cause a put or delete of a queue element to return a DB_NOTGRANTED error, if blocked. \[#15933\] + +### Recno Access Method Changes: + +1. Expose db_env_set_func_malloc, db_env_set_func_realloc, and db_env_set_func_free through the Windows API for the DB dll. \[#16045\] + +### C-specific API Changes: + +None. + +### Java-specific API Changes: + +1. Fix a bug where enabling MVCC on a database through the Java API was ignored. \[#15644\] +2. Fixed memory leak bugs in error message buffering in the Java API. \[#15843\] +3. Fix a bug where Java SecondaryConfig was not setting SecondaryMultiKeyCreator from the underlying db handle \[OTN FORUM\] +4. Fix a bug so that getStartupComplete will now return a boolean instead of an int. \[#16067\] +5. Fix a bug in the Java API, where Berkeley DB would hang on exit when using replication. \[#16142\] + +### Direct Persistence Layer (DPL), Bindings and Collections API: + +1. A new Direct Persistence Layer adds a built-in Plain Old Java Object (POJO)-based persistent object model, which provides support for complex object models without compromises in performance. For an introduction to the Direct Persistence Layer API, see Getting Started with Data Storage. \[#15936\] +2. Fixed a bug in the remove method of the Iterator instances returned by the StoredCollection.iterator method in the collections package. This bug caused ArrayIndexOutOfBoundsException in some cases when calling next, previous, hasNext or hasPrevious after calling remove. (Note that this issue does not apply to StoredIterator instances returned by the StoredCollection.storedIterator method.) This bug was reported in this forum thread: http://forums.oracle.com/forums/thread.jspa?messageID=2187896 \[#15858\] +3. Fixed a bug in the remove method of the StoredIterator instances returned by StoredCollection.storedIterator method in the collections package. If the sequence of methods next-remove-previous was called, previous would sometimes return the removed record. If the sequence of methods previous-remove-next was called, next would sometimes return the removed record. (Note that this issue does not apply to Iterator instances returned by the StoredCollection.iterator method.) \[#15909\] +4. Fixed a bug that causes a memory leak for applications where many Environment objects are opened and closed and the CurrentTransaction or TransactionRunner class is used. The problem was reported in this JE Forum thread: http://forums.oracle.com/forums/thread.jspa?messageID=1782659 \[#15444\] +5. Added StoredContainer.areKeyRangesAllowed method.  Key ranges and the methods in SortedMap and SortedSet such as subMap and subSet are now explicitly disallowed for RECNO and QUEUE databases -- they are only supported for BTREE databases.  Before, using key ranges in a RECNO or QUEUE database did not work, but was not explicitly prohibited in the Collections API. \[#15936\] + +### Tcl-specific API Changes: + +1. The Berkeley DB Tcl API does not attempt to avoid evaluating input as Tcl commands. For this reason, it may be dangerous to pass unreviewed user input through the Berkeley DB Tcl API, as the input may subsequently be evaluated as a Tcl command. To minimize the effectiveness of a Tcl injection attack, the Berkeley DB Tcl API in the 4.7 release routine resets process' effective user and group IDs to the real user and group IDs. \[#15597\] + +### RPC-specific Client/Server Changes: + +None. + +### Replication Changes: + +1. Fix a bug where a master failure resulted in multiple attempts to perform a "fast election"; subsequent elections, when necessary, now use the normal nsites value. \[#15099\] +2. Replication performance enhancements to speed up failover. \[#15490\] +3. Fix a bug where replication could self-block in a database environment configured for in-memory logging. \[#15503\] +4. Fix a bug where replication would attempt to read log file version numbers in a database configured for in-memory logging. \[#15503\] +5. Fix a bug where log files were not removed during client initialization in a database configured for in-memory logging. \[#15503\] +6. The 4.7 release no longer supports live replication upgrade from the 4.2 or 4.3 releases, only from the 4.4 and later releases. \[#15602\] +7. Fix a bug where replication could re-request missing records on every arriving record. \[#15629\] +8. Change the DB_ENV-\>rep_set_request method to use time, not the number of messages, when re-requesting missed messages on a replication client. \[#15629\] +9. Fix a minor memory leak on the master when updating a client during internal initialization. \[#15634\] +10. Fix a bug where a client error when syncing with a new replication group master could result in an inability to ever re-join the group. \[#15648\] +11. Change dbenv-\>rep_set_request to use time-based values instead of counters. \[#15682\] +12. Fix a bug where a LOCK_NOTGRANTED error could be returned from the DB_ENV-\>rep_process_message method, instead of being handled internally by replication. \[#15685\] +13. Fix a bug where the Replication Manager would reject a fresh connection from a remote site that had crashed and restarted, displaying the message: "redundant incoming connection will be ignored". \[#15731\] +14. The Replication Manager now supports dynamic negotiation of the best available wire protocol version, on a per-connection basis. \[#15783\] +15. Fix a bug, which could lead to slow performance of internal initialization under the Replication Manager, as evidenced by "queue limit exceeded" messages in verbose replication diagnostic output. \[#15788\] +16. Fix a bug where replication control message were not portable between replication clients with different endian architectures. \[#15793\] +17. Add a configuration option to turn off Replication Manager's special handling of elections in 2-site groups. \[#15873\] +18. Fix a bug making it impossible to call replicationManagerAddRemoteSite in the Java API after having called replicationManagerStart. \[#15875\] +19. Fix a bug where the DB_EVENT_REP_STARTUPDONE event could be triggered too early. \[#15887\] +20. Fix a bug where the rcvd_ts timestamp is reset when the user just changes the threshold. \[#15895\] +21. Fix a bug where the master in a 2-site replication group might wait for client acknowledgement, even when there was no client connected. \[#15927\] +22. Fix a bug, clean up and restart internal init if master log is gone. \[#16006\] +23. Fix a bug, ignore page messages that are from an old internal init. \[#16075\] \[#16059\] +24. Fix a bug where checkpoint records do not indicate a database was a named in-memory database. \[#16076\] +25. Fix a bug with in-memory replication, where we returned with the log region mutex held in an error path, leading to self-deadlock. \[#16088\] +26. Fix a bug which causes the DB_REP_CHECKPOINT_DELAY setting in rep_set_timeout() to be interpreted in seconds, rather than microseconds. \[#16153\] + +### XA Resource Manager Changes: + +1. Fix a bug where the DB_ENV-\>failchk method and replication in general could fail in database environments configured for XA. \[#15654\] + +### Locking Subsystem Changes: + +1. Fix a bug causing a lock or transaction timeout to not be set properly after the first timeout triggers on a particular lock id. \[#15847\] +2. Fix a bug that would cause a trap if DB_ENV-\>lock_id_free was passed an invalid locker id. \[#16005\] +3. Fix a bug when thread tracking is enabled where an attempt is made to release a mutex that is not lock. \[#16011\] + +### Logging Subsystem Changes: + +1. Fix a bug, handle zero-length log records doing HA sync with in-memory logs. \[#15838\] +2. Fix a bug that could cause DB_ENV-\>failcheck to leak log region memory. \[#15925\] +3. Fix a bug where the abort of a transaction that opened a database could leak log region memory. \[#15953\] +4. Fix a bug that could leak memory in the DB_ENV-\>log_archive interface if a log file was not found. \[#16013\] + +### Memory Pool Subsystem Changes: + +1. Fix multiple MVCC bugs including a race, which could *result in incorrect data being returned* to the application. \[#15653\] +2. Fixed a bug that left an active file in the buffer pool after a database create was aborted. \[#15918\] +3. Fix a bug where there could be uneven distribution of pages if a single database and multiple cache regions are configured. \[#16015\] +4. Fix a bug where DB_MPOOLFILE-\>set_maxsize was dropping the wrong mutex after open. \[#16050\] + +### Mutex Subsystem Changes: + +1. Fix a bug where mutex contention in database environments configured for hybrid mutex support could result in performance degradation. \[#15646\] +2. Set the DB_MUTEX_PROCESS_ONLY flag on all mutexes in private environments, they can't be shared and so we can use the faster, intra-process only mutex implementations \[#16025\] +3. Fix a bug so that mutexes are now removed from the environment signature if mutexes are disabled. \[#16042\] + +### Transaction Subsystem Changes: + +1. Fix a bug that could cause a checkpoint to selfblock attempting to flush a file, when the file handle was closed by another thread during the flush. \[#15692\] +2. Fix a bug that could cause DB_ENV-\>failcheck to hang if there were pending prepared transactions in the environment. \[#15925\] +3. Prepared transactions will now use the sync setting from the environment.  Default to flushing the log on commit (was nosync). \[#15995\] +4. If \_\_txn_getactive fails, we now return with the log region mutex held.  This is not a bus since \_\_txn_getactive cannot really fail.  \[#16088\] + +### Utility Changes: + +1. Update db_stat with -x option for mutex stats +2. Fix an incorrect assumption about buffer size when getting an overflow page in db_verify. \[#16064\] + +### Configuration, Documentation, Sample Application, Portability and Build Changes: + +1. Fix an installation bug where the Berkeley DB PHP header file was not installed in the correct place. +2. Merge the run-time configuration sleep and yield functions. \[#15037\] +3. Fix Handle_DEAD and other expected replication errors in the C++ sample application ReqQuoteExample.cpp. \[15568\] +4. Add support for monotonic timers. \[#15670\] +5. Fix bugs where applications using the db_env_func_map and db_env_func_unmap run-time configuration functions could not join existing database environments, or open multiple DB_ENV handles for a single environment. \[#15930\] +6. Add documentation about building Berkeley DB for VxWorks 6.x. +7. Remove the HAVE_FINE_GRAINED_LOCK_MANAGER flag, it is obsolete in 4.7. +8. Fix a bug in ex_rep, add a missing break which could cause a segment fault. +9. Fix build warnings from 64 bit Windows build. \[#16029\] +10. Fix an alignment bug on ARM Linux.  Force the assignment to use memcpy.  \[#16125\] +11. Fix a bug in the Windows specific code of ex_sequence.c, where there was an invalide printf specifier.  \[#16131\] +12. Improve the timer in ex_tpcb to use high resolution timers.  \[#16154\] +13. Mention in the documentation that env-\>open() requires DB_THREAD to be specified when using repmgr. \[#16163\] +14. Disable support for mmap on Windows CE.  The only affect is that we do not attempt to mmap small read only databases into the mpool. \[#16169\] diff --git a/docs-src/guides/upgrading/index.md b/docs-src/guides/upgrading/index.md new file mode 100644 index 000000000..7badab6bd --- /dev/null +++ b/docs-src/guides/upgrading/index.md @@ -0,0 +1,764 @@ +--- +title: "Berkeley DB Upgrade Guide" +api-name: "Berkeley DB Upgrade Guide" +source: docs/upgrading/index.html +--- +# Berkeley DB Upgrade Guide + +**Legal Notice** + +This documentation is distributed under an open source license. You may review the terms of this license at: http://www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html + +Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle. + +Other names may be trademarks of their respective owners. + +To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at: http://forums.oracle.com/forums/forum.jspa?forumID=271 + +9/9/2013 + +------------------------------------------------------------------------ + +**Table of Contents** + + [Preface](preface.md) + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + + [1. Introduction](introduction.md) + + [Library version information](introduction.md#upgrade_version) + + [2. Upgrading from previous versions of Berkeley DB](upgrade_process.md) + + [3. Upgrading Berkeley DB 4.6 applications to Berkeley DB 4.7](upgrade_4_7_toc.md) + + [Introduction](upgrade_4_7_toc.md#upgrade_4_7_intro) + + [Run-time configuration](upgrade_4_7_rtc.md) + + [Replication API](upgrade_4_7_repapi.md) + + [Tcl API](upgrade_4_7_tcl.md) + + [DB_ENV-\>set_intermediate_dir](upgrade_4_7_interdir.md) + + [Log configuration](upgrade_4_7_log.md) + + [Upgrade Requirements](upgrade_4_7_disk.md) + + [Berkeley DB 4.7.25 Change Log](changelog_4_7.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_7.md#idp50357648) + + [New Features:](changelog_4_7.md#idp50378912) + + [Database Environment Changes:](changelog_4_7.md#idp50380752) + + [Concurrent Data Store Changes:](changelog_4_7.md#idp50382592) + + [General Access Method Changes:](changelog_4_7.md#idp50381120) + + [Btree Access Method Changes:](changelog_4_7.md#idp50391248) + + [Hash Access Method Changes:](changelog_4_7.md#idp50365280) + + [Queue Access Method Changes:](changelog_4_7.md#idp50355136) + + [Recno Access Method Changes:](changelog_4_7.md#idp50346288) + + [C-specific API Changes:](changelog_4_7.md#idp50346816) + + [Java-specific API Changes:](changelog_4_7.md#idp50347072) + + [Direct Persistence Layer (DPL), Bindings and Collections API:](changelog_4_7.md#idp50347296) + + [Tcl-specific API Changes:](changelog_4_7.md#idp50386200) + + [RPC-specific Client/Server Changes:](changelog_4_7.md#idp50395072) + + [Replication Changes:](changelog_4_7.md#idp50395328) + + [XA Resource Manager Changes:](changelog_4_7.md#idp50391504) + + [Locking Subsystem Changes:](changelog_4_7.md#idp50357904) + + [Logging Subsystem Changes:](changelog_4_7.md#idp50397528) + + [Memory Pool Subsystem Changes:](changelog_4_7.md#idp50385512) + + [Mutex Subsystem Changes:](changelog_4_7.md#idp50386616) + + [Transaction Subsystem Changes:](changelog_4_7.md#idp50391632) + + [Utility Changes:](changelog_4_7.md#idp50396200) + + [Configuration, Documentation, Sample Application, Portability and Build Changes:](changelog_4_7.md#idp50412288) + + [4. Upgrading Berkeley DB 4.5 applications to Berkeley DB 4.6](upgrade_4_6_toc.md) + + [Introduction](upgrade_4_6_toc.md#upgrade_4_6_intro) + + [C API cursor handle method names](upgrade_4_6_cursor.md) + + [DB_MPOOLFILE-\>put](upgrade_4_6_memp_fput.md) + + [B_MPOOLFILE-\>set](upgrade_4_6_memp_fset.md) + + [Replication Events](upgrade_4_6_event.md) + + [DB_REP_FULL_ELECTION](upgrade_4_6_full_election.md) + + [Verbose Output](upgrade_4_6_verbose.md) + + [DB_VERB_REPLICATION](upgrade_4_6_verb.md) + + [Windows 9X](upgrade_4_6_win.md) + + [Upgrade Requirements](upgrade_4_6_disk.md) + + [Berkeley DB 4.6.21 Change Log](changelog_4_6.md) + + [4.6.21 Patches:](changelog_4_6.md#idp50449856) + + [4.6.19 Patches](changelog_4_6.md#idp50370888) + + [Database or Log File On-Disk Format Changes:](changelog_4_6.md#idp50361912) + + [New Features:](changelog_4_6.md#idp50454856) + + [Database Environment Changes:](changelog_4_6.md#idp50457960) + + [Concurrent Data Store Changes:](changelog_4_6.md#idp50459800) + + [General Access Method Changes:](changelog_4_6.md#idp50458344) + + [Btree Access Method Changes:](changelog_4_6.md#idp50475672) + + [Hash Access Method Changes:](changelog_4_6.md#idp50460536) + + [Queue Access Method Changes:](changelog_4_6.md#idp50444272) + + [Recno Access Method Changes:](changelog_4_6.md#idp50463616) + + [C++-specific API Changes:](changelog_4_6.md#idp50463872) + + [Java-specific API Changes:](changelog_4_6.md#idp50481800) + + [Java collections and bind API Changes:](changelog_4_6.md#idp50464456) + + [Tcl-specific API Changes:](changelog_4_6.md#idp50464944) + + [RPC-specific Client/Server Changes:](changelog_4_6.md#idp50465232) + + [Replication Changes:](changelog_4_6.md#idp50486584) + + [XA Resource Manager Changes:](changelog_4_6.md#idp50466136) + + [Locking Subsystem Changes:](changelog_4_6.md#idp50465496) + + [Logging Subsystem Changes:](changelog_4_6.md#idp50451848) + + [Memory Pool Subsystem Changes:](changelog_4_6.md#idp50452712) + + [Transaction Subsystem Changes:](changelog_4_6.md#idp50468064) + + [Utility Changes:](changelog_4_6.md#idp50475736) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_6.md#idp50479800) + + [5. Upgrading Berkeley DB 4.4 applications to Berkeley DB 4.5](upgrade_4_5_toc.md) + + [Introduction](upgrade_4_5_toc.md#upgrade_4_5_intro) + + [deprecated interfaces](upgrade_4_5_deprecate.md) + + [DB-\>set_isalive](upgrade_4_5_alive.md) + + [DB_ENV-\>rep_elect](upgrade_4_5_elect.md) + + [Replication method naming](upgrade_4_5_rep_set.md) + + [Replication events](upgrade_4_5_rep_event.md) + + [Memory Pool API](upgrade_4_5_memp.md) + + [DB_ENV-\>set_paniccall](upgrade_4_5_paniccall.md) + + [DB-\>set_pagesize](upgrade_4_5_pagesize.md) + + [Collections API](upgrade_4_5_collect.md) + + [--enable-pthread_self](upgrade_4_5_config.md) + + [Recno backing text source files](upgrade_4_5_source.md) + + [Application-specific logging](upgrade_4_5_applog.md) + + [Upgrade Requirements](upgrade_4_5_disk.md) + + [Berkeley DB 4.5.20 Change Log](changelog_4_5_20.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_5_20.md#idp50532016) + + [New Features:](changelog_4_5_20.md#idp50511048) + + [Database Environment Changes:](changelog_4_5_20.md#idp50513672) + + [Concurrent Data Store Changes:](changelog_4_5_20.md#idp50516704) + + [General Access Method Changes:](changelog_4_5_20.md#idp50520456) + + [Btree Access Method Changes:](changelog_4_5_20.md#idp50542544) + + [Hash Access Method Changes:](changelog_4_5_20.md#idp50536608) + + [Queue Access Method Changes:](changelog_4_5_20.md#idp50533200) + + [Recno Access Method Changes:](changelog_4_5_20.md#idp50539504) + + [C++-specific API Changes:](changelog_4_5_20.md#idp50539760) + + [Java-specific API Changes:](changelog_4_5_20.md#idp50541672) + + [Java collections and bind API Changes:](changelog_4_5_20.md#idp50542632) + + [Tcl-specific API Changes:](changelog_4_5_20.md#idp50546176) + + [RPC-specific Client/Server Changes:](changelog_4_5_20.md#idp50548752) + + [Replication Changes:](changelog_4_5_20.md#idp50547824) + + [XA Resource Manager Changes:](changelog_4_5_20.md#idp50557816) + + [Locking Subsystem Changes:](changelog_4_5_20.md#idp50534496) + + [Logging Subsystem Changes:](changelog_4_5_20.md#idp50532216) + + [Memory Pool Subsystem Changes:](changelog_4_5_20.md#idp50542056) + + [Transaction Subsystem Changes:](changelog_4_5_20.md#idp50543016) + + [Utility Changes:](changelog_4_5_20.md#idp50556608) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_5_20.md#idp50557880) + + [6. Upgrading Berkeley DB 4.3 applications to Berkeley DB 4.4](upgrade_4_4_toc.md) + + [Introduction](upgrade_4_4_toc.md#upgrade_4_4_intro) + + [DB_AUTO_COMMIT](upgrade_4_4_autocommit.md) + + [DB_DEGREE_2, DB_DIRTY_READ](upgrade_4_4_isolation.md) + + [DB_JOINENV](upgrade_4_4_joinenv.md) + + [mutexes](upgrade_4_4_mutex.md) + + [DB_MPOOLFILE-\>set_clear_len](upgrade_4_4_clear.md) + + [lock statistics](upgrade_4_4_lockstat.md) + + [Upgrade Requirements](upgrade_4_4_disk.md) + + [Berkeley DB 4.4.16 Change Log](changelog_4_4_16.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_4_16.md#idp50595920) + + [New Features:](changelog_4_4_16.md#idp50583264) + + [Database Environment Changes:](changelog_4_4_16.md#idp50583648) + + [Concurrent Data Store Changes:](changelog_4_4_16.md#idp50567656) + + [General Access Method Changes:](changelog_4_4_16.md#idp50591960) + + [Btree Access Method Changes:](changelog_4_4_16.md#idp50592384) + + [Hash Access Method Changes:](changelog_4_4_16.md#idp50595984) + + [Queue Access Method Changes:](changelog_4_4_16.md#idp50597856) + + [Recno Access Method Changes](changelog_4_4_16.md#idp50598936) + + [C++-specific API Changes:](changelog_4_4_16.md#idp50598072) + + [Java-specific API Changes:](changelog_4_4_16.md#idp50600424) + + [Java collections and bind API Changes:](changelog_4_4_16.md#idp50621112) + + [Tcl-specific API Changes:](changelog_4_4_16.md#idp50604672) + + [RPC-specific Client/Server Changes:](changelog_4_4_16.md#idp50589536) + + [Replication Changes:](changelog_4_4_16.md#idp50610200) + + [XA Resource Manager Changes:](changelog_4_4_16.md#idp50594920) + + [Locking Subsystem Changes:](changelog_4_4_16.md#idp50614600) + + [Logging Subsystem Changes:](changelog_4_4_16.md#idp50614888) + + [Memory Pool Subsystem Changes:](changelog_4_4_16.md#idp50635800) + + [Transaction Subsystem Changes:](changelog_4_4_16.md#idp50617400) + + [Utility Changes:](changelog_4_4_16.md#idp50617824) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_4_16.md#idp50621200) + + [Berkeley DB 4.4.20 Change Log](changelog_4_4_20.md) + + [Changes since Berkeley DB 4.4.16:](changelog_4_4_20.md#idp50624312) + + [7. Upgrading Berkeley DB 4.2 applications to Berkeley DB 4.3](upgrade_4_3_toc.md) + + [Introduction](upgrade_4_3_toc.md#upgrade_4_3_intro) + + [Java](upgrade_4_3_java.md) + + [DB_ENV-\>set_errcall, DB-\>set_errcall](upgrade_4_3_err.md) + + [DBcursor-\>c_put](upgrade_4_3_cput.md) + + [DB-\>stat](upgrade_4_3_stat.md) + + [DB_ENV-\>set_verbose](upgrade_4_3_verb.md) + + [Logging](upgrade_4_3_log.md) + + [DB_FILEOPEN](upgrade_4_3_fileopen.md) + + [ENOMEM and DbMemoryException](upgrade_4_3_enomem.md) + + [Replication](upgrade_4_3_repl.md) + + [Run-time configuration](upgrade_4_3_rtc.md) + + [Upgrade Requirements](upgrade_4_3_disk.md) + + [Berkeley DB 4.3.29 Change Log](changelog_4_3_29.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_3_29.md#idp50694880) + + [New Features:](changelog_4_3_29.md#idp50670248) + + [Database Environment Changes:](changelog_4_3_29.md#idp50673968) + + [Concurrent Data Store Changes:](changelog_4_3_29.md#idp50703424) + + [General Access Method Changes:](changelog_4_3_29.md#idp50690848) + + [Btree Access Method Changes:](changelog_4_3_29.md#idp50691272) + + [Hash Access Method Changes:](changelog_4_3_29.md#idp50694944) + + [Queue Access Method Changes:](changelog_4_3_29.md#idp50697104) + + [Recno Access Method Changes](changelog_4_3_29.md#idp50720352) + + [C++-specific API Changes:](changelog_4_3_29.md#idp50700784) + + [Java-specific API Changes:](changelog_4_3_29.md#idp50670632) + + [Tcl-specific API Changes:](changelog_4_3_29.md#idp50702384) + + [RPC-specific Client/Server Changes:](changelog_4_3_29.md#idp50703784) + + [Replication Changes:](changelog_4_3_29.md#idp50685776) + + [XA Resource Manager Changes:](changelog_4_3_29.md#idp50733112) + + [Locking Subsystem Changes:](changelog_4_3_29.md#idp50712384) + + [Logging Subsystem Changes:](changelog_4_3_29.md#idp50740760) + + [Memory Pool Subsystem Changes:](changelog_4_3_29.md#idp50695328) + + [Transaction Subsystem Changes:](changelog_4_3_29.md#idp50720440) + + [Utility Changes:](changelog_4_3_29.md#idp50724480) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_3_29.md#idp50724864) + + [8. Upgrading Berkeley DB 4.1 applications to Berkeley DB 4.2](upgrade_4_2_toc.md) + + [Introduction](upgrade_4_2_toc.md#upgrade_4_2_intro) + + [Java](upgrade_4_2_java.md) + + [Queue access method](upgrade_4_2_queue.md) + + [DB_CHKSUM_SHA1](upgrade_4_2_cksum.md) + + [DB_CLIENT](upgrade_4_2_client.md) + + [DB-\>del](upgrade_4_2_del.md) + + [DB-\>set_cache_priority](upgrade_4_2_priority.md) + + [DB-\>verify](upgrade_4_2_verify.md) + + [DB_LOCK_NOTGRANTED](upgrade_4_2_lockng.md) + + [Replication](upgrade_4_2_repinit.md) + + [Replication initialization](upgrade_4_2_repinit.md#idp50804696) + + [Database methods and replication clients](upgrade_4_2_repinit.md#idp50772032) + + [DB_ENV-\>rep_process_message()](upgrade_4_2_repinit.md#idp50779672) + + [Client replication environments](upgrade_4_2_nosync.md) + + [Tcl API](upgrade_4_2_tcl.md) + + [Upgrade Requirements](upgrade_4_2_disk.md) + + [Berkeley DB 4.2.52 Change Log](changelog_4_2_52.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_2_52.md#idp50822856) + + [New Features:](changelog_4_2_52.md#idp50784344) + + [Database Environment Changes:](changelog_4_2_52.md#idp50809288) + + [Concurrent Data Store Changes:](changelog_4_2_52.md#idp50822104) + + [General Access Method Changes:](changelog_4_2_52.md#idp50824288) + + [Btree Access Method Changes:](changelog_4_2_52.md#idp50825368) + + [Hash Access Method Changes:](changelog_4_2_52.md#idp50844704) + + [Queue Access Method Changes:](changelog_4_2_52.md#idp50828568) + + [Recno Access Method Changes:](changelog_4_2_52.md#idp50858440) + + [C++-specific API Changes:](changelog_4_2_52.md#idp50832248) + + [Java-specific API Changes:](changelog_4_2_52.md#idp50815840) + + [Tcl-specific API Changes:](changelog_4_2_52.md#idp50867864) + + [RPC-specific Client/Server Changes:](changelog_4_2_52.md#idp50852544) + + [Replication Changes:](changelog_4_2_52.md#idp50858528) + + [XA Resource Manager Changes:](changelog_4_2_52.md#idp50877816) + + [Locking Subsystem Changes:](changelog_4_2_52.md#idp50865088) + + [Logging Subsystem Changes:](changelog_4_2_52.md#idp50868008) + + [Memory Pool Subsystem Changes:](changelog_4_2_52.md#idp50865504) + + [Transaction Subsystem Changes:](changelog_4_2_52.md#idp50845064) + + [Utility Changes:](changelog_4_2_52.md#idp50858944) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_2_52.md#idp50892568) + + [9. Upgrading Berkeley DB 4.0 applications to Berkeley DB 4.1](upgrade_4_1_toc.md) + + [Introduction](upgrade_4_1_toc.md#upgrade_4_1_intro) + + [DB_EXCL](upgrade_4_1_excl.md) + + [DB-\>associate, DB-\>open, DB-\>remove, DB-\>rename](upgrade_4_1_fop.md) + + [DB_ENV-\>log_register](upgrade_4_1_log_register.md) + + [st_flushcommit](upgrade_4_1_log_stat.md) + + [DB_CHECKPOINT, DB_CURLSN](upgrade_4_1_checkpoint.md) + + [DB_INCOMPLETE](upgrade_4_1_incomplete.md) + + [DB_ENV-\>memp_sync](upgrade_4_1_memp_sync.md) + + [DB-\>stat.hash_nelem](upgrade_4_1_hash_nelem.md) + + [Java exceptions](upgrade_4_1_java.md) + + [C++ exceptions](upgrade_4_1_cxx.md) + + [Application-specific logging and recovery](upgrade_4_1_app_dispatch.md) + + [Upgrade Requirements](upgrade_4_1_disk.md) + + [Berkeley DB 4.1.24 and 4.1.25 Change Log](changelog_4_1_24.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_1_24.md#idp50963888) + + [Major New Features:](changelog_4_1_24.md#idp50959088) + + [General Environment Changes:](changelog_4_1_24.md#idp50962280) + + [General Access Method Changes:](changelog_4_1_24.md#idp50961984) + + [Btree Access Method Changes:](changelog_4_1_24.md#idp50964272) + + [Hash Access Method Changes:](changelog_4_1_24.md#idp50967400) + + [Queue Access Method Changes:](changelog_4_1_24.md#idp50969240) + + [Recno Access Method Changes:](changelog_4_1_24.md#idp50972088) + + [C++-specific API Changes:](changelog_4_1_24.md#idp50973928) + + [Java-specific API Changes:](changelog_4_1_24.md#idp50975768) + + [Tcl-specific API Changes:](changelog_4_1_24.md#idp50950328) + + [RPC-specific Client/Server Changes:](changelog_4_1_24.md#idp50958680) + + [Replication Changes:](changelog_4_1_24.md#idp50977144) + + [XA Resource Manager Changes:](changelog_4_1_24.md#idp50964336) + + [Locking Subsystem Changes:](changelog_4_1_24.md#idp50987264) + + [Logging Subsystem Changes:](changelog_4_1_24.md#idp50989192) + + [Memory Pool Subsystem Changes:](changelog_4_1_24.md#idp50992072) + + [Transaction Subsystem Changes:](changelog_4_1_24.md#idp50993160) + + [Utility Changes:](changelog_4_1_24.md#idp50994744) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_1_24.md#idp50997648) + + [Berkeley DB 4.1.25 Change Log](changelog_4_1_25.md) + + [10. Upgrading Berkeley DB 3.3 applications to Berkeley DB 4.0](upgrade_4_0_toc.md) + + [Introduction](upgrade_4_0_toc.md#upgrade_4_0_intro) + + [db_deadlock](upgrade_4_0_deadlock.md) + + [lock_XXX](upgrade_4_0_lock.md) + + [log_XXX](upgrade_4_0_log.md) + + [memp_XXX](upgrade_4_0_mp.md) + + [txn_XXX](upgrade_4_0_txn.md) + + [db_env_set_XXX](upgrade_4_0_env.md) + + [DB_ENV-\>set_server](upgrade_4_0_rpc.md) + + [DB_ENV-\>set_lk_max](upgrade_4_0_set_lk_max.md) + + [DB_ENV-\>lock_id_free](upgrade_4_0_lock_id_free.md) + + [Java CLASSPATH environment variable](upgrade_4_0_java.md) + + [C++ ostream objects](upgrade_4_0_cxx.md) + + [application-specific recovery](upgrade_4_0_asr.md) + + [Upgrade Requirements](upgrade_4_0_disk.md) + + [4.0.14 Change Log](changelog_4_0_14.md) + + [Major New Features:](changelog_4_0_14.md#idp51113768) + + [General Environment Changes:](changelog_4_0_14.md#idp51101344) + + [General Access Method Changes:](changelog_4_0_14.md#idp51103296) + + [Btree Access Method Changes:](changelog_4_0_14.md#idp51105152) + + [Hash Access Method Changes:](changelog_4_0_14.md#idp51109416) + + [Queue Access Method Changes:](changelog_4_0_14.md#idp51112664) + + [Recno Access Method Changes:](changelog_4_0_14.md#idp51113832) + + [C++ API Changes:](changelog_4_0_14.md#idp51115760) + + [Java API Changes:](changelog_4_0_14.md#idp51126328) + + [Tcl API Changes:](changelog_4_0_14.md#idp51116840) + + [RPC Client/Server Changes:](changelog_4_0_14.md#idp51117920) + + [XA Resource Manager Changes:](changelog_4_0_14.md#idp51118608) + + [Locking Subsystem Changes:](changelog_4_0_14.md#idp51118928) + + [Logging Subsystem Changes:](changelog_4_0_14.md#idp51103680) + + [Memory Pool Subsystem Changes:](changelog_4_0_14.md#idp51122816) + + [Transaction Subsystem Changes:](changelog_4_0_14.md#idp51109800) + + [Utility Changes:](changelog_4_0_14.md#idp51113048) + + [Database or Log File On-Disk Format Changes:](changelog_4_0_14.md#idp51125248) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_0_14.md#idp51126712) + + [11. Upgrading Berkeley DB 3.2 applications to Berkeley DB 3.3](upgrade_3_3_toc.md) + + [introduction](upgrade_3_3_toc.md#upgrade_3_3_intro) + + [DB_ENV-\>set_server](upgrade_3_3_rpc.md) + + [DB-\>get_type](upgrade_3_3_gettype.md) + + [DB-\>get_byteswapped](upgrade_3_3_getswap.md) + + [DB-\>set_malloc, DB-\>set_realloc](upgrade_3_3_alloc.md) + + [DB_LOCK_CONFLICT](upgrade_3_3_conflict.md) + + [memp_fget, EIO](upgrade_3_3_memp_fget.md) + + [txn_prepare](upgrade_3_3_txn_prepare.md) + + [--enable-dynamic, --enable-shared](upgrade_3_3_shared.md) + + [--disable-bigfile](upgrade_3_3_bigfile.md) + + [Upgrade Requirements](upgrade_3_3_disk.md) + + [12. Upgrading Berkeley DB 3.1 applications to Berkeley DB 3.2](upgrade_3_2_toc.md) + + [introduction](upgrade_3_2_toc.md#upgrade_3_2_intro) + + [DB_ENV-\>set_flags](upgrade_3_2_set_flags.md) + + [DB callback functions, app_private field](upgrade_3_2_callback.md) + + [Logically renumbering records](upgrade_3_2_renumber.md) + + [DB_INCOMPLETE](upgrade_3_2_incomplete.md) + + [DB_ENV-\>set_tx_recover](upgrade_3_2_tx_recover.md) + + [DB_ENV-\>set_mutexlocks](upgrade_3_2_mutexlock.md) + + [Java and C++ object reuse](upgrade_3_2_handle.md) + + [Java java.io.FileNotFoundException](upgrade_3_2_notfound.md) + + [db_dump](upgrade_3_2_db_dump.md) + + [Upgrade Requirements](upgrade_3_2_disk.md) + + [13. Upgrading Berkeley DB 3.0 applications to Berkeley DB 3.1](upgrade_3_1_toc.md) + + [introduction](upgrade_3_1_toc.md#upgrade_3_1_intro) + + [DB_ENV-\>open, DB_ENV-\>remove](upgrade_3_1_config.md) + + [DB_ENV-\>set_tx_recover](upgrade_3_1_set_tx_recover.md) + + [DB_ENV-\>set_feedback, DB-\>set_feedback](upgrade_3_1_set_feedback.md) + + [DB_ENV-\>set_paniccall, DB-\>set_paniccall](upgrade_3_1_set_paniccall.md) + + [DB-\>put](upgrade_3_1_put.md) + + [identical duplicate data items](upgrade_3_1_dup.md) + + [DB-\>stat](upgrade_3_1_btstat.md) + + [DB_SYSTEM_MEM](upgrade_3_1_sysmem.md) + + [log_register](upgrade_3_1_log_register.md) + + [memp_register](upgrade_3_1_memp_register.md) + + [txn_checkpoint](upgrade_3_1_txn_check.md) + + [environment configuration](upgrade_3_1_env.md) + + [Tcl API](upgrade_3_1_tcl.md) + + [DB_TMP_DIR](upgrade_3_1_tmp.md) + + [log file pre-allocation](upgrade_3_1_logalloc.md) + + [Upgrade Requirements](upgrade_3_1_disk.md) + + [14. Upgrading Berkeley DB 2.X applications to Berkeley DB 3.0](upgrade_3_0_toc.md) + + [introduction](upgrade_3_0_toc.md#upgrade_3_0_intro) + + [environment open/close/unlink](upgrade_3_0_envopen.md) + + [function arguments](upgrade_3_0_func.md) + + [DB_ENV structure](upgrade_3_0_dbenv.md) + + [database open/close](upgrade_3_0_open.md) + + [db_xa_open](upgrade_3_0_xa.md) + + [DB structure](upgrade_3_0_db.md) + + [DBINFO structure](upgrade_3_0_dbinfo.md) + + [DB-\>join](upgrade_3_0_join.md) + + [DB-\>stat](upgrade_3_0_stat.md) + + [DB-\>sync and DB-\>close](upgrade_3_0_close.md) + + [lock_put](upgrade_3_0_lock_put.md) + + [lock_detect](upgrade_3_0_lock_detect.md) + + [lock_stat](upgrade_3_0_lock_stat.md) + + [log_register](upgrade_3_0_log_register.md) + + [log_stat](upgrade_3_0_log_stat.md) + + [memp_stat](upgrade_3_0_memp_stat.md) + + [txn_begin](upgrade_3_0_txn_begin.md) + + [txn_commit](upgrade_3_0_txn_commit.md) + + [txn_stat](upgrade_3_0_txn_stat.md) + + [DB_RMW](upgrade_3_0_rmw.md) + + [DB_LOCK_NOTHELD](upgrade_3_0_lock_notheld.md) + + [EAGAIN](upgrade_3_0_eagain.md) + + [EACCES](upgrade_3_0_eacces.md) + + [db_jump_set](upgrade_3_0_jump_set.md) + + [db_value_set](upgrade_3_0_value_set.md) + + [DbEnv class for C++ and Java](upgrade_3_0_dbenv_cxx.md) + + [Db class for C++ and Java](upgrade_3_0_db_cxx.md) + + [additional C++ changes](upgrade_3_0_cxx.md) + + [additional Java changes](upgrade_3_0_java.md) + + [Upgrade Requirements](upgrade_3_0_disk.md) + + [15. Upgrading Berkeley DB 1.85 or 1.86 applications to Berkeley DB 2.0](upgrade_2_0_toc.md) + + [Introduction](upgrade_2_0_toc.md#upgrade_2_0_intro) + + [System Integration](upgrade_2_0_system.md) + + [Converting Applications](upgrade_2_0_convert.md) + + [Upgrade Requirements](upgrade_2_0_disk.md) diff --git a/docs-src/guides/upgrading/introduction.md b/docs-src/guides/upgrading/introduction.md new file mode 100644 index 000000000..724f03db4 --- /dev/null +++ b/docs-src/guides/upgrading/introduction.md @@ -0,0 +1,26 @@ +--- +title: "Chapter 1. Introduction" +api-name: "Chapter 1. Introduction" +source: docs/upgrading/introduction.html +--- +## Chapter 1. Introduction + +**Table of Contents** + + [Library version information](introduction.md#upgrade_version) + +This manual describes how to upgrade from historical versions of Berkeley DB (Berkeley DB 4.7 and older). For information on upgrading newer releases of the product (anything newer than DB 4.7), and on building and installing Berkeley DB on all of the platforms it officially supports, see the Berkeley DB Installation and Build Guide. + +## Library version information + +Each release of the Berkeley DB library has a major version number, a minor version number, and a patch number. + +The major version number changes only when major portions of the Berkeley DB functionality have been changed. In this case, it may be necessary to significantly modify applications in order to upgrade them to use the new version of the library. + +The minor version number changes when Berkeley DB interfaces have changed, and the new release is not entirely backward-compatible with previous releases. To upgrade applications to the new version, they must be recompiled and potentially, minor modifications made (for example, the order of arguments to a function might have changed). + +The patch number changes on each release. If only the patch number has changed in a release, applications do not need to be recompiled, and they can be upgraded to the new version by installing the new version of a shared library or by relinking the application to the new version of a static library. + +Internal Berkeley DB interfaces may change at any time and during any release, without warning. This means that the library must be entirely recompiled and reinstalled when upgrading to new releases of the library because there is no guarantee that modules from the current version of the library will interact correctly with modules from a previous release. + +To retrieve the Berkeley DB version information, applications should use the DB_ENV->version() function. In addition to the previous information, the DB_ENV->version() function returns a string encapsulating the version information, suitable for display to a user. diff --git a/docs-src/guides/upgrading/moreinfo.md b/docs-src/guides/upgrading/moreinfo.md new file mode 100644 index 000000000..d775c54fd --- /dev/null +++ b/docs-src/guides/upgrading/moreinfo.md @@ -0,0 +1,38 @@ +--- +title: "For More Information" +api-name: "For More Information" +source: docs/upgrading/moreinfo.html +--- +## For More Information + + [Contact Us](moreinfo.md#contact_us) + +Beyond this manual, you may also find the following sources of information useful when building a DB application: + +- Berkeley DB Installation and Build Guide + +- Getting Started with Transaction Processing for C + +- Berkeley DB Getting Started with Replicated Applications for C + +- Berkeley DB C API Reference Guide + +- Berkeley DB C++ API Reference Guide + +- Berkeley DB STL API Reference Guide + +- Berkeley DB TCL API Reference Guide + +- Berkeley DB Programmer's Reference Guide + +- Berkeley DB Getting Started with the SQL APIs + +To download the latest Berkeley DB documentation along with white papers and other collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html. + +For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html. + +### Contact Us + +You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle Berkeley DB at: http://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle Berkeley DB High Availability at: http://forums.oracle.com/forums/forum.jspa?forumID=272. + +For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe to a low-volume email announcement list for the Berkeley DB product family by sending email to: bdb-join@oss.oracle.com diff --git a/docs-src/guides/upgrading/preface.md b/docs-src/guides/upgrading/preface.md new file mode 100644 index 000000000..3ec362c95 --- /dev/null +++ b/docs-src/guides/upgrading/preface.md @@ -0,0 +1,44 @@ +--- +title: "Preface" +api-name: "Preface" +source: docs/upgrading/preface.html +--- +## Preface + +**Table of Contents** + + [Conventions Used in this Book](preface.md#conventions) + + [For More Information](moreinfo.md) + + [Contact Us](moreinfo.md#contact_us) + +Welcome to Berkeley DB (DB). This document describes how to upgrade from previous versions of Berkeley DB. + +This document reflects Berkeley DB 11*g* Release 2, which provides DB library version 11.2.5.3. + +## Conventions Used in this Book + +The following typographical conventions are used within in this manual: + +Structure names are represented in `monospaced font`, as are `method names`. For example: "`DB->open()` is a method on a `DB` handle." + +Variable or non-literal text is presented in *italics*. For example: "Go to your *DB_INSTALL* directory." + +Program examples are displayed in a `monospaced font` on a shaded background. For example: + +``` c +/* File: gettingstarted_common.h */ +typedef struct stock_dbs { + DB *inventory_dbp; /* Database containing inventory information */ + DB *vendor_dbp; /* Database containing vendor information */ + + char *db_home_dir; /* Directory containing the database files */ + char *inventory_db_name; /* Name of the inventory database */ + char *vendor_db_name; /* Name of the vendor database */ +} STOCK_DBS; +``` + +### Note + +Finally, notes of interest are represented using a note block such as this. diff --git a/docs-src/guides/upgrading/upgrade_2_0_convert.md b/docs-src/guides/upgrading/upgrade_2_0_convert.md new file mode 100644 index 000000000..e165b2ebf --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_2_0_convert.md @@ -0,0 +1,26 @@ +--- +title: "Converting Applications" +api-name: "Converting Applications" +source: docs/upgrading/upgrade_2_0_convert.html +--- +## Converting Applications + +Mapping the Berkeley DB 1.85 functionality into Berkeley DB version 2 is almost always simple. The manual page DB->open() replaces the Berkeley DB 1.85 manual pages `dbopen`(3), `btree`(3), `hash`(3) and `recno`(3). You should be able to convert each 1.85 function call into a Berkeley DB version 2 function call using just the DB->open() documentation. + +Some guidelines and things to watch out for: + +1. Most access method functions have exactly the same semantics as in Berkeley DB 1.85, although the arguments to the functions have changed in some cases. To get your code to compile, the most common change is to add the transaction ID as an argument (NULL, since Berkeley DB 1.85 did not support transactions.) + +2. You must always initialize DBT structures to zero before using them with any Berkeley DB version 2 function. (They do not normally have to be reinitialized each time, only when they are first allocated. Do this by declaring the DBT structure external or static, or by calling the C library routine `bzero`(3) or `memset`(3).) + +3. The error returns are completely different in the two versions. In Berkeley DB 1.85, \< 0 meant an error, and \> 0 meant a minor Berkeley DB exception. In Berkeley DB 2.0, \> 0 means an error (the Berkeley DB version 2 functions return `errno` on error) and \< 0 means a Berkeley DB exception. See Program returns to applications for more information. + +4. The Berkeley DB 1.85 DB-\>seq function has been replaced by cursors in Berkeley DB version 2. The semantics are approximately the same, but cursors require the creation of an extra object (the DBC object), which is then used to access the database. + + Specifically, the partial key match and range search functionality of the R_CURSOR flag in DB-\>seq has been replaced by the DB_SET_RANGE flag in DBC->get(). + +5. In version 2 of the Berkeley DB library, additions or deletions into Recno (fixed and variable-length record) databases no longer automatically logically renumber all records after the add/delete point, by default. The default behavior is that deleting records does not cause subsequent records to be renumbered, and it is an error to attempt to add new records between records already in the database. Applications wanting the historic Recno access method semantics should call the DB->set_flags() method with the DB_RENUMBER flag. + +6. Opening a database in Berkeley DB version 2 is a much heavier-weight operation than it was in Berkeley DB 1.85. Therefore, if your historic applications were written to open a database, perform a single operation, and close the database, you may observe performance degradation. In most cases, this is due to the expense of creating the environment upon each open. While we encourage restructuring your application to avoid repeated opens and closes, you can probably recover most of the lost performance by simply using a persistent environment across invocations. + +While simply converting Berkeley DB 1.85 function calls to Berkeley DB version 2 function calls will work, we recommend that you eventually reconsider your application's interface to the Berkeley DB database library in light of the additional functionality supplied by Berkeley DB version 2, as it is likely to result in enhanced application performance. diff --git a/docs-src/guides/upgrading/upgrade_2_0_disk.md b/docs-src/guides/upgrading/upgrade_2_0_disk.md new file mode 100644 index 000000000..cfc0a1dad --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_2_0_disk.md @@ -0,0 +1,8 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_2_0_disk.html +--- +## Upgrade Requirements + +You will need to upgrade your on-disk databases, as all access method database formats changed in the Berkeley DB 2.0 release. For information on converting databases from Berkeley DB 1.85 to Berkeley DB 2.0, see the db_dump185 utility and db_load utility documentation. As database environments did not exist prior to the 2.0 release, there is no question of upgrading existing database environments. diff --git a/docs-src/guides/upgrading/upgrade_2_0_system.md b/docs-src/guides/upgrading/upgrade_2_0_system.md new file mode 100644 index 000000000..392cc5617 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_2_0_system.md @@ -0,0 +1,28 @@ +--- +title: "System Integration" +api-name: "System Integration" +source: docs/upgrading/upgrade_2_0_system.html +--- +## System Integration + +1. It is possible to maintain both the Berkeley DB 1.85 and Berkeley DB version 2 libraries on your system. However, the `db.h` include file that was distributed with Berkeley DB 1.85 is not compatible with the `db.h` file distributed with Berkeley DB version 2, so you will have to install them in different locations. In addition, both the Berkeley DB 1.85 and Berkeley DB version 2 libraries are named `libdb.a`. + + As the Berkeley DB 1.85 library did not have an installation target in the Makefile, there's no way to know exactly where it was installed on the system. In addition, many vendors included it in the C library instead of as a separate library, and so it may actually be part of libc and the `db.h` include file may be installed in `/usr/include`. + + For these reasons, the simplest way to maintain both libraries is to install Berkeley DB version 2 in a completely separate area of your system. The Berkeley DB version 2 installation process allows you to install into a standalone directory hierarchy on your system. See the Berkeley DB Installation and Build Guide for more information and instructions on how to install the Berkeley DB version 2 library, include files and documentation into specific locations. + +2. Alternatively, you can replace Berkeley DB 1.85 on your system with Berkeley DB version 2. In this case, you'll probably want to install Berkeley DB version 2 in the normal place on your system, wherever that may be, and delete the Berkeley DB 1.85 include files, manual pages and libraries. + + To replace 1.85 with version 2, you must either convert your 1.85 applications to use the version 2 API or build the Berkeley DB version 2 library to include Berkeley DB 1.85 interface compatibility code. Whether converting your applications to use the version 2 interface or using the version 1.85 compatibility API, you will need to recompile or relink your 1.85 applications, and you must convert any persistent application databases to the Berkeley DB version 2 database formats. + + If you want to recompile your Berkeley DB 1.85 applications, you will have to change them to include the file `db_185.h` instead of `db.h`. (The `db_185.h` file is automatically installed during the Berkeley DB version 2 installation process.) You can then recompile the applications, linking them against the Berkeley DB version 2 library. + + For more information on compiling the Berkeley DB 1.85 compatibility code into the Berkeley DB version 2 library, see Berkeley DB Installation and Build Guide. + + For more information on converting databases from the Berkeley DB 1.85 formats to the Berkeley DB version 2 formats, see the db_dump185 utility and the db_load utility documentation. + +3. Finally, although we certainly do not recommend it, it is possible to load both Berkeley DB 1.85 and Berkeley DB version 2 into the same library. Similarly, it is possible to use both Berkeley DB 1.85 and Berkeley DB version 2 within a single application, although it is not possible to use them from within the same file. + + The name space in Berkeley DB version 2 has been changed from that of previous Berkeley DB versions, notably version 1.85, for portability and consistency reasons. The only name collisions in the two libraries are the names used by the historic dbm and hsearch interfaces, and the Berkeley DB 1.85 compatibility interfaces in the Berkeley DB version 2 library. + + If you are loading both Berkeley DB 1.85 and Berkeley DB version 2 into a single library, remove the historic interfaces from one of the two library builds, and configure the Berkeley DB version 2 build to not include the Berkeley DB 1.85 compatibility API, otherwise you could have collisions and undefined behavior. This can be done by editing the library Makefiles and reconfiguring and rebuilding the Berkeley DB version 2 library. Obviously, if you use the historic interfaces, you will get the version in the library from which you did not remove them. Similarly, you will not be able to access Berkeley DB version 2 files using the Berkeley DB 1.85 compatibility interface, since you have removed that from the library as well. diff --git a/docs-src/guides/upgrading/upgrade_2_0_toc.md b/docs-src/guides/upgrading/upgrade_2_0_toc.md new file mode 100644 index 000000000..2d2039738 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_2_0_toc.md @@ -0,0 +1,22 @@ +--- +title: "Chapter 15. Upgrading Berkeley DB 1.85 or 1.86 applications to Berkeley DB 2.0" +api-name: "Chapter 15. Upgrading Berkeley DB 1.85 or 1.86 applications to Berkeley DB 2.0" +source: docs/upgrading/upgrade_2_0_toc.html +--- +## Chapter 15. Upgrading Berkeley DB 1.85 or 1.86 applications to Berkeley DB 2.0 + +**Table of Contents** + + [Introduction](upgrade_2_0_toc.md#upgrade_2_0_intro) + + [System Integration](upgrade_2_0_system.md) + + [Converting Applications](upgrade_2_0_convert.md) + + [Upgrade Requirements](upgrade_2_0_disk.md) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 1.85 and 1.86 release interfaces to the Berkeley DB 2.0 release interfaces. They do not describe how to upgrade to the current Berkeley DB release interfaces. + +It is not difficult to upgrade Berkeley DB 1.85 applications to use the Berkeley DB version 2 library. The Berkeley DB version 2 library has a Berkeley DB 1.85 compatibility API, which you can use by either recompiling your application's source code or by relinking its object files against the version 2 library. The underlying databases must be converted, however, as the Berkeley DB version 2 library has a different underlying database format. diff --git a/docs-src/guides/upgrading/upgrade_3_0_close.md b/docs-src/guides/upgrading/upgrade_3_0_close.md new file mode 100644 index 000000000..fbd10e93f --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_close.md @@ -0,0 +1,10 @@ +--- +title: "DB->sync and DB->close" +api-name: "DB->sync and DB->close" +source: docs/upgrading/upgrade_3_0_close.html +--- +## DB-\>sync and DB-\>close + +In previous Berkeley DB releases, the DB->close() and DB->sync() methods discarded any return of DB_INCOMPLETE from the underlying buffer pool interfaces, and returned success to its caller. (The DB_INCOMPLETE error will be returned if the buffer pool functions are unable to flush all of the database's dirty blocks from the pool. This often happens if another thread is reading or writing the database's pages in the pool.) + +In the 3.X release, DB->sync() and DB->close() will return DB_INCOMPLETE to the application. The best solution is to not call DB->sync() with the DB_NOSYNC flag to the DB->close() method when multiple threads are expected to be accessing the database. Alternatively, the caller can ignore any error return of DB_INCOMPLETE. diff --git a/docs-src/guides/upgrading/upgrade_3_0_cxx.md b/docs-src/guides/upgrading/upgrade_3_0_cxx.md new file mode 100644 index 000000000..7d8dcba97 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_cxx.md @@ -0,0 +1,17 @@ +--- +title: "additional C++ changes" +api-name: "additional C++ changes" +source: docs/upgrading/upgrade_3_0_cxx.html +--- +## additional C++ changes + +The Db::set_error_model method is gone. The way to change the C++ API to return errors rather than throw exceptions is via a flag on the DbEnv or Db constructor. For example: + +``` c +int dberr; +DbEnv *dbenv = new DbEnv(DB_CXX_NO_EXCEPTIONS); +``` + +creates an environment that will never throw exceptions, and method returns should be checked instead. + +There are a number of smaller changes to the API that bring the C, C++ and Java APIs much closer in terms of functionality and usage. Please refer to the pages for upgrading C applications for further details. diff --git a/docs-src/guides/upgrading/upgrade_3_0_db.md b/docs-src/guides/upgrading/upgrade_3_0_db.md new file mode 100644 index 000000000..274f26313 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_db.md @@ -0,0 +1,35 @@ +--- +title: "DB structure" +api-name: "DB structure" +source: docs/upgrading/upgrade_3_0_db.html +--- +## DB structure + +The DB structure is now opaque for applications in the Berkeley DB 3.0 release. Accesses to any fields within that structure by the application should be replaced with method calls. The following example illustrates this using the historic type structure field. In the Berkeley DB 2.X releases, applications could find the type of an underlying database using code similar to the following: + +``` c +DB *db; +DB_TYPE type; + +type = db->type; +``` + +in the Berkeley DB 3.X releases, this should be done using the DB->get_type() method, as follows: + +``` c +DB *db; +DB_TYPE type; + +type = db->get_type(db); +``` + +The following table lists the DB fields previously used by applications and the methods that should now be used to get or set them. + +| DB field | Berkeley DB 3.X method | +|----|----| +| byteswapped | DB->get_byteswapped() | +| db_errcall | DB->set_errcall() | +| db_errfile | DB->set_errfile() | +| db_errpfx | DB->set_errpfx() | +| db_paniccall | DB-\>set_paniccall | +| type | DB->get_type() | diff --git a/docs-src/guides/upgrading/upgrade_3_0_db_cxx.md b/docs-src/guides/upgrading/upgrade_3_0_db_cxx.md new file mode 100644 index 000000000..e2577a322 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_db_cxx.md @@ -0,0 +1,41 @@ +--- +title: "Db class for C++ and Java" +api-name: "Db class for C++ and Java" +source: docs/upgrading/upgrade_3_0_db_cxx.html +--- +## Db class for C++ and Java + +The static Db::open method and the DbInfo class have been removed in the Berkeley DB 3.0 release. The way to open a database file is to use the new Db constructor with two arguments, followed by set_XXX methods to configure the Db object, and finally a call to the new (nonstatic) Db::open(). In comparing the Berkeley DB 3.0 release open method with the 2.X static open method, the second argument is new. It is a database name, which can be null. The DbEnv argument has been removed, as the environment is now specified in the constructor. The open method no longer returns a Db, since it operates on one. + +Here's a C++ example opening a Berkeley DB database using the 2.X interface: + +``` c +// Note: by default, errors are thrown as exceptions +Db *table; +Db::open("lookup.db", DB_BTREE, DB_CREATE, 0644, dbenv, 0, &table); +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +// Note: by default, errors are thrown as exceptions +Db *table = new Db(dbenv, 0); +table->open("lookup.db", NULL, DB_BTREE, DB_CREATE, 0644); +``` + +Here's a Java example opening a Berkeley DB database using the 2.X interface: + +``` c +// Note: errors are thrown as exceptions +Db table = Db.open("lookup.db", Db.DB_BTREE, Db.DB_CREATE, 0644, dbenv, 0); +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +// Note: errors are thrown as exceptions +Db table = new Db(dbenv, 0); +table.open("lookup.db", null, Db.DB_BTREE, Db.DB_CREATE, 0644); +``` + +Note that if the dbenv argument is null, the database will not exist within an environment. diff --git a/docs-src/guides/upgrading/upgrade_3_0_dbenv.md b/docs-src/guides/upgrading/upgrade_3_0_dbenv.md new file mode 100644 index 000000000..b8e225d48 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_dbenv.md @@ -0,0 +1,101 @@ +--- +title: "DB_ENV structure" +api-name: "DB_ENV structure" +source: docs/upgrading/upgrade_3_0_dbenv.html +--- +## DB_ENV structure + +The DB_ENV structure is now opaque for applications in the Berkeley DB 3.0 release. Accesses to any fields within that structure by the application should be replaced with method calls. The following example illustrates this using the historic errpfx structure field. In the Berkeley DB 2.X releases, applications set error prefixes using code similar to the following: + +``` c +DB_ENV *dbenv; + +dbenv->errpfx = "my prefix"; +``` + +in the Berkeley DB 3.X releases, this should be done using the DB_ENV->set_errpfx() method, as follows: + +``` c +DB_ENV *dbenv; + +dbenv->set_errpfx(dbenv, "my prefix"); +``` + +The following table lists the DB_ENV fields previously used by applications and the methods that should now be used to set them. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DB_ENV fieldBerkeley DB 3.X method
db_errcallDB_ENV->set_errcall()
db_errfileDB_ENV->set_errfile()
db_errpfxDB_ENV->set_errpfx()
db_lorderThis field was removed from the DB_ENV structure in the Berkeley DB 3.0 release as no application should have ever used it. Any code using it should be evaluated for potential bugs.
db_paniccallDB_ENV->set_paniccall
db_verboseDB_ENV->set_verbose() +

Note: the db_verbose field was a simple boolean toggle, the DB_ENV->set_verbose() method takes arguments that specify exactly which verbose messages are desired.

lg_maxDB_ENV->set_lg_max()
lk_conflictsDB_ENV->set_lk_conflicts()
lk_detectDB_ENV->set_lk_detect()
lk_maxdbenv->set_lk_max
lk_modesDB_ENV->set_lk_conflicts()
mp_mmapsizeDB_ENV->set_mp_mmapsize()
mp_sizeDB_ENV->set_cachesize() +

Note: the DB_ENV->set_cachesize() function takes additional arguments. Setting both the second argument (the number of GB in the pool) and the last argument (the number of memory pools to create) to 0 will result in behavior that is backward-compatible with previous Berkeley DB releases.

tx_infoThis field was used by applications as an argument to the transaction subsystem functions. As those functions take references to a DB_ENV structure as arguments in the Berkeley DB 3.0 release, it should no longer be used by any application.
tx_maxDB_ENV->set_tx_max()
tx_recoverdbenv->set_tx_recover
diff --git a/docs-src/guides/upgrading/upgrade_3_0_dbenv_cxx.md b/docs-src/guides/upgrading/upgrade_3_0_dbenv_cxx.md new file mode 100644 index 000000000..9e9ed9a44 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_dbenv_cxx.md @@ -0,0 +1,68 @@ +--- +title: "DbEnv class for C++ and Java" +api-name: "DbEnv class for C++ and Java" +source: docs/upgrading/upgrade_3_0_dbenv_cxx.html +--- +## DbEnv class for C++ and Java + +The DbEnv::appinit() method and two constructors for the DbEnv class are gone. There is now a single way to create and initialize the environment. The way to create an environment is to use the new DbEnv constructor with one argument. After this call, the DbEnv can be configured with various set_XXX methods. Finally, a call to DbEnv::open is made to initialize the environment. + +Here's a C++ example creating a Berkeley DB environment using the 2.X interface + +``` c +int dberr; +DbEnv *dbenv = new DbEnv(); + +dbenv->set_error_stream(&cerr); +dbenv->set_errpfx("myprog"); + +if ((dberr = dbenv->appinit("/database/home", + NULL, DB_CREATE | DB_INIT_LOCK | DB_INIT_MPOOL)) != 0) { + cerr << "failure: " << strerror(dberr); + exit (1); +} +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +int dberr; +DbEnv *dbenv = new DbEnv(0); + +dbenv->set_error_stream(&cerr); +dbenv->set_errpfx("myprog"); + +if ((dberr = dbenv->open("/database/home", + NULL, DB_CREATE | DB_INIT_LOCK | DB_INIT_MPOOL, 0)) != 0) { + cerr << "failure: " << dbenv->strerror(dberr); + exit (1); +} +``` + +Here's a Java example creating a Berkeley DB environment using the 2.X interface: + +``` c +int dberr; +DbEnv dbenv = new DbEnv(); + +dbenv.set_error_stream(System.err); +dbenv.set_errpfx("myprog"); + +dbenv.appinit("/database/home", + null, Db.DB_CREATE | Db.DB_INIT_LOCK | Db.DB_INIT_MPOOL); +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +int dberr; +DbEnv dbenv = new DbEnv(0); + +dbenv.set_error_stream(System.err); +dbenv.set_errpfx("myprog"); + +dbenv.open("/database/home", + null, Db.DB_CREATE | Db.DB_INIT_LOCK | Db.DB_INIT_MPOOL, 0); +``` + +In the Berkeley DB 2.X release, DbEnv had accessors to obtain "managers" of type DbTxnMgr, DbMpool, DbLog, DbTxnMgr. If you used any of these managers, all their methods are now found directly in the DbEnv class. diff --git a/docs-src/guides/upgrading/upgrade_3_0_dbinfo.md b/docs-src/guides/upgrading/upgrade_3_0_dbinfo.md new file mode 100644 index 000000000..a880d8fa0 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_dbinfo.md @@ -0,0 +1,105 @@ +--- +title: "DBINFO structure" +api-name: "DBINFO structure" +source: docs/upgrading/upgrade_3_0_dbinfo.html +--- +## DBINFO structure + +The DB_INFO structure has been removed from the Berkeley DB 3.0 release. Accesses to any fields within that structure by the application should be replaced with method calls on the DB handle. The following example illustrates this using the historic db_cachesize structure field. In the Berkeley DB 2.X releases, applications could set the size of an underlying database cache using code similar to the following: + +``` c +DB_INFO dbinfo; + +memset(dbinfo, 0, sizeof(dbinfo)); +dbinfo.db_cachesize = 1024 * 1024; +``` + +in the Berkeley DB 3.X releases, this should be done using the DB->set_cachesize() method, as follows: + +``` c +DB *db; +int ret; + +ret = db->set_cachesize(db, 0, 1024 * 1024, 0); +``` + +The DB_INFO structure is no longer used in any way by the Berkeley DB 3.0 release, and should be removed from the application. + +The following table lists the DB_INFO fields previously used by applications and the methods that should now be used to set them. Because these calls provide configuration for the database open, they must precede the call to DB->open(). Calling them after the call to DB->open() will return an error. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DB_INFO fieldBerkeley DB 3.X method
bt_compareDB->set_bt_compare()
bt_minkeyDB->set_bt_minkey()
bt_prefixDB->set_bt_prefix()
db_cachesizeDB->set_cachesize() +

Note: the DB->set_cachesize() function takes additional arguments. Setting both the second argument (the number of GB in the pool) and the last argument (the number of memory pools to create) to 0 will result in behavior that is backward-compatible with previous Berkeley DB releases.

db_lorderDB->set_lorder()
db_mallocDB->set_malloc
db_pagesizeDB->set_pagesize()
dup_compareDB->set_dup_compare()
flagsDB->set_flags() +

Note: the DB_DELIMITER, DB_FIXEDLEN and DB_PAD flags no longer need to be set as there are specific methods off the DB handle that set the file delimiter, the length of fixed-length records and the fixed-length record pad character. They should simply be discarded from the application.

h_ffactorDB->set_h_ffactor()
h_hashDB->set_h_hash()
h_nelemDB->set_h_nelem()
re_delimDB->set_re_delim()
re_lenDB->set_re_len()
re_padDB->set_re_pad()
re_sourceDB->set_re_source()
diff --git a/docs-src/guides/upgrading/upgrade_3_0_disk.md b/docs-src/guides/upgrading/upgrade_3_0_disk.md new file mode 100644 index 000000000..b3ec0e362 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_3_0_disk.html +--- +## Upgrade Requirements + +Log file formats and the Btree, Recno and Hash Access Method database formats changed in the Berkeley DB 3.0 release. (The on-disk Btree/Recno format changed from version 6 to version 7. The on-disk Hash format changed from version 5 to version 6.) Until the underlying databases are upgraded, the DB->open() method will return a `DB_OLD_VERSION` error. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_3_0_eacces.md b/docs-src/guides/upgrading/upgrade_3_0_eacces.md new file mode 100644 index 000000000..eaac8c07f --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_eacces.md @@ -0,0 +1,8 @@ +--- +title: "EACCES" +api-name: "EACCES" +source: docs/upgrading/upgrade_3_0_eacces.html +--- +## EACCES + +There was an error in previous releases of the Berkeley DB documentation that said that the lock_put and lock_vec interfaces could return EACCES as an error to indicate that a lock could not be released because it was held by another locker. The application should be searched for any occurrences of EACCES. For each of these, any that are checking for an error return from lock_put or lock_vec should have the test and any error handling removed. diff --git a/docs-src/guides/upgrading/upgrade_3_0_eagain.md b/docs-src/guides/upgrading/upgrade_3_0_eagain.md new file mode 100644 index 000000000..2897b46dd --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_eagain.md @@ -0,0 +1,12 @@ +--- +title: "EAGAIN" +api-name: "EAGAIN" +source: docs/upgrading/upgrade_3_0_eagain.html +--- +## EAGAIN + +Historically, the Berkeley DB interfaces have returned the POSIX error value EAGAIN to indicate a deadlock. This has been removed from the Berkeley DB 3.0 release in order to make it possible for applications to distinguish between EAGAIN errors returned by the system and returns from Berkeley DB indicating deadlock. + +The application should be searched for any occurrences of EAGAIN. For each of these, any that are checking for a deadlock return from Berkeley DB should be changed to check for the DB_LOCK_DEADLOCK return value. + +If, for any reason, this is a difficult change for the application to make, the `include/db.src` distribution file should be modified to translate all returns of DB_LOCK_DEADLOCK to EAGAIN. Search for the string EAGAIN in that file, there is a comment that describes how to make the change. diff --git a/docs-src/guides/upgrading/upgrade_3_0_envopen.md b/docs-src/guides/upgrading/upgrade_3_0_envopen.md new file mode 100644 index 000000000..265b87aad --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_envopen.md @@ -0,0 +1,127 @@ +--- +title: "environment open/close/unlink" +api-name: "environment open/close/unlink" +source: docs/upgrading/upgrade_3_0_envopen.html +--- +## environment open/close/unlink + +The hardest part of upgrading your application from a 2.X code base to the 3.0 release is translating the Berkeley DB environment open, close and remove calls. + +There were two logical changes in this part of the Berkeley DB interface. First, in Berkeley DB 3.0, there are no longer separate structures that represent each subsystem (for example, DB_LOCKTAB or DB_TXNMGR) and an overall DB_ENV environment structure. Instead there is only the DB_ENV references should be passed around by your application instead of passing around DB_LOCKTAB or DB_TXNMGR references. This is likely to be a simple change for most applications as few applications use the lock_XXX, log_XXX, memp_XXX or txn_XXX interfaces to create Berkeley DB environments. + +The second change is that there are no longer separate open, close, and unlink interfaces to the Berkeley DB subsystems. For example, in previous releases, it was possible to open a lock subsystem either using db_appinit or using the lock_open call. In the 3.0 release the XXX_open interfaces to the subsystems have been removed, and subsystems must now be opened using the 3.0 replacement for the db_appinit call. + +To upgrade your application, first find each place your application opens, closes and/or removes a Berkeley DB environment. This will be code of the form: + +``` c +db_appinit, db_appexit +lock_open, lock_close, lock_unlink +log_open, log_close, log_unlink +memp_open, memp_close, memp_unlink +txn_open, txn_close, txn_unlink +``` + +Each of these groups of calls should be replaced with calls to db_env_create(), DB_ENV->open(), DB_ENV->close(), and DB_ENV->remove(). + +The db_env_create() call and the call to the DB_ENV->open() method replace the db_appinit, lock_open, log_open, memp_open and txn_open calls. The DB_ENV->close() method replaces the db_appexit, lock_close, log_close, memp_close and txn_close calls. The DB_ENV->remove() call replaces the lock_unlink, log_unlink, memp_unlink and txn_unlink calls. + +Here's an example creating a Berkeley DB environment using the 2.X interface: + +``` c +/* + * db_init -- + * Initialize the environment. + */ +DB_ENV * +db_init(home) + char *home; +{ + DB_ENV *dbenv; + + if ((dbenv = (DB_ENV *)calloc(sizeof(DB_ENV), 1)) == NULL) + return (errno); + + if ((errno = db_appinit(home, NULL, dbenv, + DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | + DB_USE_ENVIRON)) == 0) + return (dbenv); + + free(dbenv); + return (NULL); +} +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +/* + * db_init -- + * Initialize the environment. + */ +int +db_init(home, dbenvp) + char *home; + DB_ENV **dbenvp; +{ + int ret; + DB_ENV *dbenv; + + if ((ret = db_env_create(&dbenv, 0)) != 0) + return (ret); + + if ((ret = dbenv->open(dbenv, home, NULL, + DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | + DB_USE_ENVIRON, 0)) == 0) { + *dbenvp = dbenv; + return (0); + } + + (void)dbenv->close(dbenv, 0); + return (ret); +} +``` + +As you can see, the arguments to db_appinit and to DB_ENV->open() are largely the same. There is some minor re-organization: the mapping is that arguments \#1, 2, 3, and 4 to db_appinit become arguments \#2, 3, 1 and 4 to DB_ENV->open(). There is one additional argument to DB_ENV->open(), argument \#5. For backward compatibility with the 2.X Berkeley DB releases, simply set that argument to 0. + +It is only slightly more complex to translate calls to XXX_open to the DB_ENV->open() method. Here's an example of creating a lock region using the 2.X interface: + +``` c +lock_open(dir, DB_CREATE, 0664, dbenv, ®ionp); +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +if ((ret = db_env_create(&dbenv, 0)) != 0) + return (ret); + +if ((ret = dbenv->open(dbenv, + dir, NULL, DB_CREATE | DB_INIT_LOCK, 0664)) == 0) { + *dbenvp = dbenv; + return (0); +} +``` + +Note that in this example, you no longer need the DB_LOCKTAB structure reference that was required in Berkeley DB 2.X releases. + +The final issue with upgrading the db_appinit call is the DB_MPOOL_PRIVATE option previously provided for the db_appinit call. If your application is using this flag, it should almost certainly use the new DB_PRIVATE flag to the DB_ENV->open() method. Regardless, you should carefully consider this change before converting to use the DB_PRIVATE flag. + +Translating db_appexit or XXX_close calls to DB_ENV->close() is equally simple. Instead of taking a reference to a per-subsystem structure such as DB_LOCKTAB or DB_TXNMGR, all calls take a reference to a DB_ENV structure. The calling sequence is otherwise unchanged. Note that as the application no longer allocates the memory for the DB_ENV structure, application code to discard it after the call to db_appexit() is no longer needed. + +Translating XXX_unlink calls to DB_ENV->remove() is slightly more complex. As with DB_ENV->close(), the call takes a reference to a DB_ENV structure instead of a per-subsystem structure. The calling sequence is slightly different, however. Here is an example of removing a lock region using the 2.X interface: + +``` c +DB_ENV *dbenv; + +ret = lock_unlink(dir, 1, dbenv); +``` + +In the Berkeley DB 3.0 release, this code fragment would be written as: + +``` c +DB_ENV *dbenv; + +ret = dbenv->remove(dbenv, dir, NULL, DB_FORCE); +``` + +The additional argument to the DB_ENV->remove() function is a configuration argument similar to that previously taken by db_appinit and now taken by the DB_ENV->open() method. For backward compatibility this new argument should simply be set to NULL. The force argument to XXX_unlink is now a flag value that is set by bitwise inclusively **OR**'ing it the DB_ENV->remove() flag argument. diff --git a/docs-src/guides/upgrading/upgrade_3_0_func.md b/docs-src/guides/upgrading/upgrade_3_0_func.md new file mode 100644 index 000000000..fc34f9fd6 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_func.md @@ -0,0 +1,65 @@ +--- +title: "function arguments" +api-name: "function arguments" +source: docs/upgrading/upgrade_3_0_func.html +--- +## function arguments + +In Berkeley DB 3.0, there are no longer separate structures that represent each subsystem (for example, DB_LOCKTAB or DB_TXNMGR), and an overall DB_ENV environment structure. Instead there is only the DB_ENV references should be passed around by your application instead of passing around DB_LOCKTAB or DB_TXNMGR references. + +Each of the following functions: + +``` c +lock_detect +lock_get +lock_id +lock_put +lock_stat +lock_vec +``` + +should have its first argument, a reference to the DB_LOCKTAB structure, replaced with a reference to the enclosing DB_ENV structure. For example, the following line of code from a Berkeley DB 2.X application: + +``` c +DB_LOCKTAB *lt; +DB_LOCK lock; + +ret = lock_put(lt, lock); +``` + +should now be written as follows: + +``` c +DB_ENV *dbenv; +DB_LOCK *lock; + +ret = lock_put(dbenv, lock); +``` + +Similarly, all of the functions: + +``` c +log_archive +log_compare +log_file +log_flush +log_get +log_put +log_register +log_stat +log_unregister +``` + +should have their DB_LOG argument replaced with a reference to a DB_ENV structure, and the functions: + +``` c +memp_fopen +memp_register +memp_stat +memp_sync +memp_trickle +``` + +should have their DB_MPOOL argument replaced with a reference to a DB_ENV structure. + +You should remove all references to DB_LOCKTAB, DB_LOG, DB_MPOOL, and DB_TXNMGR structures from your application, they are no longer useful in any way. In fact, a simple way to identify all of the places that need to be upgraded is to remove all such structures and variables they declare, and then compile. You will see a warning message from your compiler in each case that needs to be upgraded. diff --git a/docs-src/guides/upgrading/upgrade_3_0_java.md b/docs-src/guides/upgrading/upgrade_3_0_java.md new file mode 100644 index 000000000..5c41e71b0 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_java.md @@ -0,0 +1,14 @@ +--- +title: "additional Java changes" +api-name: "additional Java changes" +source: docs/upgrading/upgrade_3_0_java.html +--- +## additional Java changes + +There are several additional types of exceptions thrown in the Berkeley DB 3.0 Java API. + +DbMemoryException and DbDeadlockException can be caught independently of DbException if you want to do special handling for these kinds of errors. Since they are subclassed from DbException, a try block that catches DbException will catch these also, so code is not required to change. The catch clause for these new exceptions should appear before the catch clause for DbException. + +You will need to add a catch clause for java.io.FileNotFoundException, since that can be thrown by Db.open and DbEnv.open. + +There are a number of smaller changes to the API that bring the C, C++ and Java APIs much closer in terms of functionality and usage. Please refer to the pages for upgrading C applications for further details. diff --git a/docs-src/guides/upgrading/upgrade_3_0_join.md b/docs-src/guides/upgrading/upgrade_3_0_join.md new file mode 100644 index 000000000..ac255593c --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_join.md @@ -0,0 +1,10 @@ +--- +title: "DB->join" +api-name: "DB->join" +source: docs/upgrading/upgrade_3_0_join.html +--- +## DB-\>join + +Historically, the last two arguments to the DB->join() method were a flags value followed by a reference to a memory location to store the returned cursor object. In the Berkeley DB 3.0 release, the order of those two arguments has been swapped for consistency with other Berkeley DB interfaces. + +The application should be searched for any occurrences of DB->join(). For each of these, the order of the last two arguments should be swapped. diff --git a/docs-src/guides/upgrading/upgrade_3_0_jump_set.md b/docs-src/guides/upgrading/upgrade_3_0_jump_set.md new file mode 100644 index 000000000..f6274efa7 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_jump_set.md @@ -0,0 +1,32 @@ +--- +title: "db_jump_set" +api-name: "db_jump_set" +source: docs/upgrading/upgrade_3_0_jump_set.html +--- +## db_jump_set + +The db_jump_set interface has been removed from the Berkeley DB 3.0 release, replaced by method calls on the DB_ENV handle. + +The following table lists the db_jump_set arguments previously used by applications and the methods that should now be used instead. + +| db_jump_set argument | Berkeley DB 3.X method | +|----|----| +| DB_FUNC_CLOSE | db_env_set_func_close | +| DB_FUNC_DIRFREE | db_env_set_func_dirfree | +| DB_FUNC_DIRLIST | db_env_set_func_dirlist | +| DB_FUNC_EXISTS | db_env_set_func_exists | +| DB_FUNC_FREE | db_env_set_func_free | +| DB_FUNC_FSYNC | db_env_set_func_fsync | +| DB_FUNC_IOINFO | db_env_set_func_ioinfo | +| DB_FUNC_MALLOC | db_env_set_func_malloc | +| DB_FUNC_MAP | dbenv_set_func_map | +| DB_FUNC_OPEN | db_env_set_func_open | +| DB_FUNC_READ | db_env_set_func_read | +| DB_FUNC_REALLOC | db_env_set_func_realloc | +| DB_FUNC_RUNLINK | The DB_FUNC_RUNLINK functionality has been removed from the Berkeley DB 3.0 release, and should be removed from the application. | +| DB_FUNC_SEEK | db_env_set_func_seek | +| DB_FUNC_SLEEP | db_env_set_func_sleep | +| DB_FUNC_UNLINK | db_env_set_func_unlink | +| DB_FUNC_UNMAP | dbenv_set_func_unmap | +| DB_FUNC_WRITE | db_env_set_func_write | +| DB_FUNC_YIELD | db_env_set_func_yield | diff --git a/docs-src/guides/upgrading/upgrade_3_0_lock_detect.md b/docs-src/guides/upgrading/upgrade_3_0_lock_detect.md new file mode 100644 index 000000000..ebc9d9128 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_lock_detect.md @@ -0,0 +1,10 @@ +--- +title: "lock_detect" +api-name: "lock_detect" +source: docs/upgrading/upgrade_3_0_lock_detect.html +--- +## lock_detect + +An additional argument has been added to the lock_detect function. + +The application should be searched for any occurrences of lock_detect. For each one, a NULL argument should be appended to the current arguments. diff --git a/docs-src/guides/upgrading/upgrade_3_0_lock_notheld.md b/docs-src/guides/upgrading/upgrade_3_0_lock_notheld.md new file mode 100644 index 000000000..ed3cca5d5 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_lock_notheld.md @@ -0,0 +1,8 @@ +--- +title: "DB_LOCK_NOTHELD" +api-name: "DB_LOCK_NOTHELD" +source: docs/upgrading/upgrade_3_0_lock_notheld.html +--- +## DB_LOCK_NOTHELD + +Historically, the Berkeley DB lock_put and lock_vec interfaces could return the DB_LOCK_NOTHELD error to indicate that a lock could not be released as it was held by another locker. This error can no longer be returned under any circumstances. The application should be searched for any occurrences of DB_LOCK_NOTHELD. For each of these, the test and any error processing should be removed. diff --git a/docs-src/guides/upgrading/upgrade_3_0_lock_put.md b/docs-src/guides/upgrading/upgrade_3_0_lock_put.md new file mode 100644 index 000000000..b6b309d38 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_lock_put.md @@ -0,0 +1,10 @@ +--- +title: "lock_put" +api-name: "lock_put" +source: docs/upgrading/upgrade_3_0_lock_put.html +--- +## lock_put + +An argument change has been made in the lock_put function. + +The application should be searched for any occurrences of lock_put. For each one, instead of passing a DB_LOCK variable as the last argument to the function, the address of the DB_LOCK variable should be passed. diff --git a/docs-src/guides/upgrading/upgrade_3_0_lock_stat.md b/docs-src/guides/upgrading/upgrade_3_0_lock_stat.md new file mode 100644 index 000000000..d017fa502 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_lock_stat.md @@ -0,0 +1,8 @@ +--- +title: "lock_stat" +api-name: "lock_stat" +source: docs/upgrading/upgrade_3_0_lock_stat.html +--- +## lock_stat + +The **st_magic**, **st_version**, **st_numobjs** and **st_refcnt** fields returned from the lock_stat function have been removed, and this information is no longer available. diff --git a/docs-src/guides/upgrading/upgrade_3_0_log_register.md b/docs-src/guides/upgrading/upgrade_3_0_log_register.md new file mode 100644 index 000000000..858ba497f --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_log_register.md @@ -0,0 +1,8 @@ +--- +title: "log_register" +api-name: "log_register" +source: docs/upgrading/upgrade_3_0_log_register.html +--- +## log_register + +An argument has been removed from the log_register function. The application should be searched for any occurrences of log_register. In each of these, the DBTYPE argument (it is the fourth argument) should be removed. diff --git a/docs-src/guides/upgrading/upgrade_3_0_log_stat.md b/docs-src/guides/upgrading/upgrade_3_0_log_stat.md new file mode 100644 index 000000000..287b7d688 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_log_stat.md @@ -0,0 +1,8 @@ +--- +title: "log_stat" +api-name: "log_stat" +source: docs/upgrading/upgrade_3_0_log_stat.html +--- +## log_stat + +The **st_refcnt** field returned from the log_stat function has been removed, and this information is no longer available. diff --git a/docs-src/guides/upgrading/upgrade_3_0_memp_stat.md b/docs-src/guides/upgrading/upgrade_3_0_memp_stat.md new file mode 100644 index 000000000..32b80c562 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_memp_stat.md @@ -0,0 +1,10 @@ +--- +title: "memp_stat" +api-name: "memp_stat" +source: docs/upgrading/upgrade_3_0_memp_stat.html +--- +## memp_stat + +The **st_refcnt** field returned from the memp_stat function has been removed, and this information is no longer available. + +The **st_cachesize** field returned from the memp_stat function has been replaced with two new fields, **st_gbytes** and **st_bytes**. diff --git a/docs-src/guides/upgrading/upgrade_3_0_open.md b/docs-src/guides/upgrading/upgrade_3_0_open.md new file mode 100644 index 000000000..fc3891d35 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_open.md @@ -0,0 +1,47 @@ +--- +title: "database open/close" +api-name: "database open/close" +source: docs/upgrading/upgrade_3_0_open.html +--- +## database open/close + +Database opens were changed in the Berkeley DB 3.0 release in a similar way to environment opens. + +To upgrade your application, first find each place your application opens a database, that is, calls the db_open function. Each of these calls should be replaced with calls to db_create() and DB->open(). + +Here's an example creating a Berkeley DB database using the 2.X interface: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = db_open(DATABASE, + DB_BTREE, DB_CREATE, 0664, dbenv, NULL, &dbp)) != 0) + return (ret); +``` + +In the Berkeley DB 3.0 release, this code would be written as: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = db_create(&dbp, dbenv, 0)) != 0) + return (ret); + +if ((ret = dbp->open(dbp, + DATABASE, NULL, DB_BTREE, DB_CREATE, 0664)) != 0) { + (void)dbp->close(dbp, 0); + return (ret); +} +``` + +As you can see, the arguments to db_open and to DB->open() are largely the same. There is some re-organization, and note that the enclosing DB_ENV structure is specified when the DB object is created using the db_create() function. There is one additional argument to DB->open(), argument \#3. For backward compatibility with the 2.X Berkeley DB releases, simply set that argument to NULL. + +There are two additional issues with the db_open call. + +First, it was possible in the 2.X releases for an application to provide an environment that did not contain a shared memory buffer pool as the database environment, and Berkeley DB would create a private one automatically. This functionality is no longer available, applications must specify the DB_INIT_MPOOL flag if databases are going to be opened in the environment. + +The final issue with upgrading the db_open call is that the DB_INFO structure is no longer used, having been replaced by individual methods on the DB handle. That change is discussed in detail later in this chapter. diff --git a/docs-src/guides/upgrading/upgrade_3_0_rmw.md b/docs-src/guides/upgrading/upgrade_3_0_rmw.md new file mode 100644 index 000000000..481da12cd --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_rmw.md @@ -0,0 +1,12 @@ +--- +title: "DB_RMW" +api-name: "DB_RMW" +source: docs/upgrading/upgrade_3_0_rmw.html +--- +## DB_RMW + +The following change applies only to applications using the Berkeley DB Concurrent Data Store product. If your application is not using that product, you can ignore this change. + +Historically, the DB->cursor() method took the DB_RMW flag to indicate that the created cursor would be used for write operations on the database. This flag has been renamed to the `DB_WRITECURSOR` flag. + +The application should be searched for any occurrences of DB_RMW. For each of these, any that are arguments to the DB->cursor() function should be changed to pass in the `DB_WRITECURSOR` flag instead. diff --git a/docs-src/guides/upgrading/upgrade_3_0_stat.md b/docs-src/guides/upgrading/upgrade_3_0_stat.md new file mode 100644 index 000000000..00558df06 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_stat.md @@ -0,0 +1,8 @@ +--- +title: "DB->stat" +api-name: "DB->stat" +source: docs/upgrading/upgrade_3_0_stat.html +--- +## DB-\>stat + +The **bt_flags** field returned from the DB->stat() method for Btree and Recno databases has been removed, and this information is no longer available. diff --git a/docs-src/guides/upgrading/upgrade_3_0_toc.md b/docs-src/guides/upgrading/upgrade_3_0_toc.md new file mode 100644 index 000000000..e232fb11b --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_toc.md @@ -0,0 +1,74 @@ +--- +title: "Chapter 14. Upgrading Berkeley DB 2.X applications to Berkeley DB 3.0" +api-name: "Chapter 14. Upgrading Berkeley DB 2.X applications to Berkeley DB 3.0" +source: docs/upgrading/upgrade_3_0_toc.html +--- +## Chapter 14. Upgrading Berkeley DB 2.X applications to Berkeley DB 3.0 + +**Table of Contents** + + [introduction](upgrade_3_0_toc.md#upgrade_3_0_intro) + + [environment open/close/unlink](upgrade_3_0_envopen.md) + + [function arguments](upgrade_3_0_func.md) + + [DB_ENV structure](upgrade_3_0_dbenv.md) + + [database open/close](upgrade_3_0_open.md) + + [db_xa_open](upgrade_3_0_xa.md) + + [DB structure](upgrade_3_0_db.md) + + [DBINFO structure](upgrade_3_0_dbinfo.md) + + [DB-\>join](upgrade_3_0_join.md) + + [DB-\>stat](upgrade_3_0_stat.md) + + [DB-\>sync and DB-\>close](upgrade_3_0_close.md) + + [lock_put](upgrade_3_0_lock_put.md) + + [lock_detect](upgrade_3_0_lock_detect.md) + + [lock_stat](upgrade_3_0_lock_stat.md) + + [log_register](upgrade_3_0_log_register.md) + + [log_stat](upgrade_3_0_log_stat.md) + + [memp_stat](upgrade_3_0_memp_stat.md) + + [txn_begin](upgrade_3_0_txn_begin.md) + + [txn_commit](upgrade_3_0_txn_commit.md) + + [txn_stat](upgrade_3_0_txn_stat.md) + + [DB_RMW](upgrade_3_0_rmw.md) + + [DB_LOCK_NOTHELD](upgrade_3_0_lock_notheld.md) + + [EAGAIN](upgrade_3_0_eagain.md) + + [EACCES](upgrade_3_0_eacces.md) + + [db_jump_set](upgrade_3_0_jump_set.md) + + [db_value_set](upgrade_3_0_value_set.md) + + [DbEnv class for C++ and Java](upgrade_3_0_dbenv_cxx.md) + + [Db class for C++ and Java](upgrade_3_0_db_cxx.md) + + [additional C++ changes](upgrade_3_0_cxx.md) + + [additional Java changes](upgrade_3_0_java.md) + + [Upgrade Requirements](upgrade_3_0_disk.md) + +## introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 2.X release interfaces to the Berkeley DB 3.0 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_3_0_txn_begin.md b/docs-src/guides/upgrading/upgrade_3_0_txn_begin.md new file mode 100644 index 000000000..7e3f90f77 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_txn_begin.md @@ -0,0 +1,10 @@ +--- +title: "txn_begin" +api-name: "txn_begin" +source: docs/upgrading/upgrade_3_0_txn_begin.html +--- +## txn_begin + +An additional argument has been added to the txn_begin function. + +The application should be searched for any occurrences of txn_begin. For each one, an argument of 0 should be appended to the current arguments. diff --git a/docs-src/guides/upgrading/upgrade_3_0_txn_commit.md b/docs-src/guides/upgrading/upgrade_3_0_txn_commit.md new file mode 100644 index 000000000..244ab4f22 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_txn_commit.md @@ -0,0 +1,10 @@ +--- +title: "txn_commit" +api-name: "txn_commit" +source: docs/upgrading/upgrade_3_0_txn_commit.html +--- +## txn_commit + +An additional argument has been added to the txn_commit function. + +The application should be searched for any occurrences of txn_commit. For each one, an argument of 0 should be appended to the current arguments. diff --git a/docs-src/guides/upgrading/upgrade_3_0_txn_stat.md b/docs-src/guides/upgrading/upgrade_3_0_txn_stat.md new file mode 100644 index 000000000..37a0712e7 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_txn_stat.md @@ -0,0 +1,8 @@ +--- +title: "txn_stat" +api-name: "txn_stat" +source: docs/upgrading/upgrade_3_0_txn_stat.html +--- +## txn_stat + +The **st_refcnt** field returned from the txn_stat function has been removed, and this information is no longer available. diff --git a/docs-src/guides/upgrading/upgrade_3_0_value_set.md b/docs-src/guides/upgrading/upgrade_3_0_value_set.md new file mode 100644 index 000000000..aa419a5f3 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_value_set.md @@ -0,0 +1,18 @@ +--- +title: "db_value_set" +api-name: "db_value_set" +source: docs/upgrading/upgrade_3_0_value_set.html +--- +## db_value_set + +The db_value_set function has been removed from the Berkeley DB 3.0 release, replaced by method calls on the DB_ENV handle. + +The following table lists the db_value_set arguments previously used by applications and the function that should now be used instead. + +| db_value_set argument | Berkeley DB 3.X method | +|----|----| +| DB_MUTEX_LOCKS | dbenv_set_mutexlocks | +| DB_REGION_ANON | The DB_REGION_ANON functionality has been replaced by the DB_SYSTEM_MEM and DB_PRIVATE flags to the DB_ENV->open() function. A direct translation is not available, please review the DB_ENV->open() manual page for more information. | +| DB_REGION_INIT | dbenv_set_region_init | +| DB_REGION_NAME | The DB_REGION_NAME functionality has been replaced by the DB_SYSTEM_MEM and DB_PRIVATE flags to the DB_ENV->open() function. A direct translation is not available, please review the DB_ENV->open() manual page for more information. | +| DB_TSL_SPINS | dbenv_set_tas_spins | diff --git a/docs-src/guides/upgrading/upgrade_3_0_xa.md b/docs-src/guides/upgrading/upgrade_3_0_xa.md new file mode 100644 index 000000000..47b8db7eb --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_0_xa.md @@ -0,0 +1,12 @@ +--- +title: "db_xa_open" +api-name: "db_xa_open" +source: docs/upgrading/upgrade_3_0_xa.html +--- +## db_xa_open + +The following change applies only to applications using Berkeley DB as an XA Resource Manager. If your application is not using Berkeley DB in this way, you can ignore this change. + +The db_xa_open function has been replaced with the `DB_XA_CREATE` flag to the db_create() function. All calls to db_xa_open should be replaced with calls to db_create() with the `DB_XA_CREATE` flag set, followed by a call to the DB->open() function. + +A similar change has been made for the C++ API, where the `DB_XA_CREATE` flag should be specified to the Db constructor. All calls to the Db::xa_open method should be replaced with the `DB_XA_CREATE` flag to the Db constructor, followed by a call to the DB::open method. diff --git a/docs-src/guides/upgrading/upgrade_3_1_btstat.md b/docs-src/guides/upgrading/upgrade_3_1_btstat.md new file mode 100644 index 000000000..54b5fd78a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_btstat.md @@ -0,0 +1,12 @@ +--- +title: "DB->stat" +api-name: "DB->stat" +source: docs/upgrading/upgrade_3_1_btstat.html +--- +## DB-\>stat + +For Btree database statistics, the DB->stat() method field **bt_nrecs** has been removed, replaced by two fields: **bt_nkeys** and **bt_ndata**. The **bt_nkeys** field returns a count of the unique keys in the database. The **bt_ndata** field returns a count of the key/data pairs in the database. Neither exactly matches the previous value of the **bt_nrecs** field, which returned a count of keys in the database, but, in the case of Btree databases, could overcount as it sometimes counted duplicate data items as unique keys. The application should be searched for any uses of the **bt_nrecs** field and the field should be changed to be either **bt_nkeys** or **bt_ndata**, whichever is more appropriate. + +For Hash database statistics, the DB->stat() method field **hash_nrecs** has been removed, replaced by two fields: **hash_nkeys** and **hash_ndata**. The **hash_nkeys** field returns a count of the unique keys in the database. The **hash_ndata** field returns a count of the key/data pairs in the database. The new **hash_nkeys** field exactly matches the previous value of the **hash_nrecs** field. The application should be searched for any uses of the **hash_nrecs** field, and the field should be changed to be **hash_nkeys**. + +For Queue database statistics, the DB->stat() method field **qs_nrecs** has been removed, replaced by two fields: **qs_nkeys** and **qs_ndata**. The **qs_nkeys** field returns a count of the unique keys in the database. The **qs_ndata** field returns a count of the key/data pairs in the database. The new **qs_nkeys** field exactly matches the previous value of the **qs_nrecs** field. The application should be searched for any uses of the **qs_nrecs** field, and the field should be changed to be **qs_nkeys**. diff --git a/docs-src/guides/upgrading/upgrade_3_1_config.md b/docs-src/guides/upgrading/upgrade_3_1_config.md new file mode 100644 index 000000000..de52381bb --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_config.md @@ -0,0 +1,14 @@ +--- +title: "DB_ENV->open, DB_ENV->remove" +api-name: "DB_ENV->open, DB_ENV->remove" +source: docs/upgrading/upgrade_3_1_config.html +--- +## DB_ENV-\>open, DB_ENV-\>remove + +In the Berkeley DB 3.1 release, the **config** argument to the DB_ENV->open() and DB_ENV->remove() methods has been removed, replaced by additional methods on the DB_ENV handle. If your application calls DB_ENV->open() or DB_ENV->remove() with a NULL **config** argument, find those functions and remove the config argument from the call. If your application has non-NULL **config** argument, the strings values in that argument are replaced with calls to DB_ENV methods as follows: + +| Previous config string | Berkeley DB 3.1 version method | +|----|----| +| DB_DATA_DIR | DB_ENV->set_data_dir() | +| DB_LOG_DIR | DB_ENV->set_lg_dir() | +| DB_TMP_DIR | DB_ENV->set_tmp_dir() | diff --git a/docs-src/guides/upgrading/upgrade_3_1_disk.md b/docs-src/guides/upgrading/upgrade_3_1_disk.md new file mode 100644 index 000000000..97f8894ab --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_disk.md @@ -0,0 +1,12 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_3_1_disk.html +--- +## Upgrade Requirements + +Log file formats and the Btree, Queue, Recno and Hash Access Method database formats changed in the Berkeley DB 3.1 release. (The on-disk Btree/Recno format changed from version 7 to version 8. The on-disk Hash format changed from version 6 to version 7. The on-disk Queue format changed from version 1 to version 2.) Until the underlying databases are upgraded, the DB->open() method will return a `DB_OLD_VERSION` error. + +An additional flag, DB_DUPSORT, has been added to the DB->upgrade() method for this upgrade. Please review the DB->upgrade() documentation for further information. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_3_1_dup.md b/docs-src/guides/upgrading/upgrade_3_1_dup.md new file mode 100644 index 000000000..3663c2bd8 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_dup.md @@ -0,0 +1,10 @@ +--- +title: "identical duplicate data items" +api-name: "identical duplicate data items" +source: docs/upgrading/upgrade_3_1_dup.html +--- +## identical duplicate data items + +In previous releases of Berkeley DB, it was not an error to store identical duplicate data items, or, for those that just like the way it sounds, duplicate duplicates. However, there were implementation bugs where storing duplicate duplicates could cause database corruption. + +In this release, applications may store identical duplicate data items as long as the data items are unsorted. It is an error to attempt to store identical duplicate data items when duplicates are being stored in a sorted order. This restriction is expected to be lifted in a future release. See Duplicate data items for more information. diff --git a/docs-src/guides/upgrading/upgrade_3_1_env.md b/docs-src/guides/upgrading/upgrade_3_1_env.md new file mode 100644 index 000000000..c93c6f36e --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_env.md @@ -0,0 +1,34 @@ +--- +title: "environment configuration" +api-name: "environment configuration" +source: docs/upgrading/upgrade_3_1_env.html +--- +## environment configuration + +A set of DB_ENV configuration methods which were not environment specific, but which instead affected the entire application space, have been removed from the DB_ENV object and replaced by static functions. The following table lists the DB_ENV methods previously available to applications and the static functions that should now be used instead. + +| DB_ENV method | Berkeley DB 3.1 function | +|----|----| +| DB_ENV-\>set_func_close | db_env_set_func_close | +| DB_ENV-\>set_func_dirfree | db_env_set_func_dirfree | +| DB_ENV-\>set_func_dirlist | db_env_set_func_dirlist | +| DB_ENV-\>set_func_exists | db_env_set_func_exists | +| DB_ENV-\>set_func_free | db_env_set_func_free | +| DB_ENV-\>set_func_fsync | db_env_set_func_fsync | +| DB_ENV-\>set_func_ioinfo | db_env_set_func_ioinfo | +| DB_ENV-\>set_func_malloc | db_env_set_func_malloc | +| DB_ENV-\>set_func_map | dbenv_set_func_map | +| DB_ENV-\>set_func_open | db_env_set_func_open | +| DB_ENV-\>set_func_read | db_env_set_func_read | +| DB_ENV-\>set_func_realloc | db_env_set_func_realloc | +| DB_ENV-\>set_func_rename | db_env_set_func_rename | +| DB_ENV-\>set_func_seek | db_env_set_func_seek | +| DB_ENV-\>set_func_sleep | db_env_set_func_sleep | +| DB_ENV-\>set_func_unlink | db_env_set_func_unlink | +| DB_ENV-\>set_func_unmap | dbenv_set_func_unmap | +| DB_ENV-\>set_func_write | db_env_set_func_write | +| DB_ENV-\>set_func_yield | db_env_set_func_yield | +| DB_ENV-\>set_pageyield | dbenv_set_pageyield | +| DB_ENV-\>set_region_init | dbenv_set_region_init | +| DB_ENV-\>set_mutexlocks | dbenv_set_mutexlocks | +| DB_ENV-\>set_tas_spins | dbenv_set_tas_spins | diff --git a/docs-src/guides/upgrading/upgrade_3_1_log_register.md b/docs-src/guides/upgrading/upgrade_3_1_log_register.md new file mode 100644 index 000000000..56edf4d60 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_log_register.md @@ -0,0 +1,8 @@ +--- +title: "log_register" +api-name: "log_register" +source: docs/upgrading/upgrade_3_1_log_register.html +--- +## log_register + +The arguments to the log_register and log_unregister interfaces have changed. Instead of returning (and passing in) a logging file ID, a reference to the DB structure being registered (or unregistered) is passed. The application should be searched for any occurrences of log_register and log_unregister. For each one, change the arguments to be a reference to the DB structure being registered or unregistered. diff --git a/docs-src/guides/upgrading/upgrade_3_1_logalloc.md b/docs-src/guides/upgrading/upgrade_3_1_logalloc.md new file mode 100644 index 000000000..e85ded6c1 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_logalloc.md @@ -0,0 +1,10 @@ +--- +title: "log file pre-allocation" +api-name: "log file pre-allocation" +source: docs/upgrading/upgrade_3_1_logalloc.html +--- +## log file pre-allocation + +This change only affects Win/32 applications. + +On Win/32 platforms Berkeley DB no longer pre-allocates log files. The problem was a noticeable performance spike as each log file was created. To turn this feature back on, search for the flag DB_OSO_LOG in the source file `log/log_put.c` and make the change described there, or contact us for assistance. diff --git a/docs-src/guides/upgrading/upgrade_3_1_memp_register.md b/docs-src/guides/upgrading/upgrade_3_1_memp_register.md new file mode 100644 index 000000000..17512ab9d --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_memp_register.md @@ -0,0 +1,8 @@ +--- +title: "memp_register" +api-name: "memp_register" +source: docs/upgrading/upgrade_3_1_memp_register.html +--- +## memp_register + +An additional argument has been added to the **pgin** and **pgout** functions provided to the memp_register function. The application should be searched for any occurrences of memp_register. For each one, if **pgin** or **pgout** functions are specified, the **pgin** and **pgout** functions should be modified to take an initial argument of a **DB_ENV \***. This argument is intended to support better error reporting for applications, and may be entirely ignored by the **pgin** and **pgout** functions themselves. diff --git a/docs-src/guides/upgrading/upgrade_3_1_put.md b/docs-src/guides/upgrading/upgrade_3_1_put.md new file mode 100644 index 000000000..0a5765bdb --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_put.md @@ -0,0 +1,49 @@ +--- +title: "DB->put" +api-name: "DB->put" +source: docs/upgrading/upgrade_3_1_put.html +--- +## DB-\>put + +For the Queue and Recno access methods, when the DB_APPEND flag is specified to the DB->put() method, the allocated record number is returned to the application in the **key** DBT argument. In previous releases of Berkeley DB, this DBT structure did not follow the usual DBT conventions. For example, it was not possible to cause Berkeley DB to allocate space for the returned record number. Rather, it was always assumed that the **data** field of the **key** structure referred to memory that could be used as storage for a db_recno_t type. + +As of the Berkeley DB 3.1.0 release, the **key** structure behaves as described in the DBT C++/Java class or C structure documentation. + +Applications which are using the DB_APPEND flag for Queue and Recno access method databases will require a change to upgrade to the Berkeley DB 3.1 releases. The simplest change is likely to be to add the DB_DBT_USERMEM flag to the **key** structure. For example, code that appears as follows: + +``` c +DBT key; +db_recno_t recno; + +memset(&key, 0, sizeof(DBT)); +key.data = &recno; +key.size = sizeof(recno); +DB->put(DB, NULL, &key, &data, DB_APPEND); +printf("new record number is %lu\n", (u_long)recno); +``` + +would be changed to: + +``` c +DBT key; +db_recno_t recno; + +memset(&key, 0, sizeof(DBT)); +key.data = &recno; +key.ulen = sizeof(recno); +key.flags = DB_DBT_USERMEM; +DB->put(DB, NULL, &key, &data, DB_APPEND); +printf("new record number is %lu\n", (u_long)recno); +``` + +Note that the **ulen** field is now set as well as the flag value. An alternative change would be: + +``` c +DBT key; +db_recno_t recno; + +memset(&key, 0, sizeof(DBT)); +DB->put(DB, NULL, &key, &data, DB_APPEND); +recno = *(db_recno_t *)key->data; +printf("new record number is %lu\n", (u_long)recno); +``` diff --git a/docs-src/guides/upgrading/upgrade_3_1_set_feedback.md b/docs-src/guides/upgrading/upgrade_3_1_set_feedback.md new file mode 100644 index 000000000..392682c93 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_set_feedback.md @@ -0,0 +1,10 @@ +--- +title: "DB_ENV->set_feedback, DB->set_feedback" +api-name: "DB_ENV->set_feedback, DB->set_feedback" +source: docs/upgrading/upgrade_3_1_set_feedback.html +--- +## DB_ENV-\>set_feedback, DB-\>set_feedback + +Starting with the 3.1 release of Berkeley DB, the DB_ENV->set_feedback() and DB->set_feedback() methods may return an error value, that is, they are no longer declared as returning no value, instead they return an int or throw an exception as appropriate when an error occurs. + +If your application calls these functions, you may want to check for a possible error on return. diff --git a/docs-src/guides/upgrading/upgrade_3_1_set_paniccall.md b/docs-src/guides/upgrading/upgrade_3_1_set_paniccall.md new file mode 100644 index 000000000..08cf90865 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_set_paniccall.md @@ -0,0 +1,10 @@ +--- +title: "DB_ENV->set_paniccall, DB->set_paniccall" +api-name: "DB_ENV->set_paniccall, DB->set_paniccall" +source: docs/upgrading/upgrade_3_1_set_paniccall.html +--- +## DB_ENV-\>set_paniccall, DB-\>set_paniccall + +Starting with the 3.1 release of Berkeley DB, the DB_ENV-\>set_paniccall and DB-\>set_paniccall methods may return an error value, that is, they are no longer declared as returning no value, instead they return an int or throw an exception as appropriate when an error occurs. + +If your application calls these functions, you may want to check for a possible error on return. diff --git a/docs-src/guides/upgrading/upgrade_3_1_set_tx_recover.md b/docs-src/guides/upgrading/upgrade_3_1_set_tx_recover.md new file mode 100644 index 000000000..b7fed9371 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_set_tx_recover.md @@ -0,0 +1,18 @@ +--- +title: "DB_ENV->set_tx_recover" +api-name: "DB_ENV->set_tx_recover" +source: docs/upgrading/upgrade_3_1_set_tx_recover.html +--- +## DB_ENV-\>set_tx_recover + +The redo parameter of the function passed to DB_ENV-\>set_tx_recover used to be an integer set to any one of a number of \#defined values. In the 3.1 release of Berkeley DB, the redo parameter has been replaced by the op parameter which is an enumerated type of type db_recops. + +If your application calls DB_ENV-\>set_tx_recover, then find the function referred to by the call. Replace the flag values in that function as follows: + +| Previous flag | Berkeley DB 3.1 version flag | +|-------------------|------------------------------| +| TXN_BACKWARD_ROLL | DB_TXN_BACKWARD_ROLL | +| TXN_FORWARD_ROLL | DB_TXN_FORWARD_ROLL | +| TXN_OPENFILES | DB_TXN_OPENFILES | +| TXN_REDO | DB_TXN_FORWARD_ROLL | +| TXN_UNDO | DB_TXN_ABORT | diff --git a/docs-src/guides/upgrading/upgrade_3_1_sysmem.md b/docs-src/guides/upgrading/upgrade_3_1_sysmem.md new file mode 100644 index 000000000..664119809 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_sysmem.md @@ -0,0 +1,8 @@ +--- +title: "DB_SYSTEM_MEM" +api-name: "DB_SYSTEM_MEM" +source: docs/upgrading/upgrade_3_1_sysmem.html +--- +## DB_SYSTEM_MEM + +Using the DB_SYSTEM_MEM option on UNIX systems now requires the specification of a base system memory segment ID, using the DB_ENV->set_shm_key() method. Any valid segment ID may be specified, for example, one returned by the UNIX `ftok`(3) function. diff --git a/docs-src/guides/upgrading/upgrade_3_1_tcl.md b/docs-src/guides/upgrading/upgrade_3_1_tcl.md new file mode 100644 index 000000000..2e49a6355 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_tcl.md @@ -0,0 +1,10 @@ +--- +title: "Tcl API" +api-name: "Tcl API" +source: docs/upgrading/upgrade_3_1_tcl.html +--- +## Tcl API + +The Berkeley DB Tcl API has been modified so that the **-mpool** option to the **berkdb env** command is now the default behavior. The Tcl API has also been modified so that the **-txn** option to the **berkdb env** command implies the **-lock** and **-log** options. Tcl scripts should be updated to remove the **-mpool**, **-lock** and **-log** options. + +The Berkeley DB Tcl API has been modified to follow the Tcl standard rules for integer conversion, for example, if the first two characters of a record number are "0x", the record number is expected to be in hexadecimal form. diff --git a/docs-src/guides/upgrading/upgrade_3_1_tmp.md b/docs-src/guides/upgrading/upgrade_3_1_tmp.md new file mode 100644 index 000000000..f219a0c32 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_tmp.md @@ -0,0 +1,12 @@ +--- +title: "DB_TMP_DIR" +api-name: "DB_TMP_DIR" +source: docs/upgrading/upgrade_3_1_tmp.html +--- +## DB_TMP_DIR + +This change only affects Win/32 applications that create in-memory databases. + +On Win/32 platforms an additional test has been added when searching for the appropriate directory in which to create the temporary files that are used to back in-memory databases. Berkeley DB now uses any return value from the GetTempPath interface as the temporary file directory name before resorting to the static list of compiled-in pathnames. + +If the system registry does not return the same directory as Berkeley DB has been using previously, this change could cause temporary backing files to move to a new directory when applications are upgraded to the 3.1 release. In extreme cases, this could create (or fix) security problems if the file protection modes for the system registry directory are different from those on the directory previously used by Berkeley DB. diff --git a/docs-src/guides/upgrading/upgrade_3_1_toc.md b/docs-src/guides/upgrading/upgrade_3_1_toc.md new file mode 100644 index 000000000..f4b44da75 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_toc.md @@ -0,0 +1,46 @@ +--- +title: "Chapter 13. Upgrading Berkeley DB 3.0 applications to Berkeley DB 3.1" +api-name: "Chapter 13. Upgrading Berkeley DB 3.0 applications to Berkeley DB 3.1" +source: docs/upgrading/upgrade_3_1_toc.html +--- +## Chapter 13. Upgrading Berkeley DB 3.0 applications to Berkeley DB 3.1 + +**Table of Contents** + + [introduction](upgrade_3_1_toc.md#upgrade_3_1_intro) + + [DB_ENV-\>open, DB_ENV-\>remove](upgrade_3_1_config.md) + + [DB_ENV-\>set_tx_recover](upgrade_3_1_set_tx_recover.md) + + [DB_ENV-\>set_feedback, DB-\>set_feedback](upgrade_3_1_set_feedback.md) + + [DB_ENV-\>set_paniccall, DB-\>set_paniccall](upgrade_3_1_set_paniccall.md) + + [DB-\>put](upgrade_3_1_put.md) + + [identical duplicate data items](upgrade_3_1_dup.md) + + [DB-\>stat](upgrade_3_1_btstat.md) + + [DB_SYSTEM_MEM](upgrade_3_1_sysmem.md) + + [log_register](upgrade_3_1_log_register.md) + + [memp_register](upgrade_3_1_memp_register.md) + + [txn_checkpoint](upgrade_3_1_txn_check.md) + + [environment configuration](upgrade_3_1_env.md) + + [Tcl API](upgrade_3_1_tcl.md) + + [DB_TMP_DIR](upgrade_3_1_tmp.md) + + [log file pre-allocation](upgrade_3_1_logalloc.md) + + [Upgrade Requirements](upgrade_3_1_disk.md) + +## introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 3.0 release interfaces to the Berkeley DB 3.1 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_3_1_txn_check.md b/docs-src/guides/upgrading/upgrade_3_1_txn_check.md new file mode 100644 index 000000000..4655c193c --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_1_txn_check.md @@ -0,0 +1,10 @@ +--- +title: "txn_checkpoint" +api-name: "txn_checkpoint" +source: docs/upgrading/upgrade_3_1_txn_check.html +--- +## txn_checkpoint + +An additional argument has been added to the txn_checkpoint function. + +The application should be searched for any occurrences of txn_checkpoint. For each one, an argument of 0 should be appended to the current arguments. diff --git a/docs-src/guides/upgrading/upgrade_3_2_callback.md b/docs-src/guides/upgrading/upgrade_3_2_callback.md new file mode 100644 index 000000000..703ef07d8 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_callback.md @@ -0,0 +1,12 @@ +--- +title: "DB callback functions, app_private field" +api-name: "DB callback functions, app_private field" +source: docs/upgrading/upgrade_3_2_callback.html +--- +## DB callback functions, app_private field + +In the Berkeley DB 3.2 release, four application callback functions (the callback functions set by DB->set_bt_compare(), DB->set_bt_prefix(), DB->set_dup_compare() and DB->set_h_hash()) were modified to take a reference to a DB object as their first argument. This change allows the Berkeley DB Java API to reasonably support these interfaces. There is currently no need for the callback functions to do anything with this additional argument. + +C and C++ applications that specify their own Btree key comparison, Btree prefix comparison, duplicate data item comparison or Hash functions should modify these functions to take a reference to a DB structure as their first argument. No further change is required. + +The app_private field of the DBT structure (accessible only from the Berkeley DB C API) has been removed in the 3.2 release. It was replaced with app_private fields in the DB_ENV handles. Applications using this field will have to convert to using one of the replacement fields. diff --git a/docs-src/guides/upgrading/upgrade_3_2_db_dump.md b/docs-src/guides/upgrading/upgrade_3_2_db_dump.md new file mode 100644 index 000000000..d6cba5c15 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_db_dump.md @@ -0,0 +1,8 @@ +--- +title: "db_dump" +api-name: "db_dump" +source: docs/upgrading/upgrade_3_2_db_dump.html +--- +## db_dump + +In previous releases of Berkeley DB, the db_dump utility dumped Recno access method database keys as numeric strings. For consistency, the db_dump utility has been changed in the 3.2 release to dump record numbers as hex pairs when the data items are being dumped as hex pairs. (See the **-k** and **-p** options to the db_dump utility for more information.) Any applications or scripts post-processing the output of the db_dump utility for Recno databases under these conditions may require modification. diff --git a/docs-src/guides/upgrading/upgrade_3_2_disk.md b/docs-src/guides/upgrading/upgrade_3_2_disk.md new file mode 100644 index 000000000..defd5c477 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_3_2_disk.html +--- +## Upgrade Requirements + +Log file formats and the Queue Access Method database formats changed in the Berkeley DB 3.2 release. (The on-disk Queue format changed from version 2 to version 3.) Until the underlying databases are upgraded, the `DB_OLD_VERSION` error. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_3_2_handle.md b/docs-src/guides/upgrading/upgrade_3_2_handle.md new file mode 100644 index 000000000..7795b03fb --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_handle.md @@ -0,0 +1,8 @@ +--- +title: "Java and C++ object reuse" +api-name: "Java and C++ object reuse" +source: docs/upgrading/upgrade_3_2_handle.html +--- +## Java and C++ object reuse + +In previous releases of Berkeley DB, Java DbEnv and Db objects, and C++ DbEnv and Db objects could be reused after they were closed, by calling open on them again. This is no longer permitted, and these objects no longer allow any operations after a close. Applications reusing these objects should be modified to create new objects instead. diff --git a/docs-src/guides/upgrading/upgrade_3_2_incomplete.md b/docs-src/guides/upgrading/upgrade_3_2_incomplete.md new file mode 100644 index 000000000..7a19ed700 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_incomplete.md @@ -0,0 +1,14 @@ +--- +title: "DB_INCOMPLETE" +api-name: "DB_INCOMPLETE" +source: docs/upgrading/upgrade_3_2_incomplete.html +--- +## DB_INCOMPLETE + +There are a number of functions that flush pages from the Berkeley DB shared memory buffer pool to disk. Most of those functions can potentially fail because a page that needs to be flushed is not currently available. However, this is not a hard failure and is rarely cause for concern. In the Berkeley DB 3.2 release, the C++ API (if that API is configured to throw exceptions) and the Java API have been changed so that this failure does not throw an exception, but rather returns a non-zero error code of `DB_INCOMPLETE`. + +The following C++ methods will return `DB_INCOMPLETE` rather than throw an exception: Db::close, Db::sync, DbEnv::memp_sync, DbEnv::txn_checkpoint, and DbMpoolFile::memp_fsync. + +The following Java methods are now declared "public int" rather than "public void", and will return `Db.DB_INCOMPLETE` rather than throw an exception: `Db.close()`, `Db.sync()`, and `DbEnv.checkpoint()`. + +It is likely that the only change required by any application will be those currently checking for a `DB_INCOMPLETE` return that has been encapsulated in an exception. diff --git a/docs-src/guides/upgrading/upgrade_3_2_mutexlock.md b/docs-src/guides/upgrading/upgrade_3_2_mutexlock.md new file mode 100644 index 000000000..48d86c445 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_mutexlock.md @@ -0,0 +1,8 @@ +--- +title: "DB_ENV->set_mutexlocks" +api-name: "DB_ENV->set_mutexlocks" +source: docs/upgrading/upgrade_3_2_mutexlock.html +--- +## DB_ENV-\>set_mutexlocks + +Previous Berkeley DB releases included the db_env_set_mutexlocks function, intended for debugging, that allows applications to always obtain requested mutual exclusion mutexes without regard for their availability. This function has been replaced with dbenv_set_mutexlocks, which provides the same functionality on a per-database environment basis. Applications using the old function should be updated to use the new one. diff --git a/docs-src/guides/upgrading/upgrade_3_2_notfound.md b/docs-src/guides/upgrading/upgrade_3_2_notfound.md new file mode 100644 index 000000000..b7e29061f --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_notfound.md @@ -0,0 +1,8 @@ +--- +title: "Java java.io.FileNotFoundException" +api-name: "Java java.io.FileNotFoundException" +source: docs/upgrading/upgrade_3_2_notfound.html +--- +## Java java.io.FileNotFoundException + +The Java DbEnv.remove, Db.remove and Db.rename methods now throw java.io.FileNotFoundException in the case where the named file does not exist. Applications should be modified to catch this exception where appropriate. diff --git a/docs-src/guides/upgrading/upgrade_3_2_renumber.md b/docs-src/guides/upgrading/upgrade_3_2_renumber.md new file mode 100644 index 000000000..689c6d426 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_renumber.md @@ -0,0 +1,10 @@ +--- +title: "Logically renumbering records" +api-name: "Logically renumbering records" +source: docs/upgrading/upgrade_3_2_renumber.html +--- +## Logically renumbering records + +In the Berkeley DB 3.2 release, cursor adjustment semantics changed for Recno databases with mutable record numbers. Before the 3.2 release, cursors were adjusted to point to the previous or next record at the time the record to which the cursor referred was deleted. This could lead to unexpected behaviors. For example, two cursors referring to sequential records that were both deleted would lose their relationship to each other and would refer to the same position in the database instead of their original sequential relationship. There were also command sequences that would have unexpected results. For example, DB_AFTER and DB_BEFORE cursor put operations, using a cursor previously used to delete an item, would perform the put relative to the cursor's adjusted position and not its original position. + +In the Berkeley DB 3.2 release, cursors maintain their position in the tree regardless of deletion operations using the cursor. Applications that perform database operations, using cursors previously used to delete entries in Recno databases with mutable record numbers, should be evaluated to ensure that the new semantics do not cause application failure. diff --git a/docs-src/guides/upgrading/upgrade_3_2_set_flags.md b/docs-src/guides/upgrading/upgrade_3_2_set_flags.md new file mode 100644 index 000000000..85d1b90ec --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_set_flags.md @@ -0,0 +1,10 @@ +--- +title: "DB_ENV->set_flags" +api-name: "DB_ENV->set_flags" +source: docs/upgrading/upgrade_3_2_set_flags.html +--- +## DB_ENV-\>set_flags + +A new method has been added to the Berkeley DB environment handle, DB_ENV->set_flags(). This method currently takes three flags: DB_CDB_ALLDB, DB_NOMMAP, and DB_TXN_NOSYNC. The first of these flags, DB_CDB_ALLDB, provides new functionality, allowing Berkeley DB Concurrent Data Store applications to do locking across multiple databases. + +The other two flags, DB_NOMMAP and DB_TXN_NOSYNC, were specified to the DB_ENV->open() method in previous releases. In the 3.2 release, they have been moved to the DB_ENV->set_flags() method because this allows the database environment's value to be toggled during the life of the application as well as because it is a more appropriate place for them. Applications specifying either the DB_NOMMAP or DB_TXN_NOSYNC flags to the DB_ENV->open() method should replace those flags with calls to the DB_ENV->set_flags() method. diff --git a/docs-src/guides/upgrading/upgrade_3_2_toc.md b/docs-src/guides/upgrading/upgrade_3_2_toc.md new file mode 100644 index 000000000..98290be88 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_toc.md @@ -0,0 +1,34 @@ +--- +title: "Chapter 12. Upgrading Berkeley DB 3.1 applications to Berkeley DB 3.2" +api-name: "Chapter 12. Upgrading Berkeley DB 3.1 applications to Berkeley DB 3.2" +source: docs/upgrading/upgrade_3_2_toc.html +--- +## Chapter 12. Upgrading Berkeley DB 3.1 applications to Berkeley DB 3.2 + +**Table of Contents** + + [introduction](upgrade_3_2_toc.md#upgrade_3_2_intro) + + [DB_ENV-\>set_flags](upgrade_3_2_set_flags.md) + + [DB callback functions, app_private field](upgrade_3_2_callback.md) + + [Logically renumbering records](upgrade_3_2_renumber.md) + + [DB_INCOMPLETE](upgrade_3_2_incomplete.md) + + [DB_ENV-\>set_tx_recover](upgrade_3_2_tx_recover.md) + + [DB_ENV-\>set_mutexlocks](upgrade_3_2_mutexlock.md) + + [Java and C++ object reuse](upgrade_3_2_handle.md) + + [Java java.io.FileNotFoundException](upgrade_3_2_notfound.md) + + [db_dump](upgrade_3_2_db_dump.md) + + [Upgrade Requirements](upgrade_3_2_disk.md) + +## introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 3.1 release interfaces to the Berkeley DB 3.2 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_3_2_tx_recover.md b/docs-src/guides/upgrading/upgrade_3_2_tx_recover.md new file mode 100644 index 000000000..086366653 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_2_tx_recover.md @@ -0,0 +1,12 @@ +--- +title: "DB_ENV->set_tx_recover" +api-name: "DB_ENV->set_tx_recover" +source: docs/upgrading/upgrade_3_2_tx_recover.html +--- +## DB_ENV-\>set_tx_recover + +The **info** parameter of the function passed to DB_ENV-\>set_tx_recover is no longer needed. If your application calls DB_ENV-\>set_tx_recover, find the callback function referred to by that call and remove the **info** parameter. + +In addition, the called function no longer needs to handle Berkeley DB log records, Berkeley DB will handle them internally as well as call the application-specified function. Any handling of Berkeley DB log records in the application's callback function may be removed. + +In addition, the callback function will no longer be called with the DB_TXN_FORWARD_ROLL flag specified unless the transaction enclosing the operation successfully committed. diff --git a/docs-src/guides/upgrading/upgrade_3_3_alloc.md b/docs-src/guides/upgrading/upgrade_3_3_alloc.md new file mode 100644 index 000000000..00422368b --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_alloc.md @@ -0,0 +1,36 @@ +--- +title: "DB->set_malloc, DB->set_realloc" +api-name: "DB->set_malloc, DB->set_realloc" +source: docs/upgrading/upgrade_3_3_alloc.html +--- +## DB-\>set_malloc, DB-\>set_realloc + +There are two new methods in the Berkeley DB 3.3 release: DB_ENV->set_alloc(). These functions allow applications to specify a set of allocation functions for the Berkeley DB library to use when allocating memory to be owned by the application and when freeing memory that was originally allocated by the application. + +The new methods affect or replace the following historic methods: + +DB-\>set_malloc +The DB-\>set_malloc method has been replaced in its entirety. Applications using this method should replace the call with a call to DB->set_alloc(). + +DB-\>set_realloc +The DB-\>set_realloc method has been replaced in its entirety. Applications using this method should replace the call with a call to DB->set_alloc(). + +DB->stat() method +has been replaced. Applications using this method should do as follows: if the argument is NULL, it should simply be removed. If non-NULL, it should be replaced with a call to DB->set_alloc(). + +lock_stat +The historic **db_malloc** argument to the lock_stat function has been replaced. Applications using this function should do as follows: if the argument is NULL, it should simply be removed. If non-NULL, it should be replaced with a call to DB_ENV->set_alloc(). + +log_archive +The historic **db_malloc** argument to the log_archive function has been replaced. Applications using this function should do as follows: if the argument is NULL, it should simply be removed. If non-NULL, it should be replaced with a call to DB_ENV->set_alloc(). + +log_stat +The historic **db_malloc** argument to the log_stat function has been replaced. Applications using this function should do as follows: if the argument is NULL, it should simply be removed. If non-NULL, it should be replaced with a call to DB_ENV->set_alloc(). + +memp_stat +The historic **db_malloc** argument to the memp_stat function has been replaced. Applications using this function should do as follows: if the argument is NULL, it should simply be removed. If non-NULL, it should be replaced with a call to DB_ENV->set_alloc(). + +txn_stat +The historic **db_malloc** argument to the txn_stat function has been replaced. Applications using this function should do as follows: if the argument is NULL, it should simply be removed. If non-NULL, it should be replaced with a call to DB_ENV->set_alloc(). + +One potential incompatibility for historic applications is that the allocation functions for a database environment must now be set before the environment is opened. Historically, Berkeley DB applications could open the environment first, and subsequently call the DB-\>set_malloc and DB-\>set_realloc methods; that use is no longer supported. diff --git a/docs-src/guides/upgrading/upgrade_3_3_bigfile.md b/docs-src/guides/upgrading/upgrade_3_3_bigfile.md new file mode 100644 index 000000000..26c0c23d8 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_bigfile.md @@ -0,0 +1,8 @@ +--- +title: "--disable-bigfile" +api-name: "--disable-bigfile" +source: docs/upgrading/upgrade_3_3_bigfile.html +--- +## --disable-bigfile + +In previous releases, Berkeley DB UNIX used the --disable-bigfile configuration option for systems that could not, for whatever reason, include large file support in a particular Berkeley DB configuration. However, large file support has been integrated into the autoconf configuration tool as of version 2.50. For that reason, Berkeley DB configuration no longer supports --disable-bigfile, the autoconf standard --disable-largefile should be used instead. diff --git a/docs-src/guides/upgrading/upgrade_3_3_conflict.md b/docs-src/guides/upgrading/upgrade_3_3_conflict.md new file mode 100644 index 000000000..0d0a57037 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_conflict.md @@ -0,0 +1,8 @@ +--- +title: "DB_LOCK_CONFLICT" +api-name: "DB_LOCK_CONFLICT" +source: docs/upgrading/upgrade_3_3_conflict.html +--- +## DB_LOCK_CONFLICT + +The DB_LOCK_CONFLICT flag has been removed from the lock_detect function. Applications specifying the DB_LOCK_CONFLICT flag should simply replace it with a flags argument of 0. diff --git a/docs-src/guides/upgrading/upgrade_3_3_disk.md b/docs-src/guides/upgrading/upgrade_3_3_disk.md new file mode 100644 index 000000000..ff4baf412 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_3_3_disk.html +--- +## Upgrade Requirements + +No database formats or log file formats changed in the Berkeley DB 3.3 release. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_3_3_getswap.md b/docs-src/guides/upgrading/upgrade_3_3_getswap.md new file mode 100644 index 000000000..97e315946 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_getswap.md @@ -0,0 +1,8 @@ +--- +title: "DB->get_byteswapped" +api-name: "DB->get_byteswapped" +source: docs/upgrading/upgrade_3_3_getswap.html +--- +## DB-\>get_byteswapped + +The DB->get_byteswapped() method method can return an error in the Berkeley DB 3.3 release, and so requires an interface change. C and C++ applications calling DB->get_byteswapped() should be changed to treat the method's return as an error code, and to pass an additional second argument of type **int \*** to the method. The additional argument is used as a memory location in which to store the requested information. diff --git a/docs-src/guides/upgrading/upgrade_3_3_gettype.md b/docs-src/guides/upgrading/upgrade_3_3_gettype.md new file mode 100644 index 000000000..8fb7be160 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_gettype.md @@ -0,0 +1,8 @@ +--- +title: "DB->get_type" +api-name: "DB->get_type" +source: docs/upgrading/upgrade_3_3_gettype.html +--- +## DB-\>get_type + +The DB->get_type() method method can return an error in the Berkeley DB 3.3 release, and so requires an interface change. C and C++ applications calling DB->get_type() should be changed to treat the method's return as an error code, and to pass an additional second argument of type **DBTYPE \*** to the method. The additional argument is used as a memory location in which to store the requested information. diff --git a/docs-src/guides/upgrading/upgrade_3_3_memp_fget.md b/docs-src/guides/upgrading/upgrade_3_3_memp_fget.md new file mode 100644 index 000000000..8fcdadabd --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_memp_fget.md @@ -0,0 +1,10 @@ +--- +title: "memp_fget, EIO" +api-name: "memp_fget, EIO" +source: docs/upgrading/upgrade_3_3_memp_fget.html +--- +## memp_fget, EIO + +Previous releases of Berkeley DB returned the system error EIO when the memp_fget function was called to retrieve a page, the page did not exist, and the `DB_MPOOL_CREATE` flag was not set. In the 3.3 release, the error `DB_PAGE_NOTFOUND` is returned instead, to allow applications to distinguish between recoverable and non-recoverable errors. Applications calling the memp_fget function and checking for a return of EIO should check for `DB_PAGE_NOTFOUND` instead. + +Previous releases of Berkeley DB treated filesystem I/O failure (the most common of which the filesystem running out of space), as a fatal error, returning DB_RUNRECOVERY. When a filesystem failure happens in the 3.3 release Berkeley DB returns the underlying system error (usually EIO), but can continue to run. Applications should abort any enclosing transaction when a recoverable system error occurs in order to recover from the error. diff --git a/docs-src/guides/upgrading/upgrade_3_3_rpc.md b/docs-src/guides/upgrading/upgrade_3_3_rpc.md new file mode 100644 index 000000000..35db6ca41 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_rpc.md @@ -0,0 +1,8 @@ +--- +title: "DB_ENV->set_server" +api-name: "DB_ENV->set_server" +source: docs/upgrading/upgrade_3_3_rpc.html +--- +## DB_ENV-\>set_server + +The DB_ENV-\>set_server method has been deprecated and replaced with the `DB_ENV->set_rpc_server()` method. The DB_ENV-\>set_server method will be removed in a future release, and so applications using it should convert. The DB_ENV-\>set_server method can be easily converted to the `DB_ENV->set_rpc_server()` method by changing the name, and specifying a NULL for the added argument, second in the argument list. diff --git a/docs-src/guides/upgrading/upgrade_3_3_shared.md b/docs-src/guides/upgrading/upgrade_3_3_shared.md new file mode 100644 index 000000000..8ad54cc33 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_shared.md @@ -0,0 +1,8 @@ +--- +title: "--enable-dynamic, --enable-shared" +api-name: "--enable-dynamic, --enable-shared" +source: docs/upgrading/upgrade_3_3_shared.html +--- +## --enable-dynamic, --enable-shared + +In previous releases, Berkeley DB required separate configuration and builds to create both static and shared libraries. This has changed in the 3.3 release, and Berkeley DB now builds and installs both shared and static versions of the Berkeley DB libraries by default. This change was based on Berkeley DB upgrading to release 1.4 of the GNU Project's Libtool distribution. For this reason, Berkeley DB no longer supports the previous --enable-dynamic and --enable-shared configuration options. Instead, as Berkeley DB now builds both static and shared libraries by default, the useful options are Libtool's --disable-shared and --disable-static options. diff --git a/docs-src/guides/upgrading/upgrade_3_3_toc.md b/docs-src/guides/upgrading/upgrade_3_3_toc.md new file mode 100644 index 000000000..ea87cedef --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_toc.md @@ -0,0 +1,34 @@ +--- +title: "Chapter 11. Upgrading Berkeley DB 3.2 applications to Berkeley DB 3.3" +api-name: "Chapter 11. Upgrading Berkeley DB 3.2 applications to Berkeley DB 3.3" +source: docs/upgrading/upgrade_3_3_toc.html +--- +## Chapter 11. Upgrading Berkeley DB 3.2 applications to Berkeley DB 3.3 + +**Table of Contents** + + [introduction](upgrade_3_3_toc.md#upgrade_3_3_intro) + + [DB_ENV-\>set_server](upgrade_3_3_rpc.md) + + [DB-\>get_type](upgrade_3_3_gettype.md) + + [DB-\>get_byteswapped](upgrade_3_3_getswap.md) + + [DB-\>set_malloc, DB-\>set_realloc](upgrade_3_3_alloc.md) + + [DB_LOCK_CONFLICT](upgrade_3_3_conflict.md) + + [memp_fget, EIO](upgrade_3_3_memp_fget.md) + + [txn_prepare](upgrade_3_3_txn_prepare.md) + + [--enable-dynamic, --enable-shared](upgrade_3_3_shared.md) + + [--disable-bigfile](upgrade_3_3_bigfile.md) + + [Upgrade Requirements](upgrade_3_3_disk.md) + +## introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 3.2 release interfaces to the Berkeley DB 3.3 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_3_3_txn_prepare.md b/docs-src/guides/upgrading/upgrade_3_3_txn_prepare.md new file mode 100644 index 000000000..bc48c8b09 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_3_3_txn_prepare.md @@ -0,0 +1,8 @@ +--- +title: "txn_prepare" +api-name: "txn_prepare" +source: docs/upgrading/upgrade_3_3_txn_prepare.html +--- +## txn_prepare + +An additional argument has been added to the txn_prepare function. If your application calls txn_prepare (that is, is performing two-phase commit using Berkeley DB as a local resource manager), see the section titled *Distributed Transactions* in versions of this book that existed prior to release 4.8. diff --git a/docs-src/guides/upgrading/upgrade_4_0_asr.md b/docs-src/guides/upgrading/upgrade_4_0_asr.md new file mode 100644 index 000000000..d096bbaf9 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_asr.md @@ -0,0 +1,16 @@ +--- +title: "application-specific recovery" +api-name: "application-specific recovery" +source: docs/upgrading/upgrade_4_0_asr.html +--- +## application-specific recovery + +If you have created your own logging and recovery routines, you may need to upgrade them to the Berkeley DB 4.0 release. + +First, you should regenerate your logging, print, read and the other automatically generated routines, using the dist/gen_rec.awk tool included in the Berkeley DB distribution. + +Next, compare the template file code generated by the gen_rec.awk tool against the code generated by the last release in which you built a template file. Any changes in the templates should be incorporated into the recovery routines you have written. + +Third, if your recovery functions refer to DB_TXN_FORWARD_ROLL (that is, your code checks for that particular operation code), you should replace it with DB_REDO(op) which compares the operation code to both DB_TXN_FORWARD_ROLL and DB_TXN_APPLY. (DB_TXN_APPLY is a potential value for the operation code as of the 4.0 release.) + +Finally, if you have created your own logging and recovery routines, we recommend you contact us and ask us to review those routines for you. diff --git a/docs-src/guides/upgrading/upgrade_4_0_cxx.md b/docs-src/guides/upgrading/upgrade_4_0_cxx.md new file mode 100644 index 000000000..31443e05f --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_cxx.md @@ -0,0 +1,38 @@ +--- +title: "C++ ostream objects" +api-name: "C++ ostream objects" +source: docs/upgrading/upgrade_4_0_cxx.html +--- +## C++ ostream objects + +In the 4.0 release, the Berkeley DB C++ API has been changed to use the ISO standard C++ API in preference to the older, less portable interfaces, where available. This means the Berkeley DB methods that used to take an ostream object as a parameter now expect a std::ostream. Specifically, the following methods have changed: + +``` c +DbEnv::set_error_stream +Db::set_error_stream +Db::verify +``` + +On many platforms, the old and the new C++ styles are interchangeable; on some platforms (notably Windows systems), they are incompatible. If your code uses these methods and you have trouble with the 4.0 release, you should update code that looks like this: + +``` c +#include +#include + +void foo(Db db) { + db.set_error_stream(&cerr); +} +``` + +to look like this: + +``` c +#include +#include + +using std::cerr; + +void foo(Db db) { + db.set_error_stream(&cerr); +} +``` diff --git a/docs-src/guides/upgrading/upgrade_4_0_deadlock.md b/docs-src/guides/upgrading/upgrade_4_0_deadlock.md new file mode 100644 index 000000000..e9869f233 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_deadlock.md @@ -0,0 +1,8 @@ +--- +title: "db_deadlock" +api-name: "db_deadlock" +source: docs/upgrading/upgrade_4_0_deadlock.html +--- +## db_deadlock + +The **-w** option to the db_deadlock utility has been deprecated. Applications can get the functionality of the **-w** option by using the **-t** option with an argument of **.100000**. diff --git a/docs-src/guides/upgrading/upgrade_4_0_disk.md b/docs-src/guides/upgrading/upgrade_4_0_disk.md new file mode 100644 index 000000000..16ddfdc4c --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_0_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.0 release. No database formats changed in the Berkeley DB 4.0 release. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_0_env.md b/docs-src/guides/upgrading/upgrade_4_0_env.md new file mode 100644 index 000000000..62c908229 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_env.md @@ -0,0 +1,18 @@ +--- +title: "db_env_set_XXX" +api-name: "db_env_set_XXX" +source: docs/upgrading/upgrade_4_0_env.html +--- +## db_env_set_XXX + +The db_env_set_region_init function was removed in the 4.0 release and replaced with the `DB_REGION_INIT` flag to the DB_ENV->set_flags() method. This is an interface change: historically, the db_env_set_region_init function operated on the entire Berkeley DB library, not a single environment. The new method only operates on a single DB_ENV class handle (and any handles created in the scope of that handle). Applications calling the db_env_set_region_init function should update their calls: calls to the historic routine with an argument of 1 (0) are equivalent to calling DB_ENV->set_flags() with the `DB_REGION_INIT` flag and an argument of 1 (0). + +The db_env_set_tas_spins function was removed in the 4.0 release and replaced with the DB_ENV-\>set_tas_spins method. This is an interface change: historically, the db_env_set_tas_spins function operated on the entire Berkeley DB library, not a single environment. The new method only operates on a single DB_ENV class handle (and any handles created in the scope of that handle). Applications calling the db_env_set_tas_spins function should update their calls: calls to the historic routine are equivalent to calling DB_ENV-\>set_tas_spins with the same argument. In addition, for consistent behavior, all DB_ENV class handles opened by the application should make the same configuration call, or the value will need to be entered into the environment's DB_CONFIG file. + +Also, three of the standard Berkeley DB debugging interfaces changed in the 4.0 release. It is quite unlikely that Berkeley DB applications use these interfaces. + +The DB_ENV-\>set_mutexlocks method was removed in the 4.0 release and replaced with the `DB_NO_LOCKING` flag to the DB_ENV->set_flags() method. Applications calling the DB_ENV-\>set_mutexlocks method should update their calls: calls to the historic routine with an argument of 1 (0) are equivalent to calling `DB_NO_LOCKING` flag and an argument of 1 (0). + +The db_env_set_pageyield function was removed in the 4.0 release and replaced with the `DB_YIELDCPU` flag to the DB_ENV->set_flags() method. This is an interface change: historically, the db_env_set_pageyield function operated on the entire Berkeley DB library, not a single environment. The new method only operates on a single DB_ENV class handle (and any handles created in the scope of that handle). Applications calling the db_env_set_pageyield function should update their calls: calls to the historic routine with an argument of 1 (0) are equivalent to calling DB_ENV->set_flags() with the `DB_YIELDCPU` flag and an argument of 1 (0). In addition, all DB_ENV class handles opened by the application will need to make the same call, or the `DB_YIELDCPU` flag will need to be entered into the environment's DB_CONFIG file. + +The db_env_set_panicstate function was removed in the 4.0 release, replaced with the `DB_PANIC_ENVIRONMENT` flags to the DB_ENV->set_flags() method. (The `DB_PANIC_ENVIRONMENT` flag will cause an environment to panic, affecting all threads of control using that environment. The DB_ENV->set_flags() handle to ignore the current panic state of the environment.) This is an interface change: historically the db_env_set_panicstate function operated on the entire Berkeley DB library, not a single environment. Applications calling the db_env_set_panicstate function should update their calls, replacing the historic call with a call to DB_ENV->set_flags() and the appropriate flag, depending on their usage of the historic interface. diff --git a/docs-src/guides/upgrading/upgrade_4_0_java.md b/docs-src/guides/upgrading/upgrade_4_0_java.md new file mode 100644 index 000000000..d96ce705d --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_java.md @@ -0,0 +1,22 @@ +--- +title: "Java CLASSPATH environment variable" +api-name: "Java CLASSPATH environment variable" +source: docs/upgrading/upgrade_4_0_java.html +--- +## Java CLASSPATH environment variable + +The Berkeley DB Java class files are now packaged as jar files. In the 4.0 release, the `CLASSPATH` environment variable must change to include at least the `db.jar` file. It can optionally include the `dbexamples.jar` file if you want to run the examples. For example, on UNIX: + +``` c +export CLASSPATH="/usr/local/BerkeleyDB.4.8/lib/db.jar: \ +/usr/local/BerkeleyDB.4.8/lib/dbexamples.jar" +``` + +For example, on Windows: + +``` c +set CLASSPATH="D:\db\build_windows\Release\db.jar; +D:\db\build_windows\Release\dbexamples.jar" +``` + +For more information on Java configuration, see the Berkeley DB Installation and Build Guide. . diff --git a/docs-src/guides/upgrading/upgrade_4_0_lock.md b/docs-src/guides/upgrading/upgrade_4_0_lock.md new file mode 100644 index 000000000..eb52f8d73 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_lock.md @@ -0,0 +1,23 @@ +--- +title: "lock_XXX" +api-name: "lock_XXX" +source: docs/upgrading/upgrade_4_0_lock.html +--- +## lock_XXX + +The C API for the Berkeley DB Locking subsystem was reworked in the 4.0 release as follows: + +| Historic functional interface | Berkeley DB 4.X method | +|----|----| +| lock_detect | DB_ENV->lock_detect() | +| lock_get | DB_ENV->lock_get() | +| lock_id | DB_ENV->lock_id() | +| lock_put | DB_ENV->lock_put() | +| lock_stat | DB_ENV->lock_stat() | +| lock_vec | DB_ENV->lock_vec() | + +Applications calling any of these functions should update their calls to use the enclosing DB_ENV handle's method (easily done as the first argument to the existing call is the correct handle to use). + +In addition, the DB_ENV->lock_stat() call has been changed in the 4.0 release to take a flags argument. To leave their historic behavior unchanged, applications should add a final argument of 0 to any calls made to DB_ENV->lock_stat(). + +The C++ and Java APIs for the DbLock::put (DbLock.put) method was reworked in the 4.0 release to make the lock put interface a method of the DB_ENV handle rather than the DbLock handle. Applications calling the DbLock::put or DbLock.put method should update their calls to use the enclosing DB_ENV handle's method (easily done as the first argument to the existing call is the correct handle to use). diff --git a/docs-src/guides/upgrading/upgrade_4_0_lock_id_free.md b/docs-src/guides/upgrading/upgrade_4_0_lock_id_free.md new file mode 100644 index 000000000..64fa96f9d --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_lock_id_free.md @@ -0,0 +1,8 @@ +--- +title: "DB_ENV->lock_id_free" +api-name: "DB_ENV->lock_id_free" +source: docs/upgrading/upgrade_4_0_lock_id_free.html +--- +## DB_ENV-\>lock_id_free + +A new locker ID related API, the DB_ENV->lock_id_free() method, was added to Berkeley DB 4.0 release. Applications using the DB_ENV->lock_id() method to allocate locker IDs may want to update their applications to free the locker ID when it is no longer needed. diff --git a/docs-src/guides/upgrading/upgrade_4_0_log.md b/docs-src/guides/upgrading/upgrade_4_0_log.md new file mode 100644 index 000000000..ae2f35e7a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_log.md @@ -0,0 +1,25 @@ +--- +title: "log_XXX" +api-name: "log_XXX" +source: docs/upgrading/upgrade_4_0_log.html +--- +## log_XXX + +The C API for the Berkeley DB Logging subsystem was reworked in the 4.0 release as follows: + +| Historic functional interface | Berkeley DB 4.X method | +|----|----| +| log_archive | DB_ENV->log_archive() | +| log_file | DB_ENV->log_file() | +| log_flush | DB_ENV->log_flush() | +| log_get | DB_ENV->log_cursor() | +| log_put | DB_ENV->log_put() | +| log_register | DB_ENV-\>log_register | +| log_stat | DB_ENV->log_stat() | +| log_unregister | DB_ENV-\>log_unregister | + +Applications calling any of these functions should update their calls to use the enclosing DB_ENV class handle's method (in all cases other than the log_get call, this is easily done as the first argument to the existing call is the correct handle to use). + +Application calls to the historic log_get function must be replaced with the creation of a log file cursor (a DB_LOGC class object), using the DB_ENV->log_cursor() method to retrieve log records and calls to the DB_LOGC->close() method to destroy the cursor. It may also be possible to simplify some applications. In previous releases of Berkeley DB, the DB_CURRENT, DB_NEXT, and DB_PREV flags to the log_get function could not be used by a free-threaded DB_ENV class handle. If their DB_ENV class handle was free-threaded, applications had to create an additional, unique environment handle by separately calling DB_ENV->open(). This is no longer an issue in the log cursor interface, and applications may be able to remove the now unnecessary creation of the additional DB_ENV class object. + +Finally, the DB_ENV->log_stat() call has been changed in the 4.0 release to take a flags argument. To leave their historic behavior unchanged, applications should add a final argument of 0 to any calls made to DB_ENV->log_stat(). diff --git a/docs-src/guides/upgrading/upgrade_4_0_mp.md b/docs-src/guides/upgrading/upgrade_4_0_mp.md new file mode 100644 index 000000000..836e25372 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_mp.md @@ -0,0 +1,34 @@ +--- +title: "memp_XXX" +api-name: "memp_XXX" +source: docs/upgrading/upgrade_4_0_mp.html +--- +## memp_XXX + +The C API for the Berkeley DB Memory Pool subsystem was reworked in the 4.0 release as follows: + +| Historic functional interface | Berkeley DB 4.X method | +|----|----| +| memp_register | DB_ENV->memp_register() | +| memp_stat | DB_ENV->memp_stat() | +| memp_sync | DB_ENV->memp_sync() | +| memp_trickle | DB_ENV->memp_trickle() | +| memp_fopen | DB_ENV->memp_fcreate() | +| DB_MPOOL_FINFO: ftype | DB_MPOOLFILE->set_ftype() | +| DB_MPOOL_FINFO: pgcookie | DB_MPOOLFILE->set_pgcookie() | +| DB_MPOOL_FINFO: fileid | DB_MPOOLFILE->set_fileid() | +| DB_MPOOL_FINFO: lsn_offset | DB_MPOOLFILE->set_lsn_offset() | +| DB_MPOOL_FINFO: clear_len | DB_MPOOLFILE->set_clear_len() | +| memp_fopen | DB_MPOOLFILE->open() | +| memp_fclose | DB_MPOOLFILE->close() | +| memp_fput | DB_MPOOLFILE->put() | +| memp_fset | DB_MPOOLFILE-\>set | +| memp_fsync | DB_MPOOLFILE->sync() | + +Applications calling any of the memp_register, memp_stat, memp_sync or memp_trickle functions should update those calls to use the enclosing DB_ENV class handle's method (easily done as the first argument to the existing call is the correct DB_ENV class handle). + +In addition, the DB_ENV->memp_stat() call has been changed in the 4.0 release to take a flags argument. To leave their historic behavior unchanged, applications should add a final argument of 0 to any calls made to DB_ENV->memp_stat(). + +Applications calling the memp_fopen function should update those calls as follows: First, acquire a Cache chapter handle using the DB_ENV->memp_fcreate() method. Second, if the DB_MPOOL_FINFO structure reference passed to the memp_fopen function was non-NULL, call the Cache chapter method corresponding to each initialized field in the DB_MPOOL_FINFO structure. Third, call the DB_MPOOLFILE->open() method method to open the underlying file. If the DB_MPOOLFILE->open() method call fails, then DB_MPOOLFILE->close() method must be called to destroy the allocated handle. + +Applications calling the memp_fopen, memp_fclose, memp_fput, memp_fset, or memp_fsync functions should update those calls to use the enclosing Cache chapter handle's method. Again, this is easily done as the first argument to the existing call is the correct Cache chapter handle. With one exception, the calling conventions of the old and new interfaces are identical; the one exception is the DB_MPOOLFILE->close() method, which requires an additional flag parameter that should be set to 0. diff --git a/docs-src/guides/upgrading/upgrade_4_0_rpc.md b/docs-src/guides/upgrading/upgrade_4_0_rpc.md new file mode 100644 index 000000000..add30ecc4 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_rpc.md @@ -0,0 +1,8 @@ +--- +title: "DB_ENV->set_server" +api-name: "DB_ENV->set_server" +source: docs/upgrading/upgrade_4_0_rpc.html +--- +## DB_ENV-\>set_server + +`The DB_ENV->set_server()` method has been replaced with the `DB_ENV->set_rpc_server()` method. The `DB_ENV->set_server()` method can be easily converted to the `DB_ENV->set_rpc_server()` method by changing the name, and specifying a NULL for the added argument, second in the argument list. diff --git a/docs-src/guides/upgrading/upgrade_4_0_set_lk_max.md b/docs-src/guides/upgrading/upgrade_4_0_set_lk_max.md new file mode 100644 index 000000000..423888ccb --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_set_lk_max.md @@ -0,0 +1,8 @@ +--- +title: "DB_ENV->set_lk_max" +api-name: "DB_ENV->set_lk_max" +source: docs/upgrading/upgrade_4_0_set_lk_max.html +--- +## DB_ENV-\>set_lk_max + +The DB_ENV-\>set_lk_max method has been deprecated in favor of the DB_ENV->set_lk_max_locks(), DB_ENV->set_lk_max_lockers(), and DB_ENV->set_lk_max_objects() methods. The DB_ENV-\>set_lk_max method continues to be available, but is no longer documented and is expected to be removed in a future release. diff --git a/docs-src/guides/upgrading/upgrade_4_0_toc.md b/docs-src/guides/upgrading/upgrade_4_0_toc.md new file mode 100644 index 000000000..68398869b --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_toc.md @@ -0,0 +1,80 @@ +--- +title: "Chapter 10. Upgrading Berkeley DB 3.3 applications to Berkeley DB 4.0" +api-name: "Chapter 10. Upgrading Berkeley DB 3.3 applications to Berkeley DB 4.0" +source: docs/upgrading/upgrade_4_0_toc.html +--- +## Chapter 10. Upgrading Berkeley DB 3.3 applications to Berkeley DB 4.0 + +**Table of Contents** + + [Introduction](upgrade_4_0_toc.md#upgrade_4_0_intro) + + [db_deadlock](upgrade_4_0_deadlock.md) + + [lock_XXX](upgrade_4_0_lock.md) + + [log_XXX](upgrade_4_0_log.md) + + [memp_XXX](upgrade_4_0_mp.md) + + [txn_XXX](upgrade_4_0_txn.md) + + [db_env_set_XXX](upgrade_4_0_env.md) + + [DB_ENV-\>set_server](upgrade_4_0_rpc.md) + + [DB_ENV-\>set_lk_max](upgrade_4_0_set_lk_max.md) + + [DB_ENV-\>lock_id_free](upgrade_4_0_lock_id_free.md) + + [Java CLASSPATH environment variable](upgrade_4_0_java.md) + + [C++ ostream objects](upgrade_4_0_cxx.md) + + [application-specific recovery](upgrade_4_0_asr.md) + + [Upgrade Requirements](upgrade_4_0_disk.md) + + [4.0.14 Change Log](changelog_4_0_14.md) + + [Major New Features:](changelog_4_0_14.md#idp51113768) + + [General Environment Changes:](changelog_4_0_14.md#idp51101344) + + [General Access Method Changes:](changelog_4_0_14.md#idp51103296) + + [Btree Access Method Changes:](changelog_4_0_14.md#idp51105152) + + [Hash Access Method Changes:](changelog_4_0_14.md#idp51109416) + + [Queue Access Method Changes:](changelog_4_0_14.md#idp51112664) + + [Recno Access Method Changes:](changelog_4_0_14.md#idp51113832) + + [C++ API Changes:](changelog_4_0_14.md#idp51115760) + + [Java API Changes:](changelog_4_0_14.md#idp51126328) + + [Tcl API Changes:](changelog_4_0_14.md#idp51116840) + + [RPC Client/Server Changes:](changelog_4_0_14.md#idp51117920) + + [XA Resource Manager Changes:](changelog_4_0_14.md#idp51118608) + + [Locking Subsystem Changes:](changelog_4_0_14.md#idp51118928) + + [Logging Subsystem Changes:](changelog_4_0_14.md#idp51103680) + + [Memory Pool Subsystem Changes:](changelog_4_0_14.md#idp51122816) + + [Transaction Subsystem Changes:](changelog_4_0_14.md#idp51109800) + + [Utility Changes:](changelog_4_0_14.md#idp51113048) + + [Database or Log File On-Disk Format Changes:](changelog_4_0_14.md#idp51125248) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_0_14.md#idp51126712) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 3.3 release interfaces to the Berkeley DB 4.0 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_0_txn.md b/docs-src/guides/upgrading/upgrade_4_0_txn.md new file mode 100644 index 000000000..a357c2125 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_0_txn.md @@ -0,0 +1,26 @@ +--- +title: "txn_XXX" +api-name: "txn_XXX" +source: docs/upgrading/upgrade_4_0_txn.html +--- +## txn_XXX + +The C API for the Berkeley DB Transaction subsystem was reworked in the 4.0 release as follows: + +| Historic functional interface | Berkeley DB 4.X method | +|----|----| +| txn_abort | DB_TXN->abort() | +| txn_begin | DB_ENV->txn_begin() | +| txn_checkpoint | DB_ENV->txn_checkpoint() | +| txn_commit | DB_TXN->commit() | +| txn_discard | DB_TXN->discard() | +| txn_id | DB_TXN->id() | +| txn_prepare | DB_TXN->prepare() | +| txn_recover | DB_TXN->recover() | +| txn_stat | DB_TXN->stat() | + +Applications calling any of these functions should update their calls to use the enclosing DB_ENV class handle's method (easily done as the first argument to the existing call is the correct handle to use). + +As a special case, since applications might potentially have many calls to the txn_abort, txn_begin and txn_commit functions, those functions continue to work unchanged in the Berkeley DB 4.0 release. + +In addition, the DB_TXN->stat() call has been changed in the 4.0 release to take a flags argument. To leave their historic behavior unchanged, applications should add a final argument of 0 to any calls made to DB_TXN->stat(). diff --git a/docs-src/guides/upgrading/upgrade_4_1_app_dispatch.md b/docs-src/guides/upgrading/upgrade_4_1_app_dispatch.md new file mode 100644 index 000000000..ac45d2a09 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_app_dispatch.md @@ -0,0 +1,8 @@ +--- +title: "Application-specific logging and recovery" +api-name: "Application-specific logging and recovery" +source: docs/upgrading/upgrade_4_1_app_dispatch.html +--- +## Application-specific logging and recovery + +The application-specific logging and recovery tools and interfaces have been reworked in the 4.1 release to make it simpler for applications to use Berkeley DB to support their own logging and recovery of non-Berkeley DB objects. Specifically, the DB_ENV-\>set_recovery_init and DB_ENV-\>set_tx_recover interfaces have been removed, replaced by DB_ENV->set_app_dispatch(). Applications using either of the removed interfaces should be updated to call DB_ENV->set_app_dispatch(). For more information see Introduction to application specific logging and recovery and the DB_ENV->set_app_dispatch() documentation. diff --git a/docs-src/guides/upgrading/upgrade_4_1_checkpoint.md b/docs-src/guides/upgrading/upgrade_4_1_checkpoint.md new file mode 100644 index 000000000..0b11b8721 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_checkpoint.md @@ -0,0 +1,10 @@ +--- +title: "DB_CHECKPOINT, DB_CURLSN" +api-name: "DB_CHECKPOINT, DB_CURLSN" +source: docs/upgrading/upgrade_4_1_checkpoint.html +--- +## DB_CHECKPOINT, DB_CURLSN + +The DB_CHECKPOINT flag has been removed from the DB_LOGC->get() and DB_ENV->log_put() methods. It is very unlikely application programs used this flag. If your application used this flag, please contact us for help in upgrading. + +The DB_CURLSN flag has been removed from the DB_ENV->log_put() method. It is very unlikely application programs used this flag. If your application used this flag, please contact us for help in upgrading. diff --git a/docs-src/guides/upgrading/upgrade_4_1_cxx.md b/docs-src/guides/upgrading/upgrade_4_1_cxx.md new file mode 100644 index 000000000..2d05f99b6 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_cxx.md @@ -0,0 +1,26 @@ +--- +title: "C++ exceptions" +api-name: "C++ exceptions" +source: docs/upgrading/upgrade_4_1_cxx.html +--- +## C++ exceptions + +With default flags, the C++ DbEnv and Db classes can throw exceptions from their constructors. For example, this can happen if invalid parameters are passed in or the underlying C structures could not be created. If the objects are created in an environment that is not configured for exceptions (that is, the DB_CXX_NO_EXCEPTIONS flag is specified), errors from the constructor will be returned when the handle's open method is called. + +In addition, the behavior of the DbEnv and Db destructors has changed to simplify exception handling in applications. The destructors will now close the handle if the handle's close method was not called prior to the object being destroyed. The return value of the call is discarded, and no exceptions will be thrown. Applications should call the close method in normal situations so any errors while closing can be handled by the application. + +This change allows applications to be structured as follows: + +``` c +try { + DbEnv env(0); + env.open(/* ... */); + Db db(&env, 0); + db.open(/* ... */); + /* ... */ + db.close(0); + env.close(0); +} catch (DbException &dbe) { + // Handle the exception, the handles have already been closed. +} +``` diff --git a/docs-src/guides/upgrading/upgrade_4_1_disk.md b/docs-src/guides/upgrading/upgrade_4_1_disk.md new file mode 100644 index 000000000..7f66a9304 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_disk.md @@ -0,0 +1,12 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_1_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.1 release. + +All of the access method database formats changed in the Berkeley DB 4.1 release (Btree/Recno: version 8 to version 9, Hash: version 7 to version 8, and Queue: version 3 to version 4). **The format changes are entirely backward-compatible, and no database upgrades are needed.** Note that databases created using the 4.1 release may not be usable with earlier Berkeley DB releases. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_1_excl.md b/docs-src/guides/upgrading/upgrade_4_1_excl.md new file mode 100644 index 000000000..ae70bbccd --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_excl.md @@ -0,0 +1,8 @@ +--- +title: "DB_EXCL" +api-name: "DB_EXCL" +source: docs/upgrading/upgrade_4_1_excl.html +--- +## DB_EXCL + +The DB_EXCL flag to the DB->open() method now works for subdatabases as well as physical files, and it is now possible to use the DB_EXCL flag to check for the previous existence of subdatabases. diff --git a/docs-src/guides/upgrading/upgrade_4_1_fop.md b/docs-src/guides/upgrading/upgrade_4_1_fop.md new file mode 100644 index 000000000..fa5f35e84 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_fop.md @@ -0,0 +1,105 @@ +--- +title: "DB->associate, DB->open, DB->remove, DB->rename" +api-name: "DB->associate, DB->open, DB->remove, DB->rename" +source: docs/upgrading/upgrade_4_1_fop.html +--- +## DB-\>associate, DB-\>open, DB-\>remove, DB-\>rename + +Historic releases of Berkeley DB transaction-protected the DB->open(), DB->remove(), and DB->rename() methods, but did it in an implicit way, that is, applications did not specify the TXN handles associated with the operations. This approach had a number of problems, the most significant of which was there was no way to group operations that included database creation, removal or rename. For example, applications wanting to maintain a list of the databases in an environment in a well-known database had no way to update the well-known database and create a database within a single transaction, and so there was no way to guarantee the list of databases was correct for the environment after system or application failure. Another example might be the creation of both a primary database and a database intended to serve as a secondary index, where again there was no way to group the creation of both databases in a single atomic operation. + +In the 4.1 release of Berkeley DB, this is no longer the case. The DB->open() and DB->associate() methods now take a TXN handle returned by DB_ENV->txn_begin() as an optional argument. New DB_ENV->dbremove() and DB_ENV->dbrename() methods taking a TXN handle as an optional argument have been added. + +To upgrade, applications must add a TXN parameter in the appropriate location for the DB->open() method calls, and the DB->associate() method calls (in both cases, the second argument for the C API, the first for the C++ or Java APIs). + +Applications wanting to transaction-protect their DB->open() and DB->associate() method calls can add a NULL TXN argument and specify the DB_AUTO_COMMIT flag to the two calls, which wraps the operation in an internal Berkeley DB transaction. Applications wanting to transaction-protect the remove and rename operations must rewrite their calls to the DB->remove() and DB->rename() methods to be, instead, calls to the new DB_ENV->dbremove() and DB_ENV->dbrename() methods. Applications not wanting to transaction-protect any of the operations can add a NULL argument to their DB->open() and DB->associate() method calls and require no further changes. + +For example, an application currently opening and closing a database as follows: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = db_create(&dbp, dbenv, 0)) != 0) + goto err_handler; + +if ((ret = dbp->open(dbp, "file", NULL, DB_BTREE, + DB_CREATE, 0664)) != 0) { + (void)dbp->close(dbp); + goto err_handler; +} +``` + +could transaction-protect the DB->open() call as follows: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = db_create(&dbp, dbenv, 0)) != 0) + goto err_handler; + +if ((ret = dbp->open(dbp, + NULL, "file", NULL, DB_BTREE, DB_CREATE | + DB_AUTO_COMMIT, 0664)) != 0) { + (void)dbp->close(dbp); + goto err_handler; +} +``` + +An application currently removing a database as follows: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = db_create(&dbp, dbenv, 0)) != 0) + goto err_handler; + +if ((ret = dbp->remove(dbp, "file", NULL, 0)) != 0) + goto err_handler; +``` + +could transaction-protect the database removal as follows: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = + dbenv->dbremove(dbenv, NULL, "file", NULL, DB_AUTO_COMMIT)) != 0) + goto err_handler; +``` + +An application currently renaming a database as follows: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = db_create(&dbp, dbenv, 0)) != 0) + goto err_handler; + +if ((ret = dbp->rename(dbp, "file", NULL, "newname", 0)) != 0) + goto err_handler; +``` + +could transaction-protect the database renaming as follows: + +``` c +DB *dbp; +DB_ENV *dbenv; +int ret; + +if ((ret = dbenv->dbrename( + dbenv, NULL, "file", NULL, "newname", DB_AUTO_COMMIT)) != 0) + goto err_handler; +``` + +These examples are the simplest possible translation, and will result in behavior matching that of previous releases. For further discussion on how to transaction-protect DB->open() method calls, see Opening the databases. + +DB handles that will later be used for transaction-protected operations must be opened within a transaction. Specifying a transaction handle to operations using handles not opened within a transaction will return an error. Similarly, not specifying a transaction handle to operations using handles that were opened within a transaction will also return an error. diff --git a/docs-src/guides/upgrading/upgrade_4_1_hash_nelem.md b/docs-src/guides/upgrading/upgrade_4_1_hash_nelem.md new file mode 100644 index 000000000..c83f7ae0a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_hash_nelem.md @@ -0,0 +1,8 @@ +--- +title: "DB->stat.hash_nelem" +api-name: "DB->stat.hash_nelem" +source: docs/upgrading/upgrade_4_1_hash_nelem.html +--- +## DB-\>stat.hash_nelem + +The **hash_nelem** field of the DB->stat() method for Hash databases has been removed from the 4.1 release, this information is no longer available to applications. diff --git a/docs-src/guides/upgrading/upgrade_4_1_incomplete.md b/docs-src/guides/upgrading/upgrade_4_1_incomplete.md new file mode 100644 index 000000000..0153fda98 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_incomplete.md @@ -0,0 +1,10 @@ +--- +title: "DB_INCOMPLETE" +api-name: "DB_INCOMPLETE" +source: docs/upgrading/upgrade_4_1_incomplete.html +--- +## DB_INCOMPLETE + +The DB_INCOMPLETE error has been removed from the 4.1 release, and is no longer returned by the Berkeley DB library. Applications no longer need to check for this error return, as the underlying Berkeley DB interfaces that could historically fail to checkpoint or flush the cache and return this error can no longer fail for that reason. Applications should remove all uses of DB_INCOMPLETE. + +Additionally, the DbEnv.checkpoint and Db.sync methods have been changed from returning int to returning void. diff --git a/docs-src/guides/upgrading/upgrade_4_1_java.md b/docs-src/guides/upgrading/upgrade_4_1_java.md new file mode 100644 index 000000000..14433f27a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_java.md @@ -0,0 +1,8 @@ +--- +title: "Java exceptions" +api-name: "Java exceptions" +source: docs/upgrading/upgrade_4_1_java.html +--- +## Java exceptions + +The Java DbEnv constructor is now marked with "throws DbException". This means applications must construct DbEnv objects in a context where DbException throwables are handled (either in a try/catch block or in a method that propagates the exception up the stack). Note that previous versions of the Berkeley DB Java API could throw this exception from the constructor but it was not marked. diff --git a/docs-src/guides/upgrading/upgrade_4_1_log_register.md b/docs-src/guides/upgrading/upgrade_4_1_log_register.md new file mode 100644 index 000000000..3b2442059 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_log_register.md @@ -0,0 +1,8 @@ +--- +title: "DB_ENV->log_register" +api-name: "DB_ENV->log_register" +source: docs/upgrading/upgrade_4_1_log_register.html +--- +## DB_ENV-\>log_register + +The DB_ENV-\>log_register and DB_ENV-\>log_unregister interfaces were removed from the Berkeley DB 4.1 release. It is very unlikely application programs used these interfaces. If your application used these interfaces, please contact us for help in upgrading. diff --git a/docs-src/guides/upgrading/upgrade_4_1_log_stat.md b/docs-src/guides/upgrading/upgrade_4_1_log_stat.md new file mode 100644 index 000000000..bf26775a8 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_log_stat.md @@ -0,0 +1,8 @@ +--- +title: "st_flushcommit" +api-name: "st_flushcommit" +source: docs/upgrading/upgrade_4_1_log_stat.html +--- +## st_flushcommit + +The DB_ENV-\>log_stat "st_flushcommits" statistic has been removed from Berkeley DB, as it is now the same as the "st_scount" statistic. Any application using the "st_flushcommits" statistic should remove it, or replace it with the "st_count" statistic. diff --git a/docs-src/guides/upgrading/upgrade_4_1_memp_sync.md b/docs-src/guides/upgrading/upgrade_4_1_memp_sync.md new file mode 100644 index 000000000..3af41728e --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_memp_sync.md @@ -0,0 +1,12 @@ +--- +title: "DB_ENV->memp_sync" +api-name: "DB_ENV->memp_sync" +source: docs/upgrading/upgrade_4_1_memp_sync.html +--- +## DB_ENV-\>memp_sync + +Historical documentation for the DB_ENV->memp_sync() method stated: + +In addition, if DB_ENV->memp_sync() returns success, the value of **lsn** will be overwritten with the largest log sequence number from any page that was written by DB_ENV->memp_sync() to satisfy this request. + +This functionality was never correctly implemented, and has been removed in the Berkeley DB 4.1 release. It is very unlikely application programs used this information. If your application used this information, please contact us for help in upgrading. diff --git a/docs-src/guides/upgrading/upgrade_4_1_toc.md b/docs-src/guides/upgrading/upgrade_4_1_toc.md new file mode 100644 index 000000000..7a048c98a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_1_toc.md @@ -0,0 +1,82 @@ +--- +title: "Chapter 9. Upgrading Berkeley DB 4.0 applications to Berkeley DB 4.1" +api-name: "Chapter 9. Upgrading Berkeley DB 4.0 applications to Berkeley DB 4.1" +source: docs/upgrading/upgrade_4_1_toc.html +--- +## Chapter 9. Upgrading Berkeley DB 4.0 applications to Berkeley DB 4.1 + +**Table of Contents** + + [Introduction](upgrade_4_1_toc.md#upgrade_4_1_intro) + + [DB_EXCL](upgrade_4_1_excl.md) + + [DB-\>associate, DB-\>open, DB-\>remove, DB-\>rename](upgrade_4_1_fop.md) + + [DB_ENV-\>log_register](upgrade_4_1_log_register.md) + + [st_flushcommit](upgrade_4_1_log_stat.md) + + [DB_CHECKPOINT, DB_CURLSN](upgrade_4_1_checkpoint.md) + + [DB_INCOMPLETE](upgrade_4_1_incomplete.md) + + [DB_ENV-\>memp_sync](upgrade_4_1_memp_sync.md) + + [DB-\>stat.hash_nelem](upgrade_4_1_hash_nelem.md) + + [Java exceptions](upgrade_4_1_java.md) + + [C++ exceptions](upgrade_4_1_cxx.md) + + [Application-specific logging and recovery](upgrade_4_1_app_dispatch.md) + + [Upgrade Requirements](upgrade_4_1_disk.md) + + [Berkeley DB 4.1.24 and 4.1.25 Change Log](changelog_4_1_24.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_1_24.md#idp50963888) + + [Major New Features:](changelog_4_1_24.md#idp50959088) + + [General Environment Changes:](changelog_4_1_24.md#idp50962280) + + [General Access Method Changes:](changelog_4_1_24.md#idp50961984) + + [Btree Access Method Changes:](changelog_4_1_24.md#idp50964272) + + [Hash Access Method Changes:](changelog_4_1_24.md#idp50967400) + + [Queue Access Method Changes:](changelog_4_1_24.md#idp50969240) + + [Recno Access Method Changes:](changelog_4_1_24.md#idp50972088) + + [C++-specific API Changes:](changelog_4_1_24.md#idp50973928) + + [Java-specific API Changes:](changelog_4_1_24.md#idp50975768) + + [Tcl-specific API Changes:](changelog_4_1_24.md#idp50950328) + + [RPC-specific Client/Server Changes:](changelog_4_1_24.md#idp50958680) + + [Replication Changes:](changelog_4_1_24.md#idp50977144) + + [XA Resource Manager Changes:](changelog_4_1_24.md#idp50964336) + + [Locking Subsystem Changes:](changelog_4_1_24.md#idp50987264) + + [Logging Subsystem Changes:](changelog_4_1_24.md#idp50989192) + + [Memory Pool Subsystem Changes:](changelog_4_1_24.md#idp50992072) + + [Transaction Subsystem Changes:](changelog_4_1_24.md#idp50993160) + + [Utility Changes:](changelog_4_1_24.md#idp50994744) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_1_24.md#idp50997648) + + [Berkeley DB 4.1.25 Change Log](changelog_4_1_25.md) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.0 release interfaces to the Berkeley DB 4.1 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_2_cksum.md b/docs-src/guides/upgrading/upgrade_4_2_cksum.md new file mode 100644 index 000000000..ccc59fe24 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_cksum.md @@ -0,0 +1,8 @@ +--- +title: "DB_CHKSUM_SHA1" +api-name: "DB_CHKSUM_SHA1" +source: docs/upgrading/upgrade_4_2_cksum.html +--- +## DB_CHKSUM_SHA1 + +The flag to enable checksumming of Berkeley DB databases pages was renamed from DB_CHKSUM_SHA1 to DB_CHKSUM, as Berkeley DB uses an internal function to generate hash values for unencrypted database pages, not the SHA1 Secure Hash Algorithm. Berkeley DB continues to use the SHA1 Secure Hash Algorithm to generate hashes for encrypted database pages. Applications using the DB_CHKSUM_SHA1 flag should change that use to DB_CHKSUM; no other change is required. diff --git a/docs-src/guides/upgrading/upgrade_4_2_client.md b/docs-src/guides/upgrading/upgrade_4_2_client.md new file mode 100644 index 000000000..bd1f851fc --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_client.md @@ -0,0 +1,8 @@ +--- +title: "DB_CLIENT" +api-name: "DB_CLIENT" +source: docs/upgrading/upgrade_4_2_client.html +--- +## DB_CLIENT + +The flag to create a client to connect to a RPC server was renamed from DB_CLIENT to DB_RPCCLIENT, in order to avoid confusion between RPC clients and replication clients. Applications using the DB_CLIENT flag should change that use to DB_RPCCLIENT; no other change is required. diff --git a/docs-src/guides/upgrading/upgrade_4_2_del.md b/docs-src/guides/upgrading/upgrade_4_2_del.md new file mode 100644 index 000000000..1760cc5f3 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_del.md @@ -0,0 +1,10 @@ +--- +title: "DB->del" +api-name: "DB->del" +source: docs/upgrading/upgrade_4_2_del.html +--- +## DB-\>del + +In previous releases, the C++ Db::del and Java `Db.delete()` methods threw exceptions encapsulating the DB_KEYEMPTY error in some cases when called on Queue and Recno databases. Unfortunately, this was undocumented behavior. + +For consistency with the other Berkeley DB methods that handle DB_KEYEMPTY, this is no longer the case. Applications calling the Db::del and Java `Db.delete()` methods on Queue or Recno databases, and handling the DB_KEYEMPTY exception specially, should be modified to check for a return value of DB_KEYEMPTY instead. diff --git a/docs-src/guides/upgrading/upgrade_4_2_disk.md b/docs-src/guides/upgrading/upgrade_4_2_disk.md new file mode 100644 index 000000000..bfe45dc38 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_2_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.2 release. No database formats changed in the Berkeley DB 4.2 release. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_2_java.md b/docs-src/guides/upgrading/upgrade_4_2_java.md new file mode 100644 index 000000000..80300e606 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_java.md @@ -0,0 +1,56 @@ +--- +title: "Java" +api-name: "Java" +source: docs/upgrading/upgrade_4_2_java.html +--- +## Java + +There are a number of major changes to the Java support in Berkeley DB in this release. Despite that we have tried to make this a bridge release, a release where we don't require you to change anything. We've done this using the standard approach to deprecation in Java. If you do not compile with deprecation warnings on, your existing sources should work with this new release with only minor changes despite the large number of changes. Expect that in a future release we will remove all the deprecated API and only support the new API names. + +This is a list of areas where we have broken compatibility with the 4.1 release. In most cases it is a name change in an interface class. + +- **DbAppDispatch.app_dispatch(DbEnv,Dbt,DbLsn,int)** + + is now: **DbAppDispatch.appDispatch(DbEnv,Dbt,DbLsn,int)** + +- **DbAppendRecno.db_append_recno(Db,Dbt,int)** + + is now: **DbAppendRecno.dbAppendRecno(Db,Dbt,int)** + +- **DbBtreeCompare.bt_compare(Db,Dbt,Dbt)** + + is now: **DbBtreeCompare.compare(Db,Dbt,Dbt)** + +- **DbBtreeCompare.dup_compare(Db,Dbt,Dbt)** + + is now: **DbBtreeCompare.compareDuplicates(Db,Dbt,Dbt)** + +- **DbBtreePrefix.bt_prefix(Db,Dbt,Dbt)** + + is now: **DbBtreePrefix.prefix(Db,Dbt,Dbt)** + +- **DbSecondaryKeyCreate.secondary_key_create(Db,Dbt,Dbt,Dbt)** + + is now: **DbSecondaryKeyCreate.secondaryKeyCreate(Db,Dbt,Dbt,Dbt)** + +The 4.2 release of Berkeley DB requires at minimum a J2SE 1.3.1 certified Java virtual machine and associated classes to properly build and execute. To determine what version virtual machine you are running, enter: + +``` c +java -version +``` + +at a command line and look for the version number. If you need to deploy to a version 1.1 or 1.0 Java environment, it may be possible to do so by not including the classes in the com.sleepycat.bdb package in the Java build process (however, that workaround has not been tested by us). + +A few inconsistent methods have been cleaned up (for example, Db.close now returns void; previously, it returned an int which was always zero). The synchronized attributed has been toggled on some methods -- this is an attempt to prevent multithreaded applications from calling close or similar methods concurrently from multiple threads. + +The Berkeley DB API has up until now been consistent across all language APIs. Although consistency has is benefits, it made our Java API look strange to Java programmers. Many methods have been renamed in this release of the Java API to conform with Java naming conventions. Sometimes this renaming was simply "camel casing", sometimes it required rewording. The mapping file for these name changes is in `dist/camel.pm`. The Perl script we use to convert code to the new names is called `dist/camelize.pl`, and may help with updating Java applications written for earlier versions of Berkeley DB. + +Berkeley DB has a number of places where as a C library it uses function pointers to move into custom code for the purpose of notification of some event. In Java the best parallel is the registration of some class which implements an interface. In this version of Berkeley DB we have made an effort to make those interfaces more uniform and predictable. Specifically, DbEnvFeedback is now DbEnvFeedbackHandler, DbErrcall is DbErrorHandler and DbFeedback is DbFeedbackHandler. In every case we have kept the older interfaces and the older registration methods so as to allow for backward compatibility in this release. Expect them to be removed in future releases. + +As you upgrade to this release of Berkeley DB you will notice that we have added an entirely new layer inside the package com.sleepycat.bdb. This was formerly the Greybird project by Mark Hayes. Sleepycat Software and Mark worked together to incorporate his work. We have done this in hopes of reducing the learning curve when using Berkeley DB in a Java project. When you upgrade you should consider switching to this layer as over time the historical classes and the new bdb package classes will be more and more integrated providing a simple yet powerful interface from Java into the Berkeley DB library. + +Berkeley DB's Java API is now generated with SWIG. The new Java API is significantly faster for many operations. + +Some internal methods and constructors that were previously public have been hidden or removed. + +Packages found under com.sleepycat are considered different APIs into the Berkeley DB system. These include the core db api (com.sleepycat.db), the collections style access layer (com.sleepycat.bdb) and the now relocated XA system (com.sleepycat.xa). diff --git a/docs-src/guides/upgrading/upgrade_4_2_lockng.md b/docs-src/guides/upgrading/upgrade_4_2_lockng.md new file mode 100644 index 000000000..ec292bb37 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_lockng.md @@ -0,0 +1,16 @@ +--- +title: "DB_LOCK_NOTGRANTED" +api-name: "DB_LOCK_NOTGRANTED" +source: docs/upgrading/upgrade_4_2_lockng.html +--- +## DB_LOCK_NOTGRANTED + +In previous releases, configuring lock or transaction timeout values or calling the DB_ENV->txn_begin() method with the DB_TXN_NOWAIT flag caused database operation methods to return DB_LOCK_NOTGRANTED, or throw a DbLockNotGrantedException exception. This required applications to unnecessarily handle multiple errors or exception types. + +In the Berkeley DB 4.2 release, with one exception, database operations will no longer return DB_LOCK_NOTGRANTED or throw a DbLockNotGrantedException exception. Instead, database operations will return DB_LOCK_DEADLOCK or throw a DbDeadlockException exception. This change should require no application changes, as applications must already be dealing with the possible DB_LOCK_DEADLOCK error return or DbDeadlockException exception from database operations. + +The one exception to this rule is the DB->get() method using the DB_CONSUME_WAIT flag to consume records from a Queue. If lock or transaction timeouts are set, this method and flag combination may return DB_LOCK_NOTGRANTED or throw a DbLockNotGrantedException exception. + +Applications wanting to distinguish between true deadlocks and timeouts can configure database operation methods to return DB_LOCK_NOTGRANTED or throw a DbLockNotGrantedException exception using the DB_TIME_NOTGRANTED flag. + +The DB_ENV->lock_get() and DB_ENV->lock_vec() methods will continue to return DB_LOCK_NOTGRANTED, or throw a DbLockNotGrantedException exception as they have previously done. diff --git a/docs-src/guides/upgrading/upgrade_4_2_nosync.md b/docs-src/guides/upgrading/upgrade_4_2_nosync.md new file mode 100644 index 000000000..e60c50d95 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_nosync.md @@ -0,0 +1,8 @@ +--- +title: "Client replication environments" +api-name: "Client replication environments" +source: docs/upgrading/upgrade_4_2_nosync.html +--- +## Client replication environments + +In previous Berkeley DB releases, replication clients always behaved as if DB_TXN_NOSYNC behavior was configured, that is, clients would not write or synchronously flush their log when receiving a transaction commit or prepare message. However, applications needing a high level of transactional guarantee may need a write and synchronous flush on the client. By default in the Berkeley DB 4.2 release, client database environments write and synchronously flush their logs when receiving a transaction commit or prepare message. Applications not needing such a high level of transactional guarantee should use the environment's DB_TXN_NOSYNC flag to configure their client database environments to not do the write or flush on transaction commit, as this will increase their performance. Regardless of the setting of the DB_TXN_NOSYNC flag, clients will always write and flush on transaction prepare. diff --git a/docs-src/guides/upgrading/upgrade_4_2_priority.md b/docs-src/guides/upgrading/upgrade_4_2_priority.md new file mode 100644 index 000000000..3721bc264 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_priority.md @@ -0,0 +1,8 @@ +--- +title: "DB->set_cache_priority" +api-name: "DB->set_cache_priority" +source: docs/upgrading/upgrade_4_2_priority.html +--- +## DB-\>set_cache_priority + +In previous releases, applications set the priority of a database's pages in the Berkeley DB buffer cache with the DB-\>set_cache_priority method. This method is no longer available. Applications wanting to set database page priorities in the buffer cache should use the mempset_priority() method instead. The new call takes the same arguments and behaves identically to the old call, except that a DB_MPOOLFILE buffer cache file handle is used instead of the DB database handle. diff --git a/docs-src/guides/upgrading/upgrade_4_2_queue.md b/docs-src/guides/upgrading/upgrade_4_2_queue.md new file mode 100644 index 000000000..c0db0b34d --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_queue.md @@ -0,0 +1,17 @@ +--- +title: "Queue access method" +api-name: "Queue access method" +source: docs/upgrading/upgrade_4_2_queue.html +--- +## Queue access method + +We have discovered a problem where applications that specify Berkeley DB's encryption or data checksum features on Queue databases with extent files, the database data will not be protected. This is obviously a security problem, and we encourage you to upgrade these applications to the 4.2 release as soon as possible. + +The Queue databases must be dumped and reloaded in order to fix this problem. First build the Berkeley DB 4.2 release, then use your previous release to dump the database, and the 4.2 release to reload the database. For example: + +``` c +db-4.1.25/db_dump -P password -k database | \ +db-4.2.xx/db_load -P password new_database +``` + +Note this is **only** necessary for Queue access method databases, where extent files were configured along with either encryption or checksums. diff --git a/docs-src/guides/upgrading/upgrade_4_2_repinit.md b/docs-src/guides/upgrading/upgrade_4_2_repinit.md new file mode 100644 index 000000000..97ac32804 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_repinit.md @@ -0,0 +1,24 @@ +--- +title: "Replication" +api-name: "Replication" +source: docs/upgrading/upgrade_4_2_repinit.html +--- +## Replication + + [Replication initialization](upgrade_4_2_repinit.md#idp50804696) + + [Database methods and replication clients](upgrade_4_2_repinit.md#idp50772032) + + [DB_ENV-\>rep_process_message()](upgrade_4_2_repinit.md#idp50779672) + +### Replication initialization + +In the Berkeley DB 4.2 release, replication environments must be specifically initialized by any process that will ever do anything other than open databases in read-only mode (that is, any process which might call any of the Berkeley DB replication interfaces or modify databases). This initialization is done when the replication database environment handle is opened, by specifying the DB_INIT_REP flag to the DB_ENV->open() method. + +### Database methods and replication clients + +All of the DB object methods may now return `DB_REP_HANDLE_DEAD` when a replication client changes masters. When this happens the DB handle is no longer able to be used and the application must close the handle using the DB->close() method and open a new handle. This new return value is returned when a client unrolls a transaction in order to synchronize with the new master. Otherwise, if the application was permitted to use the original handle, it's possible the handle might attempt to access nonexistent resources. + +### DB_ENV->rep_process_message() + +The DB_ENV->rep_process_message() method has new return values and an log sequence number (LSN) associated with those return values. The new argument is **ret_lsnp**, which is the returned LSN when the DB_ENV->rep_process_message() method returns DB_REP_ISPERM or DB_REP_NOTPERM. See Transactional guarantees for more information. diff --git a/docs-src/guides/upgrading/upgrade_4_2_tcl.md b/docs-src/guides/upgrading/upgrade_4_2_tcl.md new file mode 100644 index 000000000..904aa3c82 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_tcl.md @@ -0,0 +1,8 @@ +--- +title: "Tcl API" +api-name: "Tcl API" +source: docs/upgrading/upgrade_4_2_tcl.html +--- +## Tcl API + +The Tcl API included in the Berkeley DB 4.2 release requires Tcl release 8.4 or later. diff --git a/docs-src/guides/upgrading/upgrade_4_2_toc.md b/docs-src/guides/upgrading/upgrade_4_2_toc.md new file mode 100644 index 000000000..ffffea4c9 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_toc.md @@ -0,0 +1,88 @@ +--- +title: "Chapter 8. Upgrading Berkeley DB 4.1 applications to Berkeley DB 4.2" +api-name: "Chapter 8. Upgrading Berkeley DB 4.1 applications to Berkeley DB 4.2" +source: docs/upgrading/upgrade_4_2_toc.html +--- +## Chapter 8. Upgrading Berkeley DB 4.1 applications to Berkeley DB 4.2 + +**Table of Contents** + + [Introduction](upgrade_4_2_toc.md#upgrade_4_2_intro) + + [Java](upgrade_4_2_java.md) + + [Queue access method](upgrade_4_2_queue.md) + + [DB_CHKSUM_SHA1](upgrade_4_2_cksum.md) + + [DB_CLIENT](upgrade_4_2_client.md) + + [DB-\>del](upgrade_4_2_del.md) + + [DB-\>set_cache_priority](upgrade_4_2_priority.md) + + [DB-\>verify](upgrade_4_2_verify.md) + + [DB_LOCK_NOTGRANTED](upgrade_4_2_lockng.md) + + [Replication](upgrade_4_2_repinit.md) + + [Replication initialization](upgrade_4_2_repinit.md#idp50804696) + + [Database methods and replication clients](upgrade_4_2_repinit.md#idp50772032) + + [DB_ENV-\>rep_process_message()](upgrade_4_2_repinit.md#idp50779672) + + [Client replication environments](upgrade_4_2_nosync.md) + + [Tcl API](upgrade_4_2_tcl.md) + + [Upgrade Requirements](upgrade_4_2_disk.md) + + [Berkeley DB 4.2.52 Change Log](changelog_4_2_52.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_2_52.md#idp50822856) + + [New Features:](changelog_4_2_52.md#idp50784344) + + [Database Environment Changes:](changelog_4_2_52.md#idp50809288) + + [Concurrent Data Store Changes:](changelog_4_2_52.md#idp50822104) + + [General Access Method Changes:](changelog_4_2_52.md#idp50824288) + + [Btree Access Method Changes:](changelog_4_2_52.md#idp50825368) + + [Hash Access Method Changes:](changelog_4_2_52.md#idp50844704) + + [Queue Access Method Changes:](changelog_4_2_52.md#idp50828568) + + [Recno Access Method Changes:](changelog_4_2_52.md#idp50858440) + + [C++-specific API Changes:](changelog_4_2_52.md#idp50832248) + + [Java-specific API Changes:](changelog_4_2_52.md#idp50815840) + + [Tcl-specific API Changes:](changelog_4_2_52.md#idp50867864) + + [RPC-specific Client/Server Changes:](changelog_4_2_52.md#idp50852544) + + [Replication Changes:](changelog_4_2_52.md#idp50858528) + + [XA Resource Manager Changes:](changelog_4_2_52.md#idp50877816) + + [Locking Subsystem Changes:](changelog_4_2_52.md#idp50865088) + + [Logging Subsystem Changes:](changelog_4_2_52.md#idp50868008) + + [Memory Pool Subsystem Changes:](changelog_4_2_52.md#idp50865504) + + [Transaction Subsystem Changes:](changelog_4_2_52.md#idp50845064) + + [Utility Changes:](changelog_4_2_52.md#idp50858944) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_2_52.md#idp50892568) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.1 release interfaces to the Berkeley DB 4.2 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_2_verify.md b/docs-src/guides/upgrading/upgrade_4_2_verify.md new file mode 100644 index 000000000..4d822b873 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_2_verify.md @@ -0,0 +1,10 @@ +--- +title: "DB->verify" +api-name: "DB->verify" +source: docs/upgrading/upgrade_4_2_verify.html +--- +## DB-\>verify + +In previous releases, applications calling the DB->verify() method had to explicitly discard the DB handle by calling the DB->close() method. Further, using the DB handle in other ways after calling the DB->verify() method was not prohibited by the documentation, although such use was likely to lead to problems. + +For consistency with other Berkeley DB methods, DB->verify() method has been documented in the current release as a DB handle destructor. Applications using the DB handle in any way (including calling the DB->close() method) after calling DB->verify() should be updated to make no further use of any kind of the DB handle after DB->verify() returns. diff --git a/docs-src/guides/upgrading/upgrade_4_3_cput.md b/docs-src/guides/upgrading/upgrade_4_3_cput.md new file mode 100644 index 000000000..251bb53f4 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_cput.md @@ -0,0 +1,8 @@ +--- +title: "DBcursor->c_put" +api-name: "DBcursor->c_put" +source: docs/upgrading/upgrade_4_3_cput.html +--- +## DBcursor-\>c_put + +The 4.3 release disallows the DB_CURRENT flag to the DBC->put() method after the current item referenced by the cursor has been deleted. Applications using this sequence of operations should be changed to do the put without first deleting the item. diff --git a/docs-src/guides/upgrading/upgrade_4_3_disk.md b/docs-src/guides/upgrading/upgrade_4_3_disk.md new file mode 100644 index 000000000..1c60ec9df --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_3_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.3 release. No database formats changed in the Berkeley DB 4.3 release. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_3_enomem.md b/docs-src/guides/upgrading/upgrade_4_3_enomem.md new file mode 100644 index 000000000..4bed85b1a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_enomem.md @@ -0,0 +1,16 @@ +--- +title: "ENOMEM and DbMemoryException" +api-name: "ENOMEM and DbMemoryException" +source: docs/upgrading/upgrade_4_3_enomem.html +--- +## ENOMEM and DbMemoryException + +In versions of Berkeley DB before 4.3, the error **ENOMEM** was used to indicate that the buffer in a DBT configured with DB_DBT_USERMEM was too small to hold a key or data item being retrieved. The 4.3 release adds a new error, `DB_BUFFER_SMALL`, that is returned in this case. + +The reason for the change is that the use of **ENOMEM** was ambiguous: calls such as DB->get() or DBC->get() could return **ENOMEM** either if a DBT was too small or if some resource was exhausted. + +The result is that starting with the 4.3 release, C applications should always treat **ENOMEM** as a fatal error. Code that checked for the **ENOMEM** return and allocated a new buffer should be changed to check for `DB_BUFFER_SMALL`. + +In C++ applications configured for exceptions, a DbMemoryException will continue to be thrown in both cases, and applications should check the errno in the exception to determine which error occurred. + +In Java applications, a **DbMemoryException** will be thrown when a **Dbt** is too small to hold a return value, and an **OutOfMemoryError** will be thrown in all cases of resource exhaustion. diff --git a/docs-src/guides/upgrading/upgrade_4_3_err.md b/docs-src/guides/upgrading/upgrade_4_3_err.md new file mode 100644 index 000000000..0b580417e --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_err.md @@ -0,0 +1,21 @@ +--- +title: "DB_ENV->set_errcall, DB->set_errcall" +api-name: "DB_ENV->set_errcall, DB->set_errcall" +source: docs/upgrading/upgrade_4_3_err.html +--- +## DB_ENV-\>set_errcall, DB-\>set_errcall + +The signature of the error callback passed to the DB_ENV->set_errcall() and DB->set_errcall() methods has changed in the 4.3 release. For example, if you previously had a function such as this: + +``` c +void handle_db_error(const char *prefix, char *message); +``` + +it should be changed to this: + +``` c +void handle_db_error(const DB_ENV *dbenv, + const char *prefix, const char *message); +``` + +This change adds the DB_ENV handle to provide database environment context for the callback function, and incidentally makes it clear the message parameter cannot be changed by the callback. diff --git a/docs-src/guides/upgrading/upgrade_4_3_fileopen.md b/docs-src/guides/upgrading/upgrade_4_3_fileopen.md new file mode 100644 index 000000000..b278c0c78 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_fileopen.md @@ -0,0 +1,8 @@ +--- +title: "DB_FILEOPEN" +api-name: "DB_FILEOPEN" +source: docs/upgrading/upgrade_4_3_fileopen.html +--- +## DB_FILEOPEN + +The 4.3 release removes the DB_FILEOPEN error return. Any application check for the DB_FILEOPEN error should be removed. diff --git a/docs-src/guides/upgrading/upgrade_4_3_java.md b/docs-src/guides/upgrading/upgrade_4_3_java.md new file mode 100644 index 000000000..d4430dbbe --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_java.md @@ -0,0 +1,27 @@ +--- +title: "Java" +api-name: "Java" +source: docs/upgrading/upgrade_4_3_java.html +--- +## Java + +The Berkeley DB Java API has changed significantly in the 4.3 release, in ways incompatible with previous releases. This has been done to provide a consistent Java-like API for Berkeley DB as well as to make the Berkeley DB Java API match the API in Berkeley DB Java Edition, to ease application-porting between the two libraries. + +Here is a summary of the major changes: + +- Db -\> Database +- Dbc -\> Cursor +- Dbt -\> DatabaseEntry +- DbEnv -\> Environment +- DbTxn -\> Transaction +- Db.cursor -\> Database.openCursor +- Dbc.get(..., DbConstants.DB_CURRENT) -\> Cursor.getCurrent(...) + +1. The low-level wrapper around the C API has been moved into a package called com.sleepycat.db.internal. +2. There is a new public API in the package com.sleepycat.db. +3. All flags and error numbers have been eliminated from the public API. All configuration is done through method calls on configuration objects. +4. All classes and methods are named to Java standards, matching Berkeley DB Java Edition. For example: +5. The statistics classes have "getter" methods for all fields. +6. In transactional applications, the Java API infers whether to auto-commit operations: if an update is performed on a transactional database without supplying a transaction, it is implicitly auto-committed. +7. The com.sleepycat.bdb.\* packages have been reorganized so that the binding classes can be used with the base API in the com.sleepycat.db package. The bind and collection classes are now essentially the same in Berkeley DB and Berkeley DB Java Edition. The former com.sleepycat.bdb.bind.\* packages are now the com.sleepycat.bind.\* packages. The former com.sleepycat.bdb, com.sleepycat.bdb.collections, and com.sleepycat.bdb.factory packages are now combined in the new com.sleepycat.collections package. +8. A layer of the former collections API has been removed to simplify the API and to remove the redundant implementation of secondary indices. The former DataStore, DataIndex, and ForeignKeyIndex classes have been removed. Instead of wrapping a Database in a DataStore or DataIndex, the Database object is now passed directly to the constructor of a StoredMap, StoredList, etc. diff --git a/docs-src/guides/upgrading/upgrade_4_3_log.md b/docs-src/guides/upgrading/upgrade_4_3_log.md new file mode 100644 index 000000000..8b6afa895 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_log.md @@ -0,0 +1,12 @@ +--- +title: "Logging" +api-name: "Logging" +source: docs/upgrading/upgrade_4_3_log.html +--- +## Logging + +In previous releases, the DB_ENV->set_flags() method flag DB_TXN_NOT_DURABLE specified that transactions for the entire database environment were not durable. However, it was not possible to set this flag in environments that were part of replication groups, and physical log files were still created. The 4.3 release adds support for true in-memory logging for both replication and non-replicated sites. + +Existing applications setting the DB_TXN_NOT_DURABLE flag for database environments should be upgraded to set the DB_LOG_INMEMORY flag instead. + +In previous releases, log buffer sizes were restricted to be less than or equal to the log file size; this restriction is no longer required. diff --git a/docs-src/guides/upgrading/upgrade_4_3_repl.md b/docs-src/guides/upgrading/upgrade_4_3_repl.md new file mode 100644 index 000000000..89d158c00 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_repl.md @@ -0,0 +1,12 @@ +--- +title: "Replication" +api-name: "Replication" +source: docs/upgrading/upgrade_4_3_repl.html +--- +## Replication + +The 4.3 release removes support for logs-only replication clients. Use of the DB_REP_LOGSONLY flag to the DB_ENV->rep_start() should be replaced with the DB_REP_CLIENT flag. + +The 4.3 release adds two new arguments to the DB_ENV->rep_elect() method, **nvotes** and **flags**. The **nvotes** argument sets the required number of replication group members that must participate in an election in order for a master to be declared. For backward compatibility, set the **nvotes** argument to 0. The flags argument is currently unused and should be set to 0. See DB_ENV->rep_elect() method or "Replication Elections" for more information. + +In the 4.3 release it is no longer necessary to do a database environment hot backup to initialize a replication client. All that is needed now is for the client to join the replication group. Berkeley DB will perform an internal backup from the master to the client automatically and will run recovery on the client to bring it up to date with the master. diff --git a/docs-src/guides/upgrading/upgrade_4_3_rtc.md b/docs-src/guides/upgrading/upgrade_4_3_rtc.md new file mode 100644 index 000000000..b124e00ed --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_rtc.md @@ -0,0 +1,8 @@ +--- +title: "Run-time configuration" +api-name: "Run-time configuration" +source: docs/upgrading/upgrade_4_3_rtc.html +--- +## Run-time configuration + +The signatures of the db_env_set_func_ftruncate and db_env_set_func_seek functions have been simplified to take a byte offset in one parameter rather than a page size and a page number. diff --git a/docs-src/guides/upgrading/upgrade_4_3_stat.md b/docs-src/guides/upgrading/upgrade_4_3_stat.md new file mode 100644 index 000000000..cae786074 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_stat.md @@ -0,0 +1,10 @@ +--- +title: "DB->stat" +api-name: "DB->stat" +source: docs/upgrading/upgrade_4_3_stat.html +--- +## DB-\>stat + +The 4.3 release adds transactional support to the DB->stat() method. + +Application writers can simply add a NULL **txnid** argument to the DB->stat() method calls in their application to leave the application's behavior unchanged. diff --git a/docs-src/guides/upgrading/upgrade_4_3_toc.md b/docs-src/guides/upgrading/upgrade_4_3_toc.md new file mode 100644 index 000000000..327328277 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_toc.md @@ -0,0 +1,80 @@ +--- +title: "Chapter 7. Upgrading Berkeley DB 4.2 applications to Berkeley DB 4.3" +api-name: "Chapter 7. Upgrading Berkeley DB 4.2 applications to Berkeley DB 4.3" +source: docs/upgrading/upgrade_4_3_toc.html +--- +## Chapter 7. Upgrading Berkeley DB 4.2 applications to Berkeley DB 4.3 + +**Table of Contents** + + [Introduction](upgrade_4_3_toc.md#upgrade_4_3_intro) + + [Java](upgrade_4_3_java.md) + + [DB_ENV-\>set_errcall, DB-\>set_errcall](upgrade_4_3_err.md) + + [DBcursor-\>c_put](upgrade_4_3_cput.md) + + [DB-\>stat](upgrade_4_3_stat.md) + + [DB_ENV-\>set_verbose](upgrade_4_3_verb.md) + + [Logging](upgrade_4_3_log.md) + + [DB_FILEOPEN](upgrade_4_3_fileopen.md) + + [ENOMEM and DbMemoryException](upgrade_4_3_enomem.md) + + [Replication](upgrade_4_3_repl.md) + + [Run-time configuration](upgrade_4_3_rtc.md) + + [Upgrade Requirements](upgrade_4_3_disk.md) + + [Berkeley DB 4.3.29 Change Log](changelog_4_3_29.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_3_29.md#idp50694880) + + [New Features:](changelog_4_3_29.md#idp50670248) + + [Database Environment Changes:](changelog_4_3_29.md#idp50673968) + + [Concurrent Data Store Changes:](changelog_4_3_29.md#idp50703424) + + [General Access Method Changes:](changelog_4_3_29.md#idp50690848) + + [Btree Access Method Changes:](changelog_4_3_29.md#idp50691272) + + [Hash Access Method Changes:](changelog_4_3_29.md#idp50694944) + + [Queue Access Method Changes:](changelog_4_3_29.md#idp50697104) + + [Recno Access Method Changes](changelog_4_3_29.md#idp50720352) + + [C++-specific API Changes:](changelog_4_3_29.md#idp50700784) + + [Java-specific API Changes:](changelog_4_3_29.md#idp50670632) + + [Tcl-specific API Changes:](changelog_4_3_29.md#idp50702384) + + [RPC-specific Client/Server Changes:](changelog_4_3_29.md#idp50703784) + + [Replication Changes:](changelog_4_3_29.md#idp50685776) + + [XA Resource Manager Changes:](changelog_4_3_29.md#idp50733112) + + [Locking Subsystem Changes:](changelog_4_3_29.md#idp50712384) + + [Logging Subsystem Changes:](changelog_4_3_29.md#idp50740760) + + [Memory Pool Subsystem Changes:](changelog_4_3_29.md#idp50695328) + + [Transaction Subsystem Changes:](changelog_4_3_29.md#idp50720440) + + [Utility Changes:](changelog_4_3_29.md#idp50724480) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_3_29.md#idp50724864) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.2 release interfaces to the Berkeley DB 4.3 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_3_verb.md b/docs-src/guides/upgrading/upgrade_4_3_verb.md new file mode 100644 index 000000000..06e861ae5 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_3_verb.md @@ -0,0 +1,10 @@ +--- +title: "DB_ENV->set_verbose" +api-name: "DB_ENV->set_verbose" +source: docs/upgrading/upgrade_4_3_verb.html +--- +## DB_ENV-\>set_verbose + +The 4.3 release removes support for the DB_ENV->set_verbose() method flag DB_VERB_CHKPOINT. Application writers should simply remove any use of this flag from their applications. + +The 4.3 release redirects output configured by the DB_ENV->set_verbose() method from the error output channels (see the DB_ENV->set_errfile() and DB_ENV->set_errcall() methods for more information) to the new DB_ENV->set_msgcall() and DB_ENV->set_msgfile() message output channels. This change means the error output channels are now only used for errors, and not for debugging and performance tuning messages as well as errors. Application writers using DB_ENV->set_verbose() should confirm that output is handled appropriately. diff --git a/docs-src/guides/upgrading/upgrade_4_4_autocommit.md b/docs-src/guides/upgrading/upgrade_4_4_autocommit.md new file mode 100644 index 000000000..08b57eb9c --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_autocommit.md @@ -0,0 +1,14 @@ +--- +title: "DB_AUTO_COMMIT" +api-name: "DB_AUTO_COMMIT" +source: docs/upgrading/upgrade_4_4_autocommit.html +--- +## DB_AUTO_COMMIT + +In previous Berkeley DB releases, the DB_AUTO_COMMIT flag was used in the C and C++ Berkeley DB APIs to wrap operations within a transaction without explicitly creating a transaction and passing the TXN handle as part of the operation method call. In the 4.4 release, the DB_AUTO_COMMIT flag no longer needs to be explicitly specified. + +In the 4.4 release, specifying the DB_AUTO_COMMIT flag to the DB_ENV->set_flags() method causes all database modifications in that environment to be transactional; specifying DB_AUTO_COMMIT to the DB->open() method causes all modifications to that database to be transactional; specifying DB_AUTO_COMMIT to the DB_ENV->dbremove() methods causes those specific operations to be transactional. + +No related application changes are required for this release, as the DB_AUTO_COMMIT flag is ignored where it is no longer needed. However, application writers are encouraged to remove uses of the DB_AUTO_COMMIT flag in places where it is no longer needed. + +Similar changes have been made to the Berkeley DB Tcl API. These changes are not optional, and Tcl applications will need to remove the -auto_commit flag from methods where it is no longer needed. diff --git a/docs-src/guides/upgrading/upgrade_4_4_clear.md b/docs-src/guides/upgrading/upgrade_4_4_clear.md new file mode 100644 index 000000000..2924c2065 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_clear.md @@ -0,0 +1,10 @@ +--- +title: "DB_MPOOLFILE->set_clear_len" +api-name: "DB_MPOOLFILE->set_clear_len" +source: docs/upgrading/upgrade_4_4_clear.html +--- +## DB_MPOOLFILE-\>set_clear_len + +The meaning of a 0 "clear length" argument to the DB_MPOOLFILE->set_clear_len() method changed in the Berkeley DB 4.4 release. In previous releases, specifying a length of 0 was equivalent to the default, and the entire created page was cleared. Unfortunately, this left no way to specify that no part of the page needed to be cleared. In the 4.4 release, specifying a "clear length" argument of 0 means that no part of the page need be cleared. + +Applications specifying a 0 "clear length" argument to the DB_MPOOLFILE->set_clear_len() method should simply remove the call, as the default behavior is to clear the entire created page. diff --git a/docs-src/guides/upgrading/upgrade_4_4_disk.md b/docs-src/guides/upgrading/upgrade_4_4_disk.md new file mode 100644 index 000000000..922a809a5 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_4_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.4 release. No database formats changed in the Berkeley DB 4.4 release. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_4_isolation.md b/docs-src/guides/upgrading/upgrade_4_4_isolation.md new file mode 100644 index 000000000..64bc0a376 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_isolation.md @@ -0,0 +1,8 @@ +--- +title: "DB_DEGREE_2, DB_DIRTY_READ" +api-name: "DB_DEGREE_2, DB_DIRTY_READ" +source: docs/upgrading/upgrade_4_4_isolation.html +--- +## DB_DEGREE_2, DB_DIRTY_READ + +The names of two isolation-level flags changed in the Berkeley DB 4.4 release. The DB_DEGREE_2 flag was renamed to DB_READ_COMMITTED, and the DB_DIRTY_READ flag was renamed to DB_READ_UNCOMMITTED, to match ANSI standard names for isolation levels. The historic flag names continue to work in this release, but may be removed from future releases. diff --git a/docs-src/guides/upgrading/upgrade_4_4_joinenv.md b/docs-src/guides/upgrading/upgrade_4_4_joinenv.md new file mode 100644 index 000000000..d4f0e2e03 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_joinenv.md @@ -0,0 +1,20 @@ +--- +title: "DB_JOINENV" +api-name: "DB_JOINENV" +source: docs/upgrading/upgrade_4_4_joinenv.html +--- +## DB_JOINENV + +The semantics of joining existing Berkeley DB database environments has changed in the 4.4 release. Previously: + +1. Applications joining existing environments, but not configuring some of the subsystems configured in the environment when it was created, would not be configured for those subsystems. +2. Applications joining existing environments, but configuring additional subsystems in addition to the subsystems configured in the environment when it was created, would cause additional subsystems to be configured in the database environment. + +In the 4.4 release, the semantics have been simplified to make it easier to write robust applications. In the 4.4 release: + +1. Applications joining existing environments, but not configuring some of the subsystems configured in the environment when it was created, will now automatically be configured for all of the subsystems configured in the environment. +2. Applications joining existing environments, but configuring additional subsystems in addition to the subsystems configured in the environment when it was created, will fail, as no additional subsystems can be configured for a database environment after it is created. + +In other words, the choice of subsystems initialized for a Berkeley DB database environment is specified by the thread of control initially creating the environment. Any subsequent thread of control joining the environment will automatically be configured to use the same subsystems as were created in the environment (unless the thread of control requests a subsystem not available in the environment, which will fail). Applications joining an environment, able to adapt to whatever subsystems have been configured in the environment, should open the environment without specifying any subsystem flags. Applications joining an environment, requiring specific subsystems from their environments, should open the environment specifying those specific subsystem flags. + +The DB_JOINENV flag has been changed to have no effect in the Berkeley DB 4.4 release. Applications should require no changes, although uses of the DB_JOINENV flag may be removed. diff --git a/docs-src/guides/upgrading/upgrade_4_4_lockstat.md b/docs-src/guides/upgrading/upgrade_4_4_lockstat.md new file mode 100644 index 000000000..09126b947 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_lockstat.md @@ -0,0 +1,8 @@ +--- +title: "lock statistics" +api-name: "lock statistics" +source: docs/upgrading/upgrade_4_4_lockstat.html +--- +## lock statistics + +The names of two fields in the lock statistics changed in the Berkeley DB 4.4 release. The **st_nconflicts** field was renamed to be **st_lock_wait**, and the **st_nnowaits** field was renamed to be **st_lock_nowait**. The meaning of the fields is unchanged (although the documentation has been updated to make it clear what these fields really represent). diff --git a/docs-src/guides/upgrading/upgrade_4_4_mutex.md b/docs-src/guides/upgrading/upgrade_4_4_mutex.md new file mode 100644 index 000000000..4b07d2ef4 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_mutex.md @@ -0,0 +1,12 @@ +--- +title: "mutexes" +api-name: "mutexes" +source: docs/upgrading/upgrade_4_4_mutex.html +--- +## mutexes + +The DB_ENV\>set_tas_spins and DB_ENV\>get_tas_spins methods have been renamed to DB_ENV->mutex_set_tas_spins() and DB_ENV->mutex_set_tas_spins() to match the new mutex support in the Berkeley DB 4.4 release. Applications calling the old methods should be updated to use the new method names. + +For backward compatibility, the string "set_tas_spins" is still supported in DB_CONFIG files. + +The --with-mutexalign="ALIGNMENT" compile-time configuration option has been removed from Berkeley DB configuration. Mutex alignment should now be configured at run-time, using the DB_ENV->mutex_set_align() method. diff --git a/docs-src/guides/upgrading/upgrade_4_4_toc.md b/docs-src/guides/upgrading/upgrade_4_4_toc.md new file mode 100644 index 000000000..a69da1791 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_4_toc.md @@ -0,0 +1,78 @@ +--- +title: "Chapter 6. Upgrading Berkeley DB 4.3 applications to Berkeley DB 4.4" +api-name: "Chapter 6. Upgrading Berkeley DB 4.3 applications to Berkeley DB 4.4" +source: docs/upgrading/upgrade_4_4_toc.html +--- +## Chapter 6. Upgrading Berkeley DB 4.3 applications to Berkeley DB 4.4 + +**Table of Contents** + + [Introduction](upgrade_4_4_toc.md#upgrade_4_4_intro) + + [DB_AUTO_COMMIT](upgrade_4_4_autocommit.md) + + [DB_DEGREE_2, DB_DIRTY_READ](upgrade_4_4_isolation.md) + + [DB_JOINENV](upgrade_4_4_joinenv.md) + + [mutexes](upgrade_4_4_mutex.md) + + [DB_MPOOLFILE-\>set_clear_len](upgrade_4_4_clear.md) + + [lock statistics](upgrade_4_4_lockstat.md) + + [Upgrade Requirements](upgrade_4_4_disk.md) + + [Berkeley DB 4.4.16 Change Log](changelog_4_4_16.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_4_16.md#idp50595920) + + [New Features:](changelog_4_4_16.md#idp50583264) + + [Database Environment Changes:](changelog_4_4_16.md#idp50583648) + + [Concurrent Data Store Changes:](changelog_4_4_16.md#idp50567656) + + [General Access Method Changes:](changelog_4_4_16.md#idp50591960) + + [Btree Access Method Changes:](changelog_4_4_16.md#idp50592384) + + [Hash Access Method Changes:](changelog_4_4_16.md#idp50595984) + + [Queue Access Method Changes:](changelog_4_4_16.md#idp50597856) + + [Recno Access Method Changes](changelog_4_4_16.md#idp50598936) + + [C++-specific API Changes:](changelog_4_4_16.md#idp50598072) + + [Java-specific API Changes:](changelog_4_4_16.md#idp50600424) + + [Java collections and bind API Changes:](changelog_4_4_16.md#idp50621112) + + [Tcl-specific API Changes:](changelog_4_4_16.md#idp50604672) + + [RPC-specific Client/Server Changes:](changelog_4_4_16.md#idp50589536) + + [Replication Changes:](changelog_4_4_16.md#idp50610200) + + [XA Resource Manager Changes:](changelog_4_4_16.md#idp50594920) + + [Locking Subsystem Changes:](changelog_4_4_16.md#idp50614600) + + [Logging Subsystem Changes:](changelog_4_4_16.md#idp50614888) + + [Memory Pool Subsystem Changes:](changelog_4_4_16.md#idp50635800) + + [Transaction Subsystem Changes:](changelog_4_4_16.md#idp50617400) + + [Utility Changes:](changelog_4_4_16.md#idp50617824) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_4_16.md#idp50621200) + + [Berkeley DB 4.4.20 Change Log](changelog_4_4_20.md) + + [Changes since Berkeley DB 4.4.16:](changelog_4_4_20.md#idp50624312) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.3 release interfaces to the Berkeley DB 4.4 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_5_alive.md b/docs-src/guides/upgrading/upgrade_4_5_alive.md new file mode 100644 index 000000000..17c1d2b06 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_alive.md @@ -0,0 +1,10 @@ +--- +title: "DB->set_isalive" +api-name: "DB->set_isalive" +source: docs/upgrading/upgrade_4_5_alive.html +--- +## DB-\>set_isalive + +In previous releases, the function specified to the DB_ENV->set_isalive() method did not take a flags parameter. In the Berkeley DB 4.5 release, an additional flags argument has been added: DB_MUTEX_PROCESS_ONLY. + +Applications configuring an is-alive function should add a flags argument to the function, and change the function to ignore any thread ID and return the status of just the process, when the DB_MUTEX_PROCESS_ONLY flag is specified. diff --git a/docs-src/guides/upgrading/upgrade_4_5_applog.md b/docs-src/guides/upgrading/upgrade_4_5_applog.md new file mode 100644 index 000000000..241d9d610 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_applog.md @@ -0,0 +1,20 @@ +--- +title: "Application-specific logging" +api-name: "Application-specific logging" +source: docs/upgrading/upgrade_4_5_applog.html +--- +## Application-specific logging + +In previous releases of Berkeley DB, "BEGIN" lines in the XXX.src files used to build application-specific logging support only required a log record number. In the 4.5 release, those lines require a Berkeley DB library version as well. For example, the entry: + +``` c +BEGIN mkdir 10000 +``` + +must now be: + +``` c +BEGIN mkdir 44 10000 +``` + +that is, the version of the Berkeley DB release where the log record was introduced must be included. The version is the major and minor numbers for the Berkeley DB library, with all punctuation removed. For example, Berkeley DB version 4.2 should be 42, version 4.5 should be 45. diff --git a/docs-src/guides/upgrading/upgrade_4_5_collect.md b/docs-src/guides/upgrading/upgrade_4_5_collect.md new file mode 100644 index 000000000..94b11bf36 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_collect.md @@ -0,0 +1,8 @@ +--- +title: "Collections API" +api-name: "Collections API" +source: docs/upgrading/upgrade_4_5_collect.html +--- +## Collections API + +The changes to the Collections API are compatible with prior releases, with one exception: the Iterator object returned by the StoredCollection.iterator() method can no longer be explicitly cast to StoredIterator because a different implementation class is now used for iterators. If you depend on the StoredIterator class, you must now call StoredCollection.storedIterator() instead. Note the StoredIterator.close(Iterator) static method is compatible with the new iterator implementation, so no changes are necessary if you are using that method to close iterators. diff --git a/docs-src/guides/upgrading/upgrade_4_5_config.md b/docs-src/guides/upgrading/upgrade_4_5_config.md new file mode 100644 index 000000000..4a75f7cd6 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_config.md @@ -0,0 +1,8 @@ +--- +title: "--enable-pthread_self" +api-name: "--enable-pthread_self" +source: docs/upgrading/upgrade_4_5_config.html +--- +## --enable-pthread_self + +In previous releases, the --enable-pthread_self configuration option was used to force Berkeley DB to use the POSIX pthread pthread_self function to identify threads of control (even when Berkeley DB was configured for test-and-set mutexes). In the 4.5 release, the --enable-pthread_self option has been replaced with the --enable-pthread_api option. This option has the same effect as the previous option, but configures the Berkeley DB build for a POSIX pthread application in other ways (for example, configuring Berkeley DB to use the pthread_self function). diff --git a/docs-src/guides/upgrading/upgrade_4_5_deprecate.md b/docs-src/guides/upgrading/upgrade_4_5_deprecate.md new file mode 100644 index 000000000..f7a82f9e4 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_deprecate.md @@ -0,0 +1,12 @@ +--- +title: "deprecated interfaces" +api-name: "deprecated interfaces" +source: docs/upgrading/upgrade_4_5_deprecate.html +--- +## deprecated interfaces + +Some previously deprecated interfaces were removed from the Berkeley DB 4.5 release: + +- The DB_ENV-\>set_lk_max method was removed. This method has been deprecated and undocumented since the Berkeley DB 4.0 release. +- The DB-\>stat method flags DB_CACHED_COUNT and DB_RECORDCOUNT were removed. These flags have been deprecated and undocumented since the Berkeley DB 4.1 release. +- The **-w** option to the db_deadlock utility was removed. This option has been deprecated and undocumented since the Berkeley DB 4.0 release. diff --git a/docs-src/guides/upgrading/upgrade_4_5_disk.md b/docs-src/guides/upgrading/upgrade_4_5_disk.md new file mode 100644 index 000000000..e815e6cb2 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_disk.md @@ -0,0 +1,10 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_5_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.5 release. No database formats changed in the Berkeley DB 4.5 release. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_5_elect.md b/docs-src/guides/upgrading/upgrade_4_5_elect.md new file mode 100644 index 000000000..c06a398c1 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_elect.md @@ -0,0 +1,12 @@ +--- +title: "DB_ENV->rep_elect" +api-name: "DB_ENV->rep_elect" +source: docs/upgrading/upgrade_4_5_elect.html +--- +## DB_ENV-\>rep_elect + +Two of the historic arguments for the DB_ENV->rep_elect() method have been moved from the interface to separate methods in order to make them available within the new replication manager framework. + +The **priority** parameter should now be explicitly set using the DB_ENV->rep_set_priority() method. To upgrade existing replication applications to the Berkeley DB 4.5 DB_ENV->rep_elect() interface, it may be simplest to insert a call to DB_ENV->rep_set_priority() immediately before the existing call to DB_ENV->rep_elect(). Alternatively, it may make more sense to add a single call to DB_ENV->rep_set_priority() during database environment configuration. + +The **timeout** parameter should now be explicitly set using the DB_ENV->rep_set_timeout() method. To upgrade existing replication applications to the Berkeley DB 4.5 DB_ENV->rep_elect() interface, it may be simplest to insert a call to DB_ENV->rep_set_timeout() immediately before the existing call to DB_ENV->rep_elect(). Alternatively, it may make more sense to add a single call to DB_ENV->rep_set_timeout() during database environment configuration. diff --git a/docs-src/guides/upgrading/upgrade_4_5_memp.md b/docs-src/guides/upgrading/upgrade_4_5_memp.md new file mode 100644 index 000000000..8bf6aa5b0 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_memp.md @@ -0,0 +1,12 @@ +--- +title: "Memory Pool API" +api-name: "Memory Pool API" +source: docs/upgrading/upgrade_4_5_memp.html +--- +## Memory Pool API + +As part of implementing support for multi-version concurrency control, the DB_MPOOL_DIRTY flag is now specified to the DB_MPOOLFILE->get() instead of DB_MPOOLFILE->put(), and the DB_MPOOLFILE-\>set method has been removed. In addition, a new transaction handle parameter has been added to the DB_MPOOLFILE->get() method. + +The DB_MPOOL_CLEAN flag is no longer supported. + +Applications which use the memory pool API directly should update to the new API in order to use 4.5. diff --git a/docs-src/guides/upgrading/upgrade_4_5_pagesize.md b/docs-src/guides/upgrading/upgrade_4_5_pagesize.md new file mode 100644 index 000000000..2b4d04415 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_pagesize.md @@ -0,0 +1,8 @@ +--- +title: "DB->set_pagesize" +api-name: "DB->set_pagesize" +source: docs/upgrading/upgrade_4_5_pagesize.html +--- +## DB-\>set_pagesize + +In previous releases, when creating a new database in a physical file which already contained databases, it was an error to specify a page size different from the existing databases in the file. In the Berkeley DB 4.5 release, any page size specified is ignored if the file in which the database is being created already exists. diff --git a/docs-src/guides/upgrading/upgrade_4_5_paniccall.md b/docs-src/guides/upgrading/upgrade_4_5_paniccall.md new file mode 100644 index 000000000..46e204a00 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_paniccall.md @@ -0,0 +1,10 @@ +--- +title: "DB_ENV->set_paniccall" +api-name: "DB_ENV->set_paniccall" +source: docs/upgrading/upgrade_4_5_paniccall.html +--- +## DB_ENV-\>set_paniccall + +In previous Berkeley DB releases, the DB_ENV-\>set_paniccall and DB-\>set_paniccall methods were used to register a callback function, called if the database environment failed. In the 4.5 release, this functionality has been replaced by a general-purpose event notification callback function, set with the DB_ENV->set_event_notify() method. Applications should be updated to replace DB_ENV-\>set_paniccall and DB-\>set_paniccall calls with a call to DB_ENV->set_event_notify(). This also requires the callback function itself change, as the callback signatures are different. + +The DB_ENV-\>set_paniccall and DB-\>set_paniccall calls are expected to be removed in a future release of Berkeley DB. diff --git a/docs-src/guides/upgrading/upgrade_4_5_rep_event.md b/docs-src/guides/upgrading/upgrade_4_5_rep_event.md new file mode 100644 index 000000000..482445687 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_rep_event.md @@ -0,0 +1,10 @@ +--- +title: "Replication events" +api-name: "Replication events" +source: docs/upgrading/upgrade_4_5_rep_event.html +--- +## Replication events + +One of the informational returns from the DB_ENV->rep_process_message() method found in previous releases of Berkeley DB has been changed to an event. The DB_REP_STARTUPDONE return from DB_ENV->rep_process_message() is now the DB_EVENT_REP_STARTUPDONE value to the DB_ENV->set_event_notify() callback. + +Applications should update their handling of this event as necessary. diff --git a/docs-src/guides/upgrading/upgrade_4_5_rep_set.md b/docs-src/guides/upgrading/upgrade_4_5_rep_set.md new file mode 100644 index 000000000..b0c30bd98 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_rep_set.md @@ -0,0 +1,10 @@ +--- +title: "Replication method naming" +api-name: "Replication method naming" +source: docs/upgrading/upgrade_4_5_rep_set.html +--- +## Replication method naming + +The method names DB_ENV-\>set_rep_limit, DB_ENV-\>get_rep_limit and DB_ENV-\>set_rep_transport have been changed to DB_ENV->rep_set_limit(), DB_ENV->rep_get_limit() and DB_ENV->rep_set_transport() in order to be consistent with the other replication method names. That is, the characters "set_rep" and "get_rep" have been changed to "rep_set" and "rep_get". + +Applications should modify the method names, no other change is required. diff --git a/docs-src/guides/upgrading/upgrade_4_5_source.md b/docs-src/guides/upgrading/upgrade_4_5_source.md new file mode 100644 index 000000000..e0465b4a6 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_source.md @@ -0,0 +1,12 @@ +--- +title: "Recno backing text source files" +api-name: "Recno backing text source files" +source: docs/upgrading/upgrade_4_5_source.html +--- +## Recno backing text source files + +In previous releases of Berkeley DB, Recno access method backing source text files were opened using the ANSI C fopen function with the "r" and "w" modes. This caused Windows systems to translate carriage-return and linefeed characters on input and output and could lead to database corruption. + +In the current release, Berkeley DB opens the backing source text files using the "rb" and "wb" modes, consequently carriage-return and linefeed characters will not be translated on Windows systems. + +Applications using the backing source text file feature on systems where the "r/w" and "rb/wb" modes differ should evaluate their application as part of upgrading to the 4.5 release. There is the possibility that characters have been translated or stripped and the backing source file has been corrupted. (Applications on other systems, for example, POSIX-like systems, should not require any changes related to this issue.) diff --git a/docs-src/guides/upgrading/upgrade_4_5_toc.md b/docs-src/guides/upgrading/upgrade_4_5_toc.md new file mode 100644 index 000000000..4b3107c7e --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_5_toc.md @@ -0,0 +1,86 @@ +--- +title: "Chapter 5. Upgrading Berkeley DB 4.4 applications to Berkeley DB 4.5" +api-name: "Chapter 5. Upgrading Berkeley DB 4.4 applications to Berkeley DB 4.5" +source: docs/upgrading/upgrade_4_5_toc.html +--- +## Chapter 5. Upgrading Berkeley DB 4.4 applications to Berkeley DB 4.5 + +**Table of Contents** + + [Introduction](upgrade_4_5_toc.md#upgrade_4_5_intro) + + [deprecated interfaces](upgrade_4_5_deprecate.md) + + [DB-\>set_isalive](upgrade_4_5_alive.md) + + [DB_ENV-\>rep_elect](upgrade_4_5_elect.md) + + [Replication method naming](upgrade_4_5_rep_set.md) + + [Replication events](upgrade_4_5_rep_event.md) + + [Memory Pool API](upgrade_4_5_memp.md) + + [DB_ENV-\>set_paniccall](upgrade_4_5_paniccall.md) + + [DB-\>set_pagesize](upgrade_4_5_pagesize.md) + + [Collections API](upgrade_4_5_collect.md) + + [--enable-pthread_self](upgrade_4_5_config.md) + + [Recno backing text source files](upgrade_4_5_source.md) + + [Application-specific logging](upgrade_4_5_applog.md) + + [Upgrade Requirements](upgrade_4_5_disk.md) + + [Berkeley DB 4.5.20 Change Log](changelog_4_5_20.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_5_20.md#idp50532016) + + [New Features:](changelog_4_5_20.md#idp50511048) + + [Database Environment Changes:](changelog_4_5_20.md#idp50513672) + + [Concurrent Data Store Changes:](changelog_4_5_20.md#idp50516704) + + [General Access Method Changes:](changelog_4_5_20.md#idp50520456) + + [Btree Access Method Changes:](changelog_4_5_20.md#idp50542544) + + [Hash Access Method Changes:](changelog_4_5_20.md#idp50536608) + + [Queue Access Method Changes:](changelog_4_5_20.md#idp50533200) + + [Recno Access Method Changes:](changelog_4_5_20.md#idp50539504) + + [C++-specific API Changes:](changelog_4_5_20.md#idp50539760) + + [Java-specific API Changes:](changelog_4_5_20.md#idp50541672) + + [Java collections and bind API Changes:](changelog_4_5_20.md#idp50542632) + + [Tcl-specific API Changes:](changelog_4_5_20.md#idp50546176) + + [RPC-specific Client/Server Changes:](changelog_4_5_20.md#idp50548752) + + [Replication Changes:](changelog_4_5_20.md#idp50547824) + + [XA Resource Manager Changes:](changelog_4_5_20.md#idp50557816) + + [Locking Subsystem Changes:](changelog_4_5_20.md#idp50534496) + + [Logging Subsystem Changes:](changelog_4_5_20.md#idp50532216) + + [Memory Pool Subsystem Changes:](changelog_4_5_20.md#idp50542056) + + [Transaction Subsystem Changes:](changelog_4_5_20.md#idp50543016) + + [Utility Changes:](changelog_4_5_20.md#idp50556608) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_5_20.md#idp50557880) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.4 release interfaces to the Berkeley DB 4.5 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_6_cursor.md b/docs-src/guides/upgrading/upgrade_4_6_cursor.md new file mode 100644 index 000000000..e56bcb08a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_cursor.md @@ -0,0 +1,31 @@ +--- +title: "C API cursor handle method names" +api-name: "C API cursor handle method names" +source: docs/upgrading/upgrade_4_6_cursor.html +--- +## C API cursor handle method names + +In the Berkeley DB 4.6 release, the C API DBC handle methods have been renamed for consistency with the C++ and Java APIs. The change is the removal of the leading "c\_" from the names, as follows: + +DBC-\>c_close +Renamed DBC-\>close + +DBC-\>c_count +Renamed DBC-\>count + +DBC-\>c_del +Renamed DBC-\>del + +DBC-\>c_dup +Renamed DBC-\>dup + +DBC-\>c_get +Renamed DBC-\>get + +DBC-\>c_pget +Renamed DBC-\>pget + +DBC-\>c_put +Renamed DBC-\>put + +The old DBC method names are deprecated but will continue for work for some number of future releases. diff --git a/docs-src/guides/upgrading/upgrade_4_6_disk.md b/docs-src/guides/upgrading/upgrade_4_6_disk.md new file mode 100644 index 000000000..badc6a3b5 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_disk.md @@ -0,0 +1,12 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_6_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.6 release. + +The format of Hash database pages was changed in the Berkeley DB 4.6 release, and items are now stored in sorted order. **The format changes are entirely backward-compatible, and no database upgrades are needed.** However, upgrading existing databases can offer significant performance improvements. Note that databases created using the 4.6 release may not be usable with earlier Berkeley DB releases. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_6_event.md b/docs-src/guides/upgrading/upgrade_4_6_event.md new file mode 100644 index 000000000..c6d083cbe --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_event.md @@ -0,0 +1,16 @@ +--- +title: "Replication Events" +api-name: "Replication Events" +source: docs/upgrading/upgrade_4_6_event.html +--- +## Replication Events + +It is now guaranteed the DB_EVENT_REP_STARTUPDONE event will be presented to the application after the corresponding DB_EVENT_REP_NEWMASTER event, even in the face of extreme thread-scheduling anomalies. (In previous releases, if the thread processing the NEWMASTER message was starved, and STARTUPDONE occurred soon after, the order might have been reversed.) + +In addition, the DB_EVENT_REP_NEWMASTER event is now presented to all types of replication applications: users of either the Replication Framework or the Base Replication API. In both cases, the DB_EVENT_REP_NEWMASTER event always means that a site other than the local environment has become master. + +The **envid** parameter to DB_ENV->rep_process_message() has been changed to be of type "int" rather than "int \*", and the environment ID of a new master is presented to the application along with the DB_EVENT_REP_NEWMASTER event. Replication applications should be modified to use the DB_EVENT_REP_NEWMASTER event to determine the ID of the new master. + +The **envid** parameter has been removed from the DB_ENV->rep_elect() method and a new event type has been added. The DB_EVENT_REP_ELECTED event is presented to the application at the site which wins an election. In the Berkeley DB 4.6 release, the normal result of a successful election is either the DB_EVENT_REP_NEWMASTER event (with the winner's environment ID), or the DB_EVENT_REP_ELECTED event. Only one of the two events will ever be delivered. + +The DB_REP_NEWMASTER return code has been removed from the DB_ENV->rep_process_message() method. Replication applications should be modified to use the DB_EVENT_REP_NEWMASTER and DB_EVENT_REP_ELECTED events to determine the existence of a new master. diff --git a/docs-src/guides/upgrading/upgrade_4_6_full_election.md b/docs-src/guides/upgrading/upgrade_4_6_full_election.md new file mode 100644 index 000000000..177244b7b --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_full_election.md @@ -0,0 +1,10 @@ +--- +title: "DB_REP_FULL_ELECTION" +api-name: "DB_REP_FULL_ELECTION" +source: docs/upgrading/upgrade_4_6_full_election.html +--- +## DB_REP_FULL_ELECTION + +The DB_REP_FULL_ELECTION flag historically specified to the DB_ENV->repmgr_start() method has been removed from the 4.6 release. + +In the Berkeley DB 4.6 release, a simpler and more flexible implementation of this functionality is available. Applications needing to configure the first election of a replication group differently from subsequent elections should use the DB_REP_FULL_ELECTION_TIMEOUT flag to the DB_ENV->rep_set_timeout() method to specify a different timeout for the first election. diff --git a/docs-src/guides/upgrading/upgrade_4_6_memp_fput.md b/docs-src/guides/upgrading/upgrade_4_6_memp_fput.md new file mode 100644 index 000000000..0ae8c5767 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_memp_fput.md @@ -0,0 +1,10 @@ +--- +title: "DB_MPOOLFILE->put" +api-name: "DB_MPOOLFILE->put" +source: docs/upgrading/upgrade_4_6_memp_fput.html +--- +## DB_MPOOLFILE-\>put + +The DB_MPOOLFILE->put() method takes a new parameter in the Berkeley DB 4.6 release, a page priority. This parameter allows applications to specify the page's priority when returning the page to the cache. + +Applications calling the DB_MPOOLFILE->put() method can upgrade by adding a DB_PRIORITY_UNCHANGED parameter to their calls to the DB_MPOOLFILE->put() method. This will result in no change in the application's behavior. diff --git a/docs-src/guides/upgrading/upgrade_4_6_memp_fset.md b/docs-src/guides/upgrading/upgrade_4_6_memp_fset.md new file mode 100644 index 000000000..02f8a6fab --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_memp_fset.md @@ -0,0 +1,8 @@ +--- +title: "B_MPOOLFILE->set" +api-name: "B_MPOOLFILE->set" +source: docs/upgrading/upgrade_4_6_memp_fset.html +--- +## B_MPOOLFILE-\>set + +The DB_MPOOLFILE-\>set method has been removed from the Berkeley DB 4.6 release. Applications calling this method can upgrade by removing all calls to the method. This will result in no change in the application's behavior. diff --git a/docs-src/guides/upgrading/upgrade_4_6_toc.md b/docs-src/guides/upgrading/upgrade_4_6_toc.md new file mode 100644 index 000000000..5837cfd4c --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_toc.md @@ -0,0 +1,82 @@ +--- +title: "Chapter 4. Upgrading Berkeley DB 4.5 applications to Berkeley DB 4.6" +api-name: "Chapter 4. Upgrading Berkeley DB 4.5 applications to Berkeley DB 4.6" +source: docs/upgrading/upgrade_4_6_toc.html +--- +## Chapter 4. Upgrading Berkeley DB 4.5 applications to Berkeley DB 4.6 + +**Table of Contents** + + [Introduction](upgrade_4_6_toc.md#upgrade_4_6_intro) + + [C API cursor handle method names](upgrade_4_6_cursor.md) + + [DB_MPOOLFILE-\>put](upgrade_4_6_memp_fput.md) + + [B_MPOOLFILE-\>set](upgrade_4_6_memp_fset.md) + + [Replication Events](upgrade_4_6_event.md) + + [DB_REP_FULL_ELECTION](upgrade_4_6_full_election.md) + + [Verbose Output](upgrade_4_6_verbose.md) + + [DB_VERB_REPLICATION](upgrade_4_6_verb.md) + + [Windows 9X](upgrade_4_6_win.md) + + [Upgrade Requirements](upgrade_4_6_disk.md) + + [Berkeley DB 4.6.21 Change Log](changelog_4_6.md) + + [4.6.21 Patches:](changelog_4_6.md#idp50449856) + + [4.6.19 Patches](changelog_4_6.md#idp50370888) + + [Database or Log File On-Disk Format Changes:](changelog_4_6.md#idp50361912) + + [New Features:](changelog_4_6.md#idp50454856) + + [Database Environment Changes:](changelog_4_6.md#idp50457960) + + [Concurrent Data Store Changes:](changelog_4_6.md#idp50459800) + + [General Access Method Changes:](changelog_4_6.md#idp50458344) + + [Btree Access Method Changes:](changelog_4_6.md#idp50475672) + + [Hash Access Method Changes:](changelog_4_6.md#idp50460536) + + [Queue Access Method Changes:](changelog_4_6.md#idp50444272) + + [Recno Access Method Changes:](changelog_4_6.md#idp50463616) + + [C++-specific API Changes:](changelog_4_6.md#idp50463872) + + [Java-specific API Changes:](changelog_4_6.md#idp50481800) + + [Java collections and bind API Changes:](changelog_4_6.md#idp50464456) + + [Tcl-specific API Changes:](changelog_4_6.md#idp50464944) + + [RPC-specific Client/Server Changes:](changelog_4_6.md#idp50465232) + + [Replication Changes:](changelog_4_6.md#idp50486584) + + [XA Resource Manager Changes:](changelog_4_6.md#idp50466136) + + [Locking Subsystem Changes:](changelog_4_6.md#idp50465496) + + [Logging Subsystem Changes:](changelog_4_6.md#idp50451848) + + [Memory Pool Subsystem Changes:](changelog_4_6.md#idp50452712) + + [Transaction Subsystem Changes:](changelog_4_6.md#idp50468064) + + [Utility Changes:](changelog_4_6.md#idp50475736) + + [Configuration, Documentation, Portability and Build Changes:](changelog_4_6.md#idp50479800) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.5 release interfaces to the Berkeley DB 4.6 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_4_6_verb.md b/docs-src/guides/upgrading/upgrade_4_6_verb.md new file mode 100644 index 000000000..055198d99 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_verb.md @@ -0,0 +1,8 @@ +--- +title: "DB_VERB_REPLICATION" +api-name: "DB_VERB_REPLICATION" +source: docs/upgrading/upgrade_4_6_verb.html +--- +## DB_VERB_REPLICATION + +The DB_VERB_REPLICATION flag no longer requires the Berkeley DB library be built with the --enable-diagnostic configuration option to output additional replication logging information. diff --git a/docs-src/guides/upgrading/upgrade_4_6_verbose.md b/docs-src/guides/upgrading/upgrade_4_6_verbose.md new file mode 100644 index 000000000..f9cd70a8a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_verbose.md @@ -0,0 +1,14 @@ +--- +title: "Verbose Output" +api-name: "Verbose Output" +source: docs/upgrading/upgrade_4_6_verbose.html +--- +## Verbose Output + +When an error occurs in the Berkeley DB library, an exception is thrown or an error return value is returned by the interface. In some cases, however, the exception or returned value may be insufficient to completely describe the cause of the error, especially during initial application debugging. Applications can configure Berkeley DB for verbose messages to be output when an error occurs, but it's a common cause of confusion for new users that no verbose messages are available by default. + +In the Berkeley DB 4.6 release, verbose messages are configured by default. For the C and C++ APIs, this means the default configuration when applications first create DB or DB_ENV handles is as if the DB_ENV->set_errfile() or DB->set_errfile() methods were called with the standard error output (stderr) specified as the FILE \* argument. Applications wanting no output at all can turn off this default configuration by calling the DB_ENV->set_errfile() or DB->set_errfile() methods with NULL as the FILE \* argument. Additionally, explicitly configuring the error output channel using any of the DB_ENV->set_errfile(), DB_ENV->set_errcall(), DbEnv::set_error_stream() or Db::set_error_stream() methods will also turn off this default output for the application. + +Applications which configure Berkeley DB with any error output channel should not require any changes. + +Applications which depend on having no output from the Berkeley DB library by default, should be changed to call the DB_ENV->set_errfile() or DB->set_errfile() methods with NULL as the FILE \* argument. diff --git a/docs-src/guides/upgrading/upgrade_4_6_win.md b/docs-src/guides/upgrading/upgrade_4_6_win.md new file mode 100644 index 000000000..4a673707d --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_6_win.md @@ -0,0 +1,8 @@ +--- +title: "Windows 9X" +api-name: "Windows 9X" +source: docs/upgrading/upgrade_4_6_win.html +--- +## Windows 9X + +Berkeley DB no longer supports process-shared database environments on Windows 9X platforms; the DB_PRIVATE flag must always be specified to the DB_ENV->open() method. diff --git a/docs-src/guides/upgrading/upgrade_4_7_disk.md b/docs-src/guides/upgrading/upgrade_4_7_disk.md new file mode 100644 index 000000000..334b2dce8 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_disk.md @@ -0,0 +1,14 @@ +--- +title: "Upgrade Requirements" +api-name: "Upgrade Requirements" +source: docs/upgrading/upgrade_4_7_disk.html +--- +## Upgrade Requirements + +The log file format changed in the Berkeley DB 4.7 release. + +No database formats changed in the Berkeley DB 4.7 release. + +The Berkeley DB 4.7 release does not support live replication upgrade from the 4.2 or 4.3 releases, only from the 4.4 and later releases. + +For further information on upgrading Berkeley DB installations, see Upgrading from previous versions of Berkeley DB . diff --git a/docs-src/guides/upgrading/upgrade_4_7_interdir.md b/docs-src/guides/upgrading/upgrade_4_7_interdir.md new file mode 100644 index 000000000..6c3ac3ed2 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_interdir.md @@ -0,0 +1,10 @@ +--- +title: "DB_ENV->set_intermediate_dir" +api-name: "DB_ENV->set_intermediate_dir" +source: docs/upgrading/upgrade_4_7_interdir.html +--- +## DB_ENV-\>set_intermediate_dir + +Historic releases of Berkeley DB contained an undocumented DB_ENV method named DB_ENV-\>set_intermediate_dir, which configured the creation of any intermediate directories needed during recovery. This method has been standardized as the DB_ENV->set_intermediate_dir_mode() method. + +Applications using DB_ENV-\>set_intermediate_dir should be modified to use the DB_ENV->set_intermediate_dir_mode() method instead. diff --git a/docs-src/guides/upgrading/upgrade_4_7_log.md b/docs-src/guides/upgrading/upgrade_4_7_log.md new file mode 100644 index 000000000..2ae131f2a --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_log.md @@ -0,0 +1,17 @@ +--- +title: "Log configuration" +api-name: "Log configuration" +source: docs/upgrading/upgrade_4_7_log.html +--- +## Log configuration + +In the Berkeley DB 4.7 release, the logging subsystem is configured using the DB_ENV->log_set_config() method instead of the previously used DB_ENV->set_flags() method. + +The DB_ENV->set_flags() method no longer accepts the flags DB_DIRECT_LOG, DB_DSYNC_LOG, DB_LOG_INMEMORY or DB_LOG_AUTOREMOVE. Applications should be modified to use the equivalent flags accepted by the DB_ENV->log_set_config() method. + +| Previous DB_ENV->set_flags() flag | Replacement DB_ENV->log_set_config() flag | +|----|----| +| DB_DIRECT_LOG | DB_LOG_DIRECT | +| DB_DSYNC_LOG | DB_LOG_DSYNC | +| DB_LOG_INMEMORY | DB_LOG_IN_MEMORY | +| DB_LOG_AUTOREMOVE | DB_LOG_AUTO_REMOVE | diff --git a/docs-src/guides/upgrading/upgrade_4_7_repapi.md b/docs-src/guides/upgrading/upgrade_4_7_repapi.md new file mode 100644 index 000000000..c886c8736 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_repapi.md @@ -0,0 +1,8 @@ +--- +title: "Replication API" +api-name: "Replication API" +source: docs/upgrading/upgrade_4_7_repapi.html +--- +## Replication API + +The Berkeley DB base replication API DB_ENV->rep_elect(), DB_ENV->rep_get_nsites() DB_ENV->rep_set_nsites(), DB_ENV->rep_get_priority() and DB_ENV->rep_set_priority() methods now take arguments of type u_int32_t rather than int. Applications may need to change the types of arguments to these methods, or cast arguments to these methods to avoid compiler warnings. diff --git a/docs-src/guides/upgrading/upgrade_4_7_rtc.md b/docs-src/guides/upgrading/upgrade_4_7_rtc.md new file mode 100644 index 000000000..1585cb3b6 --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_rtc.md @@ -0,0 +1,14 @@ +--- +title: "Run-time configuration" +api-name: "Run-time configuration" +source: docs/upgrading/upgrade_4_7_rtc.html +--- +## Run-time configuration + +In historic Berkeley DB releases, there were separate sleep and yield functions to be configured at run-time using the db_env_set_func_sleep and db_env_set_func_yield functions. These functions have been merged in the Berkeley DB 4.7 release. The replacement function should always yield the processor, and optionally wait for some period of time before allowing the thread to run again. + +Applications using the Berkeley DB run-time configuration interfaces should merge the functionality of their sleep and yield functions into a single configuration function. + +In the 4.7 Berkeley DB release, the db_env_set_func_map and db_env_set_func_unmap functions have been replaced. This change fixes problems where applications using the Berkeley DB run-time configuration interfaces could not open multiple DB_ENV class handles for the same database environment in a single application or join existing database environments from within multiple processes. + +Applications wanting to replace the Berkeley DB region creation functionality should replace their db_env_set_func_map and db_env_set_func_unmap calls with a call to the db_env_set_func_region_map function. Applications wanting to replace the Berkeley DB region file mapping functionality should replace their db_env_set_func_map and db_env_set_func_unmap calls with a call to the db_env_set_func_file_map function. diff --git a/docs-src/guides/upgrading/upgrade_4_7_tcl.md b/docs-src/guides/upgrading/upgrade_4_7_tcl.md new file mode 100644 index 000000000..b6cf8c24b --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_tcl.md @@ -0,0 +1,8 @@ +--- +title: "Tcl API" +api-name: "Tcl API" +source: docs/upgrading/upgrade_4_7_tcl.html +--- +## Tcl API + +The Berkeley DB Tcl API does not attempt to avoid evaluating input as Tcl commands. For this reason, it may be dangerous to pass unreviewed user input through the Berkeley DB Tcl API, as the input may subsequently be evaluated as a Tcl command. To minimize the effectiveness of a Tcl injection attack, the Berkeley DB Tcl API in the 4.7 release routine resets process' effective user and group IDs to the real user and group IDs. diff --git a/docs-src/guides/upgrading/upgrade_4_7_toc.md b/docs-src/guides/upgrading/upgrade_4_7_toc.md new file mode 100644 index 000000000..7aba5029f --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_4_7_toc.md @@ -0,0 +1,74 @@ +--- +title: "Chapter 3. Upgrading Berkeley DB 4.6 applications to Berkeley DB 4.7" +api-name: "Chapter 3. Upgrading Berkeley DB 4.6 applications to Berkeley DB 4.7" +source: docs/upgrading/upgrade_4_7_toc.html +--- +## Chapter 3. Upgrading Berkeley DB 4.6 applications to Berkeley DB 4.7 + +**Table of Contents** + + [Introduction](upgrade_4_7_toc.md#upgrade_4_7_intro) + + [Run-time configuration](upgrade_4_7_rtc.md) + + [Replication API](upgrade_4_7_repapi.md) + + [Tcl API](upgrade_4_7_tcl.md) + + [DB_ENV-\>set_intermediate_dir](upgrade_4_7_interdir.md) + + [Log configuration](upgrade_4_7_log.md) + + [Upgrade Requirements](upgrade_4_7_disk.md) + + [Berkeley DB 4.7.25 Change Log](changelog_4_7.md) + + [Database or Log File On-Disk Format Changes:](changelog_4_7.md#idp50357648) + + [New Features:](changelog_4_7.md#idp50378912) + + [Database Environment Changes:](changelog_4_7.md#idp50380752) + + [Concurrent Data Store Changes:](changelog_4_7.md#idp50382592) + + [General Access Method Changes:](changelog_4_7.md#idp50381120) + + [Btree Access Method Changes:](changelog_4_7.md#idp50391248) + + [Hash Access Method Changes:](changelog_4_7.md#idp50365280) + + [Queue Access Method Changes:](changelog_4_7.md#idp50355136) + + [Recno Access Method Changes:](changelog_4_7.md#idp50346288) + + [C-specific API Changes:](changelog_4_7.md#idp50346816) + + [Java-specific API Changes:](changelog_4_7.md#idp50347072) + + [Direct Persistence Layer (DPL), Bindings and Collections API:](changelog_4_7.md#idp50347296) + + [Tcl-specific API Changes:](changelog_4_7.md#idp50386200) + + [RPC-specific Client/Server Changes:](changelog_4_7.md#idp50395072) + + [Replication Changes:](changelog_4_7.md#idp50395328) + + [XA Resource Manager Changes:](changelog_4_7.md#idp50391504) + + [Locking Subsystem Changes:](changelog_4_7.md#idp50357904) + + [Logging Subsystem Changes:](changelog_4_7.md#idp50397528) + + [Memory Pool Subsystem Changes:](changelog_4_7.md#idp50385512) + + [Mutex Subsystem Changes:](changelog_4_7.md#idp50386616) + + [Transaction Subsystem Changes:](changelog_4_7.md#idp50391632) + + [Utility Changes:](changelog_4_7.md#idp50396200) + + [Configuration, Documentation, Sample Application, Portability and Build Changes:](changelog_4_7.md#idp50412288) + +## Introduction + +The following pages describe how to upgrade applications coded against the Berkeley DB 4.6 release interfaces to the Berkeley DB 4.7 release interfaces. This information does not describe how to upgrade Berkeley DB 1.85 release applications. diff --git a/docs-src/guides/upgrading/upgrade_process.md b/docs-src/guides/upgrading/upgrade_process.md new file mode 100644 index 000000000..3b6d3c83b --- /dev/null +++ b/docs-src/guides/upgrading/upgrade_process.md @@ -0,0 +1,110 @@ +--- +title: "Chapter 2.  Upgrading from previous versions of Berkeley DB" +api-name: "Chapter 2.  Upgrading from previous versions of Berkeley DB" +source: docs/upgrading/upgrade_process.html +--- +## Chapter 2.  Upgrading from previous versions of Berkeley DB + +The following information describes the general process of upgrading Berkeley DB installations. There are four areas to be considered when upgrading Berkeley DB applications and database environments: the application API, the database environment's region files, the underlying database formats, and, in the case of transactional database environments, the log files. The upgrade procedures required depend on whether or not the release is a major or minor release (in which either the major or minor number of the version changed), or a patch release (in which only the patch number in the version changed). Berkeley DB major and minor releases may optionally include changes in all four areas, that is, the application API, region files, database formats, and log files may not be backward-compatible with previous releases. + +Each Berkeley DB major or minor release described in this book has a chapter indicating how to upgrade to the new release. The chapter describes any API changes made in the release. Application maintainers should review the API changes and update their applications as necessary before recompiling with the new release. In addition, each chapter includes a section specifying whether the log file format or database formats changed in non-backward-compatible ways as part of the release. Because there are several underlying Berkeley DB database formats, and they do not all necessarily change in the same release, changes to a database format in a release may not affect any particular application. Further, database and log file formats may have changed but be entirely backward-compatible, in which case no upgrade will be necessary. + +A Berkeley DB patch release will never modify the API, regions, log files, or database formats in incompatible ways, and so applications need only be relinked (or, in the case of a shared library, pointed at the new version of the shared library) to upgrade to a new release. Note that internal Berkeley DB interfaces may change at any time and in any release (including patch releases) without warning. This means the library must be entirely recompiled and reinstalled when upgrading to new releases of the library because there is no guarantee that modules from one version of the library will interact correctly with modules from another release. We recommend using the same compiler release when building patch releases as was used to build the original release; in the default configuration, the Berkeley DB library shares data structures from underlying shared memory between threads of control, and should the compiler re-order fields or otherwise change those data structures between the two builds, errors may result. + +If the release is a patch release, do the following: + +1. Shut down the old version of the application. + +2. Install the new version of the application by relinking or installing a new version of the Berkeley DB shared library. + +3. Restart the application. + +Otherwise, if the application **does not** have a Berkeley DB transactional environment, the application may be installed in the field using the following steps: + +1. Shut down the old version of the application. + +2. Remove any Berkeley DB environment using the DB_ENV->remove() method or an appropriate system utility. + +3. Recompile and install the new version of the application. + +4. If necessary, upgrade the application's databases. See Database upgrade for more information. + +5. Restart the application. + +Otherwise, if the application has a Berkeley DB transactional environment, but neither the log file nor database formats need upgrading, the application may be installed in the field using the following steps: + +1. Shut down the old version of the application. + +2. Run recovery on the database environment using the DB_ENV->open() method or the db_recover utility. + +3. Remove any Berkeley DB environment using the DB_ENV->remove() method or an appropriate system utility. + +4. Recompile and install the new version of the application. + +5. Restart the application. + +If the application has a Berkeley DB transactional environment, and the log files need upgrading but the databases do not, the application may be installed in the field using the following steps: + +1. Shut down the old version of the application. + +2. Still using the old version of Berkeley DB, run recovery on the database environment using the DB_ENV->open() method, or the db_recover utility. + +3. If you used the DB_ENV->open() method to run recovery, make sure that the Berkeley DB environment is removed using the DB_ENV->remove() method or an appropriate system utility. + +4. Archive the database environment for catastrophic recovery using the `db_archive` utility as described in the Database and log file archival section in the *Berkeley DB Programmer's Reference Guide*. + +5. Recompile and install the new version of the application. + +6. Force a checkpoint using the DB_ENV->txn_checkpoint() method or the db_checkpoint utility. If you use the db_checkpoint utility, make sure to use the new version of the utility; that is, the version that came with the release of Berkeley DB to which you are upgrading. + + Note that forcing a checkpoint might result in warning messages about log files that are being skipped. This is normal, and can be safely ignored. + +7. Remove unnecessary log files from the environment using the `-d` option on the db_archive utility, or from an application which calls the DB_ENV->log_archive() method with the DB_ARCH_REMOVE flag. + + Note that removing log files in this way might result in warning messages about log files that are being skipped. This is normal, and can be safely ignored. + + Note that if you are upgrading a replicated application, then you should *not* perform this step until all of the replication sites have been upgraded to the current release level. If you run this site before all your sites are upgraded, then errors can occur in your replication activities because important version information might be lost. + +8. Restart the application. + +Otherwise, if the application has a Berkeley DB transactional environment and the databases need upgrading, the application may be installed in the field using the following steps: + +1. Shut down the old version of the application. + +2. Still using the old version of Berkeley DB, run recovery on the database environment using the DB_ENV->open() method, or the db_recover utility. + +3. If you used the DB_ENV->open() method to run recovery, make sure that the Berkeley DB environment is removed using the DB_ENV->remove() method or an appropriate system utility. + +4. Archive the database environment for catastrophic recovery using the `db_archive` utility as described in the Database and log file archival section in the *Berkeley DB Programmer's Reference Guide*. + +5. Recompile and install the new version of the application. + +6. Upgrade the application's databases. See Database upgrade for more information. + +7. Archive the database for catastrophic recovery again (using different media than before, of course). Note: This archival is not strictly necessary. However, if you have to perform catastrophic recovery after restarting the application, that recovery must be done based on the last archive you have made. If you make this second archive, you can use it as the basis of that catastrophic recovery. If you do not make this second archive, you have to use the archive you made in step 4 as the basis of your recovery, and you have to do a full upgrade on it before you can apply log files created after the upgrade to it. + +8. Force a checkpoint using the DB_ENV->txn_checkpoint() method or the db_checkpoint utility. If you use the db_checkpoint utility, make sure to use the new version of the utility; that is, the version that came with the release of Berkeley DB to which you are upgrading. + + Note that forcing a checkpoint might result in warning messages about log files that are being skipped. This is normal, and can be safely ignored. + +9. Remove unnecessary log files from the environment using the `-d` option on the db_archive utility, or from an application which calls the DB_ENV->log_archive() method with the DB_ARCH_REMOVE flag. + + Note that removing log files in this way might result in warning messages about log files that are being skipped. This is normal, and can be safely ignored. + + Note that if you are upgrading a replicated application, then you should *not* perform this step until all of the replication sites have been upgraded to the current release level. If you run this site before all your sites are upgraded, then errors can occur in your replication activities because important version information might be lost. + +10. Restart the application. + +Finally, Berkeley DB supports the live upgrade of a replication group, by allowing mixed version operation (replication sites running at the newer software version can inter-operate with older version sites). All client sites must be upgraded first; the master site must be upgraded last. In other words, at all times the master must be running the lowest version of Berkeley DB. To upgrade a replication group, you must: + +1. Bring all clients up to date with the master (that is, all clients must be brought up to the most current log record as measured by the master's log sequence number (LSN)). + +2. Perform the upgrade procedures described previously on each of the individual database environments that are part of the replication group. Each individual client may be upgraded and restarted to join the replication group. + +3. Shut down the master site and upgrade that site last. + +During live replication upgrade, while sites are running at different versions, adding new (empty) clients to the replication group is not allowed. Those empty client environments must be added after the entire group is upgraded. + +Also, all removal of log files must be suspended throughout this entire procedure, so that there is no chance of a client needing internal initialization. + +Alternatively, it may be simpler to discard the contents of all of the client database environments, upgrade the master database environment, and then re-add all of the clients to the replication group using the standard replication procedures for new sites.