structures/design_notes.html at main · caltechlibrary/structures · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>structures</title>
    <link rel="stylesheet" href="/css/site.css">
</head>
<body>
<a href="#main-content" class="visually-hidden">skip to main content</a>
<nav>
<ul>
    <li><a href="/">All Library Apps</a></li>
    <li><a href="index.html">Home</a></li>
    <li><a href="LICENSE">LICENSE</a></li>
    <li><a href="INSTALL.html">INSTALL</a></li>
    <li><a href="user_manual.html">User Manual</a></li>
    <li><a href="about.html">About</a></li>
    <!-- <li><a href="search.html">Search</a></li> -->
    <li><a href="https://github.com/caltechlibrary/structures/issues">Code Repository</a></li>
</ul>
</nav>
<section id="main-content">
<h1
id="structures-a-go-module-for-working-with-structed-json-and-yaml-records">structures,
a Go module for working with structed JSON and YAML records</h1>
<p>This is a Go module for working with structured data. It is being
developed to enhance the dataset web service’s ability to validate
records submitted via the Web API. It does this through expressing the
valid structures accepted via a JSON or YAML expression.</p>
<p>Libraries and archives work with complex structured records. It is
common to encounter lists of objects in them, example list of author,
contributor and editor objects or lists of files associated with a
record and lists of identifiers or dates. Many of these types of lists
have objects with attributes with very specific meanings, like ORCID,
ROR, ISNI, ISBN, ISSN, etc.</p>
<p>HTML5 supports a much richer web form model out of the box than prior
version of HTML. The records that are expressable in plain HTML5 web
forms are flat. There is no clear way to expression an map like object
or a list like object. In practice this is solvable by overlaying a Web
Component holding over a textarea and saving JSON, YAML or CSV data in
the text editor the Web Component wraps. In the old days you’d use a
framework for this but today with vanilla JavaScript and Web Components
the trick is easily done.</p>
<p>What is missing is an easy way to express the resulting structured
data of the web form (overloading the textarea from complex field types)
and the server that is receiving the form submission. I think this can
be solved by providing a JSON or YAML schema to represent this.</p>
<p>Doing so allows you to solve several problems at once. First the
schema expression can be added to the datasetd web service providing a
full service object storage API because it will no longer require
someone to write a validation middleware service. This gives us a simple
turn key back end system to work with.</p>
<p>The second is generating a UI from the defined schema. If the
structure is known and a suitable library of web components is also
known then the forms for curating records are relatively easy to
generate which allows the Web UI to be largely static. This keeps things
both simple and robust (static web services are easy to defend). It
integratesd well in to the environments used in libraries and archives
because the authentication and authorization layers can be deferred to
single sign-on systems like Shibboleth and a front end web service like
Apache 2 or NginX.</p>
<p>Their is a their opportunity presented. Up to now many libraries and
archives rely on third party services for simple form hosting and
submissions. Some of these systems like WuFoo forms, Google Forms,
Libguide’s form support have been around for years. If you had a form
builder that could output a record schema usable by a service like
datasetd then those forms could be kept in house avoiding exposures to
third party systems and better integrated at the eadge in a manner more
consistant with institutional or orgainzational policies.</p>
<p>The structures module in Go seeks to enable this for services and
processing programs written in Go.</p>
<h2 id="type-support">Type support</h2>
<ul>
<li>JSON data types</li>
<li>ALL HTML5 web form types</li>
<li>All identifier types support by packages like <a
href="https://github.com/caltechlibrary/metadatatools">metadatatools</a>
(a Go module inspired by Python’s IdUtils)</li>
<li>Aproximate dates (dates that are partial or fuzzy like cerca
2022)</li>
</ul>
<p>These form the basic data types common to many archival and library
records. With them we can express records used by CrossRef, DataCite,
RDM, AchivesSpace and EPrints. This facities moving data easily between
systems as well as custom systems written by development groups in
libraries and archives.</p>
<h2
id="structures-provides-both-web-and-cli-scheme-design-tools">Structures
provides both Web and cli scheme design tools</h2>
<p>At some point you need to design schemas. For simply schemas working
directly in YAMl isn’t too difficult. For more complex ones it becomes
challenges. You need a way to easily buildthis out. It would be nice to
do so as easily as it is to use a form builder. The structures module
provides a cli for doing this.</p>
</section>
</body>
</html>