Merge pull request #617 from nmaludy/feature/json-jinja-filters

Kami · web-flow · commit 0193cad62ad6 · 2017-10-02T15:50:48.000+02:00
Implement new JSON Jinja filters
diff --git a/docs/source/datastore.rst b/docs/source/datastore.rst
@@ -19,7 +19,8 @@ Key-Value pairs can also have a TTL associated with them, for automatic expiry.
    simple and fully compatible with existing API and CLI commands.
 
    If you want to store a non-string value, you can store a JSON-serialized version, and then
-   de-serialize it in your action/sensor code.
+   de-serialize it in your action/sensor code, or using the ``from_json_string``
+   Jinja filter. See the :doc:`/reference/jinja` documentation for more details.
 
    This may change in future if there is sufficient interest.
 
diff --git a/docs/source/reference/jinja.rst b/docs/source/reference/jinja.rst
@@ -12,16 +12,29 @@ with Jinja in the context of |st2|. Please refer to the `Jinja docs
 Referencing Datastore Keys in Jinja
 ------------------------------------
 
-You can use ``{{st2kv.system.foo}}`` to access key ``foo`` from datastore. Note that until
-v2.1, the expression to access key ``foo`` from datastore used to be ``{{system.foo}}``
+You can use ``{{ st2kv.system.foo }}`` to access key ``foo`` from datastore. Note that until
+v2.1, the expression to access key ``foo`` from datastore used to be ``{{ system.foo }}``
 but is now deprecated, and the leading ``st2kv.`` namespace is required.
 
+Currently, all data in the datastore is represented as strings. To represent
+complext data structures like ``dicts`` and ``lists`` the standard approach is to
+convert the data structure into JSON when storing the data, then parse it when
+retrieving the data.
+
+.. code-block:: bash
+
+    # Pass the result of this expression to the action st2.kv.set
+    {{ {'complex': 'structure', 'foo': ['x', True]} | to_json_string }}
+
+    # Read the data back in using the st2kv and from_json_string filters
+    {{ st2kv.system.foo | from_json_string }}
+
 .. _jinja-jinja-filters:
 
 Applying Filters with Jinja
 ----------------------------
 
-To use a filter ``my_filter`` on ``foo``, you use the pipe operator, e.g.: ``{{foo | my_filter}}``.
+To use a filter ``my_filter`` on ``foo``, you use the pipe operator, e.g.: ``{{ foo | my_filter }}``.
 Please pay attention to the data type and available filters for each data type. Since Jinja is a
 text templating language, all your input is converted to text and then manipulations happen on that
 value. The necessary casting at the end is done by |st2| based on information you provide in YAML
@@ -64,11 +77,28 @@ function in Mistral workflows natively supports decryption via the ``decrypt`` p
 .. note::
 
     Because of a bug in Mistral, these filters do not currently support the "pipe" operator filter
-    format (`|`) So, instead of ``'{{ _.input_str | regex_match(_.regex_pattern)}}'`` you would
+    format (`|`) So, instead of ``'{{ _.input_str | regex_match(_.regex_pattern) }}'`` you would
     call the filter like a regular function, moving the previously input value into the first
-    positional argument position: ``'{{ regex_match(_.input_str, _.regex_pattern)}}'``. This will
+    positional argument position: ``'{{ regex_match(_.input_str, _.regex_pattern) }}'``. This will
     be addressed in a future release.
 
+from_json_string
+~~~~~~~~~~~~~~~~
+
+Converts a JSON string into an object or array (opposite of ``to_json_string``).
+
+.. code-block:: bash
+
+    {{ value_key | from_json_string }}
+
+from_yaml_string
+~~~~~~~~~~~~~~~~
+
+Converts a YAML string into an object or array (opposite of ``to_yaml_string``).
+
+.. code-block:: bash
+
+    {{ value_key | from_yaml_string }}
 
 json_escape
 ~~~~~~~~~~~
@@ -77,7 +107,76 @@ Adds escape characters to JSON strings.
 
 .. code-block:: bash
 
-    {{value_key | json_escape}}
+    {{ value_key | json_escape }}
+
+jsonpath_query
+~~~~~~~~~~~~~~
+
+Provides the ability to extract data from complex ``object`` data using the
+`JSONPath <http://goessner.net/articles/JsonPath/>` query language. More specifically
+we use the ``jsonpath-rw`` library that has its own extensions, details can be
+found on the `jsonpath-rw GitHub page <https://github.com/kennknowles/python-jsonpath-rw/tree/master/jsonpath_rw>`.
+Data passed into this function should be of type ``object`` or ``array``.
+The result of this function will either be an array of results, or None if the
+query did not return any results.
+If you would like to test out your JSONPath queries prior to utilizing this filter
+an online evaluator can be found `here <http://jsonpath.com/>`.
+
+.. code-block:: bash
+
+    # Access an element in a data structure. Each level is delimited by a '.'.
+    # Each part of the query is the name of the field in the current level
+    # of the data structure.
+    #
+    # input  = {'a': {'b': {'c': 1234} } }
+    # result = [1234]
+    {{ input | jsonpath_query('a.b.c') }}
+
+    # Access an index in an array/list
+    #
+    # input  = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'cow'] }
+    # result = ['rabbit']
+    {{ input | jsonpath_query('animals[1]') }}
+
+    # Access all indexes in an array/list
+    #
+    # input  = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'cow'] }
+    # result = ['bird', 'rabbit', 'cat', 'dog', 'cow']
+    {{ input | jsonpath_query('animals[*]') }}
+    
+    # Access a range/slice of indexes in an array/list.
+    # These expressions can be read mathematically as [first, last)
+    # Meaning that the index of the first element is inclusive, and the index
+    # of the last element is exclusive (will not be included).
+    #
+    # input  = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'sheep'] }
+    # result = ['rabbit', 'cat']
+    {{ input | jsonpath_query('animals[1:3]') }}
+
+    # If you leave out the first number in the range/slice operator
+    # it will start at the beginning implicitly. It can be read as:
+    # "give me all data from the beginning to the index specified"
+    #
+    # input  = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'sheep'] }
+    # result = ['bird', 'rabbit']
+    {{ input | jsonpath_query('animals[:2]') }}
+
+    # If you leave out the last number in the range/slice operator
+    # it will go all the way to the end of the array implicitly.
+    # It can be read as: "give me all data from the index specified to the end"
+    #
+    # input  = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'sheep'] }
+    # result = ['cat', 'dog', 'sheep']
+    {{ input | jsonpath_query('animals[2:]') }}
+
+    # Access a field within every element of an array.
+    #
+    # input  = {'people': [{'first': 'James', 'last': 'd'},
+    #                      {'first': 'Jacob', 'last': 'e'},
+    #                      {'first': 'Jayden', 'last': 'f'}]}
+    # result = ['James', 'Jacob', 'Jayden']
+    {{ input | jsonpath_query('people[*].first') }}
+
 
 regex_match
 ~~~~~~~~~~~
@@ -86,8 +185,8 @@ Search for the pattern at beginning of the string. Returns True if found, False
 
 .. code-block:: bash
 
-    {{value_key | regex_match('x')}}
-    {{value_key | regex_match("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$")}}
+    {{ value_key | regex_match('x') }}
+    {{ value_key | regex_match("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$") }}
 
 regex_replace
 ~~~~~~~~~~~~~
@@ -100,8 +199,8 @@ Replaces substring that matches pattern with provided replacement value (backref
 
 .. code-block:: bash
 
-    {{value_key | regex_replace("x", "y")}}
-    {{value_key | regex_replace("(blue|white|red)", "beautiful color \\\\1")}}
+    {{ value_key | regex_replace("x", "y") }}
+    {{ value_key | regex_replace("(blue|white|red)", "beautiful color \\\\1") }}
 
 regex_search
 ~~~~~~~~~~~~
@@ -110,8 +209,8 @@ Search for pattern anywhere in the string. Returns True if found, False if not.
 
 .. code-block:: bash
 
-    {{value_key | regex_search("y")}}
-    {{value_key | regex_search("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$")}}
+    {{ value_key | regex_search("y") }}
+    {{ value_key | regex_search("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$") }}
 
 regex_substring
 ~~~~~~~~~~~~~~~
@@ -121,8 +220,8 @@ Searches for the provided pattern in a string, and returns the first matched reg
 
 .. code-block:: bash
 
-    {{value_key | regex_substring("y")}}
-    {{value_key | regex_substring("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$")}}
+    {{ value_key | regex_substring("y") }}
+    {{ value_key | regex_substring("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$") }}
 
 to_complex
 ~~~~~~~~~~
@@ -131,7 +230,7 @@ Convert data to JSON string (see ``to_json_string`` for a more flexible option)
 
 .. code-block:: bash
 
-    {{value_key | to_complex}}
+    {{ value_key | to_complex }}
 
 to_human_time_from_seconds
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -140,7 +239,7 @@ Given time elapsed in seconds, this filter converts it to human readable form li
 
 .. code-block:: bash
 
-    {{ value_key | to_human_time_from_seconds}}
+    {{ value_key | to_human_time_from_seconds }}
 
 to_json_string
 ~~~~~~~~~~~~~~
@@ -149,7 +248,7 @@ Convert data to JSON string.
 
 .. code-block:: bash
 
-    {{value_key | to_json_string}}
+    {{ value_key | to_json_string }}
 
 to_yaml_string
 ~~~~~~~~~~~~~~
@@ -158,7 +257,7 @@ Convert data to YAML string.
 
 .. code-block:: bash
 
-    {{value_key | to_yaml_string}}
+    {{ value_key | to_yaml_string }}
 
 use_none
 ~~~~~~~~
@@ -167,7 +266,7 @@ If value being filtered is None, this filter will return the string ``%*****__%N
 
 .. code-block:: bash
 
-    {{value_key | use_none}}
+    {{ value_key | use_none }}
 
 version_bump_major
 ~~~~~~~~~~~~~~~~~~
@@ -176,7 +275,7 @@ Bumps up the major version of supplied version field.
 
 .. code-block:: bash
 
-    {{version | version_bump_major}}
+    {{ version | version_bump_major }}
 
 version_bump_minor
 ~~~~~~~~~~~~~~~~~~
@@ -185,7 +284,7 @@ Bumps up the minor version of supplied version field.
 
 .. code-block:: bash
 
-    {{version | version_bump_minor}}
+    {{ version | version_bump_minor }}
 
 version_bump_patch
 ~~~~~~~~~~~~~~~~~~
@@ -194,7 +293,7 @@ Bumps up the patch version of supplied version field.
 
 .. code-block:: bash
 
-    {{version | version_bump_patch}}
+    {{ version | version_bump_patch }}
 
 version_compare
 ~~~~~~~~~~~~~~~
@@ -204,7 +303,7 @@ Compare a semantic version to another value. Returns 1 if LHS is greater or -1 i
 
 .. code-block:: bash
 
-    {{version | version_compare("0.10.1")}}
+    {{ version | version_compare("0.10.1") }}
 
 version_equal
 ~~~~~~~~~~~~~
@@ -213,19 +312,19 @@ Returns True if LHS version is equal to RHS version.
 
 .. code-block:: bash
 
-    {{version | version_equal("0.10.0")}}
+    {{ version | version_equal("0.10.0")  }}
 
 version_less_than
 ~~~~~~~~~~~~~~~~~
 
 Returns True if LHS version is lesser than RHS version. Both inputs have to follow semantic version
 syntax.
 
-E.g. ``{{“1.6.0” | version_less_than("1.7.0")}}``.
+E.g. ``{{ “1.6.0” | version_less_than("1.7.0") }}``.
 
 .. code-block:: bash
 
-    {{version | version_less_than("0.9.2")}}
+    {{ version | version_less_than("0.9.2") }}
 
 version_match
 ~~~~~~~~~~~~~
@@ -237,19 +336,19 @@ Supports operators ``>``, ``<``, ``==``, ``<=``, and ``>=``.
 
 .. code-block:: bash
 
-    {{version | version_match(">0.10.0")}}
+    {{ version | version_match(">0.10.0") }}
 
 version_more_than
 ~~~~~~~~~~~~~~~~~
 
 Returns True if LHS version is greater than RHS version. Both inputs have to follow semantic
 version syntax.
 
-E.g. ``{{"1.6.0” | version_more_than("1.7.0")}}``.
+E.g. ``{{ "1.6.0” | version_more_than("1.7.0") }}``.
 
 .. code-block:: bash
 
-    {{version | version_more_than("0.10.1")}}
+    {{ version | version_more_than("0.10.1") }}
 
 version_strip_patch
 ~~~~~~~~~~~~~~~~~~~
@@ -258,4 +357,4 @@ Drops patch version of supplied version field.
 
 .. code-block:: bash
 
-    {{version | version_strip_patch}}
+    {{ version | version_strip_patch }}
