aLIVE LIS — Livestock Information System

aLIVE LIS is an open data integration platform for the Ethiopian livestock sector. It connects disparate national livestock information systems, normalises their data to a shared Livestock Data Standard (LDS), and surfaces a unified view across five domains: population, genetics, production, health, and trade.

The platform exposes data through Apache Superset dashboards — a full-featured internal dashboard for analysts and a simplified public dashboard for citizen users.

---

Background

The foundation of aLIVE LIS was developed as part of the 2021 Livestock Information System Roadmap for Ethiopia, prepared by the Ethiopian Ministry of Agriculture (MoA), Livestock Improvement Corporation (LIC, NZ), the Alliance of Bioversity International and CIAT, and the Bill & Melinda Gates Foundation. The system architecture has since been designed to be reusable across countries.

---

Objectives

• Provide relevant, accurate, and timely digital data to key stakeholders
• Establish high interoperability between heterogeneous livestock databases
• Support research, commercial, and industrial activities
• Define requirements for data collection, validation, storage, and sharing
• Enable ownership by national ministries and cost-effective local operations
• Foster strong engagement with data suppliers and data users

---

Architecture Overview

aLIVE LIS is built on an event-streaming architecture. Source systems publish data into Apache Kafka; a set of stream-processing applications transform and standardise that data in real time before writing it to an Apache Iceberg data lake on object storage (Ozone). Apache Kyuubi provides SQL access to the lake, and Apache Superset serves the dashboards.

Source systems
     │
     ▼
Kafka Connect (source connectors + XML transformers)
     │
     ▼
Kafka Topics (raw)
     │
     ▼
Kafka Streams apps  ──►  Kafka Topics (LDS-normalised)
     │
     ▼
Spark / Iceberg data lake
     │
     ▼
Kyuubi (SQL gateway)  ──►  Superset dashboards

Key components in this repository

Module	Role
lis-streams-app	Kafka Streams app — transforms ET-LITS raw data into the Livestock Data Standard
ess-streams-app	Kafka Streams app — imports and processes ESS data
et-lits-xml-to-struct	Kafka Connect plugin — converts XML payloads to Kafka Connect Struct (FasterXML)
aadgg-connector	Kafka Connect plugin — sources data from the AADGG MySQL database
kafka-connect-extras	Kafka Connect plugin — general-purpose custom Single Message Transforms (SMTs)
et-lits-gen	Test data generator — produces synthetic ET-LITS XML records at scale
dashboard-data	Utility — Liquibase migrations and data load to PostgreSQL / Kyuubi
lis-monitor	Utility — Kafka Connect and Kubernetes metrics → email alerts
iceberg-maintainer	Utility — Apache Iceberg table maintenance via Hive JDBC
schema-migrations	Utility — registers Avro schemas with Apicurio Schema Registry
backup	Utility — backs up databases and files to SSH or S3
spark-cleaner	Utility — removes stale files generated by Spark jobs
charts/kyuubi	Helm chart — customised Kyuubi deployment for Kubernetes

Schema registry: Apicurio Registry (not Confluent). Serialisation uses Avro 1.11.4.

---

Integrated Source Systems

System	Description
ET-LITS	Livestock Identification and Traceability System
DOVAR2	Disease Outbreak & Vaccination Reporting System 2
ADNIS	Animal Disease Notification & Investigation System
AADGG	Africa Asia Dairy Genetic Gains
NLMIS	National Livestock Market Information System

---

Prerequisites

Tool	Version	Notes
JDK (Temurin)	11 (most modules), 21 (lis-monitor), 1.8 (iceberg-maintainer)	mise manages versions automatically
Maven	3.9+	mise manages the version automatically
Docker	20+	Required to build container images
mise	latest	Automatic tool-version switching per directory

Install mise and run mise install in the repository root; it will activate the correct Java and Maven versions for each module automatically.

---

Building

This is a mono-repo. Every module is an independent Maven project built from its own directory.

Build a single module

cd <module-directory>
mvn clean package          # compile, test, and package
mvn -DskipTests clean package  # skip tests (faster)

Build a Docker image

Modules that ship as Docker images include a build-image.sh (and optionally a build.sh) script:

cd lis-streams-app
./build.sh            # build the jar
./build-image.sh      # build and push the Docker image

Images are based on eclipse-temurin:11 and pushed to the project's private Docker registry.

Packaging formats

• Kafka Streams apps (lis-streams-app, ess-streams-app): tar.gz + Docker image
• Kafka Connect plugins (et-lits-xml-to-struct, aadgg-connector, kafka-connect-extras): tar.gz only (deployed to the Kafka Connect worker)
• Utilities (dashboard-data, lis-monitor, iceberg-maintainer, etc.): tar.gz + Docker image

---

Running the System

Running the full platform on Kubernetes is handled by the companion repository:

alive-lis-ops — Kustomize-based Kubernetes manifests and configuration for deploying all services.

Refer to that repository for environment configuration, secret management, and deployment instructions.

---

Release Process

Releases use a package-name@version tag convention (e.g. lis-monitor@1.0.1).

1. Update the version in the module's pom.xml and remove the -SNAPSHOT suffix.
2. Commit the version change (do not push yet).
3. Create a tag: git tag lis-monitor@1.0.1
4. Add a new commit that restores the -SNAPSHOT suffix with an incremented version.
5. Push commits and the tag.
6. Trigger the Jenkins pipeline for the new tag.

Windows users: keep line endings as LF. Files that end up inside Docker images will fail at runtime if CRLF line endings are introduced. Configure your editor and Git accordingly.

---

Contributing

See CONTRIBUTING.md for code organisation guidelines, tagging conventions, and the release workflow.

---

License

Distributed under the Apache License 2.0.

---

Contact

Development Gateway Inc. https://developmentgateway.org/contact/