Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
- 'project/build.properties'
- 'msgpack-core/**'
- 'msgpack-jackson/**'
- 'msgpack-jackson3/**'
docs:
- '**.md'
- '**.txt'
Expand Down Expand Up @@ -81,8 +82,20 @@ jobs:
- name: Test
env:
TEST_JAVA_HOME: ${{ steps.target-jdk.outputs.path }}
run: ./sbt test
run: |
# msgpack-jackson3 (Jackson 3) requires JDK 17+; older lanes test the
# Java 8 compatible modules only
if [[ ${{ matrix.java }} -lt 17 ]]; then
./sbt msgpack-core/test msgpack-jackson/test
else
./sbt test
fi
- name: Universal Buffer Test
env:
TEST_JAVA_HOME: ${{ steps.target-jdk.outputs.path }}
run: ./sbt test -J-Dmsgpack.universal-buffer=true
run: |
if [[ ${{ matrix.java }} -lt 17 ]]; then
./sbt msgpack-core/test msgpack-jackson/test -J-Dmsgpack.universal-buffer=true
else
./sbt test -J-Dmsgpack.universal-buffer=true
fi
12 changes: 11 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,17 @@ jobs:
PGP_PASSPHRASE: ${{ secrets.PGP_PASSPHRASE }}
TEST_JAVA_HOME: ${{ steps.jdk8.outputs.path }}
run: |
./sbt publishSigned
./sbt msgpack-core/publishSigned msgpack-jackson/publishSigned
# msgpack-jackson3 (Jackson 3) requires JDK 17+
- uses: actions/setup-java@v5
with:
java-version: 17
distribution: temurin
- name: Build bundle for msgpack-jackson3 (Jackson 3)
env:
PGP_PASSPHRASE: ${{ secrets.PGP_PASSPHRASE }}
run: |
./sbt msgpack-jackson3/publishSigned
- name: Release to Sonatype
env:
SONATYPE_USERNAME: '${{ secrets.SONATYPE_USERNAME }}'
Expand Down
10 changes: 6 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

MessagePack-Java is a binary serialization library that provides a fast and compact alternative to JSON. The project consists of two main modules:
- **msgpack-core**: Standalone MessagePack implementation with no external dependencies
- **msgpack-jackson**: Jackson integration for object mapping capabilities
- **msgpack-jackson**: Jackson 2.x integration in maintenance mode (`org.msgpack:jackson-dataformat-msgpack`, Java 8+)
- **msgpack-jackson3**: Jackson 3.x integration for object mapping capabilities (`org.msgpack.jackson3:jackson-dataformat-msgpack`, Java 17+)

## Essential Development Commands

Expand Down Expand Up @@ -49,7 +50,8 @@ The main entry point is the `MessagePack` factory class which creates:

Key locations:
- Core interfaces: `msgpack-core/src/main/java/org/msgpack/core/`
- Jackson integration: `msgpack-jackson/src/main/java/org/msgpack/jackson/dataformat/`
- Jackson 2 integration: `msgpack-jackson/src/main/java/org/msgpack/jackson/dataformat/`
- Jackson 3 integration: `msgpack-jackson3/src/main/java/org/msgpack/jackson3/dataformat/`

### Buffer Management System
MessagePack uses an efficient buffer abstraction layer:
Expand All @@ -68,8 +70,8 @@ The msgpack-jackson module provides:
### Testing Structure
- **msgpack-core tests**: Written in Scala (always use the latest Scala 3 version) using AirSpec framework
- Location: `msgpack-core/src/test/scala/`
- **msgpack-jackson tests**: Written in Java using JUnit
- Location: `msgpack-jackson/src/test/java/`
- **msgpack-jackson / msgpack-jackson3 tests**: Written in Java using JUnit
- Location: `msgpack-jackson/src/test/java/`, `msgpack-jackson3/src/test/java/`

## Important JVM Options

Expand Down
15 changes: 12 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,16 @@ For using DirectByteBuffer (off-heap memory access methods) in JDK17, you need t

### Integration with Jackson ObjectMapper (jackson-databind)

msgpack-java supports serialization and deserialization of Java objects through [jackson-databind](https://github.com/FasterXML/jackson-databind).
For details, see [msgpack-jackson/README.md](https://github.com/msgpack/msgpack-java/blob/develop/msgpack-jackson/README.md). The template-based serialization mechanism used in v06 is deprecated.
msgpack-java supports serialization and deserialization of Java objects through [jackson-databind](https://github.com/FasterXML/jackson-databind). The template-based serialization mechanism used in v06 is deprecated.

Two artifacts are published depending on which Jackson major version your project uses:

| Jackson version | groupId | artifactId | Java package | Requirements |
| --- | --- | --- | --- | --- |
| Jackson 3.x | `org.msgpack.jackson3` | [`jackson-dataformat-msgpack`](https://github.com/msgpack/msgpack-java/blob/main/msgpack-jackson3/README.md) | `org.msgpack.jackson3.dataformat` | Java 17+ |
| Jackson 2.x | `org.msgpack` | [`jackson-dataformat-msgpack`](https://github.com/msgpack/msgpack-java/blob/main/msgpack-jackson/README.md) (maintenance mode) | `org.msgpack.jackson.dataformat` | Java 8+ |

The Jackson 2.x artifact keeps the same Maven coordinates and Java package as before, so existing users need no changes. The Jackson 3.x support is published under a new groupId (`org.msgpack.jackson3`) with a new Java package, following the same convention as Jackson itself (`com.fasterxml.jackson.dataformat` → `tools.jackson.dataformat`); renaming the Maven coordinates and the Java package together allows both artifacts to coexist on the same classpath for incremental migration. See each module's README for install instructions and usage details.

- [Release Notes](https://github.com/msgpack/msgpack-java/blob/develop/RELEASE_NOTES.md)

Expand Down Expand Up @@ -145,5 +153,6 @@ If some sporadic error happens (e.g., Sonatype timeout), rerun `sonaRelease` aga

```
msgpack-core # Contains packer/unpacker implementation that never uses third-party libraries
msgpack-jackson # Contains jackson-dataformat-java implementation
msgpack-jackson # org.msgpack:jackson-dataformat-msgpack: Jackson 2.x integration (maintenance mode)
msgpack-jackson3 # org.msgpack.jackson3:jackson-dataformat-msgpack: Jackson 3.x integration (Java 17+)
```
80 changes: 75 additions & 5 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,22 @@ val buildSettings = Seq[Setting[?]](
Test / compile := Def.uncached((Test / compile).dependsOn(Test / jcheckStyle).value)
)

val junitJupiter = "org.junit.jupiter" % "junit-jupiter" % "5.14.4" % "test"
val junitVintage = "org.junit.vintage" % "junit-vintage-engine" % "5.14.4" % "test"
val junitJupiter = "org.junit.jupiter" % "junit-jupiter" % "5.14.4" % "test"
val junitVintage = "org.junit.vintage" % "junit-vintage-engine" % "5.14.4" % "test"
val junitInterface = "com.github.sbt" % "junit-interface" % "0.13.3" % "test"

// Project settings
val isJava17Plus: Boolean = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is meant to skip msgpack-jackson3, which compiles with --release 17, when the target JDK is older. But java.specification.version is the JDK running sbt, not the one compiling. Since #998 those are two different JDKs whenever TEST_JAVA_HOME is set, and sbt 2 requires its own to be 17+, so this is always true and the module is never skipped.

Before #998 there was no TEST_JAVA_HOME and one JDK did both, so it worked.

TEST_JAVA_HOME=<jdk8> ./sbt clean compile fails with javac: invalid flag: --release. The clean matters, stale classes hide it.

Minor: the comment says "the Jackson 3 module (msgpack-jackson)", which is the Jackson 2 module now.

val v = sys.props.getOrElse("java.specification.version", "1.8")
// getOrElse(false): non-numeric versions (e.g. early-access "17-ea") fail safe
// by not compiling the Jackson 3 module (msgpack-jackson) rather than making an
// optimistic guess.
if (v.startsWith("1."))
false
else
scala.util.Try(v.toInt >= 17).getOrElse(false)
}

lazy val root = Project(id = "msgpack-java", base = file("."))
.settings(
buildSettings,
Expand All @@ -127,7 +139,14 @@ lazy val root = Project(id = "msgpack-java", base = file("."))
publish := {},
publishLocal := {}
)
.aggregate(msgpackCore, msgpackJackson)
.aggregate(
Seq[ProjectReference](msgpackCore, msgpackJackson) ++ (
if (isJava17Plus)
Seq[ProjectReference](msgpackJackson3, msgpackJackson3Benchmark)
else
Nil
): _*
)

lazy val msgpackCore = Project(id = "msgpack-core", base = file("msgpack-core"))
.enablePlugins(SbtOsgi)
Expand Down Expand Up @@ -170,21 +189,72 @@ lazy val msgpackCore = Project(id = "msgpack-core", base = file("msgpack-core"))
)
)

// Jackson 2.x module. Keeps the same Maven coordinates (org.msgpack:jackson-dataformat-msgpack)
// and Java package as the 0.9.x line, so existing Jackson 2.x users need no changes.
// Maintenance mode: Jackson 2.x dependency bumps and bug fixes only.
lazy val msgpackJackson = Project(id = "msgpack-jackson", base = file("msgpack-jackson"))
.enablePlugins(SbtOsgi)
.settings(
buildSettings,
name := "jackson-dataformat-msgpack",
description := "Jackson extension that adds support for MessagePack",
description := "Jackson 2.x extension that adds support for MessagePack",
OsgiKeys.bundleSymbolicName := "org.msgpack.msgpack-jackson",
OsgiKeys.exportPackage := Seq("org.msgpack.jackson", "org.msgpack.jackson.dataformat"),
libraryDependencies ++=
Seq(
"com.fasterxml.jackson.core" % "jackson-databind" % "2.22.1",
junitJupiter,
junitVintage,
"org.apache.commons" % "commons-math3" % "3.6.1" % "test"
"com.github.sbt.junit" % "jupiter-interface" % JupiterKeys.jupiterVersion.value % "test",
"org.apache.commons" % "commons-math3" % "3.6.1" % "test"
),
testOptions += Tests.Argument(TestFrameworks.JUnit, "-v")
)
.dependsOn(msgpackCore)

// Jackson 3.x module. Published under a new groupId (org.msgpack.jackson3) with a new
// Java package (org.msgpack.jackson3.dataformat), following JLBP-6: the Maven coordinates
// and the Java package are renamed together, so this artifact can coexist on the same
// classpath with the Jackson 2.x artifact (org.msgpack:jackson-dataformat-msgpack).
lazy val msgpackJackson3 = Project(id = "msgpack-jackson3", base = file("msgpack-jackson3"))
.enablePlugins(SbtOsgi)
.settings(
buildSettings,
organization := "org.msgpack.jackson3",
name := "jackson-dataformat-msgpack",
// sbt derives build output paths from moduleName, which would collide with the
// Jackson 2 module publishing the same artifactId; use the project id instead
outputPath := s"${platform.value}/u/msgpack-jackson3",
description := "Jackson 3.x extension that adds support for MessagePack",
OsgiKeys.bundleSymbolicName := "org.msgpack.jackson3.jackson-dataformat-msgpack",
OsgiKeys.exportPackage := Seq("org.msgpack.jackson3", "org.msgpack.jackson3.dataformat"),
OsgiKeys.importPackage := Seq("!android.os", "!sun.*"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bnd drops any referenced package that matches nothing in this list, so with only exclusions the bundle gets no Import-Package header at all and can't resolve tools.jackson.* in OSGi. msgpack-jackson doesn't set this key and gets a correct header.

msgpack-jackson3 references neither sun.misc nor android.os, so the line can just go. My bug from #987.

Suggested change
OsgiKeys.importPackage := Seq("!android.os", "!sun.*"),

Test / fork := true,
javacOptions := Seq("--release", "17"),
doc / javacOptions := Seq("--release", "17", "-Xdoclint:none"),
libraryDependencies ++=
Seq("tools.jackson.core" % "jackson-databind" % "3.1.2", junitInterface),
testOptions += Tests.Argument(TestFrameworks.JUnit, "-v")
)
.dependsOn(msgpackCore)

// JMH benchmarks for the Jackson 3.x module. Kept in a separate, unpublished project as
// sbt-jmh recommends: JmhPlugin adds jmh-core and the code generators to
// libraryDependencies unscoped, so enabling it on a published module would leak them into
// that module's POM as compile dependencies of every consumer.
lazy val msgpackJackson3Benchmark = Project(
id = "msgpack-jackson3-benchmark",
base = file("msgpack-jackson3-benchmark")
).enablePlugins(JmhPlugin)
.settings(
buildSettings,
description := "JMH benchmarks for the Jackson 3.x MessagePack integration",
publish / skip := true,
javacOptions := Seq("--release", "17"),
Jmh / javaOptions ++=
Seq(
"--add-opens=java.base/java.nio=ALL-UNNAMED",
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
)
)
.dependsOn(msgpackJackson3)
4 changes: 3 additions & 1 deletion msgpack-jackson/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.msgpack/jackson-dataformat-msgpack/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.msgpack/jackson-dataformat-msgpack/)
[![Javadoc](https://www.javadoc.io/badge/org.msgpack/jackson-dataformat-msgpack.svg)](https://www.javadoc.io/doc/org.msgpack/jackson-dataformat-msgpack)

This Jackson extension library is a component to easily read and write [MessagePack](http://msgpack.org/) encoded data through jackson-databind API.
This Jackson 2.x extension library is a component to easily read and write [MessagePack](http://msgpack.org/) encoded data through jackson-databind API.

**Maintenance mode:** this module keeps the same Maven coordinates (`org.msgpack:jackson-dataformat-msgpack`) and Java package (`org.msgpack.jackson.dataformat`) as before, so existing Jackson 2.x users need no changes. It receives Jackson 2.x dependency bumps and bug fixes only. For Jackson 3.x, use [`org.msgpack.jackson3:jackson-dataformat-msgpack`](../msgpack-jackson3/) (Java 17+), which uses a different groupId and Java package so both artifacts can coexist on the same classpath.

It extends standard Jackson streaming API (`JsonFactory`, `JsonParser`, `JsonGenerator`), and as such works seamlessly with all the higher level data abstractions (data binding, tree model, and pluggable extensions). For the details of Jackson-annotations, please see https://github.com/FasterXML/jackson-annotations.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -585,13 +585,14 @@ public JsonStreamContext getParsingContext()
@Override
public JsonLocation currentTokenLocation()
{
return new JsonLocation(ioContext.contentReference(), tokenPosition, -1, -1);
// Byte offset is exposed as columnNr since JsonLocation has no field for it
return new JsonLocation(ioContext.contentReference(), tokenPosition, -1, -1, (int) tokenPosition);
}

@Override
public JsonLocation currentLocation()
{
return new JsonLocation(ioContext.contentReference(), currentPosition, -1, -1);
return new JsonLocation(ioContext.contentReference(), currentPosition, -1, -1, (int) currentPosition);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.After;
import org.junit.Before;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
Expand Down Expand Up @@ -49,7 +49,7 @@ public class MessagePackDataformatTestBase
protected TinyPojo tinyPojo;
protected ComplexPojo complexPojo;

@Before
@BeforeEach
public void setup()
{
factory = new MessagePackFactory();
Expand Down Expand Up @@ -100,7 +100,7 @@ public void setup()
complexPojo.values = Arrays.asList("one", "two", "three");
}

@After
@AfterEach
public void teardown()
{
if (in != null) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
//
// MessagePack for Java
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
package org.msgpack.jackson3.dataformat.benchmark;

import org.msgpack.jackson3.dataformat.MessagePackFactory;
import org.msgpack.jackson3.dataformat.MessagePackMapper;
import org.msgpack.jackson3.dataformat.benchmark.model.MediaItem;
import org.msgpack.jackson3.dataformat.benchmark.model.MediaItems;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import tools.jackson.databind.ObjectMapper;
import tools.jackson.databind.json.JsonMapper;

@State(Scope.Thread)
public class BenchmarkState
{
public final ObjectMapper msgpackMapper = MessagePackMapper.builder(new MessagePackFactory()).build();
public final ObjectMapper jsonMapper = JsonMapper.builder().build();

public final byte[] msgpackBytes;
public final byte[] jsonBytes;

public BenchmarkState()
{
try {
MediaItem item = MediaItems.stdMediaItem();
msgpackBytes = msgpackMapper.writeValueAsBytes(item);
jsonBytes = jsonMapper.writeValueAsBytes(item);
}
catch (Exception e) {
throw new RuntimeException(e);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
//
// MessagePack for Java
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
package org.msgpack.jackson3.dataformat.benchmark;

import org.msgpack.jackson3.dataformat.benchmark.model.MediaItem;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.SECONDS)
@State(Scope.Thread)
@Fork(2)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 5, time = 1)
public class MsgpackReadBenchmark
{
private final BenchmarkState state = new BenchmarkState();

@Benchmark
public Object readPojoMsgpack() throws Exception
{
return state.msgpackMapper.readValue(state.msgpackBytes, MediaItem.class);
}

@Benchmark
public Object readPojoJson() throws Exception
{
return state.jsonMapper.readValue(state.jsonBytes, MediaItem.class);
}
}
Loading
Loading