Possible output formats for integration

[devicieRK]

Hi David,
Love this project and am in the process of using it in anger.
Had a couple of thoughts that would be helpful for our use cases.

The tools give pretty much all the details we would need to do some complicated build items across out multi language repo but it can be tough to consume.

Any opinion on adding a JSON output (or similar) option to things like poly deps / poly libs / poly diff / poly info?

From this I can feed lists of changed / affected bricks when we do our build caching with jq or similar.
Another handy one would be an option for poly libs to return non-zero if anything does not match this could be a useful lint.

Regards,
Rob

[DavidVujic]

Hi @devicieRK,

The poly diff command has a --short option that is made for CI and automation. It will produce a list of the affected projects for changed bricks. This information can be used to determine what projects to build. Here's the docs.

Maybe that will help you?

The poly libs command should return with exit code 1 if there are missing libraries detected already. Or did I misunderstand something?

[devicieRK]

Thanks for the rapid response David.

Poly libs returns 1 if they are missing (and poly check I believe also). Ideally though I would like to (optionally) get an error code if the versions are mismatched, currently highlighted in the text but with an exit of 0

[DavidVujic]

Now I understand what you meant, thank you!

When adding this feature (the library versions table), I thought about returning an exit code 1 for version mismatch. But then I realized some projects might have different versions intentionally. And I don't want the tool to block that scenario. That's why I went for a visual output only.

I like your ideas, and will think about a way to run the commands with a strict option to return an error if the versions are not the same.

[DavidVujic]

The poly diff command also has a --bricks option, to list the changed bricks only. This can be combined with the --short option too.

But if you want to know which projects should be deployed in an automation process, I would recommend to use poly diff --short.

[DavidVujic]

I like your ideas about JSON, and will give it some thought! I would still recommend to use git tags in combination with the poly diff command and with the --short option specifically for automation. You will get a comma-separated string of project names and I believe that would cover a lot of automation scenarios.

Can you tell me more about how you would use the output?

[devicieRK]

For poly diff I think I am doing it wrong, but I have tried with a tag on the first commit in main and I can see the poly diff uses the command but no difs show.

At this point I actually need to get a list of all dependencies in a project for the way I am building atm.
I am using earthly that creates a build container and currently I copy all Bases and Components in to the container to make sure it works but this will get tough as it scales.
edit for clarity: So the build container is isolated to allow for consistency across build environments (local, ci etc) and all dependencies must be added to the container explicitly. each item that is added to the container can be cached to improve performance but will be re added if it has changes (this is based on the container layers). Currently I transfer the Project files then the bases and components folders in their entirety to allow for the build. What would be good would be to just pass over the base and components that the current project uses and as such a json/csv etc list of all deps would be useful. edit

I realise this particular case is an edge case that I can work around (some hacky script to extract from pyproject.toml comes to mind) so take this more as a general suggestion.

Thaks for all your efforts

[DavidVujic]

The diff command should report on the changes since a specific tag, that sounds like a bug in the tool or a tag pattern that is unexpected maybe?

In the workspace.toml, there is a section where you can specify the tag patterns the poly tool should use:

[tool.polylith.tag.patterns]
stable = "stable-*"
release = "v[0-9]*"

As an example, If you create a tag named stable-1, and change code during development, it should report on the diff compared to that tag pattern. To get the tags to the local machine I usually run git fetch --all (not sure if that is the proper way).

[DavidVujic]

About the Earthly setup, would it be possible to pass in your project as a wheel (or sdist) into the container? This is done by running the poetry build-project --directory projects/the-project command (if you use Poetry). The wheel contains all the needed bases and components for that project. You could install the wheel in the container, and it will fetch all the third-party dependencies from the wheel.

The "build" step is the recommended way to collect all the needed parts for a specific Polylith project.

I'm not an Earthly user, and might have misunderstood how things are set up with it.

[devicieRK]

Thanks again David,
I am using earthly to build the wheel and/or sdist as the main benefit is the reproducible environment. Essentially my python build and test environment is inside the build container, So I need to pass in everything it needs to create the wheel. As mentioned, I am sure I can worm around this in a few ways so not particularly concerned about this case.