Config's `log_options` resets after restart

[chelog]

Hello! I am trying to change logging driver for Docker containers to use Loki with following property value:

docker:
  // ... other options
  log_config:
    driver: loki
    config:
      loki-url: http://127.0.0.1:3100/loki/api/v1/push
      loki-retries: 5
      loki-batch-size: 400

But right after I restart the Wings instance it immediately populates config section with default properties which break integration:

  log_config:
    type: loki
    config:
      compress: "false"
      loki-batch-size: "400"
      loki-retries: "5"
      loki-url: http://127.0.0.1:3100/loki/api/v1/push
      max-file: "1"
      max-size: 5m
      mode: non-blocking

I think it has to do something with setting defaults for the config file and opened a PR matching the one from Pterodactyl Wings, however unfortunately one of Pelican contributors said that they eventually reverted the change due to breaking the config generation.

Is there a way to override logging driver with Wings config?

[rmartinoscar]

Hey loki isn't a supported logger by docker so it gets reverted.
https://github.com/moby/moby/tree/v28.2.1/daemon/logger

And even if it was you would have something more like this.

docker:
  // ... other options
  log_config:
    driver: loki
    config:
      options:
        loki-url: http://127.0.0.1:3100/loki/api/v1/push
        oki-retries: 5
        loki-batch-size: 400

[chelog]

Loki driver is an official plugin for Docker and works for rest of my infrastructure

You suggested the config that works for docker containers but I think Wings uses different config interface, because there are no type field in docker logging field and this looks like default local Docker logging driver alongside with its options

UPD: Thanks for pointing that out, I double-checked my config and I mistakenly set the options property instead of config. Updated the issue description, this still breaks the container startup due to excessive properties in logging config

[Mawiguk0]

Does pelican wings actually support changing the logging driver?
I tried some combinations for that config for a local running fluentd service but had no success. When starting the server with the panel it instantly errors our because it was not able to read the last line of log.

[Mawiguk0]

@chelog do you still have problems with another logging driver?

[chelog]

I thought that is what this section is made for, never been able to configure it though as I've only ever used the Loki one which is compatible with default local driver but additionally pushes the logs to a Loki instance

I am still experiencing the issue that is described in the first message - the config is populated with default values, and as far as I understand, this behavior is not intentional. I was not able to verify that this property is designed to change the logging driver but it strongly implies it due to the options being very similar to the logging section of docker compose file, and I would really love it to support that :)

[rmartinoscar]

Loki driver is an official plugin for Docker and works for rest of my infrastructure

Its Officially supported by Grafana not the docker team.

You suggested the config that works for docker containers but I think Wings uses different config interface, because there are no type field in docker logging field and this looks like default local Docker logging driver alongside with its options

type is driver

[Mawiguk0]

@rmartinoscar is pelican really capable of working with other drivers than local? Is this confirmed somewhere else?

[Mawiguk0]

🤔
Okay I tried to set up the fluentd logging driver/type whatsoever and get following result because of this exact problem @chelog is describing:

ERROR: [Oct 21 07:38:40.781] failed to handle server crash error=failed to start server after crash detection: environment/docker: failed to run pre-boot process: environment/docker: failed to create container: Error response from daemon: unknown log opt 'compress' for fluentd log driver server=58f2e1db-eb79-426f-8cdb-51f479ebea3f

Stacktrace:
Error response from daemon: unknown log opt 'compress' for fluentd log driver
environment/docker: failed to create container
github.com/pelican-dev/wings/environment/docker.(*Environment).Create
        github.com/pelican-dev/wings/environment/docker/container.go:273
github.com/pelican-dev/wings/environment/docker.(*Environment).OnBeforeStart
        github.com/pelican-dev/wings/environment/docker/power.go:41
github.com/pelican-dev/wings/environment/docker.(*Environment).Start
        github.com/pelican-dev/wings/environment/docker/power.go:101
github.com/pelican-dev/wings/server.(*Server).HandlePowerAction
        github.com/pelican-dev/wings/server/power.go:134
github.com/pelican-dev/wings/server.(*Server).handleServerCrash
        github.com/pelican-dev/wings/server/crash.go:106
github.com/pelican-dev/wings/server.(*Server).OnStateChange.func1
        github.com/pelican-dev/wings/server/server.go:388
runtime.goexit
        runtime/asm_amd64.s:1700

environment/docker: failed to run pre-boot process
failed to start server after crash detection
github.com/pelican-dev/wings/server.(*Server).handleServerCrash
        github.com/pelican-dev/wings/server/crash.go:106
github.com/pelican-dev/wings/server.(*Server).OnStateChange.func1
        github.com/pelican-dev/wings/server/server.go:388
runtime.goexit
        runtime/asm_amd64.s:1700

This is indeed a really messy approach of letting the config write defaults for a hardcoded logging driver. Fluentd is a well supported docker logging driver I guess!?

config.yml before restarting the wings service:

...
docker:
...
  log_config:
    type: fluentd
    config:
      fluentd-address: unix://run/fluentd.sock
...

config.yml after restarting the wings service:

...
docker:
...
  log_config:
    type: fluentd
    config:
      compress: "false"
      fluentd-address: unix://run/fluentd.sock
      max-file: "1"
      max-size: 5m
      mode: non-blocking
...

And in the logs, the Docker daemon unsurprisingly says:

Error response from daemon: unknown log opt 'compress' for fluentd log driver
Yes, it is indeed unknown. even to me it is unkown, since I never configured it.

Questions:
Is it (pelican+wings) even designed to work with a custom logging driver?
Why is ‘fixing’ the config even a thing? The idea of an application writing its own config seems completely unnecessary.

When I receive a good answer, I´m happy to dig into the code and open a PR to fix this - if wanted. 😸

[chelog]

Considering what @rmartinoscar said, this is what this section is designed for. The config's type field is here to choose a driver for docker containers and config is passed to config.options

So yes, this default options populating behavior is what pterodactyl/panel#5008 describing as not intended. I would gladly contribute to the project if I hadn't have to learn golang from zero 😅

[Mawiguk0]

I changed the code of config/config_docker.go so that it does not override the defaults in the section
before:

LogConfig struct {
		Type   string            `default:"local" json:"type" yaml:"type"`
		Config map[string]string `default:"{\"max-size\":\"5m\",\"max-file\":\"1\",\"compress\":\"false\",\"mode\":\"non-blocking\"}" json:"config" yaml:"config"`
	} `json:"log_config" yaml:"log_config"`

after:

	LogConfig struct {
		Type   string            `default:"local" json:"type" yaml:"type"`
		Config map[string]string `default:"{\"max-size\":\"5m\",\"max-file\":\"1\",\"compress\":\"false\",\"mode\":\"non-blocking\"}" json:"config" yaml:"config"`
	} `json:"log_config" yaml:"log_config"`
}

func (c DockerConfiguration) ContainerLogConfig() container.LogConfig {
	if c.LogConfig.Type == "" {
		return container.LogConfig{}
	}

	// Filter config options based on the logging driver type
	// The "local" driver has specific options that aren't compatible with other drivers
	config := make(map[string]string)
	for k, v := range c.LogConfig.Config {
		// Only include these options if using the "local" driver
		if c.LogConfig.Type != "local" {
			// Exclude local-specific options for non-local drivers like fluentd, json-file, etc.
			if k == "max-size" || k == "max-file" || k == "compress" || k == "mode" {
				continue
			}
		}
		config[k] = v
	}

I´ll open a pull request once I was able to verify it on our gameservers infrastructure @ sxlan.de