Skip to content

Errno::EROFS thrown when run from a docker container with a RO filesystem #1464

Open
Open
Errno::EROFS thrown when run from a docker container with a RO filesystem#1464
Labels
agent-ruby
@smedrick

Description

@smedrick
smedrick
opened on Jul 2, 2024

Describe the bug

When running our Rails app on a docker container with a read-only Filesystem mount, we get

`/home/overlay` is not writable.
/usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:117:in `rescue in filesystem_access': There was an error accessing `/app`. (Bundler::GenericSystemCallError)
The underlying system error is Errno::EROFS: Read-only file system @ dir_s_mkdir - /app/bundler20240702-392135-36ubeq392135
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:102:in `filesystem_access'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:681:in `tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:241:in `user_home'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:40:in `default_bundle_dir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:459:in `default_bundle_dir'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:33:in `<class:StacktraceBuilder>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:25:in `<module:ElasticAPM>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:23:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/agent.rb:24:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm.rb:37:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic-apm.rb:21:in `<top (required)>'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `block in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:175:in `require'
	from /app/config/application.rb:15:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require_application!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands/console/console_command.rb:95:in `perform'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/base.rb:69:in `perform'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command.rb:46:in `invoke'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>'
/usr/share/ruby/tmpdir.rb:89:in `mkdir': Read-only file system @ dir_s_mkdir - /app/bundler20240702-392135-36ubeq392135 (Errno::EROFS)
	from /usr/share/ruby/tmpdir.rb:89:in `block in mktmpdir'
	from /usr/share/ruby/tmpdir.rb:135:in `create'
	from /usr/share/ruby/tmpdir.rb:87:in `mktmpdir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:315:in `tmp'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:682:in `block in tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:103:in `filesystem_access'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:681:in `tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:241:in `user_home'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:40:in `default_bundle_dir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:459:in `default_bundle_dir'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:33:in `<class:StacktraceBuilder>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:25:in `<module:ElasticAPM>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:23:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/agent.rb:24:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm.rb:37:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic-apm.rb:21:in `<top (required)>'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `block in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:175:in `require'
	from /app/config/application.rb:15:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require_application!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands/console/console_command.rb:95:in `perform'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/base.rb:69:in `perform'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command.rb:46:in `invoke'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>'

I suspect this is the commit that breaks compatibility with a RO FS:
23b4e4d

and browsing Issue history on the Bundler repo, it appears default_bundle_dir was not designed with the intent that it would be used outside of the install process.

Steps to reproduce

  1. Create Rails app with latest version of apm agent
  2. Build docker image with pre-bundled Rails app
  3. Deploy docker image to Kubernetes cluster using helm directive
        securityContext:
          readOnlyRootFilesystem: true

Expected behavior

Able to run bundle exec rails c or bundle exec rails s without encountering exception

Environment

  • OS: Red Hat Universal Base Image 8
  • Ruby version: 2.6.4
  • Framework and version: Rails 5.2.8, Bundler 2.2.33
  • APM Server version:
  • Agent version: 4.7.3

Additional context

Add any other context about the problem here.

  • Agent config options

    Click to expand 
      enabled: false
  log_path: log/elastic-apm-<%= ENV['HOSTNAME'] %>.log
  service_node_name: <%= ENV['HOSTNAME'] %>

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-ruby

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions