Runned RAILS_ENV=production script/delayed_job status in live server. Returns delayed_job: running [pid 17880]
Probably method to apply into rails console on production: RAILS_ENV=production script/delayed_job restart
Important: I don't how this command would affect DB records.
Referential info:
when :working
'locked_at is not null'
when :failed
'last_error is not null'
when :pending
'attempts = 0'
end
- Enqueued would be the total number of delayed jobs, i.e. Delayed::Job.count
- Working jobs are those that have been locked by the delayed_job process and are currently being worked.
- Failed are those that have a last_error
- Pending are those jobs that have never been attempted.
Recently there are delayed jobs with this failure message:
PG::UnableToSend: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: SELECT "activities".* FROM "activities" WHERE "activities"."id" = $1 LIMIT $2
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:625:in `exec_prepared'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:625:in `block (2 levels) in exec_cache'
/data/app/vendor/bundle/ruby/2.3.0/gems/activesupport-5.1.4/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'
/data/app/vendor/bundle/ruby/2.3.0/gems/activesupport-5.1.4/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'
/data/app/vendor/bundle/ruby/2.3.0/gems/activesupport-5.1.4/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:624:in `block in exec_cache'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract_adapter.rb:612:in `block (2 levels) in log'
/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract_adapter.rb:611:in `block in log'
/data/app/vendor/bundle/ruby/2.3.0/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal/hooks/active_support_notifications.rb:36:in `instrument'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract_adapter.rb:603:in `log'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:623:in `exec_cache'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `execute_and_clear'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_query'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/querying.rb:39:in `find_by_sql'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation.rb:667:in `exec_queries'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation.rb:546:in `load'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation.rb:255:in `records'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:518:in `find_take'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:100:in `take'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:468:in `find_one'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:450:in `find_with_ids'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/relation/finder_methods.rb:66:in `find'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/querying.rb:3:in `find'
/data/app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.1.4/lib/active_record/core.rb:171:in `find'
/data/app/app/jobs/activity_process_job.rb:8:in `perform'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal/integrations/delayed_job_plugin.rb:49:in `block in invoke_with_instrumentation'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal.rb:251:in `block in monitor_transaction'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal.rb:567:in `instrument'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal.rb:250:in `monitor_transaction'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal/integrations/delayed_job_plugin.rb:36:in `invoke_with_instrumentation'
/data/app/vendor/bundle/ruby/2.3.0/gems/appsignal-2.4.3/lib/appsignal/integrations/delayed_job_plugin.rb:9:in `block (2 levels) in <class:DelayedJobPlugin>'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:61:in `block in initialize'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:79:in `block in add'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:66:in `execute'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/backend/base.rb:78:in `invoke_job'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/usr/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
/usr/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:230:in `block in run'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:229:in `run'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:61:in `block in initialize'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:66:in `execute'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:213:in `block in work_off'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:212:in `times'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:212:in `work_off'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:175:in `block (4 levels) in start'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:174:in `block (3 levels) in start'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:61:in `block in initialize'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:66:in `execute'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:173:in `block (2 levels) in start'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:172:in `loop'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:172:in `block in start'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:61:in `block in initialize'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:79:in `block in add'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:66:in `execute'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:171:in `start'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/command.rb:137:in `run'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/command.rb:125:in `block in run_process'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons/application.rb:270:in `block in start_proc'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons/daemonize.rb:84:in `call_as_daemon'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons/application.rb:274:in `start_proc'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons/application.rb:300:in `start'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons/controller.rb:56:in `run'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons.rb:197:in `block in run_proc'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons/cmdline.rb:92:in `catch_exceptions'
/data/app/vendor/bundle/ruby/2.3.0/gems/daemons-1.2.5/lib/daemons.rb:196:in `run_proc'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/command.rb:123:in `run_process'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/command.rb:104:in `block in daemonize'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/command.rb:102:in `times'
/data/app/vendor/bundle/ruby/2.3.0/gems/delayed_job-4.1.3/lib/delayed/command.rb:102:in `daemonize'
script/delayed_job:5:in `<top (required)>'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:74:in `load'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:74:in `kernel_load'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:28:in `run'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/cli.rb:424:in `exec'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/cli.rb:27:in `dispatch'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/cli.rb:18:in `start'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/exe/bundle:30:in `block in <top (required)>'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/var/lib/gems/2.3.0/gems/bundler-1.16.5/exe/bundle:22:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Looks like something wrong with /data/app/app/jobs/activity_process_job.rb:8:in 'perform'
Same error but saw no clue: collectiveidea/delayed_job#628