Resque.enqueue 在第二次运行时失败

【问题标题】：Resque.enqueue failing on second runResque.enqueue 在第二次运行时失败
【发布时间】：2011-09-02 12:20:15
【问题描述】：

我正在尝试将应用程序从 Rails 3.0.3 移植到 Rails 3.1rc...我认为我没有错过任何东西，就配置而言。该过程在 Rails 3.0.X 中完美运行，而不是在 3.1rc 中。

在控制台中，我这样做：

Resque.enqueue(EncodeSong, Song.find(20).id, Song.find(20).unencoded_url)

到目前为止一切正常。 Resque-web 报告没有失败的作业。而且，我从模块 EncodeSong 中得到了两个“放置”。

但是，再次运行 Resque.enqueue(EncodeSong, Song.find(20).id, Song.find(20).unencoded_url) 将在 resque-web 中返回以下错误 (以下）。为了使错误消失，我必须关闭正在运行的进程： QUEUE=* rake environment resque:work 并在控制台窗口中重新运行它。但是在第一次尝试 Resque.enqueue() 之后问题又回来了。

Class
    EncodeSong
Arguments

    20
    "https://bucket_name.s3.amazonaws.com/unencoded/users/1/songs/test.mp3"

Exception
    PGError
Error
    server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/postgresql_adapter.rb:272:in `exec'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/postgresql_adapter.rb:272:in `block in clear_cache!'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/postgresql_adapter.rb:271:in `each_value'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/postgresql_adapter.rb:271:in `clear_cache!'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/postgresql_adapter.rb:299:in `disconnect!'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_pool.rb:191:in `block in disconnect!'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `each'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `disconnect!'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activesupport-3.1.0.rc1/lib/active_support/core_ext/module/synchronization.rb:35:in `block in disconnect_with_synchronization!'
    /Users/Chris/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activesupport-3.1.0.rc1/lib/active_support/core_ext/module/synchronization.rb:34:in `disconnect_with_synchronization!'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_pool.rb:407:in `remove_connection'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_specification.rb:116:in `remove_connection'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_specification.rb:79:in `establish_connection'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_specification.rb:60:in `establish_connection'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/activerecord-3.1.0.rc1/lib/active_record/connection_adapters/abstract/connection_specification.rb:55:in `establish_connection'
    /Users/Chris/Sites/site_name/lib/tasks/resque.rake:17:in `block (2 levels) in <top (required)>'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/worker.rb:355:in `call'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/worker.rb:355:in `run_hook'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/worker.rb:162:in `perform'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/worker.rb:130:in `block in work'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/worker.rb:116:in `loop'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/worker.rb:116:in `work'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/resque-1.16.1/lib/resque/tasks.rb:27:in `block (2 levels) in <top (required)>'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:205:in `call'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:205:in `block in execute'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:200:in `each'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:200:in `execute'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
    /Users/Chris/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:151:in `invoke_with_call_chain'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/task.rb:144:in `invoke'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:112:in `invoke_task'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:90:in `block (2 levels) in top_level'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:90:in `each'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:90:in `block in top_level'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:129:in `standard_exception_handling'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:84:in `top_level'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:62:in `block in run'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:129:in `standard_exception_handling'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/lib/rake/application.rb:59:in `run'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/gems/rake-0.9.0/bin/rake:31:in `<top (required)>'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/bin/rake:19:in `load'
    /Users/Chris/.rvm/gems/ruby-1.9.2-p136@railspre/bin/rake:19:in `<main>'

这是我的其余相关代码：

/config/initializers/resque.rb

require 'resque'

uri = URI.parse(APP_CONFIG['redis_to_go_url'])
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

# Load all jobs at /app/jobs
Dir["#{Rails.root}/app/jobs/*.rb"].each { |file| require file }

/app/jobs/encode_song.rb

module EncodeSong
  @queue = :encode_song

  def self.perform(media_id, s3_file_url)
    begin
      media = Song.find(media_id)
      puts 'foo11111'
      puts media.id
    rescue
      puts "Error #{$!}"
    end
  end
end

lib/tasks/resque.rake

require 'resque/tasks'

task "resque:setup" => :environment do
  ENV['QUEUE'] = '*'

  # ONLY on Heroku, since they are still running PostgreSql 8 on their shared plan.
  # This block of code is not needed on PostgreSql 9, as tested on local environment.
  # Issue: My best guess is that master resque process establishes connection to db,
  # while loading rails app classes, models, etc, and that connection becomes corrupted
  # in fork()ed process (on exit?). Possible fix is to reestablish the connection the AR
  # after a Resque fork.
  Resque.after_fork do |job|
    ActiveRecord::Base.establish_connection
  end

end

desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"

不太确定，但可能与这个issue 有点关系。我的猜测是，主 resque 进程建立与 db 的连接，同时加载 rails 应用程序类、模型等，并且该连接在 fork()ed 进程中损坏（退出时？）。

任何帮助/指导将不胜感激。

编辑：

如果我从 lib/tasks/resque.rake 中删除以下块：

  Resque.after_fork do |job|
    ActiveRecord::Base.establish_connection
  end

然后在控制台中运行 Resque.enqueue(EncodeSong, Song.find(20).id, Song.find(20).unencoded_url)

我收到一个新错误（在运行 QUEUE=* rake environment resque:work 的控制台中）：

Error PGError: ERROR:  prepared statement "a3" already exists
: SELECT  "songs".* FROM "songs"  WHERE "songs"."id" = $1 LIMIT 1

看来，这可能是适配器的错误？这里可能是错的。你的想法？

【问题讨论】：

    标签： ruby-on-rails redis resque


    【解决方案1】：

    这已经为我解决了：

    require 'resque/tasks'

task "resque:setup" => :environment do
  ENV['QUEUE'] = '*'

  Resque.before_fork = Proc.new { ActiveRecord::Base.establish_connection }
end

    来源：carhartl comment on Github

    【讨论】：

    • 谢谢，我也读到过关于在某处使用before_fork 的信息。不知道与after_fork相比有什么意义
    • 您能否为您的代码添加文件名，以便我们知道该放在哪里？谢谢！
    • 我把这个放在lib/tasks/resque.rake。
    • 最好使用after_fork 挂钩，以便在子进程而不是父进程上建立新连接。 （PS：我是resque的维护者）
    猜你喜欢
    • 1970-01-01
    • 2020-03-28
    • 1970-01-01
    • 2016-03-08
    • 1970-01-01
    • 2020-08-22
    • 2017-05-20
    • 2021-01-20
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode