Leon's Blogging

Coding blogging for hackers.

用 Sidekiq 處理 Background Job

| Comments

當遇到一些需要等待比較長時間處理的動作時，就可以將這工作，丟到背景去處理
給用戶們更好的時候體驗。

Sidekiq:

Pros

  • runs thread per worker (uses much less memory);
  • less forking (works faster);
  • more options out of the box.

Cons

  • [huge] requires thread-safety of your code and all dependencies. If you run thread-unsafe code with threads, you’re asking for trouble;
  • works on some rubies better than others (jruby and rubinius are recommended, efficiency on MRI is decreased due to GVL (global VM lock)).

每個 worker 是一個 thread，要注意 thread-safety 的問題
Your worker code does need to be thread-safe.

一台機器，只能有一個 redis server，但是 redis 可以有多個 database
redis 的 database 可以共享給不同 server shared

Redis

Sidekiq是搭配Redis來儲存Job，而Redis是一套高性能的 In-Memory Key-Value 儲存系統。

1

brew install redis

啟動

1
2
3
4
5
6

redis-server #is the Redis Server itself.
redis-sentinel #is the Redis Sentinel executable (monitoring and failover).
redis-cli #is the command line interface utility to talk with Redis.
redis-benchmark #is used to check Redis performances.
redis-check-aof #are useful in the rare event of corrupted data files.
redis-check-dump #are useful in the rare event of corrupted data files.

安裝

1
2

gem 'sidekiq'
gem 'sinatra', :require => nil #用來呈現 Web UI

設定

Active Job

可用 ruby 內建的 Active Job，來產生 job。

1
2

rails g job test
# => app/jobs/test_job.rb

設定在 config/application.rb
或是各個環境去做設定 config/environments/development.rb

1

config.active_job.queue_adapter = :sidekiq

app/jobs/test_job.rb

1
2
3
4
5
6
7

class TestJob < ActiveJob::Base
  queue_as :default

  def perform(*args)
    # Do something later
  end
end

Worker

或是直接用 sidekiq 的 worker。

app/works/test_worker.rb

1
2
3
4
5
6
7
8

class TestWorker < ActiveJob::Base
  include Sidekiq::Worker
  sidekiq_options :queue => :default, :retry => 3

  def perform(*args)
    # Do something later
  end
end

other

設定 redis client 和 server

  • 將 worker push 到哪個 redis
  • 從哪個 redis pull worker 下來

config/sidekiq.rb

1
2
3
4
5
6
7
8
9
10
11
12

redis_server = '127.0.0.1'
redis_port = 6379
redis_db_num = 0
redis_namespace = "sidekiq_moai_#{Rails.env}"

Sidekiq.configure_server do |config|
  config.redis = { url: "redis://#{redis_server}:#{redis_port}/#{redis_db_num}", namespace: redis_namespace }
end

Sidekiq.configure_client do |config|
  config.redis = { url: "redis://#{redis_server}:#{redis_port}/#{redis_db_num}", namespace: redis_namespace, size: 25 }
end

config/initializers/sidekiq.yml

1
2
3
4
5
6
7
8
9
10
11
12

:concurrency: 1
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
  - default
  - [myqueue, 2]
development:
  :concurrency: 1
staging:
  :concurrency: 10
production:
  :concurrency: 20

config/routes.rb

1
2

require 'sidekiq/web'
mount Sidekiq::Web => '/sidekiq'

啟動

1
2
3
4

bundle exec sidekiq
#會自動去找 sidekiq.yml

bundle exec sidekiq -C ./config/sidekiq.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

bundle exec sidekiq -q default -c 1

#-c, --concurrency INT            processor threads to use
#-d, --daemon                     Daemonize process
#-e, --environment ENV            Application environment
#-g, --tag TAG                    Process tag for procline
#-i, --index INT                  unique process index on this machine
#-q, --queue QUEUE[,WEIGHT]       Queues to process with optional weights
#-r, --require [PATH|DIR]         Location of Rails application with workers or file to require
#-t, --timeout NUM                Shutdown timeout
#-v, --verbose                    Print more verbose output
#-C, --config PATH                path to YAML config file
#-L, --logfile PATH               path to writable logfile
#-P, --pidfile PATH               path to pidfile
#-V, --version                    Print version and exit
#-h, --help                       Show help

1
2
3
4
5
6
7

#job
TestJob.perform_later()

#worker
TesrWork.perform_async()

#建議參數不要直接塞 object

備註

清除 Queue 中的 Job，避免Queue中存放了太多蠢蠢欲動的僵屍Job，一方面也要防止耗費大量時間的Job再起浪費資源。

1
2

redis-cli -n flushdb
redis-cli flushall

官方文件：
Active Job Basics
Active Job Basics 中文

參考文件：
Web server / Application server 傻傻分不清楚 ？ Thread-Safe的理解與分析
Sharing Sidekiq between two apps
multiple project using sidekiq on same machine
Use Sidekiq on a separate servers
Sidekiq 异常的监控 使用 Monit＋Mina 监控服务器
【译】使用Rails 4.2+ 测试异步邮件系统
Sidekiq in Rails
How to Integrate Sidekiq With ActiveJob
Redis學習手冊(目錄)
消息队列之active_job结合sidekiq(一)
使用Upstart + Inspeqtor 管理你的Sidekiq(监控、崩溃自动重启、邮件通知)
Redis實戰

gem：
sidekiq
sidekiq-status
sidekiq-failures
exception_notification

Comments