Tuesday 21 October 2014

Process monitoring with God in Rails

God: A Ruby framework for process management.

The core issue we face before using the explanation of the use of process managment tools/gems:

While I have implemented sidekiq to monitor the threads in our application for sending emails in certain time frames. The sidekiq stops if server reboots and the queued processes goes in a indefinite period of time without execution. So this needs the sidekiq should be restarted automatically when it stops.

God is a framework which can automate the process.

FEATURES
  • Config file is written in Ruby
  • Easily write your own custom conditions in Ruby
  • Supports both poll and event based conditions
  • Different poll conditions can have different intervals
  • Integrated notification system (write your own too!)
  • Easily control non-daemonizing scripts
Installation

Add gem 'god' in Gemfile .
Run $ bundle install

Or

Run $ [sudo] install gem god


Version

$ god –version

God Learning Curve

Let us start with basic examples, how to write codes to execute the processes using God.

Let us create a new file god_demo.rb in application folder.

Write the following code in the same file

loop do
puts 'Hello'
sleep 1
end

Now we'll write a god config file that tells god about our process. Place it in the same directory and call it god_demo .god:



God.watch do |w|
w.name = "god_demo"
w.start = "ruby /full/path/to/god_demo.rb"
w.keepalive
end

God configuration should start with a God.watch block. A watch in god represents a process that we want to watch and control. Each watch must have, at minimum, a unique name and a command that tells god how to start the process. The keepalive declaration denotes god to keep this process alive. If the process is not running when god starts, it will be started. If the process dies, it will be restarted.

To run god, we give it the configuration file we wrote with -c. To see what's going on, we can ask it to run foreground with -D:

There are two ways that god can monitor your process. The first and better way is with process events. Not every system supports it, but those that do will automatically use it. With events, god will know immediately when a process exits. For those systems without process event support, god will use a polling mechanism. The output you see throughout this section will show both ways.

This is the example in http://godrb.com/

# Events

I [2011-12-10 15:24:34] INFO: Loading simple.god
I [2011-12-10 15:24:34] INFO: Syslog enabled.
I [2011-12-10 15:24:34] INFO: Using pid file directory: /Users/tom/.god/pids
I [2011-12-10 15:24:34] INFO: Started on drbunix:///tmp/god.17165.sock
I [2011-12-10 15:24:34] INFO: simple move 'unmonitored' to 'init'
I [2011-12-10 15:24:34] INFO: simple moved 'unmonitored' to 'init'
I [2011-12-10 15:24:34] INFO: simple [trigger] process is not running (ProcessRunning)
I [2011-12-10 15:24:34] INFO: simple move 'init' to 'start'
I [2011-12-10 15:24:34] INFO: simple start: ruby /Users/tom/dev/mojombo/god/simple.rb
I [2011-12-10 15:24:34] INFO: simple moved 'init' to 'start'
I [2011-12-10 15:24:34] INFO: simple [trigger] process is running (ProcessRunning)
I [2011-12-10 15:24:34] INFO: simple move 'start' to 'up'
I [2011-12-10 15:24:34] INFO: simple registered 'proc_exit' event for pid 23298
I [2011-12-10 15:24:34] INFO: simple moved 'start' to 'up'

# Polls

I [2011-12-07 09:40:18] INFO: Loading simple.god
I [2011-12-07 09:40:18] INFO: Syslog enabled.
I [2011-12-07 09:40:18] INFO: Using pid file directory: /Users/tom/.god/pids
I [2011-12-07 09:40:18] INFO: Started on drbunix:///tmp/god.17165.sock
I [2011-12-07 09:40:18] INFO: simple move 'unmonitored' to 'up'
I [2011-12-07 09:40:18] INFO: simple moved 'unmonitored' to 'up'
I [2011-12-07 09:40:18] INFO: simple [trigger] process is not running (ProcessRunning)
I [2011-12-07 09:40:18] INFO: simple move 'up' to 'start'
I [2011-12-07 09:40:18] INFO: simple start: ruby /Users/tom/dev/mojombo/god/simple.rb
I [2011-12-07 09:40:19] INFO: simple moved 'up' to 'up'
I [2011-12-07 09:40:19] INFO: simple [ok] process is running (ProcessRunning)
I [2011-12-07 09:40:24] INFO: simple [ok] process is running (ProcessRunning)
I [2011-12-07 09:40:29] INFO: simple [ok] process is running (ProcessRunning)




Wednesday 8 October 2014

Custom routing for restricting sidekiq pages in rails.

After integrating sidekiq with rails . It provides the web access to manage the processes/jobs . User can view Scheduled jobs from "Scheduled" tab.

Here is the screenshot of the sidekiq page.  This will be more easier to understand with this page.


Here we can see the jobs are being listed in the table. These jobs are in queue to be scheduled.

These listed jobs have some individual checkbox to select each jobs. We can delete the jobs at any point. Also deschedule from the site . Below is the "Delete" button to delete the jobs .


If we will view it to pubic it will be miss-utilized.

What we can do?

We can restrict the url from routes. Below is the routes for sidekiq.

require 'sidekiq/web'

Myapp::Application.routes.draw do
     mount Sidekiq::Web, at: '/sidekiq'
end

This is the general route for sidekiq. All the routes are generated from the same route configurations.

If we will do  $rake routes   in our rails console, we will get the following routes
 sidekiq_web     /sidekiq           Sidekiq::Web

How to restrict?

Example 1-

We can use a lambda call for the specific user group or emails. For example :

authenticate :user, lambda { |u| u.is_super_admin? || u.email == 'your@email.com'} do
    mount Sidekiq::Web, at: '/sidekiq'

end

Here it is restricting through the user group as if the user is super admin then it will show to the user.
Also it can be opened to a specific user email.

Example 2

Through Restful Authentication

Checks a User model instance that responds to admin?

#lib/admin_constraint.rb
class AdminConstraint
    def matches?(request)
        return false unless request.session[:user_id]
        user = User.find request.session[:user_id]
        user && user.admin?
    end
end

#config/routes.rb
require "admin_constraint"
mount Sidekiq::Monitor::Engine => '/sidekiq', :constraints => AdminConstraint.new



Also we can follow all the examples explained in
 Sidekiq wiki : https://github.com/mperham/sidekiq/wiki/Monitoring

Thanks :)

Wednesday 24 September 2014

Getting labels from fedex

Here we will start on getting rates from fedex with rails..

 require 'fedex' 
fedex = Fedex::Shipment.new(:key => 'xxx', 
       :password => 'xxxx', 
       :account_number => 'xxxx', 
       :meter => 'xxx', 
       :mode => 'production') 

Generating a shipping label with PDF Format

To create a label for a shipment:

label = fedex.label(:filename => "label_dir/label.pdf",
:shipper=>shipper,
:recipient =>recipient,
:packages =>packages,
:service_type =>"FEDEX_GROUND",
:shipping_options=> shipping_options)

shipper is the test shipper we defined in the previous blog . Recipient is the
same test recipient also packages .

Generate a shipping label in any available format

Change the filename extension and pass a label_specification hash. 

For example:

example_spec = {
:image_type => "EPL2",
:label_stock_type => "STOCK_4X6"
}

label = fedex.label(:filename =>"my_dir/example_epl2.pcx",
:shipper=>shipper,
:recipient =>recipient,
:packages =>packages,
:service_type =>"FEDEX_GROUND",
:shipping_options=> shipping_options, 
:label_specification => example_spec)

Generate shipping labels for multi-package shipments (MPS)

Customer can have multiple packages for a single pick-up, destination and payer 

can be combined into a single MPS shipment. The first label will provide a
master tracking number which must be used in the subsequent calls for
the remaining packages in the shipment.

Parameters for the first label:

label = fedex.label(
:filename => file_name,
:shipper => shipper,
:recipient => recipient,
:packages => packages,
:service_type => service_type,
:shipping_details => shipping_details,
:shipping_charges_payment => shipping_charges_payment,
:customs_clearance_detail => customs_clearance_detail,
:mps => {:package_count =>
package_count, :total_weight => total_weight, :sequence_number =>
'1'}

)

Parameters for labels 2 through 'n':

fedex.label(
:filename => file_name,
:shipper => shipper,
:recipient => recipient,
:packages => packages,
:service_type => service_type,
:shipping_details =>shipping_details,
:shipping_charges_payment =>shipping_charges_payment,
:customs_clearance_detail =>customs_clearance_detail,
:mps => { :master_tracking_id =>{
:tracking_id_type => 'FEDEX', 
:tracking_number=>tracking_number},
:package_count =>package_count,
:total_weight => total_weight,
:sequence_number => 'n'      }   )


Tracking a shipment

To track a shipment:

results = fedex.track(:tracking_number=> "1234567890123")

# => [#<Fedex::TrackingInformation>]

# Pull the first result from the returned array

tracking_info = results.first

tracking_info.tracking_number

# => "1234567890123"

tracking_info.status

# => "Delivered"

tracking_info.events.first.description

# => "On FedEx vehicle for
delivery"

Deleting a shipment
If you dont want to use a fedex label, you should delete the shipment. Fedex will not charge you .

For deleting fedex shipment :
fedex.delete(:tracking_number => "1121123123345")

Address verification with fedex


address = {
:address     => "5 Elm Street",
:city        => "Littleton",
:state       => "CO",
:postal_code => "80125",
:country     => "USA"
}

address_result = fedex.validate_address(:address => address)


# =>  #<Fedex::Address:0xb2e60ac
@changes=["NORMALIZED", "HOUSE_OR_BOX_NUMBER_NOT_FOUND"],
@score=0, @confirmed=false, @available=true, @status="UNDETERMINED",
@residential=false, @business=false, @company=nil, @street_lines="5
ELM ST", @city="LITTLETON", @state="CO",
@province_code="CO", @postal_code="80125",
@country_code="US", @options={:score=>"0",
:changes=>["NORMALIZED",
"HOUSE_OR_BOX_NUMBER_NOT_FOUND"],
:residential_status=>"UNDETERMINED",
:delivery_point_validation=>"UNCONFIRMED",
:address=>{:street_lines=>"5 ELM ST",
:city=>"LITTLETON", :state_or_province_code=>"CO",
:postal_code=>"80125", :country_code=>"US"},
:removed_non_address_data=>nil}>


address_result.residential

# => false

address_result.score

# => 0

address_result.postal_code

# => "80125"

These are the operations working for fedex with this gem. Hope this will help to understand the integration of fedex with rails . Thanks









Thursday 18 September 2014

Fedex API integration in rails

Introduction to Fedex
FedEx Corporation is an American global courier delivery services company headquartered in Memphis, Tennessee.


With FedEx Web Services, we can integrate FedEx Express®, FedEx Ground® and FedEx Freight® shipping services into our business systems, retail website or order management system. FedEx Web Services is the next generation shipping API for integrating software applications with FedEx Systems to create shipping labels, facilitate returns, track shipments, obtain rate quotes and generate reports.

Fedex API integration in rails 4

We can integrate Fedex API with gem 'fedex' in rails .

It uses Fedex API Shipment version 13. This version uses the Non-SOAP Web Services so there is no need to download the Fedex WSDL files, note however that you will need to apply for development/production credentials.

Installation

Include gem 'fedex' in Gemfile.rb .

Run $bundle install .

How to use?
Basically we use some shipping destination where to ship and the sender's destinations from where we want to ship. We need to provide package informations . The package informations consists of

Package Dimensions with parameters

  • height
  • length
  • width
These informations need to give in inches or centimeters .

Package Weight parameter
  • weight with lbs

We will send a request to the fedex API to get rate from fedex. Later we will get a request to get label from fedex. The label will give a report with bar code with valid fedex tracking number.

Using Rails gem how will acomplish this will described here .....

Defining a Test Shiper

shipper = { :name => "Test Fedex Sender",
:company => "Home",
:phone_number => "121-333-2332",
:address => "Main Street",
:city => "Littleton",
:state => "CO",
:postal_code => "80125",
:country_code => "US" }

Defining a test Recipient

recipient = { :name => "Test Fedex Recipient",
:company => "Home",
:phone_number => "555-555-5555",
:address => "Main Street",
:city => "Boulder",
:state => "CO",
:postal_code => "80304",
:country_code => "US",
:residential => "false" }

Defining the packages; multiple packages in a single shipment are also allowed (all the dimensions must be integers).

packages = []
packages << {
:weight => {:units => "LB", :value => 2},
:dimensions => {:length => 10, :width => 5, :height => 4, :units => "IN" }
}
packages << {
:weight => {:units => "LB", :value => 6},
:dimensions => {:length => 5, :width => 5, :height => 4, :units => "IN" }
}

By default the packaging type will be “YOUR PACKAGING” and the dropoff type will be “REGULAR PICKUP”

Initializing shipping options

shipping_options = {
:packaging_type => "YOUR_PACKAGING",
:drop_off_type => "REGULAR_PICKUP"
}

Creating a Fedex::Shipment object using your FedEx credentials; mode should be either production or development depending on what Fedex environment you want to use.

Include fedex in your ruby file
require 'fedex'

Initializing Fedex::Shipment

fedex = Fedex::Shipment.new(:key => 'xxx',
:password => 'xxxx',
:account_number => 'xxxx',
:meter => 'xxx',
:mode => 'development')




Getting Shipment Rates

rate = fedex.rate(:shipper=>shipper,
:recipient => recipient,
:packages => packages,
:service_type => "FEDEX_GROUND",
:shipping_options => shipping_options)


This will provide a response as
# Complete response
$ <Fedex::Rate:0x1019ba5f8
@total_net_charge="34.03",
@total_surcharges="1.93",
@total_billing_weight="8.0 LB",
@total_taxes="0.0",
@rate_type="PAYOR_ACCOUNT_PACKAGE",
@total_base_charge="32.1",
@total_freight_discounts=nil,
@total_net_freight="32.1",
@rate_zone="51">

$ rate.total_net_charge => "34.03"


Fedex provides multiple total values; total_net_charge is the final amount we are looking for.


We will then find the suitable way to pay the fedex price and continue shipping.

In the next blogs we will read about the fedex label, cancelling fedex shipment, verifying address with fedex.