当前访客身份:游客 [ 登录 | 加入 OSCHINA ]

代码分享

当前位置:
代码分享 » Ruby/Rails  » 网络编程
分享到: 
收藏 +0
1
使用Ruby和Nokogiri的网页爬虫例子,导出RSS种子
标签: Ruby 爬虫 Nokogiri

代码片段(1) [全屏查看所有代码]

1. [代码][Ruby]代码     跳至 [1] [全屏预览]

# encoding: utf-8
require 'thread'
require 'nokogiri'
require 'open-uri'
require 'rss/maker'

$result=Queue.new
def extract_readme_header(no,name,url) 
  frame = Nokogiri::HTML(open(url))
  return unless frame
  readme=$url+frame.css('frame')[1]['src']
  return unless readme
  open(readme) do |f|
    doc = Nokogiri::HTML(f.read)
    text=doc.css("div#content div#filecontents p")[0..4].map { |c| c.content }.join(" ").strip
    return if text.length==0
    if text !~ /(rails)|(activ_)/i
      puts "========= #{no} #{name} : #{text[0..50]}"
      date = f.last_modified
      $result << [no,name,readme,date,text]
    end
  end
rescue
  puts $!.to_s
end

def make_rss(items)
  RSS::Maker.make("2.0") do |m|
    m.channel.title = "GtitHub recently updated projects"
    m.channel.link = "http://localhost"
    m.channel.description = "GitHub recently updated projects"
    m.items.do_sort = true
    items.each do |no,name,url,date,descr|
      i = m.items.new_item
      i.title = name
      i.link = url
      i.description=descr
      i.date = date
    end 
  end
end

############################## M A I N ########################

############# Scan list of recent project

lth=[]
$url="http://rdoc.info" 
puts "get url #{$url}..."
doc = Nokogiri::HTML(open($url))
doc.css('ul.libraries')[1].css('li').each_with_index do |li,i|
  aname =li.css('a').first
  name=aname.content
  purl=$url+aname['href']
  lth << Thread.new(i,name,purl) { |j,n,u| extract_readme_header(j,n,u)  }
end

################ wait all readme are read

lth.each { |th| th.join() }

################ dequeue results and sort them by date descending

result=[]
result << $result.shift while $result.size>0
result.sort!  { |a,b| a[0] <=> b[0] }


################ format results in rss

File.open("RubyFeeds.rss","w") do |file|
  file.write make_rss(result)
end


开源中国-程序员在线工具:Git代码托管 API文档大全(120+) JS在线编辑演示 二维码 更多»

发表评论 回到顶部 网友评论(1)

  • 1楼:哈哈__哈哈 发表于 2013-08-31 02:29 回复此评论
    正在看这个,会普通抓取,兄台会Nokogiri的session么?
    我想抓新浪微博的,这个我想模拟登录
开源从代码分享开始 分享代码
小卒过河的其它代码 全部(191)...