python 爬蟲程式 -1

-

做一個抓取 104人力銀行 新鮮人找工作版 所有工作及公司資料的程式

首先要抓搜尋結果頁面的資訊


  1. import requests
  2. from bs4 import BeautifulSoup as bs
  3. search_url = "https://www.104.com.tw/area/freshman/search?keyword=%E6%A9%9F%E5%99%A8%E4%BA%BA&area=6001001000,6001002000&jobcategory=2007000000&industry=&page=1&sortField=APPEAR_DATE&sortMode=DESC"
  4. head = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
  5.         'Accept-Language':'zh-TW,zh;q=0.8,en-US;q=0.6,en;q=0.4'} 
  6. res = requests.get(search_url, headers = head)
  7. print(res.encoding)
  8. res.encoding = 'utf8'
  9. soup = bs(res.text)
  10.  
Out[1]:ISO-8859-1
用request傳回網頁內容
res.encoding可得知網頁編碼方式，一開始我根據網站上寫utf-8編碼，跟python設定編碼一致，想說沒問題，之後才找到如何知道是用甚麼編碼的方法，print出來才知道被python判斷成ISO-8859-1，那就自行設定成utf-8
BeautifulSoup 是一個能解構及分析的一套很好用的工具，將res轉成文字"res.text"，丟給 BeautifulSoup

抓取一個工作資料


用select擷取出我們要的資訊在哪個區域

job_box = soup.select(".job_box")[0]
job_box.select('.joblist_cont')[0]

job_name = joblist.select('a')[0]['title']    #job name
com_name = joblist.select('a')[1]['title']    #company name
edu = joblist.select(".edu")[0].text.replace('\t','').replace('\r\n','')   #學歷
area = joblist.select(".area")[0].text.replace('\t','').replace('\r\n','') #工作地點在哪個市區
job_url = "https://www.104.com.tw" + joblist.select('a')[0]['href']        #職缺網頁
com_url = "https://www.104.com.tw" + joblist.select('a')[1]['href']        #公司簡介網頁

開啟職缺網頁及公司簡介網頁並取得資料
def load_web(url):
    web_res = requests.get(url)
    web_res.encoding = 'utf8'
    #print(job_web_res.text)
    time.sleep(random.uniform(0,2))
    web_bs = bs(web_res.text)
    return web_bs

job_web_bs = load_web(job_url)
com_web_bs = load_web(com_url)
   
# number of people apply for a job
num = num_of_people(job_web_bs)       
#job information
job_info = load_job_info(job_web_bs)
# company information
com_info = load_com_info(com_web_bs)
用hash sort的方式儲存我們要的資料
  1. job_info = {}   #先建一個空間
  2. title1 = '工作內容'
  3. title2 = '工作說明'
  4. explanation = 'xxxxxxx'
  5. if title1 not in job_info.keys():           #當不存在'工作內容'這個索引值，建立'工作內容'這個索引值
  6.     job_info[title1] = {}
  7. if title2 not in job_info[title1].keys():   #當不存在'工作說明'這個索引值，建立'工作說明'這個索引值
  8.     job_info[title1][title2] = {}
  9. job_info[title1][title2] = explanation      #將'xxxxxxx'放在 job_info[工作內容][工作說明]裡

舉例：職缺網頁
# job info
def load_job_info(job_web_bs):
    job_info = {}
    main_info = job_web_bs.select("section")[0]
    title1 = main_info.select('h2')[0].text      #工作內容
    title2 = '工作說明'
    explanation = main_info.select('p')[0].text.replace('\r', ' ')
    
    if title1 not in job_info.keys():
        job_info[title1] = {}
    if title2 not in job_info[title1].keys():
        job_info[title1][title2] = {}
    job_info[title1][title2] = explanation
            
    for i in range(len(main_info.select('dt'))):
        title2 = main_info.select('dt')[i].text.replace('\r','').replace('\t','').replace('：','')
        explanation = main_info.select('dd')[i].text.replace(' ','').replace('\n','')                       
        if title2 not in job_info[title1].keys():
            job_info[title1][title2] = {}
        job_info[title1][title2] = explanation
               
    # job condition
    condition = job_web_bs.select("section")[1]
    title1 = condition.select('h2')[0].text      #工作條件
    if title1 not in job_info.keys():
        job_info[title1] = {}
        
    for i in range(len(condition.select('dt'))):
        title2 = condition.select('dt')[i].text.replace('：','')
        explanation = condition.select('dd')[i].text.replace(' ','')  
        if title2 not in job_info[title1].keys():
            job_info[title1][title2] = {}
        job_info[title1][title2] = explanation
        
    # work welfare
    welfare = job_web_bs.select("section")[2]
    title1 = welfare.select('h2')[0].text        #公司福利
    explanation = welfare.select('p')[0].text.replace('\r\u3000','\t').replace('\r','')
    if title1 not in job_info.keys():
        job_info[title1] = {}
    job_info[title1] = explanation
    return job_info




寫檔案的格式問題：

上網查excel不是用utf-8編碼，python寫檔時也有錯誤訊息，寫說'cp950' codec can't encode character '\u30fb' in position 0: illegal multibyte sequence，所以就全部經由cp950編碼一次測試，如果有字元不能被編碼那就以空白字元代替
要先將string轉為list，在對特定字元做設定，之後再轉回string格式

wl = '"工作內容","工作說明","xxxx"'
for j in range(len(wl)):
    try:
        wl[j].encode('cp950').decode('cp950')
    except:
        tmp = list(wl)
        tmp[j] = " "
        wl = ''.join(tmp)

完整程式碼：

  1. import requests
  2. from bs4 import BeautifulSoup as bs
  3. import time
  4. import pandas as pd
  5. import chardet
  6. import random
  7.  
  8. # get title company there's link
  9. def load_web(url):
  10.     web_res = requests.get(url)
  11.     web_res.encoding = 'utf8'
  12.     #print(job_web_res.text)
  13.     time.sleep(random.uniform(0,2))
  14.     web_bs = bs(web_res.text)
  15.     return web_bs
  16.  
  17. def writeline(array):
  18.     wl = '"' + array[0] + '"'
  19.     for i in range(1, len(array)):
  20.         wl = wl + ',"' + array[i] + '"'
  21.     return wl
  22.  
  23. def get_all_page(beautifulsoup_text):
  24.     page = beautifulsoup_text.select('.next_page')[0]
  25.     tt = page.text.replace('\t','').split('\n')
  26.     num = tt[1].find('共')
  27.     allpage = int(tt[1][num+2])
  28.     return allpage
  29.  
  30. class job_class(object):
  31.     
  32.     def __init__(self, joblist):
  33.         self.job_name = joblist.select('a')[0]['title']    #job name
  34.         self.com_name = joblist.select('a')[1]['title']    #company name
  35.         self.edu = joblist.select(".edu")[0].text.replace('\t','').replace('\r\n','')
  36.         self.area = joblist.select(".area")[0].text.replace('\t','').replace('\r\n','')
  37.         self.job_url = "https://www.104.com.tw" + joblist.select('a')[0]['href']
  38.         self.com_url = "https://www.104.com.tw" + joblist.select('a')[1]['href']
  39.         
  40.     # job info
  41.     def load_job_info(job_web_bs):
  42.         job_info = {}
  43.         main_info = job_web_bs.select("section")[0]
  44.         title1 = main_info.select('h2')[0].text      #工作內容
  45.         title2 = '工作說明'
  46.         explanation = main_info.select('p')[0].text.replace('\r', ' ')
  47.         
  48.         if title1 not in job_info.keys():
  49.             job_info[title1] = {}
  50.         if title2 not in job_info[title1].keys():
  51.             job_info[title1][title2] = {}
  52.         job_info[title1][title2] = explanation
  53.                 
  54.         for i in range(len(main_info.select('dt'))):
  55.             title2 = main_info.select('dt')[i].text.replace('\r','').replace('\t','').replace('：','')
  56.             explanation = main_info.select('dd')[i].text.replace(' ','').replace('\n','')                       
  57.             if title2 not in job_info[title1].keys():
  58.                 job_info[title1][title2] = {}
  59.             job_info[title1][title2] = explanation
  60.                    
  61.         # job condition
  62.         condition = job_web_bs.select("section")[1]
  63.         title1 = condition.select('h2')[0].text      #工作條件
  64.         if title1 not in job_info.keys():
  65.             job_info[title1] = {}
  66.             
  67.         for i in range(len(condition.select('dt'))):
  68.             title2 = condition.select('dt')[i].text.replace('：','')
  69.             explanation = condition.select('dd')[i].text.replace(' ','')  
  70.             if title2 not in job_info[title1].keys():
  71.                 job_info[title1][title2] = {}
  72.             job_info[title1][title2] = explanation
  73.             
  74.         # work welfare
  75.         welfare = job_web_bs.select("section")[2]
  76.         title1 = welfare.select('h2')[0].text        #公司福利
  77.         explanation = welfare.select('p')[0].text.replace('\r\u3000','\t').replace('\r','')
  78.         if title1 not in job_info.keys():
  79.             job_info[title1] = {}
  80.         job_info[title1] = explanation
  81.         return job_info
  82.     
  83.     # company info
  84.     def load_com_info(com_web_bs):
  85.         com_info = {}
  86.         com_info_bs = com_web_bs.select(".intro")[0]
  87.         title1 = com_info_bs.select('h2')[0].text       #公司介紹
  88.         
  89.         if title1 not in com_info.keys():
  90.             com_info[title1] = {}
  91.         for i in range(len(com_info_bs.select('dt'))):
  92.             title2 = com_info_bs.select('dt')[i].text.replace('\u3000','').replace('：','')
  93.             explanation = com_info_bs.select('dd')[i].text.replace('\r','').replace('\n','')
  94.             if title2 not in com_info[title1].keys():
  95.                 com_info[title1][title2] = {}
  96.             com_info[title1][title2] = explanation
  97.             
  98.         com_info_bs = com_web_bs.select(".intro")[1]   
  99.         title1 = com_info_bs.select('h2')[0].text        #公司簡介
  100.         explanation = com_info_bs.select('p')[0].text.replace('\r','\n')
  101.         if title1 not in com_info.keys():
  102.             com_info[title1] = {}
  103.         com_info[title1] = explanation
  104.         
  105.         title1 = com_info_bs.select('h2')[1].text #主要商品／服務項目
  106.         explanation = com_info_bs.select('p')[1].text.replace('\r','')
  107.         if title1 not in com_info.keys():
  108.             com_info[title1] = {}
  109.         com_info[title1] = explanation
  110.         return com_info
  111.     
  112.     # number of people apply for a job
  113.     def num_of_people(job_web_bs):
  114.         number_of_people_bs = job_web_bs.select('.sub')[0]
  115.         number_of_people = number_of_people_bs.select('a')[0].text
  116.         return number_of_people
  117.         
  118.  
  119. def load_104_newpeople_main(search_url, filename = './test.csv'):
  120.     print('write to file ', filename)
  121.  
  122.     
  123.     job_list = [[   ''    , '工作內容', '條件要求', '公司福利'],
  124.                 ['工作說明', '職務類別', '工作待遇', '工作性質', '上班地點', '管理責任', '出差外派', '上班時段', '休假制度', '可上班日', '需求人數'], 
  125.                 ['接受身份', '工作經歷', '學歷要求', '科系要求', '語文條件', '擅長工具', '工作技能', '具備駕照', '其他條件']]
  126.     com_list = [[   ''    , '公司介紹', '公司簡介', '主要商品／服務項目'],
  127.                 ['產業類別', '產業描述', '員工', '資\xa0本\xa0額', '聯\xa0絡\xa0人', '公司地址', '電話', '傳真', '公司網址']]
  128.     all_headlist = ['職務名稱','公司名稱','學歷', '地區', '應徵人數',
  129.                 '工作說明', '職務類別', '工作待遇', '工作性質', '上班地點', '管理責任', '出差外派', '上班時段', '休假制度', '可上班日', '需求人數', '接受身份', '工作經歷', '學歷要求', '科系要求', '語文條件', '擅長工具', '工作技能', '具備駕照', '其他條件', '公司福利',
  130.                 '產業類別', '產業描述', '員工', '資本額', '聯絡人', '公司地址', '電話', '傳真', '公司網址', '公司簡介', '主要商品／服務項目']
  131.     
  132.     head = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
  133.             'Accept-Language':'zh-TW,zh;q=0.8,en-US;q=0.6,en;q=0.4'}
  134.     
  135.     res = requests.get(search_url, headers = head)
  136.     #print(res.encoding)
  137.     res.encoding = 'utf8'
  138.     soup = bs(res.text)
  139.     
  140.     wl = writeline(all_headlist)  
  141.                           
  142.     fw = open(filename, 'w')
  143.     fw.writelines(wl + '\n')
  144.     
  145.     allpage = get_all_page(soup)+1
  146.     
  147.     for page in range(1, allpage):
  148.         
  149.         if page == 1:
  150.             print('page:',page)
  151.             print(search_url)
  152.             
  153.         else:
  154.             print('page:', page)
  155.             #url = 'https://www.104.com.tw/area/freshman/search?area=6001001000%2C6001002000&jobcategory=2007000000&industry=&keyword=%E6%A9%9F%E5%99%A8%E4%BA%BA&page=2&sortField=APPEAR_DATE&sortMode=DESC'
  156.             findpage = 'page='
  157.             local = search_url.find(findpage)
  158.             tmp = list(search_url)
  159.             tmp[local + len(findpage)] = str(page)
  160.             search_url = ''.join(tmp)
  161.     
  162.             res = requests.get(search_url, headers = head)
  163.             #print(res.encoding)
  164.             res.encoding = 'utf8'
  165.             soup = bs(res.text)
  166.             print(search_url)
  167.             
  168.         job_box = soup.select(".job_box")[0]
  169.         
  170.         #job_url = ''
  171.         #com_url = ''
  172.         for joblis in job_box.select('.joblist_cont'):
  173.             
  174.             job = job_class(joblis)
  175.             print(job.job_name)            
  176.         
  177.         #        print("job_url: ", job_url)
  178.         #        print("com url: ", com_url)
  179.             global job_info
  180.             global com_info
  181.             job_web_bs = load_web(job.job_url)
  182.             com_web_bs = load_web(job.com_url)
  183.             
  184.             # number of people apply for a job
  185.             num = job_class.num_of_people(job_web_bs)       
  186.             #job info
  187.             job_info = job_class.load_job_info(job_web_bs)
  188.             # company
  189.             com_info = job_class.load_com_info(com_web_bs)
  190.              
  191.             once_data = []
  192.             once_data.append(job.job_name)
  193.             once_data.append(job.com_name)
  194.             once_data.append(job.edu)
  195.             once_data.append(job.area)
  196.             once_data.append(num)
  197.             # job
  198.             for j in range(len(job_list[1])):
  199.                 try:
  200.                     once_data.append(job_info[job_list[0][1]][job_list[1][j]])
  201.                 except:
  202.                     once_data.append('')       
  203.             for j in range(len(job_list[2])):
  204.                 try:
  205.                     once_data.append(job_info[job_list[0][2]][job_list[2][j]])
  206.                 except:
  207.                     once_data.append('')   
  208.             once_data.append(job_info[job_list[0][3]])    
  209.             
  210.             # company
  211.             for j in range(len(com_list[1])):
  212.                 try:
  213.                     once_data.append(com_info[com_list[0][1]][com_list[1][j]])
  214.                 except:
  215.                     once_data.append('')       
  216.             once_data.append(com_info[com_list[0][2]])
  217.             once_data.append(com_info[com_list[0][3]])
  218.         
  219.             # do wl
  220.             wl = writeline(once_data)
  221.             for j in range(len(wl)):
  222.                 try:
  223.                     wl[j].encode('cp950').decode('cp950')
  224.                 except:
  225.                     tmp = list(wl)
  226.                     tmp[j] = " "
  227.                     wl = ''.join(tmp)
  228.             #write to file
  229.             fw.writelines(wl + '\n')
  230.             break
  231.             
  232.     fw.close()
  233.     print('output file succesful')
  234.     return 'succesful'
  235.  
  236. if __name__ == '__main__':
  237.     filename = './test.csv'
  238.     search_url = "https://www.104.com.tw/area/freshman/search?keyword=%E6%A9%9F%E5%99%A8%E4%BA%BA&area=6001001000,6001002000&jobcategory=2007000000&industry=&page=1&sortField=APPEAR_DATE&sortMode=DESC"
  239.  
  240.     load_104_newpeople_main(search_url, filename)


留言

  1. Beachman2018年4月12日 晚上7:42

    太厲害了！最近才想用python寫一個可以固定擷取特定公司每日是否有釋放新職缺，並就工作性質分類。看到你的BLOG發現都寫好了XD

    回覆刪除
    回覆
    1. Amin2018年6月8日 晚上10:38

      台北班 Python + Excel VBA 金融資訊爬蟲課程 https://docs.google.com/forms/d/e/1FAIpQLSdL6NzcFFoitBT6zfkLWpOPghHjwOB9pXBWp5SJgduFM0EPDg/viewform

      刪除
  2. 橘2019年3月4日 晚上10:29

    很有用的教學！剛好要爬104，值得參考

    回覆刪除
  3. B2021年6月10日 晚上11:21

    作者已經移除這則留言。

    回覆刪除
    回覆
    1. B2021年6月10日 晚上11:26

      不好意思想請教板大是用什麼環境去跑的
      我用 jupyter 跑上面的完整程式碼好像跑不出來🥴

      刪除
    2. 阿翔2021年7月23日 凌晨1:14

      可能對方網頁格式有改版吧，這種事常有

      刪除

張貼留言

這個網誌中的熱門文章

python nn 聲音辨識 -1 傅立葉轉換

-
辨識環境聲音判斷出這是甚麼聲音 主要想法是用頻域赫茲分布做判斷 首先先錄三種聲音的聲音檔，我每一種聲音重複錄製九個sample，所以有27個wav檔 讀取wav檔案，作正規劃處理，在做傅立葉轉換 一開始由於自己做的fft_domyself() function跟其他聲音處理軟體做出來的不一樣，數值非常小，之後找到有內建的spectrogram() function，使用也是數值很小，最後才想到要20log()，但由於內建的覺得不適合自由調整，所以還是修改自己的function。 import matplotlib.pyplot as plt from scipy.fftpack import fft from scipy.io import wavfile # get the api import numpy as np from scipy import signal import math import pyaudio import wave import time #buf=[] #讀入單一檔案測試 def init(): fs, data = wavfile.read('0106.wav') # load the data b=[(ele/2**16.) for ele in data] # this is 16-bit track, b is now normalized on [-1,1) c = fft(b) # calculate fourier transform (complex numbers list) d = len(c)/2 # you only need half of the fft list (real signal symmetry) plt.subplot(211) plt.plot(b,'b') plt.title('time line') plt.subplot(212) plt.plot(abs(c[:(d-1)]),'r') plt.title('fft') plt.show() return fs, b #取樣率 dat...
閱讀完整內容

android 定時通知(永久長期的) 本篇只講AlarmManager使用

-
本篇使用AlarmManager達到準時執行某程式，跟鬧鐘的方式一樣，我之前用過IntentService、Service、Handler，都不能長期使用。 Servicer簡單缺點整理： IntentService：只要關閉activity背景保存視窗，IntentService背景所執行的程式就被強迫結束，好像可以重開(我沒試過)。 Service、Handler：雖然關閉activity會繼續執行，但他是重新在開啟一次service，這到還好，但當手機螢幕關閉處於待機狀態時，service會停止運作暫停在那邊，等你喚醒再繼續執行，所以用這個加timer做定時通知會整個亂掉，timer會停下來不數了。 AlarmManager主要架構是 跟系統註冊鬧鐘甚麼時間點 → 時間到系統會發送一個鬧鈴 → 你的activity接收那個鬧鈴後執行你的程式 當你註冊好幾個鬧鐘已經晚於現在時間，他會馬上通通都執行完 先建立接收鬧鈴的程式 在Manifest.xml 中加入接收系統鬧鈴 <application> <activity android:name=".MainActivity"> </activity> <!-- 當鬧鈴時間到達時要執行的程式 --> <receiver android:name=".AlarmReceiver"> <intent-filter> <action android:name="activity_app" /> </intent-filter> </receiver> </application> 建立一個class檔，執行接收到鬧鈴後的程式 public class AlarmReceiver extends BroadcastReceiver { @Override public void onReceive(Context context, Intent intent) { ...
閱讀完整內容

C# 模擬鍵盤滑鼠控制電腦

-
滑鼠模擬 指定一個螢幕像素的座標位置 int x = 1255; int y = 161; Cursor.Position = new Point(x, y); ClassMouse.LeftClick(); //按下滑鼠左鍵 鍵盤模擬 官方文件 的範例是用小算盤程式，計算111*11= public class demo { [DllImport("USER32.DLL", CharSet = CharSet.Unicode)] public static extern IntPtr FindWindow(string lpClassName, string lpWindowName); // Activate an application window. [DllImport("USER32.DLL")] public static extern bool SetForegroundWindow(IntPtr hWnd); // Send a series of key presses to the Calculator application. public static void Calculator() { // Get a handle to the Calculator application. The window class // and window name were obtained using the Spy++ tool. IntPtr calculatorHandle = FindWindow("ApplicationFrameWindow", "小算盤"); // Verify that Calculator is a running process. if (calculatorHandle == IntPtr.Zero) { MessageBox.Show("Calculator is not running....
閱讀完整內容

raspberrypi 開機自動執行程式 與 在terminal開啟第二個terminal執行python

-
raspberrypi 開啟另一個terminal執行程式 在terminal輸入： lxterminal -e python3 demo.py 參考自： https://www.raspberrypi.org/forums/viewtopic.php?t=65607 ------------------------------------------------------------------------------------------------------------------------------------------------------- raspberrypi 開機自動執行程式 在terminal中輸入以下指令，開啟檔案： sudo nano /etc/rc.local 以下是加入執行python程式 #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. # Print the IP address _IP=$(hostname -I) || true if [ "$_IP" ]; then    printf "My IP address is %s\n" "$_IP" fi sudo python /home/pi/demo.py  <==加入在這邊 exit 0 參考自： http://dirtypig8.blogspot.tw/2016/12/python.html#!/tcmbck
閱讀完整內容

python pyautogui 簡介

-
pyautogui 是 python的一個模組，他能控制鍵盤和滑鼠的操作，能利用它做自動化操作，像是用在軟體測試、重複性動作等。 pyautogui 暫停幾秒鐘 >>>pyautogui.PAUSE = 1.5 fail-safe (預設false) 當滑鼠一道螢幕左上角時，觸發 pyautogui 的failsafeexception 異常 >>>pyautogui.FAILSAFE = True 螢幕上的座標軸 (0,0) →→→→(1920,0)    ↓    ↓    ↓    ↓ (0,1080) 取得螢幕解析度 >>>import pyautogui >>>pyautogui.size() (1920, 1080) >>>width, height = pyautogui.size() 滑鼠部分 moveTo(width, height, duration) example移動到(300, 400)位置 pyautogui.moveTo(300, 400)                 #立即到 pyautogui.moveTo(300, 400, duration = 1.5) #花1.5秒移到 moeRel(width, height, duration) 移到相對座標，以當前滑鼠座標為基準點 pyautogui.moveRel(0, 100, duration=0.25) pyautogui.moveRel(-100, 0, duration=0.25) position() 獲得滑鼠當前的座標位置 >>>pyautogui.position() (300, 400) click(width, height, button) 滑鼠點擊 button = 'left', 'middle', 'right'。 pyautogui.click()       ...
閱讀完整內容