Mobile packages spider with mitmproxy

Powerful crawl tool mitmproxy

Posted by CodingMrWang on July 13, 2018

This post is first created by CodingMrWang, 作者 @Zexian Wang ,please keep the original link if you want to repost it.

Mitmproxy

  • Mitmproxy is a packet capture program that supports HTTP and HTTPS.

related links

Install with pip

The easiest way to install mitmproxy is to install it with pip, you can directly install with the following Command

sudo pip3 install mitmproxy

if you are using python2

pip install "mitmproxy==0.18.2"

Certificate configuration

If you want to catch HTTPS requests, you have to configure the certificates, after you install mitmproxy, it will create some certificates for you, only if you trust the certificates, you could catch HTTPS requests, or you cannot catch them.

Fristly, create the certificates:

mitmdump

Under the .mitmproxy directory, you can find six certificates.

.mitmproxy

Double click mitmproxy-ca-cert.pem, then find mitmproxy Certificate, open the settings and set always trust.

certificate

IOS:

use your Iphone connect to the same wifi with your computer, open the setting of wifi, switch http proxies to manual, then enter your computer’s ip address and port number 8080. After that, open your brower on your Iphone, go to mitm.it, click the apple icon and download the certificate, then trust it.

phone

After that, run the mitmproxy:

mitmproxy

Then you can see all requests to your Iphone would be shown on your terminal.

mitmproxy Some Hot Keys:

  • q: quit
  • z: clean the screen
  • f: filter

Handle packages with self designed python code

Besides that, you can catch all requests and handle them with python code:

# -*- coding=utf-8 -*-
import os
import hashlib
import re
import MySQLdb

def response(flow):
    request = flow.request
    response = flow.response

    # feeds data
    if 'xxx' in request.url:
        file_name = re.findall(r'\d+', request.url)[0]
        try:
            insertIntodb(**kwargs)
        except Exception as e:
            print(e)

def insertIntodb(**kwargs):
    db = MySQLdb.connect(
        host = 'xx.xx.xx.xx',
        user = 'xxx',
        passwd = 'xxxx',
        db = 'xxx',
        port = xxx,
        charset='utf8')
    cursor = db.cursor()
    # SQL 插入语句
    sql = "Your sql command"
    try:
       # 执行sql语句
       cursor.execute(sql)
       # 提交到数据库执行
       db.commit()
    except Exception as e:
       # Rollback in case there is any error
        print (e)
        print('db error')
        db.rollback()
    # 关闭数据库连接
    db.close()

After finishing your python code, run:

mitmdump -s /path/to/yourcode.py