????
Your IP : 3.147.70.194
3
�c� � @ s� d dl mZmZmZ d dlmZ d dlmZ d dlm Z
m Z e
e_ e e_ dgZ
G dd� de�ZG dd� de�ZG d d
� d
e�Zd S ) � )�absolute_import�division�unicode_literals)�str)�urllib)�parse�request�RobotFileParserc @ sZ e Zd ZdZddd�Zdd� Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dS )r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
� c C s, g | _ d | _d| _d| _| j|� d| _d S )NFr )�entries�
default_entry� disallow_all� allow_all�set_url� last_checked)�self�url� r �!/usr/lib/python3.6/robotparser.py�__init__ s
zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r )r r r r �mtime& s zRobotFileParser.mtimec C s ddl }|j � | _dS )zYSets the time the robots.txt file was last fetched to the
current time.
r N)�timer )r r r r r �modified/ s zRobotFileParser.modifiedc C s& || _ tjj|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r r r �urlparse�host�path)r r r r r r 7 s zRobotFileParser.set_urlc C s| yt jj| j�}W nH t jjk
rZ } z(|jdkr:d| _n|jdkrJd| _W Y dd}~X n X |j � }| j
|j d�j � � dS )z4Reads the robots.txt URL and feeds it to the parser.� � Ti� Nzutf-8)r r )
r r Zurlopenr �errorZ HTTPError�coder
r �readr �decode�
splitlines)r �f�err�rawr r r r" < s
zRobotFileParser.readc C s, d|j kr | jd kr(|| _n | jj|� d S )N�*)�
useragentsr r �append)r �entryr r r �
_add_entryI s
zRobotFileParser._add_entryc C st d}t � }�xN|D �]D}|sL|dkr0t � }d}n |dkrL| j|� t � }d}|jd�}|dkrj|d|� }|j� }|sxq|jdd�}t|�dkr|d j� j� |d< tjj |d j� �|d< |d dkr�|dkr�| j|� t � }|j
j |d � d}q|d dk�r*|dk�rX|j j t
|d d �� d}q|d d
kr|dkr|j j t
|d d �� d}qW |dk�rp| j|� dS ) z�Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r � �#N�:z
user-agentZdisallowFZallowT)�Entryr, �find�strip�split�len�lowerr r �unquoter) r* � rulelines�RuleLine)r �lines�stater+ �line�ir r r r R sJ
zRobotFileParser.parsec C s� | j r
dS | jrdS tjjtjj|��}tjjdd|j|j|j |j
f�}tjj |�}|s\d}x"| j D ]}|j
|�rd|j|�S qdW | jr�| jj|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr
�/)r
r r r r r6 Z
urlunparser ZparamsZqueryZfragment�quoter �
applies_to� allowancer )r � useragentr Z
parsed_urlr+ r r r � can_fetch� s
zRobotFileParser.can_fetchc C s dj dd� | jD ��S )Nr
c S s g | ]}t |�d �qS )�
)r )�.0r+ r r r �
<listcomp>� s z+RobotFileParser.__str__.<locals>.<listcomp>)�joinr )r r r r �__str__� s zRobotFileParser.__str__N)r
)
�__name__�
__module__� __qualname__�__doc__r r r r r" r, r rB rG r r r r r s
3c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r8 zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s* |dkr| rd}t jj|�| _|| _d S )Nr
T)r r r> r r@ )r r r@ r r r r � s zRuleLine.__init__c C s | j dkp|j| j �S )Nr( )r �
startswith)r �filenamer r r r? � s zRuleLine.applies_toc C s | j r
dp dd | j S )NZAllowZDisallowz: )r@ r )r r r r rG � s zRuleLine.__str__N)rH rI rJ rK r r? rG r r r r r8 � s r8 c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
S ) r0 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d S )N)r) r7 )r r r r r � s zEntry.__init__c C sP g }x | j D ]}|jd|dg� q W x | jD ]}|jt|�dg� q,W dj|�S )Nz User-agent: rC r
)r) �extendr7 r rF )r �ret�agentr; r r r rG � s z
Entry.__str__c C sF |j d�d j� }x.| jD ]$}|dkr*dS |j� }||krdS qW dS )z2check if this entry applies to the specified agentr= r r( TF)r3 r5 r) )r rA rP r r r r? � s zEntry.applies_toc C s$ x | j D ]}|j|�r|jS qW dS )zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)r7 r? r@ )r rM r; r r r r@ � s
zEntry.allowanceN)rH rI rJ rK r rG r? r@ r r r r r0 � s
r0 N)Z
__future__r r r Zfuture.builtinsr Zfuture.backportsr Zfuture.backports.urllibr �_parser Z_request�__all__�objectr r8 r0 r r r r �<module> s