⚝
One Hat Cyber Team
⚝
Your IP:
216.73.216.144
Server IP:
157.245.143.252
Server:
Linux www 6.11.0-9-generic #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024 x86_64
Server Software:
nginx/1.26.0
PHP Version:
8.3.11
Buat File
|
Buat Folder
Eksekusi
Dir :
~
/
usr
/
lib
/
python3.12
/
urllib
/
__pycache__
/
View File Name :
robotparser.cpython-312.pyc
CRh$ d Z ddlZddlZddlZdgZ ej dd Z G d d Z G d d Z G d d Z y) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt NRobotFileParser RequestRatezrequests secondsc Z e Zd ZdZddZd Zd Zd Zd Zd Z d Z d Z d Z d Z d Zd Zy)r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. c z g | _ g | _ d | _ d| _ d| _ | j | d| _ y )NFr )entriessitemaps default_entry disallow_all allow_allset_url last_checkedselfurls )/usr/lib/python3.12/urllib/robotparser.py__init__zRobotFileParser.__init__ s; ! ! S c | j S )zReturns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r r s r mtimezRobotFileParser.mtime% s r c 6 ddl }|j | _ y)zYSets the time the robots.txt file was last fetched to the current time. r N)timer )r r s r modifiedzRobotFileParser.modified. s IIKr c p || _ t j j | dd \ | _ | _ y)z,Sets the URL referring to a robots.txt file. N)r urllibparseurlparsehostpathr s r r zRobotFileParser.set_url6 s- %|| 4 4S 9!A > 49r c t j j | j }|j }| j |j d j y# t j j $ rT}|j dv rd| _ n4|j dk\ r |j dk r d| _ Y d}~yY d}~yY d}~yY d}~yd}~ww xY w)z4Reads the robots.txt URL and feeds it to the parser.zutf-8)i i Ti i N) r requesturlopenr readr decode splitlineserror HTTPErrorcoder r )r frawerrs r r% zRobotFileParser.read; s 9&&txx0A &&(C JJszz'*557 8 ||%% &xx:%$(!S SXX^!% &4 " &s )A* * C;CCc d|j v r| j || _ y y | j j | y N*) useragentsr r append)r entrys r _add_entryzRobotFileParser._add_entryH s= %"" "!!)%*" * LL &r c d}t }| j |D ] }|s4|dk( r t }d}n"|dk( r | j | t }d}|j d }|dk\ r|d| }|j }|sh|j dd }t | dk( s|d j j |d< t j j |d j |d< |d dk( rB|dk( r| j | t }|j j |d d}*|d dk( r3|dk7 s9|j j t |d d d}e|d d k( r3|dk7 st|j j t |d d d}|d d k( r?|dk7 s|d j j rt! |d |_ d}|d d k( r|dk7 s|d j d }t | dk( rk|d j j rJ|d j j r)t% t! |d t! |d |_ d}|d dk( s| j( j |d |dk( r| j | yy)zParse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. r r #N:z user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryr r4 findstripsplitlenlowerr r unquoter1 r2 rulelinesRuleLineisdigitintdelayr req_rater )r linesstater3 lineinumberss r r zRobotFileParser.parseQ s DA: !GE EaZOOE* !GE E #AAvBQx::<D::c1%D4yA~ q'--///1Q ,,..tAw}}?Q7l* z . %$$++DG4 E!W * z ..xQ/GH !!W' z ..xQ/FG !!W - z 7==?224*-d1g,EK !!W. z"&q'--"4 LA -'!*2B2B2D2L2L2N ' 0 0 2 : : <-8WQZ #gVWj/-ZEN !!W ) MM((a1o p A: OOE " r c b | j ry| j ry| j syt j j t j j | }t j j dd|j |j |j |j f }t j j | }|sd}| j D ]& }|j | s|j | c S | j r| j j | S y)z=using the parsed robots.txt decide if useragent can fetch urlFT r; )r r r r r r rC urlunparser! paramsqueryfragmentquoter applies_to allowancer )r useragentr parsed_urlr3 s r can_fetchzRobotFileParser.can_fetch s >> \\**6<<+?+?+DE ll%%r"Z__ j . . 0C0C'E Fll %C\\E * s++ " %%//4 4r c | j sy | j D ]! }|j | s|j c S | j r| j j S y N)r r rV rH r r rX r3 s r crawl_delayzRobotFileParser.crawl_delay sT zz|\\E * {{" " %%++ +r c | j sy | j D ]! }|j | s|j c S | j r| j j S y r\ )r r rV rI r r] s r request_ratez RobotFileParser.request_rate sT zz|\\E * ~~% " %%.. .r c 4 | j sy | j S r\ )r r s r site_mapszRobotFileParser.site_maps s }}}} r c | j }| j || j gz }dj t t | S )Nz )r r joinmapstr)r r s r __str__zRobotFileParser.__str__ s@ ,, ) !3!3 44G{{3sG,--r N)rP )__name__ __module__ __qualname____doc__r r r r r% r4 r rZ r^ r` rb rg r r r r sE !(? 9'G#R: .r c " e Zd ZdZd Zd Zd Zy)rE zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c |dk( r|sd}t j j t j j | }t j j | | _ || _ y )NrP T)r r rQ r rU r! rW )r r! rW s r r zRuleLine.__init__ sP 2:i I||&&v||'<'<T'BCLL&&t, "r c Z | j dk( xs |j | j S r/ )r! startswith)r filenames r rV zRuleLine.applies_to s% yyC A8#6#6tyy#AAr c B | j rdnddz | j z S )NAllowDisallowz: )rW r! r s r rg zRuleLine.__str__ s >>zTADIIMMr N)rh ri rj rk r rV rg rl r r rE rE s 1#BNr rE c ( e Zd ZdZd Zd Zd Zd Zy)r= z?An entry has one or more user-agents and zero or more rulelinesc < g | _ g | _ d | _ d | _ y r\ )r1 rD rH rI r s r r zEntry.__init__ s r c g }| j D ] }|j d| | j |j d| j | j 7| j }|j d|j d|j |j t t | j dj | S )Nz User-agent: z Crawl-delay: zRequest-rate: r; ) r1 r2 rH rI requestssecondsextendre rf rD rd )r retagentrates r rg z Entry.__str__ s __E JJ eW- . % :: ! JJ tzzl3 4 == $==D JJ a ~F G 3sDNN+,yy~ r c |j d d j }| j D ] }|dk( r y|j }||v s y y)z2check if this entry applies to the specified agentr; r r0 TF)r@ rB r1 )r rX r} s r rV zEntry.applies_to sS OOC(+113 __E|KKME ! % r c d | j D ]! }|j | s|j c S y)zZPreconditions: - our agent applies to this entry - filename is URL decodedT)rD rV rW )r rq rL s r rW zEntry.allowance s- NNDx(~~% # r N)rh ri rj rk r rg rV rW rl r r r= r= s I r r= ) rk collections urllib.parser urllib.request__all__ namedtupler r rE r= rl r r <module>r sU $k$$]4FG ~. ~.BN N$( (r