how to parse robot.txt using requests

TheRubberDucky answered on April 2, 2022 Popularity 2/10 Helpfulness 3/10

answer how to parse robot.txt using requests

how to parse robot.txt using requests

Comment

import requests

result_data_set = {"Disallowed":[], "Allowed":[]}

r = requests.get(url + "/robots.txt").text 

for line in result.split("\n"):
    if line.startswith('Allow'):    # this is for allowed url
        result_data_set["Allowed"].append(line.split(': ')[1].split(' ')[0])    # to neglect the comments or other junk info
    elif line.startswith('Disallow'):    # this is for disallowed url
        result_data_set["Disallowed"].append(line.split(': ')[1].split(' ')[0])    # to neglect the comments or other junk info

print (result_data_set)

Popularity 2/10 Helpfulness 3/10 Language whatever

Source: Grepper

Tags: using whatever

Link to this answer
Share Copy Link

Contributed on Apr 02 2022

TheRubberDucky

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

how to parse robot.txt using requests

Contents

More Related Answers

how to parse robot.txt using requests

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.