Python-da DOM-dagi elementlar va resurslar bilan ishlash (BeautifulSoup va Scrapy)

1. BeautifulSoup bilan ishlash

1.1. BeautifulSoup o'rnatish

pip install beautifulsoup4
pip install lxml

1.2. Asosiy importlar va HTML yuklash

from bs4 import BeautifulSoup

html = """
<html>
    <head><title>Misol Sahifa</title></head>
    <body>
        <h1>Sarlavha</h1>
        <p class="paragraf">Bu oddiy paragraf.</p>
        <a href="https://example.com">Link</a>
    </body>
</html>
"""

# HTMLni parse qilish
soup = BeautifulSoup(html, "lxml")

1.3. DOM elementlarni qidirish

Element bo‘yicha qidirish:

print(soup.title)  # <title>Misol Sahifa</title>
print(soup.h1.text)  # Sarlavha

Class orqali qidirish:

pythonCopy codeparagraf = soup.find("p", class_="paragraf")
print(paragraf.text)  # Bu oddiy paragraf.

Havolalarni olish:

link = soup.find("a")
print(link['href'])  # https://example.com

1.4. Bir nechta elementlarni olish

from bs4 import BeautifulSoup

html = """
<ul>
    <li>Element 1</li>
    <li>Element 2</li>
    <li>Element 3</li>
</ul>
"""

soup = BeautifulSoup(html, "lxml")
elements = soup.find_all("li")
for element in elements:
    print(element.text)

2. Scrapy bilan ishlash

2.1. Scrapy o'rnatish

pip install scrapy

2.2. Yangi loyiha yaratish

scrapy startproject myproject

2.3. Scrapy'ning asosiy tuzilmasi

spiders: Skriptingni amalga oshiradigan fayllar joylashgan papka.
items.py: Topilgan ma'lumotlarni saqlash uchun strukturani belgilash.
pipelines.py: Ma'lumotlarni saqlash yoki tozalash uchun ishlatiladi.

2.4. Oddiy spider yaratish

Spider faylini yaratish:

cd myproject
scrapy genspider example example.com

Spiderni yozish:

import scrapy

class ExampleSpider(scrapy.Spider):
    name = 'example'
    start_urls = ['https://example.com']

    def parse(self, response):
        # Sarlavhani chiqarish
        title = response.xpath("//title/text()").get()
        print(f"Sarlavha: {title}")

2.5. XPath va CSS Selector bilan qidirish

XPath orqali:

response.xpath("//h1/text()").get()  # H1 tagini olish

CSS Selector orqali:

response.css("h1::text").get()  # H1 tagini olish

2.6. Ma'lumotlarni yig'ish va saqlash

JSON yoki CSV faylga yozish:

scrapy crawl example -o results.json

Pipeline orqali ma'lumotni qayta ishlash: pipelines.py faylini tahrirlash:

class MyProjectPipeline:
    def process_item(self, item, spider):
        # Ma'lumotni tozalash
        item['title'] = item['title'].strip()
        return item

2.7. Middleware va konfiguratsiya

So‘rovlar tezligini boshqarish: settings.py faylida:

DOWNLOAD_DELAY = 2

User-Agent sozlash:

 DEFAULT_REQUEST_HEADERS = {
    'User-Agent': 'my-scrapy-bot'
}

3. BeautifulSoup va Scrapy qachon ishlatiladi?

BeautifulSoup: Statik HTML fayllarni tahlil qilishda va kichik loyihalarda qulay.
Scrapy: Dinamik ma'lumotlar va katta miqyosli web scraping uchun samarali vosita

1. BeautifulSoup bilan ishlashni kengaytirish

1.1. Mahalliy HTML faylni ochish va yuklash

Agar siz HTML faylni mahalliy diskdan o'qishni xohlasangiz:

from bs4 import BeautifulSoup

with open("example.html", "r", encoding="utf-8") as file:
    content = file.read()

soup = BeautifulSoup(content, "lxml")

# Sahifaning title'ini olish
print(soup.title.text)

1.2. HTML strukturasini tozalash

Ba'zan HTML sahifalar yaxshi tuzilmagan bo‘lishi mumkin. BeautifulSoup bu kabi vaziyatlarda juda qulay.

messy_html = "<html><head><title>Title</title></head><body><p>Text"
soup = BeautifulSoup(messy_html, "lxml")

# Tozalangan HTML
print(soup.prettify())

1.3. Bir nechta class yoki atributlar bilan qidirish

Class va atributlar orqali filtr qo‘yish:

html = """
<div class="content main">
    <p data-id="1">Birinchi paragraf</p>
    <p data-id="2" class="highlight">Ikkinchi paragraf</p>
    <p data-id="3">Uchinchi paragraf</p>
</div>
"""
soup = BeautifulSoup(html, "lxml")

# Ma'lum atribut orqali qidirish
highlighted = soup.find("p", {"class": "highlight"})
print(highlighted.text)  # Ikkinchi paragraf

# Atributlar asosida qidirish
specific = soup.find("p", {"data-id": "2"})
print(specific.text)  # Ikkinchi paragraf

1.4. Elementlarni qo'shish, o‘zgartirish va o‘chirish

Element qo'shish:

pythonCopy codenew_tag = soup.new_tag("h2")
new_tag.string = "Yangi Sarlavha"
soup.body.append(new_tag)
print(soup.prettify())

Elementni o‘chirish:

soup.find("p", {"data-id": "1"}).decompose()
print(soup.prettify())

2. Scrapy bilan chuqurlashgan mavzular

2.1. AJAX ma'lumotlarini olish

Ba'zan ma'lumotlar AJAX orqali yuklanadi. Scrapy bilan so'rov yuborish orqali bu ma'lumotlarni olish mumkin:

import scrapy

class AjaxSpider(scrapy.Spider):
    name = "ajax_example"
    start_urls = ["https://example.com/ajax"]

    def parse(self, response):
        # JSON ma'lumotlarni olish
        data = response.json()
        for item in data['results']:
            yield {
                'name': item['name'],
                'price': item['price']
            }

2.2. Dinamik ma'lumotlarni olish (Selenium bilan birga ishlash)

Scrapy dinamik yuklangan sahifalarni olish uchun Selenium bilan birga ishlatilishi mumkin:

from scrapy import Spider
from selenium import webdriver
from selenium.webdriver.common.by import By

class SeleniumSpider(Spider):
    name = "selenium_spider"

    def __init__(self):
        self.driver = webdriver.Chrome()

    def start_requests(self):
        urls = ["https://example.com"]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        self.driver.get(response.url)
        elements = self.driver.find_elements(By.TAG_NAME, "h1")
        for element in elements:
            print(element.text)

2.3. Ma'lumotlarni qayta ishlash va filtrlash

Scrapy'da ma'lumotlarni filtrlash uchun ItemLoader dan foydalaning:

from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst, MapCompose

class MySpider(scrapy.Spider):
    name = "filtered_spider"
    start_urls = ["https://example.com"]

    def parse(self, response):
        loader = ItemLoader(item={}, response=response)
        loader.add_xpath("title", "//title/text()", TakeFirst())
        loader.add_xpath("price", "//span[@class='price']/text()", MapCompose(str.strip))
        yield loader.load_item()

2.4. Proksi va User-Agent o'zgartirish

Bloklanishni oldini olish uchun proksi yoki User-Agent ni sozlash:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}

Proksi sozlash:

HTTP_PROXY = 'http://your_proxy_here'
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

3. BeautifulSoup va Scrapy bo‘yicha qo‘shimcha maslahatlar

3.1. Scrapy bilan katta hajmdagi ma'lumotlarni boshqarish

Katta hajmdagi ma'lumotlarni yig‘ish uchun:

So'rovlarni vaqtni nazorat qilish: DOWNLOAD_DELAY
Bir vaqtning o'zida bajariladigan so'rovlar soni: CONCURRENT_REQUESTS

settings.py:

DOWNLOAD_DELAY = 1.5  # Har bir so‘rov orasidagi vaqt
CONCURRENT_REQUESTS = 8  # Bir vaqtning o‘zida 8 so‘rov

3.2. BeautifulSoup va Scrapy integratsiyasi

Agar veb-saytning ba'zi qismlari Scrapy uchun dinamik bo‘lsa, uni BeautifulSoup bilan qo‘shib ishlatish mumkin:

from bs4 import BeautifulSoup
import scrapy

class CombinedSpider(scrapy.Spider):
    name = "combined"

    def start_requests(self):
        urls = ["https://example.com"]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        soup = BeautifulSoup(response.text, "lxml")
        title = soup.find("title").text
        print(f"Sarlavha: {title}")

Quyida BeautifulSoup va Scrapy kutubxonalarini o'rganishda foydali bo'ladigan asosiy kalitlar (eng ko'p ishlatiladigan funksiyalar va xususiyatlar)

BeautifulSoup Kalitlari

1. HTMLni yuklash va tahlil qilish

BeautifulSoup(html, parser) - HTML matnini parse qilish uchun ishlatiladi.
- parser: "lxml", "html.parser" yoki "xml"

2. Elementlarni olish va qidirish

soup.find(name, attrs) - Birinchi mos keluvchi elementni topadi.
soup.find_all(name, attrs, limit) - Barcha mos elementlarni topadi.
- name: Teg nomi (masalan: "div", "p", "a")
- attrs: Atributlar ({"class": "my-class"}, {"id": "my-id"})
- limit: Topiladigan elementlar soni.
soup.select(css_selector) - CSS selector orqali elementlarni qidiradi.

3. Ma'lumot olish

element.text - Element ichidagi matnni qaytaradi.
element.get(attribute) - Element atributini qaytaradi (masalan: href, src).
element.attrs - Barcha atributlarni dictionary sifatida qaytaradi.

4. HTMLni o'zgartirish

soup.new_tag(tag_name, **attrs) - Yangi element yaratish.
element.append(new_tag) - Yangi elementni mavjud elementga qo‘shish.
element.decompose() - Elementni DOMdan o‘chirish.

5. Navigatsiya

soup.head - <head> elementini olish.
soup.body - <body> elementini olish.
element.parent - Elementning ota elementi.
element.children - Elementning to‘g‘ridan-to‘g‘ri bolalari (iterator).
element.descendants - Hamma bolalarini (rekursiv) olish.

Scrapy Kalitlari

1. Spider tuzilishi

name - Spiderning nomi.
start_urls - Boshlang‘ich URL'lar ro‘yxati.
parse(self, response) - Har bir URL uchun asosiy qayta ishlovchi funksiya.

2. So‘rov va javoblar

response.xpath(xpath_selector) - XPath orqali ma'lumotni olish.
response.css(css_selector) - CSS selector orqali ma'lumotni olish.
.get() - Birinchi mos keluvchi elementni qaytaradi.
.getall() - Barcha mos keluvchi elementlarni ro‘yxat sifatida qaytaradi.
.attrib - Elementning atributlarini qaytaradi (masalan: href, src).

3. Ma'lumotni saqlash

yield {"key": "value"} - Ma'lumotlarni chiqarish.
scrapy crawl spider_name -o filename.json - JSON formatida ma'lumotni saqlash.

4. So‘rov yuborish

scrapy.Request(url, callback) - Yangi URL'ga so‘rov yuborish.
scrapy.FormRequest(url, formdata, callback) - Forma ma'lumotlari bilan POST so‘rov yuborish.

5. Middleware va sozlash

DOWNLOAD_DELAY - So‘rovlar orasidagi kechikish (soniya).
CONCURRENT_REQUESTS - Bir vaqtning o‘zida bajariladigan so‘rovlar soni.
DEFAULT_REQUEST_HEADERS - So‘rov uchun maxsus boshqaruv (User-Agent va boshqalar).

Qo'shimcha Kalitlar

XPath

//tag - Hamma <tag> elementlarini topadi.
//tag[@attr='value'] - Ma'lum atributga ega elementni topadi.
//tag/text() - Element ichidagi matnni qaytaradi.

CSS Selectors

tag.class - Berilgan classdagi elementlar.
tag#id - Berilgan IDga ega elementlar.
tag[attr=value] - Berilgan atribut qiymatiga ega elementlar.

PreviousPython yordamida HTTP so'rovlarini skanerlash NextSQL Injection va Python

Last updated 1 day ago

from bs4 import BeautifulSoup html = """ <html> <head><title>Misol Sahifa</title></head> <body> <h1>Sarlavha</h1> <p class="paragraf">Bu oddiy paragraf.</p> <a href="https://example.com">Link</a> </body> </html> """ # HTMLni parse qilish soup = BeautifulSoup(html, "lxml")

from bs4 import BeautifulSoup html = """ <ul> <li>Element 1</li> <li>Element 2</li> <li>Element 3</li> </ul> """ soup = BeautifulSoup(html, "lxml") elements = soup.find_all("li") for element in elements: print(element.text)

import scrapy class ExampleSpider(scrapy.Spider): name = 'example' start_urls = ['https://example.com'] def parse(self, response): # Sarlavhani chiqarish title = response.xpath("//title/text()").get() print(f"Sarlavha: {title}")

from bs4 import BeautifulSoup with open("example.html", "r", encoding="utf-8") as file: content = file.read() soup = BeautifulSoup(content, "lxml") # Sahifaning title'ini olish print(soup.title.text)

html = """ <div class="content main"> <p data-id="1">Birinchi paragraf</p> <p data-id="2" class="highlight">Ikkinchi paragraf</p> <p data-id="3">Uchinchi paragraf</p> </div> """ soup = BeautifulSoup(html, "lxml") # Ma'lum atribut orqali qidirish highlighted = soup.find("p", {"class": "highlight"}) print(highlighted.text) # Ikkinchi paragraf # Atributlar asosida qidirish specific = soup.find("p", {"data-id": "2"}) print(specific.text) # Ikkinchi paragraf

import scrapy class AjaxSpider(scrapy.Spider): name = "ajax_example" start_urls = ["https://example.com/ajax"] def parse(self, response): # JSON ma'lumotlarni olish data = response.json() for item in data['results']: yield { 'name': item['name'], 'price': item['price'] }

from scrapy import Spider from selenium import webdriver from selenium.webdriver.common.by import By class SeleniumSpider(Spider): name = "selenium_spider" def __init__(self): self.driver = webdriver.Chrome() def start_requests(self): urls = ["https://example.com"] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): self.driver.get(response.url) elements = self.driver.find_elements(By.TAG_NAME, "h1") for element in elements: print(element.text)

from scrapy.loader import ItemLoader from scrapy.loader.processors import TakeFirst, MapCompose class MySpider(scrapy.Spider): name = "filtered_spider" start_urls = ["https://example.com"] def parse(self, response): loader = ItemLoader(item={}, response=response) loader.add_xpath("title", "//title/text()", TakeFirst()) loader.add_xpath("price", "//span[@class='price']/text()", MapCompose(str.strip)) yield loader.load_item()

from bs4 import BeautifulSoup import scrapy class CombinedSpider(scrapy.Spider): name = "combined" def start_requests(self): urls = ["https://example.com"] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): soup = BeautifulSoup(response.text, "lxml") title = soup.find("title").text print(f"Sarlavha: {title}")