Website's search function with Python requests

Question

Cannot successfully implement website's search function using python requests. Performing search at the website results with https://bidbaits.ru/search/#used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0&orderBy=2, but performing it with requests return nothing:

r = requests.get('https://bidbaits.ru/search/#used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0&orderBy=2')
print(r.text)

Also, how is it possible to send query as a parameter?

question from:https://stackoverflow.com/questions/65920704/websites-search-function-with-python-requests

Answer 1

This page uses JavaScript to display elements on page but requests/BeautifulSoup can't run JavaScript.

But using DevTools in Chrome/Firefox you can see JavaScript (AJAX/XHR) reads data from other url

https://bidbaits.ru/searchresults/?page=1&seo=SEARCH&orderBy=2&used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0

which works with requests if you use header Referer.
(I checked it manually running code with different headers)

I use module json only to display it with indentations. You can work directly with dictionary data.

import requests

#s = requests.Session()

headers = {
    #'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0',
    #'Accept': 'application/json, text/javascript, */*; q=0.01',
    #'X-Requested-With': 'XMLHttpRequest',  # AJAX
    'Referer': 'https://bidbaits.ru/',
}

url = 'https://bidbaits.ru/searchresults/?page=1&seo=SEARCH&orderBy=2&used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0'
r = requests.get(url, headers=headers)    

data = r.json() 

#print(data)

import json

print(json.dump(data, indent=2))

Result:

{
  "totalCount": "4",
  "totalCountMsg": "u041du0430u0439u0434u0435u043du043e <span class="b-bsr_count">4</span> u0432u0430u0440u0438u0430u043du0442u0430",
  "items": [
    {
      "itemId": "544665",
      "name": "Jackson Cyarl Blade 48F 4g Sinking Ama Gaeru Rain Frog",
      "price": "700",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/544665/",
      "imageUrl": "//c.bidbaits.ru/items/544665/thumb/1580816722-0419.JPG",
      "userName": "Survivor-Demon",
      "userRating": "658",
      "userUrl": "/users/survivor-demon/",
      "typeId": "3",
      "bets": "0 u0441u0442u0430u0432u043eu043a",
      "isTrusted": true,
      "timeLeft": 235373
    },
    {
      "itemId": "735811",
      "name": "Jackson Cyarl 20mm 1,8g sinking",
      "price": "450",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/735811/",
      "imageUrl": "//c.bidbaits.ru/items/735811/thumb/1600933789-5099.jpg",
      "userName": "Dmitr79",
      "userRating": "22",
      "userUrl": "/users/dmitr79/",
      "typeId": "3",
      "bets": "0 u0441u0442u0430u0432u043eu043a",
      "isTrusted": false,
      "timeLeft": 572273
    },
    {
      "itemId": "840409",
      "name": "Jackson Cyarl 25S (1,8 u0433.) OT",
      "price": "700",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/840409/",
      "imageUrl": "//c.bidbaits.ru/items/840409/thumb/1611249724-4935.JPG",
      "userName": "Ima2021",
      "userRating": "6",
      "userUrl": "/users/ima2021/",
      "typeId": "3",
      "bets": "0 u0441u0442u0430u0432u043eu043a",
      "isTrusted": false,
      "timeLeft": 2050658
    },
    {
      "itemId": "840416",
      "name": "Jackson Cyarl 25S (1,8 u0433.) PA",
      "price": "700",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/840416/",
      "imageUrl": "//c.bidbaits.ru/items/840416/thumb/1611250255-9944.JPG",
      "userName": "Ima2021",
      "userRating": "6",
      "userUrl": "/users/ima2021/",
      "typeId": "3",
      "bets": "0 u0441u0442u0430u0432u043eu043a",
      "isTrusted": false,
      "timeLeft": 2051635
    }
  ],
  "currentPage": 1,
  "hasNextPage": false,
  "isSEO": false,
  "pageTitle": "u041fu043eu0438u0441u043a: cyarl",
  "

---

Working directly with data

data = r.json() 

#print(data)
#import json
#print(json.dumps(data, indent=2))

for item in data['items']:
    print('name:', item['name'])
    print('price:', item['price'])    
    print('---')

Result:

name: Jackson Cyarl Blade 48F 4g Sinking Ama Gaeru Rain Frog
price: 700
---
name: Jackson Cyarl 20mm 1,8g sinking
price: 450
---
name: Jackson Cyarl 25S (1,8 г.) OT
price: 700
---
name: Jackson Cyarl 25S (1,8 г.) PA
price: 700
---