I’m using selenium to open the webpage https://www.optionistics.com/quotes/stock-option-chains/GOOG It was loaded successfully at first.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from selenium.webdriver.firefox.options import Options
options = Options()
options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()),options=options)
driver.get("https://www.optionistics.com/quotes/stock-option-chains/GOOG")
And then I came across the human verification after I changed the settings in the web page (for example, the “report date”) and the form submitted to fetch new data. The problem is, I can’t pass the check myself by clicking checkbox. After click, it would pend for a while before refreshd and showed the checkbox again. I clicked and it won’t let me through
I’m not meant to bypass it. I just want to get myself verified. Is the Cloudflare expecting something from the header?
CloudFlare has an issue with you if you’ve previously failed human checks or are using a contained browser, with or without VPN and ad blocking extensions. They check your browsing history, and what other human checks you may or may not have failed, just by you using the internet. If they can’t set a validation cookie for tracking you between other CloudFlare services, they may treat you as a bot and hang your connection.
I typically fail the “Select all the motorcycles” test because they show me scooters all the time. By doing that, my computer gets marked as suspicious, slowing down some of my connections that need to go through CloudFlare, sometimes just hanging my connection for a little time.
A serious question does arise, though. Have you tried using a regular chromium type browser to see its headers sent to CloudFlare to identify what might need to be sent?
Yes, all the sessions with regular browsers are working as expected. The problem only occurs in the browser launched by selenium. And the check didn’t appear at all when I changed to undetected_chromedriver, even including some automations that indeed by bot.