Skip to content

add support for scraping with surf (tls impersonation)#6806

Draft
feederbox826 wants to merge 1 commit intostashapp:developfrom
feederbox826:scrape-surf
Draft

add support for scraping with surf (tls impersonation)#6806
feederbox826 wants to merge 1 commit intostashapp:developfrom
feederbox826:scrape-surf

Conversation

@feederbox826
Copy link
Copy Markdown
Member

adds https://github.com/enetx/surf support

this will let us bypass basic bot detection tests and hopefully the amount of external python dependencies. This will likely succeed even where cloudscraper fails

It is not documented, just a proof of concept and if the concept/implementation needs work

can be tested with https://github.com/feederbox826/scrapers/blob/main/scrapers/tls-fprint.yml

add

driver:
  useSurf: true

you can look up the returned fingerprint at https://ja3.zone/

expected peetprint is 1d4ffe9b0e34acac0bd883fa7f79d7b5

Code was sloppily done by copying above implementation and combining it with surf readme. User-Agent is explicitly dropped since it would defeat our anti-fingerprinting efforts and other headers excluded (open for discussion)

@discourse-stashapp
Copy link
Copy Markdown

This pull request has been mentioned on Stash Forum. There might be relevant details there:

https://discourse.stashapp.cc/t/jav-english-scraper-how-do-i-get-it-to-work/6669/8

@feederbox826 feederbox826 marked this pull request as draft April 11, 2026 04:51
@feederbox826
Copy link
Copy Markdown
Member Author

needs golang version bump, will wait for other backend merges to hit, but would help greatly simplify scrapers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants