feat: add InBerlinWohnen provider#367
Conversation
| async function fetchPage(page) { | ||
| const pageUrl = new URL(url); | ||
| if (page > 1) pageUrl.searchParams.set('page', String(page)); | ||
| const response = await fetch(pageUrl, { headers: { Accept: 'text/html' } }); |
There was a problem hiding this comment.
why do you use fetch instead of puppeteer? Fetch is known to be caught by bot detection pretty easily
There was a problem hiding this comment.
Good point. I replaced the direct listing-page fetches with the shared CloakBrowser. The provider now declares requiresBrowser, receives the job browser in getListings(url, browser), and reuses it across all result pages and detail requests, including the configured proxy and bot-detection handling. Structurally incomplete pages now fail the crawl instead of looking like an empty result.
| }, | ||
| normalize, | ||
| filter: applyBlacklist, | ||
| activeTester: isListingActive, |
There was a problem hiding this comment.
would you mind also implementing fetchDetails?
There was a problem hiding this comment.
Implemented. fetchDetails reuses the shared browser and has explicit parsers for Berlinovo, degewo, GESOBAU, Gewobag, HOWOGE, and WBM. STADT UND LAND currently exposes no useful narrative detail block, so it keeps the structured snapshot description. Unknown hosts, bot detection, and parser failures always return the original listing. The live smoke test verifies one current page for every enrichable partner domain.
| const newCache = new Set(); | ||
| for (const entry of allEntries) { | ||
| newCache.add(toHash(entry?.title, entry?.price, entry?.address)); | ||
| newCache.add(toHash(entry?.job_id, entry?.title, entry?.price, entry?.address)); |
There was a problem hiding this comment.
What do you want to accomplish here?
There was a problem hiding this comment.
The goal is to keep cross-provider similarity deduplication within one job while preventing one job from hiding the same listing from another job. Persistent listing uniqueness is already scoped by (job_id, hash), so the in-memory cache now follows the same boundary. I expanded the tests and fixed the related hard-delete path: hydration and eviction include job_id, addresses use the same normalization as storage, and deleting an already hidden duplicate no longer evicts the active listing cache entry.
|
Addressed all three review threads in |
| } | ||
|
|
||
| if (!browser && matchedProvider.config.getListings == null) { | ||
| if (!browser && (matchedProvider.config.getListings == null || matchedProvider.config.requiresBrowser)) { |
There was a problem hiding this comment.
Why is this needed? I know you're using claude.. While this is fine, you should tell it not to push in vars/methods for the sake of doing so.
please remove the requiredBrowser again from where you/claude added it. The shared browser instance should be available in your provider whether or not you make use out of it ;)
The implementation of your provider is already much more complicated than the rest.
Thanks.
…r to getListings The shared browser is now launched once per job and passed to getListings unconditionally, so a provider can use it or ignore it. Drops the requiresBrowser flag from the provider config, the pipeline executioner, the job execution service, the type doc, and the two tests that asserted the flag-gated behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the patience on this, and for the clear steer. And yes, built with Claude, no secret there. Done: I removed requiresBrowser again. The shared browser now just gets passed into getListings, whether the provider uses it or not. Also stripped it from the tests and the type doc. On the provider being heavier than the rest, fair point. What's left is mostly the pagination and the fetchDetails you asked for. If any part still feels like too much, tell me which and I'll trim it. |
2 participants
Summary
inberlinwohnen.deas a native Fredy providerImplementation
The provider reuses one shared browser for pagination and optional detail enrichment, preserving Fredy's proxy, timeout, and bot-detection behavior. It normalizes stable partner IDs, canonical allowlisted deeplinks, total rent, size, rooms, address, image, provider details, WBS, and availability.
fetchDetailsuses explicit selectors for Berlinovo, degewo, GESOBAU, Gewobag, HOWOGE, and WBM. STADT UND LAND and any failed or unsupported detail extraction retain the structured InBerlinWohnen snapshot description.Similarity hashes now include
jobId, preserving cross-provider deduplication within a job without suppressing the same listing in another job. Cache hydration, address normalization, hard-delete eviction, and hidden-duplicate handling use the same identity.Verification
npm run test:offline: 41 files, 252 tests passed, 1 live-only test skippednpm test -- test/provider/inberlinwohnen.test.js --testTimeout=180000: 5 live tests passed, 13 offline-only tests skippednpm run lint: passednpm run format:check: passednpm run build:frontend: passed