Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ scheduler (every N minutes) or manual trigger via POST /api/jobs/:id/run
|---|---|---|
| Event bus | `lib/services/events/event-bus.js` | Plain `EventEmitter`; events: `jobs:runAll`, `jobs:runOne`, `jobs:status` |
| SSE broker | `lib/services/sse/sse-broker.js` | Per-userId `Set<ServerResponse>`; heartbeat every 25s; pushes job status to UI |
| Similarity cache | `lib/services/similarity-check/` | In-memory SHA-256 Set; refreshes hourly; cross-provider dedup by title+price+address |
| Similarity cache | `lib/services/similarity-check/` | In-memory SHA-256 Set; refreshes hourly; per-job cross-provider dedup by title+price+address |
| SqliteConnection | `lib/services/storage/SqliteConnection.js` | Singleton, WAL mode; `execute()`, `query()`, `withTransaction()` |
| Migrations | `lib/services/storage/migrations/` | Numbered JS files each exporting `up(db)`; checksum-tracked in `schema_migrations` |
| Extractor | `lib/services/extractor/` | Orchestrates Puppeteer + Cheerio; shared browser instance per job |
Expand Down Expand Up @@ -117,4 +117,4 @@ Tools: `list_jobs`, `get_job`, `list_listings`, `get_listing`, `get_current_date
- After building the task, run the tests
- New features must be tested
- New features must be properly documented with JsDoc
- You do **not** commit any changes, you do **not** create a new branch unless I told you so
- You do **not** commit any changes, you do **not** create a new branch unless I told you so
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
Finding an apartment or house in Germany can be stressful and
time-consuming.\
**Fredy** makes it easier: it automatically scrapes **ImmoScout24,
Immowelt, Immonet, eBay Kleinanzeigen, and WG-Gesucht** and notifies you
Immowelt, Immonet, eBay Kleinanzeigen, WG-Gesucht, and InBerlinWohnen** and notifies you
instantly via **Slack, Telegram, Email, ntfy, discord and more** when new
listings appear.

Expand All @@ -41,7 +41,7 @@ same listing twice.
## ✨ Key Features

- 🏠 Scrapes **ImmoScout24, Immowelt, Immonet, eBay Kleinanzeigen,
WG-Gesucht**
WG-Gesucht, InBerlinWohnen**
- ⚡ Instant notifications: Slack, Telegram, Email (SendGrid,
Mailjet), ntfy, discord
- 🔎 Uses the **ImmoScout Mobile API** (reverse engineered)
Expand Down
7 changes: 6 additions & 1 deletion lib/FredyPipelineExecutioner.js
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,11 @@ class FredyPipelineExecutioner {
*/
execute() {
return Promise.resolve(urlModifier(this._providerConfig.url, this._providerConfig.sortByDateParam))
.then(this._providerConfig.getListings?.bind(this) ?? this._getListings.bind(this))
.then((url) =>
this._providerConfig.getListings
? this._providerConfig.getListings.call(this, url, this._browser)
: this._getListings(url),
)
.then(this._normalize.bind(this))
.then(this._filter.bind(this))
.then(this._findNew.bind(this))
Expand Down Expand Up @@ -439,6 +443,7 @@ class FredyPipelineExecutioner {
const filteredIds = [];
const keptListings = listings.filter((listing) => {
const similar = this._similarityCache.checkAndAddEntry({
jobId: this._jobKey,
title: listing.title,
address: listing.address,
price: listing.price,
Expand Down
Loading