Initial commit of Dataminr Pulse integration for elastic#17886
Initial commit of Dataminr Pulse integration for elastic#17886pgifford-dataminr wants to merge 2 commits intoelastic:mainfrom
Conversation
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
packages/dataminr_pulse/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
| "expires_at": auth_result.new_state.expires_at, | ||
| "last_fetch_timestamp": fetch_ts, | ||
| "retry_count": 0, | ||
| "want_more": false |
There was a problem hiding this comment.
🟡 Medium stream/stream.yml.hbs:241
Line 241 hardcodes want_more: false regardless of whether the API returns a nextPage cursor, so pagination stops after the first page even when more data is available. This causes multi-page alerts to be fetched across multiple execution intervals instead of within the same run. Consider setting want_more: has(api_body.nextPage) && api_body.nextPage != "" to continue fetching pages until the cursor is exhausted.
- "want_more": false🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/dataminr_pulse/data_stream/alerts/agent/stream/stream.yml.hbs around line 241:
Line 241 hardcodes `want_more: false` regardless of whether the API returns a `nextPage` cursor, so pagination stops after the first page even when more data is available. This causes multi-page alerts to be fetched across multiple execution intervals instead of within the same run. Consider setting `want_more: has(api_body.nextPage) && api_body.nextPage != ""` to continue fetching pages until the cursor is exhausted.
|
/test |
andrewkroh
added
dashboard
|
The issues raised by the review bot are valid. Please address them. |
4 participants
This is a new PR that covers the same changes as this PR. The original submitter on that PR is no longer at the company so I am opening this new PR and the other one can be closed.
This is the very first integration for Dataminr Pulse with Elastic for cyber alerts.
We have created an integration that uses Dataminr Pulse V4 APIs and pull Dataminr alerts periodically into Elastic.
The integration creates a data stream with Dataminr alert fields mapped to ECS specific and custom fields. The ReadMe lists all the field mappings done for the integration.
The integration also provides a couple of dashboards to view the results.
Checklist
changelog.ymlfile.How to test this PR locally
One can run static, pipeline, asset and policy tests locally to verify the integration.
Commands:
elastic-package test static
elastic-package test pipeline --data-streams alerts
elastic-package test asset
elastic-package test policy
Screenshots
Screenshot of Integration set up page:
Screenshot of dashboards:
