Clean indicators option fails when performing step - Start clean up of CrowdStrike malware family indicator events from MISP.

[jezkerwin]

I'm running the latest version of misp_import.py (did a git pull this morning), when I'm running misp_import.py --clean-indicators I'm getting the following error messages when the step get to:

Start clean up of CrowdStrike malware family indicator events from MISP.

[2024-01-05 11:06:34,077] INFO     processor/main       Start clean up of CrowdStrike malware family indicator events from MISP.
[2024-01-05 11:06:47,850] ERROR    processor/main       ('Connection broken: IncompleteRead(418397409 bytes read, 772955021 more expected)', IncompleteRead(418397409 bytes read, 772955021 more expected))
Traceback (most recent call last):
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/urllib3/response.py", line 712, in _error_catcher
    yield
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/urllib3/response.py", line 833, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(418397409 bytes read, 772955021 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/urllib3/response.py", line 934, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/urllib3/response.py", line 905, in read
    data = self._raw_read(amt)
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/urllib3/response.py", line 833, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/urllib3/response.py", line 729, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(418397409 bytes read, 772955021 more expected)', IncompleteRead(418397409 bytes read, 772955021 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/misp/MISP-tools/misp_import.py", line 183, in perform_local_cleanup
    importer.clean_crowdstrike_events(args.clean_reports, args.clean_indicators, args.clean_actors)
  File "/home/misp/MISP-tools/cs_misp_import/importer.py", line 222, in clean_crowdstrike_events
    perform_threaded_family_delete()
  File "/home/misp/MISP-tools/cs_misp_import/importer.py", line 148, in perform_threaded_family_delete
    executor.map(self.misp_client.delete_event, self.misp_client.search(eventinfo="Malware Family:%"))
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/pymisp/api.py", line 2715, in search
    response = self._prepare_request('POST', url, data=query)
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/pymisp/api.py", line 3761, in _prepare_request
    return self.__session.send(prepped, timeout=self.timeout, **settings)
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/requests/sessions.py", line 747, in send
    r.content
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/home/misp/MISP-tools/venv/lib64/python3.9/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(418397409 bytes read, 772955021 more expected)', IncompleteRead(418397409 bytes read, 772955021 more expected))
('Connection broken: IncompleteRead(418397409 bytes read, 772955021 more expected)', IncompleteRead(418397409 bytes read, 772955021 more expected))

I'm running the latest version of MISP v2.4.183 and all the requirements like falconpy and pymisp are up to date with the latest version.
The misp_import.ini file is all left to the defaults except for changing the relevant values to access the CS and MISP APIs.

[jshcodes]

Hi @jezkerwin -

I'm unable to recreate this but am still trying a few ideas.

The error above originates from urllib3 and appears to be a communications issue. Is there a proxy or some other network hardware in the environment that could be impacting import?