OSError: [Errno 24] Too many open files: 'photos/icnZ2R8PcDs.jpg'

[dbl001]

What do recommend setting max_open_files to?

images = [Image.open("photos/"+filepath) for filepath in tqdm(img_names[:5000])]
image_names = img_names[:5000]
image_embeddings = img_embeddings[:5000]

54%|███████████████████▍                | 2693/5000 [00:00<00:00, 13545.87it/s]
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 images = [Image.open("photos/"+filepath) for filepath in tqdm(img_names[:5000])]
      2 image_names = img_names[:5000]
      3 image_embeddings = img_embeddings[:5000]

Input In [4], in <listcomp>(.0)
----> 1 images = [Image.open("photos/"+filepath) for filepath in tqdm(img_names[:5000])]
      2 image_names = img_names[:5000]
      3 image_embeddings = img_embeddings[:5000]

File ~/tensorflow-metal/lib/python3.8/site-packages/PIL/Image.py:2968, in open(fp, mode, formats)
   2965     filename = fp
   2967 if filename:
-> 2968     fp = builtins.open(filename, "rb")
   2969     exclusive_fp = True
   2971 try:

OSError: [Errno 24] Too many open files: 'photos/icnZ2R8PcDs.jpg'

% ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       11136
-n: file descriptors                8192
(base) davidlaxer@x86_64-apple-darwin13 notebooks % 

[MaartenGr]

What do recommend setting max_open_files to?

I am not entirely sure what you are referring to. What do you mean with max_open_files?

Checking the code that you posted it seems that you are taking a subset of 5000 images and trying to open them all at once. In Concept, you only need to give the path to the images and not the images themselves as it will indeed result in many issues with holding all those images in memory. I would highly recommend batching them instead as is being done by Concept:

Concept/concept/_model.py

Lines 197 to 231 in d270607

	def _embed_images(self,
	images: List[str]) -> np.ndarray:
	""" Embed the images
	Not entirely sure why but the RAM ramps up
	if I do not close the images between batches.
	So I make a copy out of those and simply
	close them in between without touching the original
	images.
	Arguments:
	images: A list of paths to each image
	Returns:
	embeddings: The image embeddings
	"""
	# Prepare images
	batch_size = 128
	nr_iterations = int(np.ceil(len(images) / batch_size))
	# Embed images per batch
	embeddings = []
	for i in tqdm(range(nr_iterations)):
	start_index = i * batch_size
	end_index = (i * batch_size) + batch_size
	images_to_embed = [Image.open(filepath) for filepath in images[start_index:end_index]]
	img_emb = self.embedding_model.encode(images_to_embed, show_progress_bar=False)
	embeddings.extend(img_emb.tolist())
	# Close images
	for image in images_to_embed:
	image.close()
	return np.array(embeddings)

[dbl001]

Agreed. I was running the code example in this notebook: https://github.com/MaartenGr/Concept/blob/main/notebooks/Concept.ipynb

…

On May 9, 2022, at 10:42 PM, Maarten Grootendorst ***@***.***> wrote:  What do recommend setting max_open_files to? I am not entirely sure what you are referring to. What do you mean with max_open_files? Checking the code that you posted it seems that you are taking a subset of 5000 images and trying to open them all at once. In Concept, you only need to give the path to the images and not the images themselves as it will indeed result in many issues with holding all those images in memory. I would highly recommend batching them instead as is being done by Concept: https://github.com/MaartenGr/Concept/blob/d270607d6ea4d789a42d54880ab4a0c977bb69ce/concept/_model.py#L197-L231 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[MaartenGr]

Ah okay! I'll make sure to update the notebook in the next release.