"idf.py create-project" fails with "PermissionError: [Errno 13] Permission denied: ..."

[oskarboer]

Created dev environment with nix --experimental-features 'nix-command flakes' develop github:mirrexagon/nixpkgs-esp-dev#esp32-idf

And tried to create an empty project causes permission errors:

[capreme@thinkpad:~/Projects/tmp_fresh]$ idf.py create-project hello
WARNING: Python interpreter "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/bin/python3.12" used to start idf.py is not from installed venv "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/python-env"
Executing action: create-project
Traceback (most recent call last):
  File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 855, in <module>
    main()
  File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 745, in main
    cli(argv, prog_name=PROG, complete_var=SHELL_COMPLETE_VAR)
  File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1720, in invoke
    return _process_result(rv)
           ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1657, in _process_result
    value = ctx.invoke(self._result_callback, value, **ctx.params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 640, in execute_tasks
    task(ctx, global_args, task.action_args)
  File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 212, in __call__
    self.callback(self.name, context, global_args, **action_args)
  File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf_py_actions/create_ext.py", line 76, in create_new
    func_action_map[action](target_path, action_args['name'])
  File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf_py_actions/create_ext.py", line 49, in create_project
    os.rename(os.path.join(main_folder, 'main.c'), os.path.join(main_folder, '.'.join((name, 'c'))))
PermissionError: [Errno 13] Permission denied: '/home/capreme/Projects/tmp_fresh/hello/main/main.c' -> '/home/capreme/Projects/tmp_fresh/hello/main/hello.c'

idf --version works and reports a version

[capreme@thinkpad:~/Projects/tmp_fresh]$ idf.py --version
WARNING: Python interpreter "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/bin/python3.12" used to start idf.py is not from installed venv "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/python-env"
fatal: not a git repository: '/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/.git'
WARNING: Git version unavailable, reading from source
ESP-IDF v5.4.0

Running all on Nixos uname -a: Linux thinkpad 6.12.12 #1-NixOS SMP PREEMPT_DYNAMIC Sat Feb 1 17:39:40 UTC 2025 x86_64 GNU/Linux

Might be related issue 53, but in my case issue is with main.c file not CMakeLists.txt and I assume that it happends at a different step.

[backspace-phill]

Okay I have the same Problem but I think I got it to work by just giving my user read and write permission for the whole folder:

chmod -R +rw hello

after that you can build the project.

[Scot-Survivor]

Any updates to this?

[lukeblume]

you need to run it with sudo

[Scot-Survivor]

This tool shouldn't need super user privileges, what does it do that requires it?

[lukeblume]

nope you are correct. I commented that when I had gotten rid of that error but not yet build the project succesfully.
These two commands ended up fixing it for me:
sudo chown -R <user>:users <path_to_project_file>
chmod -R u+rw <path to project folder>

[mirrexagon]

The problem seems to be that idf.py create-project copies a sample project from $IDF_PATH/tools/templates/sample_project. Because $IDF_PATH is in the Nix store, all the files and directories are read-only.

idf.py tries to ensure that permissions aren't copied: https://github.com/espressif/esp-idf/blob/v5.4.1/tools/idf_py_actions/create_ext.py#L45-L46
But that only seems to affect files, not directories. When I try idf.py create-project, the main directory within the created project directory is still read-only (and has a last modified date in 1970, which is also normal for Nix store files).

#54 presumably fixed this previously. I will need to see if reapplying that fixes it again, or if something else needs to be done.

[mirrexagon]

The functions used to do the copying have changed since #54, so that exact fix doesn't work.

espressif/esp-idf@524f44d looks like they had this problem on Windows and fixed it there. The commit message mentions there is a test for copying from read-only ESP-IDF - I wonder if it is succeeding for them but still this happens for us, or the situation is a bit different (eg. permissions tested are a bit different from our case).

The function they use to do the copying (shutil.copytree()) has documentation that explicitly says it copies directory permissions and times: https://docs.python.org/3.12/library/shutil.html#shutil.copytree

[mirrexagon]

I see that I'm not the only Nixer to find this out: python/cpython#44602 (comment)

[mirrexagon]

Based on the report of a known working commit of this repo and git bisect, it seems like the issue started after upgrading from ESP-IDF 5.3 to 5.4 (f3434d1).

When I get more time I will look at what changed in ESP-IDF and what can be done about it. If anyone wants to look, look for changes to https://github.com/espressif/esp-idf/blob/v5.4.1/tools/idf_py_actions/create_ext.py#L41-L52 between 5.3.1 and 5.4. I assume it is whatever changed between what worked in #54 and now.

[Sped0n]

Have done some investigation myself, and it seems this issue is related to shutil.copytree. I tried running the corresponding test with pytest, and it failed on my work computer at least. However, when I switched to the implementation of v5.3 (which use distutils), it passed.

However, distutils was removed in python 3.12, and this is the reason they changed the implementation.

---

Edited

Right now I don't have a particularly good solution. The closest approach I can think of is the following monkey patch, but it's still quite tricky.

diff --git a/tools/idf_py_actions/create_ext.py b/tools/idf_py_actions/create_ext.py
index 00d9fa2881..ec41d46e24 100644
--- a/tools/idf_py_actions/create_ext.py
+++ b/tools/idf_py_actions/create_ext.py
@@ -2,14 +2,57 @@
 # SPDX-License-Identifier: Apache-2.0
 import os
 import re
+import shutil
 import sys
-from shutil import copyfile
-from shutil import copytree
 from typing import Dict

 import click
 from idf_py_actions.tools import PropertyDict

+def custom_copytree(src: str, dst: str, dirs_exist_ok: bool = False) -> str:
+    if not os.path.exists(src):
+        raise FileNotFoundError(f"Source directory '{src}' does not exist.")
+    if not os.path.isdir(src):
+        raise NotADirectoryError(f"Source path '{src}' is not a directory.")
+    if (not dirs_exist_ok) and os.path.exists(dst) and os.listdir(dst):
+        raise FileExistsError(
+            f"Destination directory '{dst}' already exists and is not empty and dirs_exist_ok is False."
+        )
+
+    try:
+        os.makedirs(dst, exist_ok=True)
+    except FileExistsError:
+        raise NotADirectoryError(f"Destination path '{dst}' exists and is not a directory.")
+    except OSError as e:
+        raise shutil.Error(f"Could not create destination directory '{dst}': {e}")
+
+    names = os.listdir(src)
+    errors = []
+    for name in names:
+        srcname = os.path.join(src, name)
+        dstname = os.path.join(dst, name)
+        try:
+            if os.path.islink(srcname):
+                if os.path.isdir(srcname):  # This follows the link
+                    custom_copytree(srcname, dstname, dirs_exist_ok=True)
+                else:
+                    shutil.copyfile(srcname, dstname)
+            elif os.path.isdir(srcname):
+                custom_copytree(srcname, dstname, dirs_exist_ok=True)
+            else:
+                shutil.copyfile(srcname, dstname)
+        except shutil.Error as e:
+            if hasattr(e, 'args') and e.args and isinstance(e.args[0], list):
+                errors.extend(e.args[0])
+            else:
+                errors.append((srcname, dstname, str(e)))
+        except OSError as why:
+            errors.append((srcname, dstname, str(why)))
+
+    if errors:
+        raise shutil.Error(errors)
+    return dst
+

 def get_type(action: str) -> str:
     return action.split('-')[1]
@@ -39,11 +82,9 @@ def is_empty_and_create(path: str, action: str) -> None:


 def create_project(target_path: str, name: str) -> None:
-    copytree(
+    custom_copytree(
         os.path.join(os.environ['IDF_PATH'], 'tools', 'templates', 'sample_project'),
         target_path,
-        # 'copyfile' ensures only data are copied, without any metadata (file permissions)
-        copy_function=copyfile,
         dirs_exist_ok=True,
     )
     main_folder = os.path.join(target_path, 'main')
@@ -53,11 +94,9 @@ def create_project(target_path: str, name: str) -> None:


 def create_component(target_path: str, name: str) -> None:
-    copytree(
+    custom_copytree(
         os.path.join(os.environ['IDF_PATH'], 'tools', 'templates', 'sample_component'),
         target_path,
-        # 'copyfile' ensures only data are copied, without any metadata (file permissions)
-        copy_function=copyfile,
         dirs_exist_ok=True,
     )
     os.rename(os.path.join(target_path, 'main.c'), os.path.join(target_path, '.'.join((name, 'c'))))

At least related test are passed.

/media/spedon/eden/esp-idf/tools/test_build_system on git master [!?] via py v3.11.12                                                                                         took 2s
❯ pytest -k test_create_project
================================================================================ test session starts =================================================================================
platform linux -- Python 3.11.12, pytest-8.3.5, pluggy-1.5.0
rootdir: /media/spedon/eden/esp-idf/tools/test_build_system
configfile: pytest.ini
plugins: embedded-1.16.1, ignore-test-results-0.3.0, timeout-2.4.0, rerunfailures-15.1
collected 103 items / 101 deselected / 2 selected

test_common.py::test_create_project
----------------------------------------------------------------------------------- live log call ------------------------------------------------------------------------------------
2025-05-16 13:54:20 INFO Check that command for creating new project will fail if the target folder is not empty.
2025-05-16 13:54:20 INFO Check that command for creating new project will fail if the target path is file.
PASSED                                                                                                                                                                         [ 50%]
test_common.py::test_create_project_with_idf_readonly
----------------------------------------------------------------------------------- live log call ------------------------------------------------------------------------------------
2025-05-16 13:54:22 INFO Check that command for creating new project will success if the IDF itself is readonly.
PASSED                                                                                                                                                                         [100%]

================================================================================== warnings summary ==================================================================================
test_common.py::test_create_project
  test_common.py:218: PytestExperimentalApiWarning: record_xml_attribute is an experimental feature
    def test_create_project(idf_py: IdfPyFunc, idf_copy: Path) -> None:

test_common.py::test_create_project_with_idf_readonly
  test_common.py:234: PytestExperimentalApiWarning: record_xml_attribute is an experimental feature
    @pytest.mark.skipif(sys.platform == 'darwin', reason='macos runner is a shell executor, it would break the file system')

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== 2 passed, 101 deselected, 2 warnings in 3.98s ====================================================================

I will try to create a stable reproduction from the idf's docker image. If can, I will try to report this issue to upstream.

[Sped0n]

should be fixed by espressif/esp-idf@b3f24a9