It takes a long time to enum IVssExamineWriterMetadata::Components files

[HuAaron]

I pulled the latest master branch of VssApi code, and run a sample test. I found that it took very long time to list the system writer component files.
I guess for IReadOnlyList it was lazy load, every time we got item it invoke COM method?

Thanks!

Here is the sample code:

using System.Diagnostics;
using System.Runtime.InteropServices;
using Vanara.PInvoke;
using Vanara.PInvoke.VssApi;

namespace ConsoleApp1
{
    internal class Program
    {
        [System.Runtime.InteropServices.DllImport("ole32.dll")]
        public static extern int CoInitializeSecurity(IntPtr pVoid, int
            cAuthSvc, IntPtr asAuthSvc, IntPtr pReserved1, int level,
            int impers, IntPtr pAuthList, int dwCapabilities, IntPtr
            pReserved3);
        static void Main(string[] args)
        {
            try
            {
                // Invoke CoInitializeSecurity to get system writer listed.
                Marshal.ThrowExceptionForHR(CoInitializeSecurity(IntPtr.Zero, -1, IntPtr.Zero, IntPtr.Zero,
                    1, 3, IntPtr.Zero, 0x00, IntPtr.Zero));
                Console.WriteLine("Initializing VSS Backup Components...");
                IVssBackupComponents ppBack;
                HRESULT hr = VssFactory.CreateVssBackupComponents(out ppBack);
                hr.ThrowIfFailed();
                Console.WriteLine("Successfully created IVssBackupComponents instance.");
                ppBack.InitializeForBackup(null);
                ppBack.SetBackupState(true, true, VSS_BACKUP_TYPE.VSS_BT_FULL, false);
                ppBack.SetContext(VSS_SNAPSHOT_CONTEXT.VSS_CTX_BACKUP);
                IVssAsync getAsync = ppBack.GatherWriterMetadata();
                getAsync.Wait();
                getAsync.QueryStatus(out hr);
                hr.ThrowIfFailed();
                foreach (IVssExamineWriterMetadata writeMeta in ppBack.WriterMetadata)
                {
                    writeMeta.GetIdentity(out var pidInstance,
                            out var pidWriter,
                            out var WriteName,
                            out var pInstanceName,
                            out var usage,
                            out var source);
                    Console.WriteLine("Get compoment for writer:{0}", WriteName);

                    foreach (IVssWMComponent component in writeMeta.Components)
                    {
                        Console.WriteLine(DateTime.Now.ToString() + "start add expand path for writer:{0}", WriteName);
                        Stopwatch stopwatch = new Stopwatch();
                        stopwatch.Start();
                        for (int k = 0; k < component.Files.Count; k++)
                        {
                            // DO nothing here;
                            // For system writer it took very very long time.
                        }
                        stopwatch.Stop();
                        TimeSpan elapsedTime = stopwatch.Elapsed;
                        Console.WriteLine(DateTime.Now.ToString() + "add expand path for writer:{0}, file count:{1}, took:{2} seconds",
                            WriteName, component.Files.Count, elapsedTime.TotalSeconds);
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error initializing VSS: {0}" + ex);
            }
        }
    }
}

For normal case, it should take about 26 seconds. But here it took 400+ seconds.

[dahall]

By "normal case" do you mean that similar code in C++ took 26 seconds or are you just extrapolating from the time/file of the other writers? Each writer is distinct in the way that processes files, so extrapolations really aren't valid. If this is running slower than C++ code, then I really don't have an explanation.

[HuAaron]

By "normal case" do you mean that similar code in C++ took 26 seconds or are you just extrapolating from the time/file of the other writers? Each writer is distinct in the way that processes files, so extrapolations really aren't valid. If this is running slower than C++ code, then I really don't have an explanation.

For normal case, I tested in AlphaVSS (https://github.com/alphaleonis/AlphaVSS), not c++ code and it took nearly 26 seconds.

VssComponentDescriptor.cs line 65:

         foreach (VssWMFileDescriptor file in component.Files)
         {
            affectedPaths.Add(file.GetExpandedPath());
            affectedVolumes.Add(file.GetAffectedVolume(host));
         }

I also tested it with Windows built-in diskshadow.exe tool and took even less time.
DiskShadow:

set verbose on
begin backup
add volume C:
create 
end backup

I guess that using speicified class VssWMFileDescriptor but not interface IVssWMComponent that would not lazy execute the com+ result?

[dahall]

In looking at the implementation, it appears the only overhead is converting from the COM interface pointer in C++ to the .NET COM wrapper (RCW) that makes it accessible in .NET languages. This will create some overhead for sure, but it shouldn't be significant. I looked at the AlphaVSS code and it is does almost the exact same thing. I'm out of ideas.

[HuAaron]

I found it took less than 1 second.

code:

using Alphaleonis.Win32.Vss;
using System.Diagnostics;
using System.Runtime.InteropServices;
using Vanara.PInvoke;
using Vanara.PInvoke.VssApi;

namespace ConsoleApp1
{
    internal class Program
    {
        [System.Runtime.InteropServices.DllImport("ole32.dll")]
        public static extern int CoInitializeSecurity(IntPtr pVoid, int
            cAuthSvc, IntPtr asAuthSvc, IntPtr pReserved1, int level,
            int impers, IntPtr pAuthList, int dwCapabilities, IntPtr
            pReserved3);
        static void Main(string[] args)
        {
            Marshal.ThrowExceptionForHR(CoInitializeSecurity(IntPtr.Zero, -1, IntPtr.Zero, IntPtr.Zero,
    1, 3, IntPtr.Zero, 0x00, IntPtr.Zero));
            try
            {

                Console.WriteLine("============================= TestAplaphAsync =============================");
                TestAplaphAsync();

                Console.WriteLine("============================= TestVanara =============================");
                TestVanara();
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error initializing VSS: {0}" + ex);
            }
        }

        static void TestVanara()
        {
            // Invoke CoInitializeSecurity to get system writer listed.
            //Marshal.ThrowExceptionForHR(CoInitializeSecurity(IntPtr.Zero, -1, IntPtr.Zero, IntPtr.Zero,
            //    1, 3, IntPtr.Zero, 0x00, IntPtr.Zero));
            Console.WriteLine("Initializing VSS Backup Components...");
            //IVssBackupComponents ppBack;
            HRESULT hr = VssFactory.CreateVssBackupComponents(out var ppBack);
            hr.ThrowIfFailed();
            Console.WriteLine("Successfully created IVssBackupComponents instance.");
            ppBack.InitializeForBackup(null);
            ppBack.SetBackupState(true, true, VSS_BACKUP_TYPE.VSS_BT_FULL, false);
            ppBack.SetContext(VSS_SNAPSHOT_CONTEXT.VSS_CTX_BACKUP);
            IVssAsync getAsync = ppBack.GatherWriterMetadata();
            getAsync.Wait();
            getAsync.QueryStatus(out hr);
            hr.ThrowIfFailed();
            foreach (var writeMeta in ppBack.WriterMetadata)
            {
                writeMeta.GetIdentity(out var pidInstance,
                        out var pidWriter,
                        out var WriteName,
                        out var pInstanceName,
                        out var usage,
                        out var source);
                Console.WriteLine("Get compoment for writer:{0}", WriteName);

                foreach (var component in writeMeta.Components)
                {
  
                    Console.WriteLine(DateTime.Now.ToString() + "start add expand path for writer:{0}", WriteName);
                    Stopwatch stopwatch = new Stopwatch();
                    stopwatch.Start();
                    for (int k = 0; k < component.Files.Count; k++)
                    {
                        // DO nothing here;
                        // For system writer it took very very long time.
                    }
                    stopwatch.Stop();
                    TimeSpan elapsedTime = stopwatch.Elapsed;
                    Console.WriteLine(DateTime.Now.ToString() + "add expand path for writer:{0}, file count:{1}, took:{2} seconds",
                        WriteName, component.Files.Count, elapsedTime.TotalSeconds);
                }
            }
        }
        static void TestAplaphAsync()
        {
           var s_implementation = VssFactoryProvider.Default.GetVssFactory();
            var m_backupComponents = s_implementation.CreateVssBackupComponents();
            m_backupComponents.InitializeForBackup(null);
            m_backupComponents.SetBackupState(true, true, VssBackupType.Full, false);
            m_backupComponents.SetContext(VssSnapshotContext.Backup);
            // var  comps = m_backupComponents.WriterMetadata;
            m_backupComponents.GatherWriterMetadataAsync(default).Wait();

            foreach (var writeMeta in m_backupComponents.WriterMetadata)
            {
                foreach (var component in writeMeta.Components)
                {
                    var WriteName = writeMeta.WriterName;
                  
                    Console.WriteLine(DateTime.Now.ToString() + "start add expand path for writer:{0}", WriteName);
                    Stopwatch stopwatch = new Stopwatch();
                    stopwatch.Start();
                    for (int k = 0; k < component.Files.Count; k++)
                    {
                        // DO nothing here;
                        // For system writer it took very very long time.
                    }
                    stopwatch.Stop();
                    TimeSpan elapsedTime = stopwatch.Elapsed;
                    Console.WriteLine(DateTime.Now.ToString() + "add expand path for writer:{0}, file count:{1}, took:{2} seconds",
                        WriteName, component.Files.Count, elapsedTime.TotalSeconds);
                }
            }
        }
    }
}

[dahall]

So effectively, for "System Writer", the Count property call takes 688 second? Is that what you would deduce too?

[HuAaron]

So effectively, for "System Writer", the Count property call takes 688 second? Is that what you would deduce too?

No. Not the Count property took so much time, but the "for loop" did. Code that took long time:

for (int k = 0; k < component.Files.Count; k++){}.
If using component.Files.ToList() also took a long time.

I also debuged step by step and did some research. I wonder if the issue is related to lazy execute.

Vanara component.Files type is IEnumerable.

Alpha component.Files type is List.

I checked this blog: https://dev.to/bennnym/c-ienumerable-vs-list-and-array-1mdc.
IEnumerables delay execution of the query until the code is looped over or a value type is extracted.

Is this the cause of slow enum?

Thanks!