Introduction

Lexical.FileProvider is a class library with IFileProvider implementations and utilities.

Lexical.FileProvider.Package contains classes for opening package files recursively as folders. It manages opening, closing, caching and streaming package content. (Web, NuGet, Git)

mydata.zip/
mydata.zip/Folder/
mydata.zip/Folder/data.zip/
mydata.zip/Folder/data.zip/Lexical.Localization.Tests.dll/
mydata.zip/Folder/data.zip/Lexical.Localization.Tests.dll/Lexical.Localization.Tests.localization.ini
mydata.zip/Folder/data.zip/Lexical.Localization.Tests.dll/Lexical.Localization.Tests.localization.json
...

Lexical.FileProvider.Package.Abstractions contains abstractions for the package file provider. This is needed to be able to support different file formats. (Web, NuGet, Git)

Lexical.FileProvider.Root enables the use of file-system paths with IFileProvider implementations. (Web, NuGet, Git)

C:\dir\file
/home/user/dir/file

Lexical.FileProvider.Zip contains IFileProvider that opens .zip file contents. (Web, NuGet, Git)

Lexical.FileProvider.Dll contains IFileProvider that opens resources of .dll files. (Web, NuGet, Git)

Lexical.FileProvider.SharpCompress wraps SharpCompress .zip, .rar, .tar, .7z, .gz file formats into IFileProvider implementations. (Web, NuGet, Git)

Lexical.FileProvider.SharpZipLib wraps SharpZipLib .bzip2 and .Z file formats into IFileProvider implementations. (Web, NuGet, Git)

Lexical.FileProvider.Utils contains IFileProvider related utilities such as FileScanner. (Web, NuGet, Git)

Contents:

Lexical.FileProvider.Dll

DllFileProvider.OpenFile(string) opens .dll files. This library uses Mono.Cecil NuGet package.

DllFileProvider fileProvider = DllFileProvider.OpenFile("Lexical.Localization.Tests.dll");

Another constructor is DllFileProvider.UseStream(Stream).

// Open some stream
Stream stream = new FileStream("Lexical.Localization.Tests.dll", FileMode.Open);

// Use stream as dll file.
DllFileProvider fileProvider = DllFileProvider.UseStream(stream).AddDisposable(stream);

List all embedded resources.

foreach (string embeddedResource in fileProvider.ListAllPaths().OrderBy(s=>s))
    Console.WriteLine(embeddedResource);

Looks like this.

Lexical.Localization.Tests.localization.ini
Lexical.Localization.Tests.localization.json
Lexical.Localization.Tests.Localization.localization.ini
Lexical.Localization.Tests.Localization.localization-en.ini
Lexical.Localization.Tests.Localization.localization-fi.ini
...

File provider must be disposed after use.

fileProvider.Dispose();

Belated disposing can be added to the file provider. The disposable will be disposed once the fileprovider and all its streams are closed.

// Create file provider
DllFileProvider fileProvider = DllFileProvider.OpenFile("Lexical.Localization.Tests.dll");
// Add disposable for belated dispose
fileProvider.AddBelatedDispose(new _Disposable_());
// Open stream
Stream s = fileProvider
        .GetFileInfo("Lexical.Localization.Tests.Localization.localization-en.ini")
        .CreateReadStream();
// Dispose file provider
fileProvider.Dispose();
// Dispose the open stream  --  _Disposable_ is disposed here.
s.Dispose();

Package Loader

Lexical.FileProvider.PackageLoader.Dll can be used as a component of PackageFileLoader, making it possible to drill down into embedded resources of .dll files. Lexical.FileProvider.PackageLoader.Exe can open embedded resources of managed .exe files.

// Create root file provider
PhysicalFileProvider root = new PhysicalFileProvider(Directory.GetCurrentDirectory());

// Create package options
IPackageFileProviderOptions options =
    new PackageFileProviderOptions()
    .AddPackageLoaders(
        Lexical.FileProvider.PackageLoader.Dll.Singleton, 
        Lexical.FileProvider.PackageLoader.Exe.Singleton
    );

// Create package file provider
IPackageFileProvider fileProvider = new PackageFileProvider(root, options).AddDisposable(root);

Links

Lexical.FileScanner

Lexical.FileProviders.FileScanner is a configurable scanner that searches for files and folders. It is configurable to match file paths against wildcards, regular expressions, and glob patterns. It uses all threads concurrently, which makes it effective for opening package files.

// Create root file provider
PhysicalFileProvider root = new PhysicalFileProvider(Directory.GetCurrentDirectory());

// Create package options
IPackageFileProviderOptions options =
    new PackageFileProviderOptions()
    .AddPackageLoaders(Dll.Singleton, Exe.Singleton, Zip.Singleton, Rar.Singleton, _7z.Singleton,
                       Tar.Singleton, GZip.Singleton, BZip2.Singleton, Lzw.Singleton);

// Create package file provider
using (var fileProvider = new PackageFileProvider(root, options).AddDisposable(root))
{
    // Glob scan with multiple threads
    string[] files = new FileScanner(fileProvider)
        .AddGlobPattern("**.zip/**.resources")
        .OrderBy(s => s)
        .ToArray();

    // Print
    foreach (String filename in files)
        Console.WriteLine(filename);
}

Errors that are thrown by file provider are captured. .SetErrorTarget(IProducerConsumerCollection<Exception>) sets a target where captured errors are to be placed. 

int count = 0;
ConcurrentBag<Exception> errors = new ConcurrentBag<Exception>();
foreach (string filepath in new FileScanner(fileProvider).AddGlobPattern("**").SetErrorTarget(errors))
{
    // Print errors
    Exception e;
    if (errors.TryTake(out e)) Console.Error.WriteLine(e);

    Console.WriteLine(filepath);
    if (++count == 100) break;
}
fileProvider.Dispose();

Lexical.FileProvider.Package

PackageFileProvider is a file provider that can open different package file formats, such as .zip and .dll.

// Create root file provider
RootFileProvider root = new RootFileProvider();

// Create package options
IPackageFileProviderOptions options = 
    new PackageFileProviderOptions()
    .SetAllowOpenFiles(false)
    .AddPackageLoaders(
        Lexical.FileProvider.PackageLoader.Dll.Singleton,
        Lexical.FileProvider.PackageLoader.Exe.Singleton,
        Lexical.FileProvider.PackageLoader.Zip.Singleton,
        Lexical.FileProvider.PackageLoader._Zip.Singleton,
        Lexical.FileProvider.PackageLoader.Rar.Singleton,
        Lexical.FileProvider.PackageLoader._7z.Singleton,
        Lexical.FileProvider.PackageLoader.Tar.Singleton,
        Lexical.FileProvider.PackageLoader.GZip.Singleton,
        Lexical.FileProvider.PackageLoader.BZip2.Singleton,
        Lexical.FileProvider.PackageLoader.Lzw.Singleton
    );

// Create package file provider
PackageFileProvider fileProvider = new PackageFileProvider(root, options);

List package contents. Extension method in Lexical.FileProvider.Utils .ListAllFileInfoAndPath() visits IFileInfo and corresponding paths recursively.

string path = Directory.GetCurrentDirectory();

foreach ((IFileInfo, String) pair in fileProvider.ListAllFileInfoAndPath(path).OrderBy(pair => pair.Item2))
    Console.WriteLine(pair.Item2 + (pair.Item1.IsDirectory ? "/" : ""));

Packages are opened as folders.

mydata.zip/
mydata.zip/Folder/
mydata.zip/Folder/data.zip/
mydata.zip/Folder/data.zip/Lexical.Localization.Tests.dll/
mydata.zip/Folder/data.zip/Lexical.Localization.Tests.dll/Lexical.Localization.Tests.localization.ini
mydata.zip/Folder/data.zip/Lexical.Localization.Tests.dll/Lexical.Localization.Tests.localization.json
...

Disposing

File provider must be disposed after use. The root provider too.

fileProvider.Dispose();
root.Dispose();

Root file provider can be attached to be disposed along with the PackageFileProvider. 

// Create root
IFileProvider root = new RootFileProvider();
// Create package provider and attach root
PackageFileProvider fileProvider = new PackageFileProvider(root).AddDisposable(root);
// Disposes both fileProvider and its root
fileProvider.Dispose();

Disposable can be attached with a delegate Func<PackageFileProvider, object>.

// Create package provider and attach root to be disposed
PackageFileProvider fileProvider = new PackageFileProvider( new RootFileProvider() ).AddDisposable( p=>p.FileProvider );

Options

Reference to options is given at construction.

PackageFileProviderOptions options = new PackageFileProviderOptions();
PackageFileProvider fileProvider = new PackageFileProvider(root, options);

Or, if options are not provided, then PackageFileProvider creates new a with default values.

PackageFileProvider fileProvider = new PackageFileProvider(root);
IPackageFileProviderOptions options = fileProvider.Options;

Options must be populated with instances of IPackageLoader. These handle how different file formats are opened. Package loader creates new IFileProvider instances as needed.

PackageFileProvider fileProvider = new PackageFileProvider(root);
fileProvider.Options.AddPackageLoaders(
        Lexical.FileProvider.PackageLoader.Dll.Singleton,
        Lexical.FileProvider.PackageLoader.Zip.Singleton
);

If one-liner is needed, the newly created options can be configured in a lambda function .ConfigureOptions(o=>o....).

PackageFileProvider fileProvider = new PackageFileProvider(root)
    .ConfigureOptions(o => o.AddPackageLoaders(Dll.Singleton, Zip.Singleton));

.AsReadonly() locks the options into an immutable read-only state.

IPackageFileProviderOptions shared_options =
    new PackageFileProviderOptions()
    .AddPackageLoaders(Lexical.FileProvider.PackageLoader.Zip.Singleton)
    .AsReadonly();

Package loading

There are four ways how packages are loaded:

  1. Reading from an open file.
  2. Streaming from parent file provider with an open stream.
  3. Taking a snapshot copy into memory.
  4. Taking a snapshot copy into temp file.

Options.SetMemorySnapshotLength(int) sets the maximum size of a memory snapshot. If package file is smaller or equal to this number, then the file can be loaded into a memory snapshot. If package file is larger, then the package will not be loaded into memory. If this value is set to 0, then no packages are loaded into memory.

fileProvider.Options.SetMemorySnapshotLength(1048576);

.SetAllowOpenFiles(bool) configures the policy of whether it is allowed to keep open file handles. If this value is false, package files are always copied into snapshots, either to memory or to temp file.

fileProvider.Options.SetAllowOpenFiles(true);

.SetReuseFailedResult(bool) configures the policy of whether it is allowed to reuse failed load result. If the policy is true and load has failed, then the error is remembered and reused if package is accessed again. If false, then reload is attempted on every load. Failed load result can be evicted just as successful load results.

fileProvider.Options.SetReuseFailedResult(true);

.SetErrorHandler(Func<PackageEvent, bool>) configures an overriding exception handler. This is used for considering whether an exception is expectable, such as problems with file formats. When this handler returns true, then the exception is suppressed (not thrown). When false, then the exception is let to be thrown to caller.

fileProvider.Options.SetErrorHandler( pe => pe.LoadError is PackageException.LoadError );

Temp File Provider

ITempFileProvider must be assigned to utilize temp files. Some package loaders need them to work properly. TempFileProvider.Default is a singleton instance that uses the current user's default temp folder.

fileProvider.SetTempFileProvider(TempFileProvider.Default);

TempFileProvider is constructed with custom TempFileOptions for more detailed behaviour.

// Create temp options
TempFileProviderOptions tempFileOptions = new TempFileProviderOptions {
    Directory = "%tmp%",
    Prefix = "package-",
    Suffix = ".tmp"
};
// Create temp file provider
ITempFileProvider tempProvider = new TempFileProvider(tempFileOptions);
// Assign temp file provider
fileProvider
    .SetTempFileProvider(tempProvider)
    .AddDisposable(tempProvider);

Options.SetTempFileSnapshotLength(long) sets the maximum size of a temp file snapshot. If this value is set to 0, then package file provider will not take any temp file copies.

fileProvider.Options.SetTempFileSnapshotLength(1073741824);

Observing

.Subscribe(IObserver<PackageEvent>) subscribes for package loading, error and evicting events.

IDisposable handle = fileProvider.Subscribe(new MyObserver());

Observable receives notifications.

class MyObserver : IObserver<PackageEvent>
{
    public void OnCompleted()
        => Console.WriteLine("End of subscription");

    public void OnError(Exception error)
        => Console.WriteLine(error);

    public void OnNext(PackageEvent value)
        => Console.WriteLine(value);
}

Subsription is canceled by disposing the handle.

handle.Dispose();

Logging

.AddLogger(ILogger) adds an ILogger as observable. It writes log entries about loading and eviciting of packages.

IServiceCollection serviceCollection = new ServiceCollection().AddLogging(builder => builder.AddConsole());
ILogger logger = serviceCollection.BuildServiceProvider().GetService<ILogger<PackageFileProvider>>();
fileProvider.AddLogger(logger);

Looks like this.

info: Lexical.FileProvider.Package.PackageFileProvider[0]
      Folder/mydata.zip/Lexical.Localization.Tests.dll was loaded.
info: Lexical.FileProvider.Package.PackageFileProvider[0]
      Folder/mydata.zip/Lexical.Localization.Tests.dll was evicted.

Evicting

Packages are evicted automatically when memory constraints from IPackageFileProviderOptions are met. However, they can be evicted manually as well.

.GetPackageInfos() and .GetPackageInfo(string) gives info about cached packages.

// Open cascading packages
fileProvider
    .GetFileInfo("mydata.zip/Folder/mydata.zip/Lexical.Localization.Tests.dll/Lexical.Localization.Tests.localization.ini")
    .CreateReadStream()
    .Dispose();

// List all open packages
foreach (PackageInfo pi in fileProvider.GetPackageInfos())
    Console.WriteLine($"{pi.FilePath} {pi.State}");

// Specific package
PackageInfo _pi = fileProvider.GetPackageInfo("mydata.zip/Folder/mydata.zip");

Cached packages can be evicted manually with .Evict(string). Evict will fail if there is an open stream to a package.

fileProvider.Evict("mydata.zip/Folder/mydata.zip/Lexical.Localization.Tests.dll");
fileProvider.Evict("mydata.zip/Folder/mydata.zip");
fileProvider.Evict("mydata.zip");

.EvictAll() releases all non-locked cached packages from memory and temp files.

fileProvider.EvictAll();

.StartEvictTimer(TimeSpan, TimeSpan) starts a timer that checks periodically if package hasn't been accessed for a while, and evicts them.

fileProvider.StartEvictTimer(TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(5));

Dependency Injection

There are many ways how a DI container can be put together. The following example demonstrates how configuration is read from config.json. Modifications are reloaded and will be forwarded to PackageFileProvider and TempFileProvider instances. 

ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
// Add config.json
configurationBuilder
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddJsonFile("config.json", optional: false, reloadOnChange: true);
// Build config
IConfiguration configuration = configurationBuilder.Build();

//// Start DI Configuration
IServiceCollection serviceCollection = new ServiceCollection();
// Add the configuration that has already been loaded
serviceCollection.AddSingleton<IConfiguration>(configuration);
// Add IOptions support
serviceCollection.AddOptions();

// Add logger
serviceCollection.AddLogging(builder => builder.AddConsole());

ITempFileProvider is initialized to read configuration from "TempFiles" section into monitorable TempFileProviderOptions poco class. TempFileProviderOptionsMonitor is a workaround that allows to not expose IOptionsMonitor to the implementing class.

// Add instantiation of TempFileProvider
serviceCollection.AddTransient<ITempFileProvider, TempFileProvider>();
// Bind configuration section to IOptions<TempFileProviderOptions>
serviceCollection.Configure<TempFileProviderOptions>(configuration.GetSection("TempFiles"));
// Workaround, convert IOptionsMonitor<TempFileProviderOptions> to TempFileProviderOptions.
serviceCollection.AddTransient<TempFileProviderOptions, TempFileProviderOptionsMonitor>();
// Register Options validator
serviceCollection.AddSingleton(TempFileProviderOptionsValidator.Singleton);

PackageFileProvider is registered as a service of IFileProvider. Configurations are read from node "PackageFileProvider" into record class PackageFileProviderOptionsRecord. PackageFileProviderOptionsMonitor is a class that converts package loader type strings into instances of IPackageLoader. PackageFileProviderOptionsValidator validates every field of the options record. RootFileProvider is used in this example as the source file provider.

// Bind configuration section to IOptions<PackageFileProviderOptionsRecord>
serviceCollection.Configure<PackageFileProviderOptionsRecord>(configuration.GetSection("PackageFileProvider"));
// Adapt IOptions<PackageFileProviderOptionsRecord> to IPackageFileProviderOptions
serviceCollection.AddTransient<IPackageFileProviderOptions, PackageFileProviderOptionsMonitor>();
// Register Options validator
serviceCollection.AddSingleton(PackageFileProviderOptionsValidator.Level2);
// Add root file provider at current dir
serviceCollection.AddSingleton<RootFileProvider>(new RootFileProvider());
// Add service PackageFileProvider as IFileProvider
serviceCollection.AddSingleton<IFileProvider>(s =>
   new PackageFileProvider(s.GetService<RootFileProvider>(), s.GetService<IPackageFileProviderOptions>(), s.GetService<ITempFileProvider>())
        .StartEvictTimer(s.GetService<IOptionsMonitor<PackageFileProviderOptionsRecord>>())
        .AddLogger(s.GetService<ILogger<PackageFileProvider>>())
);

Let's give it a go.

using (var service = serviceCollection.BuildServiceProvider())
{
    // Get the singleton package file provider (don't dispose here)
    IFileProvider fp = service.GetService<IFileProvider>();

    // List all file paths
    foreach (string filepath in fp.ListAllPaths("."))
        Console.WriteLine(filepath);
}

The example config.json looks like this. Package loaders are assembly qualified type names. The second parameter is assembly name. It can be left out if the assemblies are already loaded to the AppDomain.

{
  "PackageFileProvider": {
    "AllowOpenFiles": "True",
    "ReuseFailedResult": "True",
    "MaxMemorySnapshotLength": 1073741824,
    "MaxTempSnapshotLength": 1099511627776,
    "PackageLoaders": [
      "Lexical.FileProvider.PackageLoader.Dll, Lexical.FileProvider.Dll",
      "Lexical.FileProvider.PackageLoader.Exe, Lexical.FileProvider.Dll",
      "Lexical.FileProvider.PackageLoader.Zip, Lexical.FileProvider",
      "Lexical.FileProvider.PackageLoader.Rar, Lexical.FileProvider.SharpCompress",
      "Lexical.FileProvider.PackageLoader._7z, Lexical.FileProvider.SharpCompress",
      "Lexical.FileProvider.PackageLoader.Tar, Lexical.FileProvider.SharpCompress",
      "Lexical.FileProvider.PackageLoader.GZip, Lexical.FileProvider.SharpCompress",
      "Lexical.FileProvider.PackageLoader.BZip2, Lexical.FileProvider.SharpZipLib",
      "Lexical.FileProvider.PackageLoader.Lzw, Lexical.FileProvider.SharpZipLib"
    ],
    "CacheEvictTime": "15"
  },
  "TempFiles": {
    "Directory": "%tmp%",
    "Prefix": "package-",
    "Suffix": ".tmp"
  }
}

The full example.

//// Start configuration
ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
// Add config.json
configurationBuilder.SetBasePath(Directory.GetCurrentDirectory()).AddJsonFile("config.json", optional: false, reloadOnChange: true);
// Build config
IConfiguration configuration = configurationBuilder.Build();

//// Start DI Configuration
IServiceCollection serviceCollection = new ServiceCollection();
// Add the configuration that has already been loaded
serviceCollection.AddSingleton<IConfiguration>(configuration);
// Add IOptions support
serviceCollection.AddOptions();

//// Add logger
serviceCollection.AddLogging(builder => builder.AddConsole());

//// Configure Temp File Provider
// Add instantiation of TempFileProvider
serviceCollection.AddTransient<ITempFileProvider, TempFileProvider>();
// Bind configuration section to IOptions<TempFileProviderOptions>
serviceCollection.Configure<TempFileProviderOptions>(configuration.GetSection("TempFiles"));
// Workaround, convert IOptionsMonitor<TempFileProviderOptions> to TempFileProviderOptions.
serviceCollection.AddTransient<TempFileProviderOptions, TempFileProviderOptionsMonitor>();
// Register Options validator
serviceCollection.AddSingleton(TempFileProviderOptionsValidator.Singleton);

//// Configure PackageFileProvider
// Bind configuration section to IOptions<PackageFileProviderOptionsRecord>
serviceCollection.Configure<PackageFileProviderOptionsRecord>(configuration.GetSection("PackageFileProvider"));
// Adapt IOptions<PackageFileProviderOptionsRecord> to IPackageFileProviderOptions
serviceCollection.AddTransient<IPackageFileProviderOptions, PackageFileProviderOptionsMonitor>();
// Register Options validator
serviceCollection.AddSingleton(PackageFileProviderOptionsValidator.Level2);
// Add root file provider at current dir
serviceCollection.AddSingleton<RootFileProvider, RootFileProvider>();
// Add service PackageFileProvider as IFileProvider
serviceCollection.AddSingleton<IFileProvider>(s =>
   new PackageFileProvider(s.GetService<RootFileProvider>(), s.GetService<IPackageFileProviderOptions>(), s.GetService<ITempFileProvider>())
        .StartEvictTimer(s.GetService<IOptionsMonitor<PackageFileProviderOptionsRecord>>())
        .AddLogger(s.GetService<ILogger<PackageFileProvider>>())
);

//// Give it a go
using (var service = serviceCollection.BuildServiceProvider())
{
    // Get the singleton package file provider (don't dispose here)
    IFileProvider fp = service.GetService<IFileProvider>();

    // List all file paths
    foreach (string filepath in fp.ListAllPaths("."))
        Console.WriteLine(filepath);
}

Links

Lexical.FileProvider.Package.Abstractions

Lexical.FileProviders.Package.Abstractions is a class library that contains interfaces for implementing package loaders for the IPackageFileProvider. IPackageLoader interface is divided into multiple interfaces.

IPackageLoader is the interface classes that load package files into file providers. csharp /// <summary> /// Interace for loaders that read package files, such as .zip, as <see cref="IFileProvider"/>s. /// /// The implementing class must implement one or more of the following sub-interfaces: /// <list type="bullet"> /// <item><see cref="IPackageLoaderOpenFile"/></item> /// <item><see cref="IPackageLoaderLoadFile"/></item> /// <item><see cref="IPackageLoaderUseStream"/></item> /// <item><see cref="IPackageLoaderLoadFromStream"/></item> /// <item><see cref="IPackageLoaderUseBytes"/></item> /// </list> /// </summary> public interface IPackageLoader { /// <summary> /// The file extension(s) this format can open. /// /// The string is a regular expression. /// For example "\.zip" or "\.zip|\.7z|\.tar\.gz" /// /// Pattern will be used as case insensitive, so the case doesn't matter, but lower is preferred. /// /// Do not add named groups. For example "(?&lt;name&gt;..)". /// /// Unnamed groups are, however, allowed. For example: "\.zip(\.tmp)?" /// </summary> String FileExtensionPattern { get; } }
IPackageLoaderOpenFile is an interface for reading content from open files. csharp /// <summary> /// Package loader that can open a package file as <see cref="IFileProvider"/>. /// </summary> public interface IPackageLoaderOpenFile : IPackageLoader { /// <summary> /// Open a package file and keep it open until the file provider is disposed. /// Return <see cref="IFileProvider"/> that represents the contents of the open file. /// /// The caller is responsible for disposing the returned file provider if it implements <see cref="IDisposable"/>. /// </summary> /// <param name="filepath">data to read from</param> /// <param name="packageInfo">(optional) Information about packge that is being opened</param> /// <returns>file provider</returns> /// <exception cref="Exception">If there was unexpected error, such as IOException</exception> /// <exception cref="InvalidOperationException">If this load method is not supported.</exception> /// <exception cref="IOException">Problem with io stream</exception> /// <exception cref="PackageException.LoadError">The when file format is erronous, package will not be opened as directory.</exception> IFileProvider OpenFile(string filepath, IPackageLoadInfo packageInfo = null); }
IPackageLoaderLoadFile is an interface for loading from files. csharp /// <summary> /// Package loader that cab load a package file completely. /// </summary> public interface IPackageLoaderLoadFile : IPackageLoader { /// <summary> /// Load a package file completely. The implementation must close the file before the call returns. /// Return <see cref="IFileProvider"/> that represents the contents of the open file. /// /// The caller is responsible for disposing the returned file provider if it implements <see cref="IDisposable"/>. /// </summary> /// <param name="filepath">data to read from</param> /// <param name="packageInfo">(optional) Information about packge that is being opened</param> /// <returns>file provider</returns> /// <exception cref="Exception">If there was unexpected error, such as IOException</exception> /// <exception cref="InvalidOperationException">If this load method is not supported.</exception> /// <exception cref="IOException">Problem with io stream</exception> /// <exception cref="PackageException.LoadError">The when file format is erronous, package will not be opened as directory.</exception> IFileProvider LoadFile(string filepath, IPackageLoadInfo packageInfo = null); }
IPackageLoaderUseStream is an interface for using stream as open content source. csharp /// <summary> /// Package loader that can open <see cref="Stream"/> to access contents of a package file. /// </summary> public interface IPackageLoaderUseStream : IPackageLoader { /// <summary> /// Use an open <paramref name="stream"/> to read contents from a package file. /// Return a <see cref="IFileProvider"/> that represent the contents. /// /// The returned file provider takes ownership of the stream, and must close the <paramref name="stream"/> along with the provider. /// /// <paramref name="stream"/> must be readable and seekable, <see cref="Stream.CanSeek"/> must be true. /// /// The caller is responsible for disposing the returned file provider if it implements <see cref="IDisposable"/>. /// /// Note, open stream cannot be read concurrently from two threads and must be locked with mutually exclusive lock if two reads attempted. /// </summary> /// <param name="stream">stream to read data from. Stream must be disposed along with the returned file provider.</param> /// <param name="packageInfo">(optional) Information about packge that is being opened</param> /// <returns>file provider that can be disposable</returns> /// <exception cref="Exception">If there was unexpected error, such as IOException</exception> /// <exception cref="InvalidOperationException">If this load method is not supported.</exception> /// <exception cref="IOException">Problem with io stream</exception> /// <exception cref="PackageException.LoadError">The when file format is erronous, package will not be opened as directory.</exception> IFileProvider UseStream(Stream stream, IPackageLoadInfo packageInfo = null); }
IPackageLoaderLoadFromStream is an interface for loading content from stream. csharp /// <summary> /// Package loader that can load a package completely from an open <see cref="Stream"/>. /// </summary> public interface IPackageLoaderLoadFromStream : IPackageLoader { /// <summary> /// Read package completely from <paramref name="stream"/> and return representation of contents as <see cref="IFileProvider"/>. /// The implementation and the returned <see cref="IFileProvider"/> does not take ownership of the stream. /// /// The returned file provider can be left to be garbage collected and doesn't need to be disposed. /// </summary> /// <param name="stream">stream to read data from. Stream doesn't need to be closed by callee, but is allowed to do so.</param> /// <param name="packageInfo">(optional) Information about packge that is being opened</param> /// <returns>file provider</returns> /// <exception cref="Exception">If there was unexpected error, such as IOException</exception> /// <exception cref="InvalidOperationException">If this load method is not supported.</exception> /// <exception cref="IOException">Problem with io stream</exception> /// <exception cref="PackageException.LoadError">The when file format is erronous, package will not be opened as directory.</exception> IFileProvider LoadFromStream(Stream stream, IPackageLoadInfo packageInfo = null); }
IPackageLoaderUseBytes is an interface for loading content from stream. csharp /// <summary> /// Package loader that can load a package completely from an bytes. /// </summary> public interface IPackageLoaderUseBytes : IPackageLoader { /// <summary> /// Load file provider from bytes. /// /// The caller is responsible for disposing the returned file provider if it implements <see cref="IDisposable"/>. /// </summary> /// <param name="data">data to read from</param> /// <param name="packageInfo">(optional) Information about packge that is being opened</param> /// <returns>file provider</returns> /// <exception cref="Exception">If there was unexpected error, such as IOException</exception> /// <exception cref="InvalidOperationException">If this load method is not supported.</exception> /// <exception cref="IOException">Problem with io stream</exception> /// <exception cref="PackageException.LoadError">The when file format is erronous, package will not be opened as directory.</exception> IFileProvider UseBytes(byte[] data, IPackageLoadInfo packageInfo = null); }
IPackageLoadInfo is an interface for loading content from stream. csharp /// <summary> /// Optional hints about the package that is being loaded. /// </summary> public interface IPackageLoadInfo { /// <summary> /// (optional) Path within package file provider. /// </summary> string Path { get; } /// <summary> /// (Optional) Last modified UTC date time. /// </summary> DateTimeOffset? LastModified { get; } /// <summary> /// File length, or -1 if unknown /// </summary> long Length { get; } }

ITempFileProvider

Temp file provider is attached to package file provider options.

// Create temp options
TempFileProviderOptions tempFileOptions = new TempFileProviderOptions { Directory = "%tmp%", Prefix = "package-", Suffix = ".tmp" };

// Create temp provider
ITempFileProvider tempFileProvider = new TempFileProvider(tempFileOptions);

// Try to create temp file
using (var tempFile = tempFileProvider.CreateTempFile())
    Console.WriteLine(tempFile.Filename);

// Attach temp provider
fileProvider.SetTempFileProvider(tempFileProvider).Options.SetTempFileSnapshotLength(1073741824);
ITempProvider is an interface for creating temp files. csharp /// <summary> /// Temporary file provider. /// </summary> public interface ITempFileProvider : IDisposable { /// <summary> /// Create a new unique 0-bytes temp file that is not locked. /// </summary> /// <exception cref="IOException">if file creation failed</exception> /// <exception cref="ObjectDisposedException">if provider is disposed</exception> /// <returns>handle with a filename. Caller must dispose after use, which will delete the file if it still exists.</returns> ITempFileHandle CreateTempFile(); } /// <summary> /// A handle to a temp file name. /// /// Dispose the handle to delete it. /// /// If temp file is locked, Dispose() throws an <see cref="IOException"/>. /// /// Failed deletion will still be marked as to-be-deleted. /// There is another delete attempt when the parent <see cref="ITempFileProvider"/> is disposed. /// </summary> public interface ITempFileHandle : IDisposable { /// <summary> /// Filename to 0 bytes temp file. /// </summary> String Filename { get; } }

Links

Lexical.FileProvider.Root

RootFileProvider is a file provider that enables OS wide file paths such as volumes.

RootFileProvider fileProvider = new RootFileProvider();

Constructor delegate can be given as argument. This is useful for configuring PhysicalFilePath exclusion filters.

RootFileProvider fileProvider = new RootFileProvider(
    path => new PhysicalFileProvider(path, ExclusionFilters.None)
);

.GetDirectoryContents("") returns all drive letters on windows and the root "/" on linux.

foreach (IFileInfo fileinfo in fileProvider.GetDirectoryContents(null))
    Console.WriteLine(fileinfo.Name);

C:
D:
E:
/

.GetDirectoryContents("C:/") and .GetDirectoryContents("C:\") returns the root of that drive letter.

foreach (IFileInfo fileinfo in fileProvider.GetDirectoryContents("C:/"))
    Console.WriteLine(fileinfo.Name);
foreach (IFileInfo fileinfo in fileProvider.GetDirectoryContents("C:\\"))
    Console.WriteLine(fileinfo.Name);

Relative paths, such as "." and ".." are relative to the current working directory.

foreach (IFileInfo fileinfo in fileProvider.GetDirectoryContents("."))
    Console.WriteLine(fileinfo.Name);

Windows network drives "\\server\name/path" are accessed with back-slashes.

foreach (IFileInfo fileinfo in fileProvider.GetDirectoryContents(@"\\192.168.8.200\shared"))
    Console.WriteLine(fileinfo.Name);

FileScanner can scan relative drive and file on the computer.

int count = 0;
foreach (string filepath in new FileScanner(fileProvider).AddGlobPattern("**").SetReturnDirectories(true))
{
    Console.WriteLine(filepath);
    if (++count == 100) break;
}
fileProvider.Dispose();

And network drives too. Note that file scanner uses '/' as internal separator, therefore the network path must have '/' as first folder separator.

foreach (string filepath in new FileScanner(fileProvider).AddWildcard(@"\\192.168.8.200\shared/*"))
    Console.WriteLine(filepath);

Links

Lexical.FileProvider.SharpCompress

Lexical.FileProviders.SharpCompress is a class library that wraps SharpCompress into IFileProvider.

The new _ZipFileProvider(string) constructs a file provider that reads .zip contents. Other classes are RarFileProvider, _7zFileProvider, TarFileProvider and GZipFileProvider. Content can be read concurrently as file provider opens new file handles as needed.

_ZipFileProvider fileProvider_zip = new _ZipFileProvider("mydata.zip");
RarFileProvider fileProvider_rar = new RarFileProvider("mydata.rar");
TarFileProvider fileProvider_tar = new TarFileProvider("mydata.tar");
_7zFileProvider fileProvider_7z = new _7zFileProvider("mydata.7z");
GZipFileProvider fileProvider_gz = new GZipFileProvider("mydata.tar.gz", "mydata.tar");

Another constructor argument is new _ZipFileProvider(Stream). Note however, that unlike file based constructor, this stream based constructor has only one file pointer, content can be accessed only by one thread at a time. Concurrent threads have to wait.

// Open some stream
Stream stream = new FileStream("mydata.zip", FileMode.Open);

// Use stream as zip file.
_ZipFileProvider fileProvider = new _ZipFileProvider(stream).AddDisposable(stream);

File provider must be disposed after use.

fileProvider.Dispose();

Belated disposing can be added to the file provider. The disposable will be disposed once the fileprovider and all its streams are closed.

// Create file provider
_ZipFileProvider fileProvider = new _ZipFileProvider("mydata.zip");
// Add disposable for belated dispose
fileProvider.AddBelatedDispose(new _Disposable_());
// Open stream
Stream s = fileProvider
        .GetFileInfo("Lexical.Localization.Tests.dll")
        .CreateReadStream();
// Dispose file provider
fileProvider.Dispose();
// Dispose the open stream  --  _Disposable_ is disposed here.
s.Dispose();

Package Loader

Lexical.FileProvider.PackageLoader._Zip, .Rar, ._7z, .Tar and .GZip can be used as a component of PackageFileLoader, making it possible to drill down into archive files recursively.

// Create root file provider
PhysicalFileProvider root = new PhysicalFileProvider(Directory.GetCurrentDirectory());

// Create package options
IPackageFileProviderOptions options =
    new PackageFileProviderOptions()
    .AddPackageLoaders(
        Lexical.FileProvider.PackageLoader._Zip.Singleton, 
        Lexical.FileProvider.PackageLoader.Rar.Singleton,
        Lexical.FileProvider.PackageLoader._7z.Singleton,
        Lexical.FileProvider.PackageLoader.Tar.Singleton,
        Lexical.FileProvider.PackageLoader.GZip.Singleton
    );

// Create package file provider
IPackageFileProvider fileProvider = new PackageFileProvider(root, options).AddDisposable(root);

// Read compressed file
using (Stream document = fileProvider.GetFileInfo("document.txt.gz/document.txt").CreateReadStream())
{
    byte[] data = FileUtils.ReadFully(document);
    string text = Encoding.UTF8.GetString(data);
    Console.WriteLine(text);
}

Links

Lexical.FileProvider.SharpZipLib

Lexical.FileProviders.SharpZipLib is a class library that wraps SharpZipLib into IFileProvider.

The new BZip2FileProvider(string, string) constructs a file provider that reads .bzip2 content. Other class is LzwFileProvider. Content can be read concurrently as file provider opens new file handles as needed.

BZip2FileProvider fileProvider_bzip2 = new BZip2FileProvider("mydata.tar.bzip2", "mydata.tar");
LzwFileProvider fileProvider_lzw = new LzwFileProvider("mydata.tar.Z", "mydata.tar");

Another constructor variant is new BZip2FileProvider(byte[], string). 

// Read data
byte[] data;
using (Stream stream = new FileStream("mydata.tar.bzip2", FileMode.Open))
    data = FileUtils.ReadFully(stream);

// Use stream as zip file.
BZip2FileProvider fileProvider = new BZip2FileProvider(data, "mydata.tar");

File provider must be disposed after use.

fileProvider.Dispose();

Belated disposing can be added to the file provider. The disposable will be disposed once the fileprovider and all its streams are closed.

// Create file provider
BZip2FileProvider fileProvider = new BZip2FileProvider("mydata.tar.bzip2", "mydata.tar");
// Add disposable for belated dispose
fileProvider.AddBelatedDispose(new _Disposable_());
// Open stream
Stream s = fileProvider
        .GetFileInfo("mydata.tar")
        .CreateReadStream();
// Dispose file provider
fileProvider.Dispose();
// Dispose the open stream  --  _Disposable_ is disposed here.
s.Dispose();

Package Loader

Lexical.FileProvider.PackageLoader.BZip2 and .Lzw can be used as a component of PackageFileLoader, making it possible to drill down into archive files recursively.

// Create root file provider
PhysicalFileProvider root = new PhysicalFileProvider(Directory.GetCurrentDirectory());

// Create package options
IPackageFileProviderOptions options =
    new PackageFileProviderOptions()
    .AddPackageLoaders(
        Lexical.FileProvider.PackageLoader.Tar.Singleton,
        Lexical.FileProvider.PackageLoader.BZip2.Singleton,
        Lexical.FileProvider.PackageLoader.Lzw.Singleton
    );

// Create package file provider
IPackageFileProvider fileProvider = new PackageFileProvider(root, options).AddDisposable(root);

// Read compressed file
using (Stream document = fileProvider.GetFileInfo("document.txt.Z/document.txt").CreateReadStream())
{
    byte[] data = FileUtils.ReadFully(document);
    string text = Encoding.UTF8.GetString(data);
    Console.WriteLine(text);
}

Links

Lexical.FileProviders.Utils

Lexical.FileProviders.Utils is a namespace that introduces extension methods.

Extension method .ListAllPaths() visits IFileProvider and returns all subpaths. This method uses one thread.

// Scan all subpaths
foreach (string subpath in fileProvider.ListAllPaths())
    Console.WriteLine(subpath);

Extension method .ListAll() visits IFileProvider and returns all IFileInfos. 

// Scan all file infos
foreach (IFileInfo fi in fileProvider.ListAll())
    Console.WriteLine(fi.Name);

Extension method .ListAllFileInfoAndPath() visits IFileProvider and returns both subpath and IFileInfo in a tuple.

// Scan all file infos and their full subpaths
foreach ((IFileInfo, String) pair in fileProvider.ListAllFileInfoAndPath().OrderBy(pair => pair.Item2))
    Console.WriteLine(pair.Item2 + (pair.Item1.IsDirectory ? "/" : ""));

Links

Lexical.FileProvider.Zip

new ZipFileProvider(string) constructs a file provider that reads .zip contents. Content can be read concurrently as file provider opens new file handles as needed.

ZipFileProvider fileProvider = new ZipFileProvider("mydata.zip");

Another constructor is new ZipFileProvider(Stream). Note however, that as stream has only one file pointer, the content can be accessed only by one thread at a time. Concurrent threads have to wait.

// Open some stream
Stream stream = new FileStream("mydata.zip", FileMode.Open);

// Use stream as zip file.
ZipFileProvider fileProvider = new ZipFileProvider(stream).AddDisposable(stream);

File provider must be disposed after use.

fileProvider.Dispose();

Belated dispose can be added to the file provider. They are disposed once the fileprovider is disposed and all its streams.

// Create file provider
ZipFileProvider fileProvider = new ZipFileProvider("mydata.zip");
// Add disposable for belated dispose
fileProvider.AddBelatedDispose(new _Disposable_());
// Open stream
Stream s = fileProvider
        .GetFileInfo("Lexical.Localization.Tests.dll")
        .CreateReadStream();
// Dispose file provider
fileProvider.Dispose();
// Dispose the open stream  --  _Disposable_ is disposed here.
s.Dispose();

Package Loader

Lexical.FileProvider.PackageLoader.Zip can be used as a component of PackageFileLoader, making possible to drill down into .zip files.

// Create root file provider
PhysicalFileProvider root = new PhysicalFileProvider(Directory.GetCurrentDirectory());

// Create package options
IPackageFileProviderOptions options =
    new PackageFileProviderOptions()
    .AddPackageLoaders(Lexical.FileProvider.PackageLoader.Zip.Singleton);

// Create package file provider
IPackageFileProvider fileProvider = new PackageFileProvider(root, options).AddDisposable(root);

Lexical.FileProvider.PackageLoader.Zip can be constructed to open other file extensions that are of zip file format, such as .nupkg.

IPackageFileProviderOptions options = new PackageFileProviderOptions()
    .AddPackageLoaders( new Lexical.FileProvider.PackageLoader.Zip("\\.nupkg") );

Links