Jump to content
  • 0

OneDrive Files On Demand (fka sparse files)


ElectroStrong

Question

I am in the process of evaluating DrivePool as a replacement to the Windows RAID mirroring.  One of the problems that I've run into revolves around the fact that I can not longer use the OneDrive Files on Demand feature when the OneDrive folder is located in a drive pool.  This is consistent with feedback that I've seen in the reddit community in which other users have stated that they have the same issue.

This is a deal-breaker for me - I use multiple machines and sparse files allow for me to access these files without storing them locally on my drive configuration.  With Windows RAID, it worked without issues as you can imagine.  My main focus around using DrivePool is to have a redundant copy of data in case of a drive failure so the other use cases of DrivePool, such as SSD caching and multi-drive configurations are not as useful to me as having the ability to support sparse files with OneDrive.

Is this a known issue?  Is there any plans to add support for this capability?  I'd appreciate any input on this topic so I can make a decision on what my path forward is for my use case.

Link to comment
Share on other sites

3 answers to this question

Recommended Posts

  • 0

Yeah, it's a known issue.  It has to do with how the "on demand" feature works, and that it's not supported on the emulated drive that StableBit CloudDrive. 

Also, it's not just a sparse file, it's a special type of file, that requires file system support.  And because we use an emulated drive, we'd have to reverse engineer that and add support for it.  

A possible solution is to create a StableBit CloudDrive disk on the pool and use that. But I can understand not wanting to do that. 

Link to comment
Share on other sites

  • 0
On 1/2/2023 at 8:17 PM, Christopher (Drashna) said:

Yeah, it's a known issue.  It has to do with how the "on demand" feature works, and that it's not supported on the emulated drive that StableBit CloudDrive. 

Also, it's not just a sparse file, it's a special type of file, that requires file system support.  And because we use an emulated drive, we'd have to reverse engineer that and add support for it.  

A possible solution is to create a StableBit CloudDrive disk on the pool and use that. But I can understand not wanting to do that. 

Thanks for the response Christopher.  I apologize for the confusion with the terminology - you are correct that they are not sparse files but they are considered reparse points.

It looks like Microsoft has documented the interaction with the placeholder files here:

https://learn.microsoft.com/en-us/windows/win32/cfapi/build-a-cloud-file-sync-engine

There is also additional information on reparse points which are used for the placeholder files:

https://learn.microsoft.com/en-us/windows/win32/fileio/reparse-points

Finally - there are a bunch of examples on how to manage reparse points with a device driver perspective that could be used to establish a base design.  A basic example in C identifies some of the core concepts and methods that need to be implemented, which will probably vary significantly from your device driver design, but the net result is that this is a well documented feature that shouldn't require a lot of reverse engineering.  In fact, the core to the reparse points logic is in ntifs.h which is included in the Windows SDK:

https://github.com/tpn/winsdk-10/blob/master/Include/10.0.16299.0/km/ntifs.h

An example that I asked our favorite little AI tool to generate an example, which obviously has flaws in it but shows some of the core functions to call, is here as well:

#include <ntddk.h>
#include <ntdddisk.h>
#include <mountdev.h>

// Reparse point buffer structure
typedef struct _REPARSE_DATA_BUFFER {
    ULONG  ReparseTag;
    USHORT ReparseDataLength;
    USHORT Reserved;
    union {
        struct {
            USHORT SubstituteNameOffset;
            USHORT SubstituteNameLength;
            USHORT PrintNameOffset;
            USHORT PrintNameLength;
            ULONG  Flags;
            WCHAR  PathBuffer[1];
        } SymbolicLinkReparseBuffer;
        struct {
            USHORT SubstituteNameOffset;
            USHORT SubstituteNameLength;
            USHORT PrintNameOffset;
            USHORT PrintNameLength;
            WCHAR  PathBuffer[1];
        } MountPointReparseBuffer;
        struct {
            UCHAR DataBuffer[1];
        } GenericReparseBuffer;
    };
} REPARSE_DATA_BUFFER, *PREPARSE_DATA_BUFFER;

#define REPARSE_DATA_BUFFER_HEADER_SIZE   FIELD_OFFSET(REPARSE_DATA_BUFFER, GenericReparseBuffer)

#define IOCTL_DISK_GET_REPARSE_POINT   CTL_CODE(IOCTL_DISK_BASE, 0x0003, METHOD_BUFFERED, FILE_ANY_ACCESS)
#define IOCTL_DISK_SET_REPARSE_POINT   CTL_CODE(IOCTL_DISK_BASE, 0x0004, METHOD_BUFFERED, FILE_ANY_ACCESS)
#define IOCTL_DISK_DELETE_REPARSE_POINT   CTL_CODE(IOCTL_DISK_BASE, 0x0005, METHOD_BUFFERED, FILE_ANY_ACCESS)

typedef struct _DEVICE_EXTENSION {
    PDEVICE_OBJECT DeviceObject;
    PDEVICE_OBJECT LowerDeviceObject;
    PUNICODE_STRING DeviceName;
} DEVICE_EXTENSION, *PDEVICE_EXTENSION;

// Reparse point filter functions
NTSTATUS ReparsePointFilter_DispatchCreate(PDEVICE_OBJECT DeviceObject, PIRP Irp);
NTSTATUS ReparsePointFilter_DispatchIoctl(PDEVICE_OBJECT DeviceObject, PIRP Irp);
NTSTATUS ReparsePointFilter_DispatchClose(PDEVICE_OBJECT DeviceObject, PIRP Irp);

NTSTATUS DriverEntry(PDRIVER_OBJECT driverObject, PUNICODE_STRING registryPath) {
    PDEVICE_OBJECT deviceObject = NULL;
    UNICODE_STRING deviceName;
    UNICODE_STRING linkName;
    PDEVICE_EXTENSION deviceExtension;
    NTSTATUS status;

    RtlInitUnicodeString(&deviceName, L"\\Device\\MyVirtualDisk");
    RtlInitUnicodeString(&linkName, L"\\DosDevices\\MyVirtualDisk");

// Create the device object
status = IoCreateDevice(driverObject, sizeof(DEVICE_EXTENSION), &deviceName, FILE_DEVICE_DISK, 0, TRUE, &deviceObject);
if (!NT_SUCCESS(status)) {
    return status;
}

// Initialize the device extension
deviceExtension = (PDEVICE_EXTENSION) deviceObject->DeviceExtension;
deviceExtension->DeviceObject = deviceObject;
deviceExtension->DeviceName = &deviceName;

// Create the symbolic link
status = IoCreateSymbolicLink(&linkName, &deviceName);
if (!NT_SUCCESS(status)) {
    IoDeleteDevice(deviceObject);
    return status;
}

// Register the dispatch functions
driverObject->MajorFunction[IRP_MJ_CREATE] = ReparsePointFilter_DispatchCreate;
driverObject->MajorFunction[IRP_MJ_DEVICE_CONTROL] = ReparsePointFilter_DispatchIoctl;
driverObject->MajorFunction[IRP_MJ_CLOSE] = ReparsePointFilter_DispatchClose;

// Set the device object as ready
deviceObject->Flags |= DO_DIRECT_IO;
deviceObject->Flags &= ~DO_DEVICE_INITIALIZING;

return STATUS_SUCCESS;
}

NTSTATUS ReparsePointFilter_DispatchCreate(PDEVICE_OBJECT DeviceObject, PIRP Irp) {
PDEVICE_EXTENSION deviceExtension = (PDEVICE_EXTENSION) DeviceObject->DeviceExtension;
PIO_STACK_LOCATION stack = IoGetCurrentIrpStackLocation(Irp);
PFILE_OBJECT fileObject = stack->FileObject;

// Check if the file being opened is a reparse point
if (fileObject->FileName.Length >= sizeof(WCHAR) &&
    fileObject->FileName.Buffer[fileObject->FileName.Length/sizeof(WCHAR) - 1] == L'$') {

    // Do something with the reparse point, such as redirecting the file to a different location
    // ...

    // Complete the IRP and return success
    Irp->IoStatus.Status = STATUS_SUCCESS;
    IoCompleteRequest(Irp, IO_NO_INCREMENT);
    return STATUS_SUCCESS;
}

// Forward the IRP to the next driver in the stack
return IoPassIrpDown(DeviceObject, Irp);
}

NTSTATUS ReparsePointFilter_DispatchIoctl(PDEVICE_OBJECT DeviceObject, PIRP Irp) {
PIO_STACK_LOCATION stack = IoGetCurrentIrpStackLocation(Irp);
ULONG ioControlCode = stack->Parameters.DeviceIoControl.IoControlCode;
PVOID inputBuffer = Irp->AssociatedIrp.SystemBuffer;
ULONG inputBufferLength = stack->Parameters.DeviceIoControl.InputBufferLength;
PVOID outputBuffer = Irp->AssociatedIrp.SystemBuffer;
ULONG outputBufferLength = stack->Parameters.DeviceIoControl.OutputBufferLength;
ULONG bytesReturned = 0;
NTSTATUS status = STATUS_SUCCESS;

switch (ioControlCode) {
    case IOCTL_DISK_GET_REPARSE_POINT:
        // Retrieve the reparse point data for the specified file
        // ...

        bytesReturned = sizeof(REPARSE_DATA_BUFFER);
        break;

    case IOCTL_DISK_SET_REPARSE_POINT:
        // Set the reparse point data for the specified file
        // ...
        break;

    case IOCTL_DISK_DELETE_REPARSE_POINT:
        // Delete the reparse point for the specified file
        // ...
        break;

    default:
        // Forward the IRP to the next driver in the stack
        return IoPassIrpDown(DeviceObject, Irp);
}

// Complete the IRP and return success
Irp->IoStatus.Information = bytesReturned;
Irp->IoStatus.Status = status;
IoCompleteRequest(Irp, IO_NO_INCREMENT);
return status;
}

NTSTATUS ReparsePointFilter_DispatchClose(PDEVICE_OBJECT DeviceObject, PIRP Irp) {
// Do any necessary cleanup when a file handle is closed
// ...

// Forward the IRP to the next driver in the stack
return IoPassIrpDown(DeviceObject, Irp);
}

This example does implement the IRP concept for the device stack as well.

There are also examples as to how the Cloud API filter driver works in Windows which also supports reparse points:

https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/CloudMirror

There's quite a bit of documentation out there on this topic from what I've seen...is this candidly a feature that does not have as much importance as opposed to other features to be added?  Reparse points are used in other situations beyond this - they are also used in volume mountpoints and another common use for reparse points is backup files that are archived that are then retrieved (not using the Cloud API from Microsoft for example).

I apologize for the length of this post - just trying to help out as I think this feature would be used by others - it's just the most obvious hole I have in my current setup and figured I'd inquire.

 

 

Link to comment
Share on other sites

  • 0

Another comment on this topic - I have discovered the following when a OneDrive folder is located on a drivepool (in addition to not supporting Files On Demand/Reparse Points):

  1. DOS/PowerShell commands in a folder that is considered a OneDrive folder do _not_ allow for basic DOS commands.  An example, if you try to use a "move" command it will fail with "access denied".  When the file is copied to a folder outside of the OneDrive folder on a drivepool, the system will allow the move command.  This also affects tools like ffmpeg.  Interestingly, the Windows Explorer does support the movement of files through cut and paste...but not PowerShell/Cmd.
  2. OneDrive's sync engine does not seem to catch some files - I have files now in my folder structure that are setup for syncing but are not syncing as expected.
  3. Windows Backup, when trying to create a system image backup and save that in a drivepool, gives an error (The bootfile is too small to support persistent snapshots) when the VSS provider closes the snapshot.  This is caused by the drivepool as backing up to a drive that is not a drivepool succeeds.

It looks like the virtual driver that is used has a lot of limitations.  Are these also known issues as well?

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...