Scuro
-
Posts
41 -
Joined
-
Last visited
-
Days Won
1
Posts posted by Scuro
-
-
Done. I have submitted the service logs as requested.
I also verified that Mariadb generated the same errors as before when I moved it back to drivepool.
It otherwise runs without issue when the database is not used on drivepool.
-
Yes I am on latest RC.
QuoteIf you're on the RC build, and you're still seeing this issue then stop the service, enable file system logging, start the service and wait for it to fail.
Would it be okay to turn mariadb service off, enable logging, turn mariadb back on with error reported, stop mariadb, and then send you the logs? It's used for a small website that I just had it throw errors saying it hit it's max connections which it has never done before. I am trying to avoid more downtime.
EDIT: Database logs show same errors reported many times before the max connection outage:
2018-03-01 11:15:49 9192 [ERROR] InnoDB: Operating system error number 24 in a file operation. 2018-03-01 11:15:49 9192 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html 2018-03-01 11:15:49 9192 [ERROR] InnoDB: File \\.\D:: 'DeviceIoControl(IOCTL_STORAGE_QUERY_PROPERTY)' returned OS error 224.
-
Does anyone have experience with a SQL database on drivepool? When checking logs after moving my database to the drivepool I noticed some errors:
2018-03-01 10:32:23 7100 [Note] InnoDB: innodb_page_size=32768 2018-03-01 10:32:23 7100 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions 2018-03-01 10:32:23 7100 [Note] InnoDB: Uses event mutexes 2018-03-01 10:32:23 7100 [Note] InnoDB: Compressed tables use zlib 1.2.3 2018-03-01 10:32:23 7100 [Note] InnoDB: Number of pools: 1 2018-03-01 10:32:23 7100 [Note] InnoDB: Using SSE2 crc32 instructions 2018-03-01 10:32:23 7100 [Note] InnoDB: Initializing buffer pool, total size = 896M, instances = 1, chunk size = 128M 2018-03-01 10:32:23 7100 [Note] InnoDB: Completed initialization of buffer pool 2018-03-01 10:32:23 7100 [Note] InnoDB: Highest supported file format is Barracuda. 2018-03-01 10:32:25 7100 [Note] InnoDB: 128 out of 128 rollback segments are active. 2018-03-01 10:32:25 7100 [Note] InnoDB: Creating shared tablespace for temporary tables 2018-03-01 10:32:25 7100 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2018-03-01 10:32:25 7100 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB. 2018-03-01 10:32:25 7100 [ERROR] InnoDB: Operating system error number 24 in a file operation. 2018-03-01 10:32:25 7100 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html 2018-03-01 10:32:25 7100 [ERROR] InnoDB: File \\.\D:: 'DeviceIoControl(IOCTL_STORAGE_QUERY_PROPERTY)' returned OS error 224. 2018-03-01 10:32:25 7100 [Note] InnoDB: Waiting for purge to start 2018-03-01 10:32:25 7100 [Note] InnoDB: 5.7.20 started; log sequence number 33980985041 2018-03-01 10:32:25 6684 [Note] InnoDB: Loading buffer pool(s) from D:\ServerFiles\MariaDB\ib_buffer_pool 2018-03-01 10:32:25 7100 [Note] Plugin 'FEEDBACK' is disabled. 2018-03-01 10:32:25 7100 [Note] Server socket created on IP: '::'. 2018-03-01 10:32:26 7100 [Note] Reading of all Master_info entries succeded 2018-03-01 10:32:26 7100 [Note] Added new Master_info '' to hash table 2018-03-01 10:32:26 7100 [Note] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: ready for connections. Version: '10.2.12-MariaDB-log' socket: '' port: 3306 mariadb.org binary distribution 2018-03-01 10:32:27 6684 [Note] InnoDB: Buffer pool(s) load completed at 180301 10:32:27
That error is reported every time the service is started if the database is on drivepool.
EDIT: The SQL service still runs but I am not sure how safe it is.
-
Just as an update, I believe I found the root cause of the hard lockups.
IO errors in event log showed IO hardware failures on almost every drive. I replaced two that had the most errors thinking that they were failing. The IO errors and lockups continued. After doing further searching in event logs, I found that the IO errors all started an hour after I had replaced the PSU on the server. This leads me to what I would now believe a faulty PSU, not drivepool (which I had updated around the same time). I will be returning the PSU and getting a different model to verify.
On another note however, I still can't get drivepool to balance the new drive even though drivepool knows that it is not balanced. Troubleshooting is still ongoing with stablebit.I've disabled write caching on the drives to avoid corruption of files until crash is fully diagnosed.
-
I appreciate the help. I have sent the troubleshoot report in.
Do you have any VMs or SQL databases on drivepool? The hardlocks appeared to happen after any intense IO operations from either or of these services. It may be better for me to remove these services form the pool regardless.
Still having issues with the drives not balancing correctly.
-
8 hours ago, Christopher (Drashna) said:
The event you've posted most likely has nothing to do with the issue that you're seeing. If you have the "not more often than every X hours" option enabled, then you'll see this.
That said, could you enable tracing and try to enable file system logging and initiate rebalancing?
http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_CollectionAnd after that, run the StableBit Troubleshooter?
http://wiki.covecube.com/StableBit_TroubleshooterUse "3451" as the Contact ID
I will run that for you and submit the troubleshooter for you shortly, but I have been having serious issues since upgrading to the RC builds. Any program that puts IO stress on the pool is hard locking windows 10 after throwing disk errors into event history ("The IO operation at logical block address 0x1 for Disk 2 (PDO name: \Device\00000039) was retried."). Disk 2 is drivepool.
I had first thought a drive may be dying but after replacing two drives that sometimes showed errors in event viewer during these lockups but a lockup happened again this morning.
I have (had as I now moved them off drivepool) a hyper-v VM and a Maria database in drivepool. Any extensive IO operations by these and windows would hard lock (no BSOD, computer would be unresponsive to everything). Is there a recommended stable version for me to try downgrading to?
-
Got a new drive and noticed that no matter how many times I tell it to balance, it won't balance fully.
Service log shows no errors but it does say "0:10:38.3: Information: 0 : [Rebalance] Cannot balance. Immediate re-balance throttled."
Regardless, if I try a manual rebalance it only runs for a few minutes and then shows the same balance as shown in picture.
I am running the latest release client.
-
Thanks. It looks like after updating it fixed the problem.
-
Here is a picture of the problem:
-
Is it for duplication, specifically, or balancing?
Folder duplication.
From the manual of Stablebit it supports duplication of files in use if you have it selected to real time.
Using Duplicate files later, DrivePool cannot duplicate files that are in use. Any files that need duplicating and are in use are skipped. Also, duplicating files later will take longer, the larger your pool becomes.
Duplicating files in real-time doesn't have these 2 limitations. Because of this, duplicating files in real-time is the best option for most people.
https://stablebit.com/Support/DrivePool/1.X/Manual?Section=Folder%20Duplication
Also, what version of StableBit DrivePool are you using, specifically?I made sure that I stated it in my first post. I am on the current beta 2.2.0.798
-
I have a mysql database set to be duplicated real time but I keep getting warnings that \ServerFiles\MySQL\user.MYD File is in use.
I thought drivepool supported duplication in real time for files in use? I am on the current beta 2.2.0.798
-
For some reason drivepool was not using duplicate folders (redundant folders for safety) to balance the drives. After I removed duplicate options and reapplied, drivepool balanced correctly.
-
Log sent in PM
-
Never mind. I found out that the problem was caused by duplicated folders. For some reason it would not balance correctly till the duplicated folders were removed and reapplied.
-
Drive pool used to be balanced correctly for the longest time. Then I added a few more drives and it's gone downhill since. No matter what I try to do, it won't balance the drives evenly. I checked logs and didnt see any errors. I tried resetting all balance options to default as well as turning all plugins off except disk space equalizer. No luck. Drivepool obviously sees the balance issue but when I tell it to rebalance, it just immediately stops and continues saying that it is unbalanced. Any help?
MySQL (Mariadb) errors only when using drivepool.
in General
Posted
Thanks for your testing Christopher. I have more information that you could maybe test.
It's 100% replicable for me and is not permissions. The folder has the exact same permissions while on the drivepool, that it has off of it.
It is not run as system, but as a service account, and I gave that service account the correct permissions to the directory. Same permissions on multiple test directories on different drives do not result in the errors if the directory is not in the pool.
Mariadb was also a duplicated directory when it was on the pool.
Be warned that this has caused massive unstability for me. I have since removed mariadb + www (was also a drivepool duplicated directory) directories off of drivepool. The IO errors, php/apache2 errors, and the database errors have all stopped after removing them from the pool.