in

System.Data.SQLite

An open source ADO.NET provider for the SQLite database engine

Database file is locked in a multithreading scenario

Last post 07-29-2011 2:04 AM by tral. 19 replies.
Page 1 of 2 (20 items) 1 2 Next >
Sort Posts: Previous Next
  • 06-04-2008 3:12 PM

    Database file is locked in a multithreading scenario

    Hello,

    First let me take this opportunity to thank you for the great job you are doing with this provider!

    As for our problem, perhaps it is a result of not understanding correctly - the way SQLite behaves in a multithreading scenario, but we have an issue when multiple threads are attempting to perform inserts on the same table.

    each insert should be very quick but when the two are run together we get the dreaded "The database file is locked" mesage.

     

    I have added a sample application that replicates this scenario (using provider V1.0.49.0 as well as 1.0.48.0).

     

    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Data.SQLite;
    using System.Threading;

    namespace SqliteTest
    {
        class Program
        {
            private static volatile bool ms_bContinueRunning = true;

            static void Main(string[ args)
            {
                bool bTableCreated = false;

                //first we create the test db and table
                Console.WriteLine("Creating Test DB...");
                try
                {
                    using (SQLiteConnection connection = new SQLiteConnection(string.Format("Data Source = {0};", AppDomain.CurrentDomain.BaseDirectory + "TestDB.DB")))
                    {
                        connection.Open();

                        using (SQLiteCommand commandCreate = connection.CreateCommand())
                        {
                            commandCreate.CommandText = "create table TestTable(aTest INTEGER, bTest INTEGER, cTest INTEGER, dTest INTEGER, eTest INTEGER)";
                            commandCreate.CommandType = System.Data.CommandType.Text;
                            commandCreate.ExecuteNonQuery();

                            bTableCreated = true;
                            Console.WriteLine("Test DB created successfully");
                        }
                    }
                }
                catch (Exception ex)
                {
                    bTableCreated = false;

                    if (ex != null)
                    {
                        if (ex.InnerException != null) ex = ex.InnerException;
                        Console.WriteLine("Error while creating table: " + ex.ToString());
                    }
                }

                if (bTableCreated)
                {
                    //start 2 concurrent threads
                    Console.WriteLine("Starting Test insert threads...");
                    Console.WriteLine();

                    Thread[ threadsInsertTest = new Thread[2];

                    for (int iThreadCounter = 0; iThreadCounter < 2; iThreadCounter++)
                    {
                        threadsInsertTest[iThreadCounter] = new Thread(delegate() { InsertSqlite(); });
                        threadsInsertTest[iThreadCounter].IsBackground = true;
                        threadsInsertTest[iThreadCounter].Name = string.Format("Thread Insert Test #{0}", iThreadCounter);
                        threadsInsertTest[iThreadCounter].Start();
                    }

                    Console.ReadKey();

                    try
                    {
                        ms_bContinueRunning = false;

                        for (int iThreadCounter = 0; iThreadCounter < 2; iThreadCounter++)
                        {
                            if (!threadsInsertTest[iThreadCounter].Join(5000)) threadsInsertTest[iThreadCounter].Abort();
                        }
                    }
                    finally
                    {
                        Console.WriteLine("Done, Exitting....");
                        Thread.Sleep(1500);
                    }
                }
                else Console.ReadKey();
            } 

            static public void InsertSqlite()
            {
                string sSQLCommand = "INSERT INTO TestTable(aTest, bTest, cTest, dTest, eTest) VALUES(1, 2, 3, 4, 5)";

                try
                {
                    using (SQLiteConnection connection = new SQLiteConnection(string.Format("Data Source = {0};", AppDomain.CurrentDomain.BaseDirectory + "TestDB.DB")))
                    {
                        connection.Open();

                        while (ms_bContinueRunning)
                        {
                            for (int iLoopCounter = 0; iLoopCounter < 10; iLoopCounter++)
                            {
                                using (SQLiteCommand command = connection.CreateCommand())
                                {
                                    command.CommandText = sSQLCommand;
                                    command.CommandType = System.Data.CommandType.Text;
                                    int iResult = command.ExecuteNonQuery();
                                }
                            }
                        }
                    }
                }
                catch (SQLiteException sqlitex)
                {
                    if (sqlitex != null)
                    {
                        Console.WriteLine("SQLite Exeption: " + sqlitex.ToString());
                    }
                    else Console.WriteLine("Unknown SQLite Error occurred");
                }
                catch (Exception ex)
                {
                    if (ex != null)
                    {
                        if (ex.InnerException != null) ex = ex.InnerException;
                        Console.WriteLine("General Exception: " + ex.ToString());
                    }
                    else Console.WriteLine("Unknown General Error occurred");
                }
            }           
        }
    }

    and the error we get is :

    SQLite Exeption: System.Data.SQLite.SQLiteException: The database file is locked

    database is locked
       at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
       at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
       at SqliteTest.Program.InsertSqlite() in C:\Users\tomk\Documents\Visual Studio
     2005\Projects\CSSQLiteLockTest\CSSQLiteLockTest\Program.cs:line 101

    The error is not caused by a 30 seconds timeout (which is the default) - we measured that and it happens instantaneously with sending the command, and none of the inserts should take that long anyway... Is this a problem with the code or with the DB/Provider?

    Thanks in advance for any assistance.

  • 06-05-2008 9:58 AM In reply to

    • Sam_
    • Top 10 Contributor
    • Joined on 02-14-2007
    • Washington DC
    • Posts 87

    Re: Database file is locked in a multithreading scenario


    We have noticed similar problems where multi-threaded and especially multi-process executions take exponentially longer than a single-threaded model with occasional locked errors. 

    No solution or suggestions though, just my condolences.  :(

     

    Sam 

    We're hiring! B-Line Medical is seeking .NET Developers for exciting positions in medical product development in MD/DC. Work with a variety of technologies in a relaxed team environment. See ads on http://careerbuilder.com.
  • 06-06-2008 3:37 AM In reply to

    Re: Database file is locked in a multithreading scenario

     I think this is a problem with the code.  you're implicitly opening and commiting 10 transactions on each iteration of each thread.  SQLite enforces ACID complicance via disk syncs so this is an awful lot of overhead.  More over, you're re-creating your command every time you try to run it.  Again this is an un-needed performance hit.  I suggest restructuring your  InsertSQLite function as such:

     

    static public void InsertSqlite()
    {
        string sSQLCommand = "INSERT INTO TestTable(aTest, bTest, cTest, dTest, eTest) VALUES(1, 2, 3, 4, 5)";
        SQLiteConnection conn = new SQLiteConnection(string.Format("Data Source = {0};", AppDomain.CurrentDomain.BaseDirectory + "TestDB.DB"));
        SQLiteTransaction trans =  null;
        SQLiteCommand cmd = null;
        try
        {
            conn.Open();
           
            cmd = conn.CreateCommand
            cmd.CommandText = sSQLCommand;
            cmd.CommandType = System.Data.CommandType.Text;
           
            while (ms_bContinueRunning)
            {
                trans = conn.BeginTransaction();
                cmd.Transaction = trans;
                for (int iLoopCounter = 0; iLoopCounter < 10; iLoopCounter++)
                {
                        int iResult = command.ExecuteNonQuery();
                }
                trans.Commit();
                trans = null;
            }
        }
        catch (SQLiteException sqlitex)
        {
            if (sqlitex != null)
            {
                Console.WriteLine("SQLite Exeption: " + sqlitex.ToString());
            }
            else Console.WriteLine("Unknown SQLite Error occurred");
        }
        catch (Exception ex)
        {
            if (ex != null)
            {
                if (ex.InnerException != null) ex = ex.InnerException;
                Console.WriteLine("General Exception: " + ex.ToString());
            }
            else Console.WriteLine("Unknown General Error occurred");
        }
        finally
        {
            if(cmd != null)
                cmd.Dispose();

            if(trans != null)
                trans.Dispose();

            conn.Dispose();
        }
    }

  • 06-06-2008 6:26 PM In reply to

    Re: Database file is locked in a multithreading scenario

    Hey Jeffrey,

    While I appreciate your comment, I think you have misunderstood. The code above is obviously not the code in our application but merely a simple way to recreate the bug. We are using transactions in our full code, BUT not in every scenario (single inserts for example) transactions are relevant and for all we have noticed this bug exists on whatever multithreaded implementation we are using.

    The code above only aims to recreate the bug in the simplest, fastest way so it is more easily debugable for the dev's. In any other implementation it may take a while before it appears (minutes to hours sometimes).

  • 06-07-2008 1:56 PM In reply to

    Re: Database file is locked in a multithreading scenario

     have you considered the possiblity that your transaction is creating a deadlock?  I know a lot of people forget that database access = locking.

  • 06-07-2008 2:20 PM In reply to

    Re: Database file is locked in a multithreading scenario

    Yup, but that is not what's happening. The transaction shouldn't be more than a second or two (instantaneous actually). Have you tried running the sample program? I think the results pretty much speak for themselves - if this was a problem with locking then the 30 seconds timeout would be the reason for the exception (which i would be ok with and look for ways to improve our code). 

    BUT, if you'll run the sample program you'll be able to see that the exception occurs immediately when the execute command is run. NO 30 sec. timeout - even if our code was horrible (and it's not - pretty decent actually, we have it running on a bunch of other DB's including PostgreSQL, ORACLE, MS SQL, MySQL, and it works VERY good on them), the 30 sec. timeout should be the cause for the exception.

    Or are we missing something here?

  • 06-07-2008 2:54 PM In reply to

    Re: Database file is locked in a multithreading scenario

    I'll be looking into this for the next release.  It shouldn't give up right away even if the underlying SQLite engine does.  It should timeout after the specifed command timeout.

    Robert

     

  • 06-07-2008 3:12 PM In reply to

    Re: Database file is locked in a multithreading scenario

    Thanks Robert, I appreciate all your hard work on this.

  • 06-30-2008 12:31 AM In reply to

    Re: Database file is locked in a multithreading scenario

    Hi Robert,

    Is there a chance for a fix in the 1.0.50.X release? I don't mean to nag or to put any pressure, just to know if we should be watching out for these versions :)

  • 06-30-2008 9:55 AM In reply to

    Re: Database file is locked in a multithreading scenario

    Sorry, I forgot about this one when I did the release.  I'll get it fixed ASAP.

  • 06-30-2008 11:57 AM In reply to

    Re: Database file is locked in a multithreading scenario

    it's ok - thank you very much for working on it.
  • 07-14-2008 1:41 AM In reply to

    • Yannik
    • Not Ranked
    • Joined on 07-14-2008
    • Posts 2

    Re: Database file is locked in a multithreading scenario

    Hi there

    I was wondering if this one was fixed as I can reproduce a similar issue in 1.0.51.0?

    Thank you very much.

     

    PS. Terrific provider!

  • 07-14-2008 8:30 AM In reply to

    Re: Database file is locked in a multithreading scenario

    What's the error message?

    Also, you guys have to remember that in a multi-threaded scenario where more than one thread is writing to the database, there is no "queue" so that the threads play nicely with one another and wait in line.  It's very easy for one thread to "starve" another thread, and gain write access to the database multiple times before the other thread ever gets a chance to squeeze a write in.  In such a case its theoretically possible that one thread will get starved for more than the CommandTimeout, and fail.

  • 07-14-2008 9:12 AM In reply to

    • Yannik
    • Not Ranked
    • Joined on 07-14-2008
    • Posts 2

    Re: Database file is locked in a multithreading scenario

     Here's our scenario (all in the same app):

    we first try to open a pooled connection, give up after 5 secs (connect timeout=5),

    then we try to open a non pooled connection for about 20 ms, and this one fails

    almost immediately. this is a code that was put in place for another sql, as

    sometimes the connection pool could get corrupted.


    so, in this case, the sqlite provider cannot open a connection in 5 secs, we try to

    open a non pooled one for 20 secs, but it immediately fails with the following

    exception:

    System.Data.SQLite.SQLiteException: The database file is locked
    database is locked
       at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql,

    SQLiteStatement previous, String& strRemain)
       at System.Data.SQLite.SQLiteCommand.BuildNextCommand()
       at System.Data.SQLite.SQLiteCommand.GetStatement(Int32 index)
       at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
       at System.Data.SQLite.SQLiteConnection.Open()


    Sorry, I should have said earlied that I was trying to open a non pooled connection after a pooled one; I just forgot about this code and I thought the initial connection was failing immediately whereas it was failing after 5 secs, and given the load at that time, it was normal.

    So, we're fine ;)

    Thanks.

  • 07-22-2008 1:24 AM In reply to

    Re: Database file is locked in a multithreading scenario

    Dear Robert,

    Thanks for all your hard work. I'm using the provider since version 1.0.44.0.
    Recently I tried to switch to version 1.0.52.0 and encountered the kind of problems
    as discussed in this thread and in thread http://sqlite.phxsoftware.com/forums/p/977/4161.aspx.

    To prevent database locking exception I wrote my own wrapper with the .Net lock mechanism.
    My program uses two threads and multiple transactions. Luckely I had written a NUnit test to
    see if every thing was working as I expected.
    This all was ok when using version 1.0.44.0.

    When I encountered the database locked exception again in 1.0.52.0 I was a bit surprised.
    Like said above I'm sure there is one and only connection open at a time.
    I've tested multiple versions to find the first version with this problem.
    My tests succeeded in the versions 1.0.48.0 till 1.0.50.0A.
    The test failed in versions 1.0.51.0 and 1.0.52.0.

    The multithreaded test without using transactions succeeds in all versions I've tested.

    If I have more time I will rewrite and post the test so it doesn't depend on my code.

    Best regards,


    IJsbrand


    The stack trace I got was:

    --ApplicationException
       at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql, SQLiteStatement previous, String& strRemain)
       at System.Data.SQLite.SQLiteCommand.BuildNextCommand()
       at System.Data.SQLite.SQLiteCommand.GetStatement(Int32 index)
       at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
       at System.Data.SQLite.SQLiteConnection.Open()
     

Page 1 of 2 (20 items) 1 2 Next >
Powered by Community Server (Commercial Edition), by Telligent Systems