The eflow blog » mysql

Ignoring Foreign Key Checks in Liquibase

Ben — Wed, 10 Feb 2010 01:17:53 +0000

This seems fairly simple in hindsight, but I had a difficult time finding an answer to it, so I putting it down here for posterity’s sake.

SET FOREIGN_KEY_CHECKS=0

I put this as the first changeset in a liquibase xml file. This disables the foreign key and constraint checking in mysql. This would work for other db’s as well, but I haven’t looked up their syntax just yet.

Updating UUID Column in the Large Obs Table

Ben — Thu, 07 Jan 2010 15:43:04 +0000

We have well over 50 million rows in our obs table. Using the simple “update obs set uuid = uuid() where uuid is null” would bomb out with a “The total number of locks exceeds the lock table size” error.

I ran this for a few hours to get through all rows. (Note that it overwrites any uuids that are currently there)

Copy this into a sql script and run it with “source” in mysql at command line.

drop procedure update_uuids;
delimiter //
create procedure update_uuids() begin
set @max = (select max(obs_id) from obs);
set @x = 1;
repeat
set @y = @x + 100000;
update obs set uuid = uuid() where obs_id >=@x and obs_id <@y;
set @x = @y;
until @x > @max
end repeat;
end
//
delimiter ;
call update_uuids();

OpenMRS Database Creation Error – Errcode 17

Ben — Mon, 19 Oct 2009 09:12:31 +0000

A programmer here at AMPATH (Gilbert Tuwei) had some issues installing mysql. The error message was very unhelpful and the solution was equally puzzling, so I want to put here to for the mighty google to find.

Tuwei was running v5.0.17 of MySQL. Upgrading to version 5.1.31 made the problem disappear.

WARN – InitializationFilter$InitializationCompletion$1.run(823) |2009-10-02 15:24:21,000| Error while trying to update to the latest database version
org.openmrs.util.DatabaseUpdateException: There was an error while updating the database to the latest. file: liquibase-update-to-latest.xml. Error: Migration failed for change set liquibase-update-to-latest.xml::2::upul:
Reason:
java.sql.SQLException: Can’t create/write to file ‘C:\WINDOWS\TEMP\#sql_fe4_0.MYI’ (Errcode: 17):
Caused By: Precondition Error
at org.openmrs.util.DatabaseUpdater.executeChangelog(DatabaseUpdater.java:145)
at org.openmrs.web.filter.initialization.InitializationFilter$InitializationCompletion$1.run(InitializationFilter.java:817)
at java.lang.Thread.run(Unknown Source)
Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set liquibase-update-to-latest.xml::2::upul:
Reason:
java.sql.SQLException: Can’t create/write to file ‘C:\WINDOWS\TEMP\#sql_fe4_0.MYI’ (Errcode: 17):
Caused By: Precondition Error
at liquibase.ChangeSet.execute(ChangeSet.java:204)
at liquibase.parser.visitor.UpdateVisitor.visit(UpdateVisitor.java:26)
at org.openmrs.util.DatabaseUpdater$1OpenmrsUpdateVisitor.visit(DatabaseUpdater.java:177)
at liquibase.parser.ChangeLogIterator.run(ChangeLogIterator.java:41)
at org.openmrs.util.DatabaseUpdater.executeChangelog(DatabaseUpdater.java:201)
at org.openmrs.util.DatabaseUpdater.executeChangelog(DatabaseUpdater.java:142)
… 2 more
Caused by: liquibase.exception.PreconditionErrorException: Precondition Error
at liquibase.preconditions.ForeignKeyExistsPrecondition.check(ForeignKeyExistsPrecondition.java:36)
at liquibase.preconditions.NotPrecondition.check(NotPrecondition.java:18)
at liquibase.preconditions.AndPrecondition.check(AndPrecondition.java:21)
at liquibase.ChangeSet.execute(ChangeSet.java:169)
… 7 more

MySQL “Errno” Mystery Explained!

Ben — Wed, 15 Apr 2009 19:22:32 +0000

One of MySQL’s biggest annoyances is the lack of descriptive error messages. Often when dropping a table, index, foreign key you get a cryptic “errno 150 or errno152″ error message.

Well, now when you come across those simply go to the command line and type:

show engine innodb status;

Scroll up a ways and see a very helpful message about exactly which other foreign key or index is causing the blockage! Why can’t MySQL put this information into error message to begin with??

Default MySQL Database to InnoDB Storage Engine

Ben — Tue, 31 Mar 2009 12:13:06 +0000

Currently tables created in MySQL default to MyISam. This is unacceptable for us at Openmrs. You can set a property in your my.cnf file called “default-storage-engine”, but I needed a solution that didn’t involve implementers having to modify their system config files.

I changed the jdbc connection url string to set a session property that sets InnoDB as the default engine:

jdbc:mysql://localhost:3306/databasename?autoReconnect=true&sessionVariables=storage_engine=InnoDB

NOTE! Requires mysql jdbc connecter version 3.1.18 or greater. See http://dev.mysql.com/doc/refman/5.0/en/connector-j.html for documentation and then http://dev.mysql.com/downloads/connector/j/ to download.

The error message I was getting before upgrading the mysql jdbc connector was:

java.sql.SQLException: Server connection failure during transaction. Due to underlying exception: ‘java.sql.SQLException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ” at line 1′.

** BEGIN NESTED EXCEPTION **

java.sql.SQLException
MESSAGE: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ” at line 1

STACKTRACE:

java.sql.SQLException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ” at line 1
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2921)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1570)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2972)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2902)
at com.mysql.jdbc.Statement.executeUpdate(Statement.java:929)
at com.mysql.jdbc.Connection.setSessionVariables(Connection.java:5077)
at com.mysql.jdbc.Connection.initializePropsFromServer(Connection.java:3680)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2684)
at com.mysql.jdbc.Connection.(Connection.java:1474)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:264)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Installation troubles with MySQLdb-python

Ben — Thu, 29 May 2008 18:34:03 +0000

I’m in the middle of moving OpenMRS’s web presence from a shared virtual machine to a new dedicated box. The only trouble I ran into was with the mysql python module (again).

For posterity’s sake, I’m going to write my solution here.

easy_install mysql-python

I received this error when trying to install the MySQLdb-python module into python on centOS:

_mysql.c:2808: error: expected declaration specifiers before ‘init_mysql’
_mysql.c:2886: error: expected ‘{’ at end of input
error: Setup script exited with error: command ‘gcc’ failed with exit status 1

The fix I remembered. I just had to install the python development packages:

yum install python-devel

I then had another very odd and strangely familiar error:

In file included from _mysql.c:35:
/usr/include/mysql/my_config.h:15:28: error: my_config_i386.h: No such file or directory
error: command ‘gcc’ failed with exit status 1

Googling around gave me nothing helpful…in fact the results looked strangely familiar as well. I couldn’t remember the solution to this one though.

Eventually, I actually read the error message and realized that our new vm isn’t an x64 box. For some reason there was a my_config_x64.h file in /usr/include/mysql but there wasn’t a my_config_i386.h. Luckily, that file was present on the old server. Copying that file from /usr/include/mysql on the old box to the new was the cure.

Speeding up OpenMRS with KickFire

Ben — Thu, 17 Apr 2008 22:04:20 +0000

A new company named Kickfire launched at the MySQL 2008 Conference. Their product is a rackmounted server intended to be used as a high-end database (and server, if need be). Their innovation is a cpu specific to sql commands: the “SQL chip”. It works in similar fashion to a graphics specific cpu or router cpu. Instead of taking in the normal assembly commands, the sql chip understands 10 or so sql type of commands: insert, update, join, etc. This change drops the number of commands being run on the chip by leaps and bounds. There are a few other changes they make, but you can learn those in their video. They claim there is a 10x decrease in query time.

Kickfire is currently in beta. If you would like to join the beta program, I can put you in touch with the right people. Unfortunately, all beta boxes will be returned to Kickfire. Fortunately, OpenMRS implementers may have an opportunity at that point for a discounted machine. If you’re looking into buying a high end server, you definitely want to add the Kickfire box to your list of options.

Bending Connector/J to Our Liking

Ben — Wed, 16 Apr 2008 22:55:40 +0000

I attended a session today at the MySQLConf2008 by Mark Matthews on adding extensions to the mysql jdbc connector. There are some exciting implications as an alternative to the BIRT ODA Plugin project. The idea of that project is to create a series of REST calls to only require the end user of the BIRT reporting application to have a high level knowledge of OpenMRS objects (instead of low level knowledge of the OpenMRS data model).

Sometime last year Burke laid out his dream BIRT (or other reporting frameworks) connection. It entailed writing a custom jdbc connector that didn’t act on the database, but instead acted on our API. The “available tables” as the user would see it in BIRT would actually be a subset of our higher level Java objects.

In the session Mark discussed the new intercept-ability of the MySQL Connector/J (new in version 5.1ga of the connector). The relevant/interesting addition to the connector is the “statement interceptor”. We could use this to intercept the select calls from BIRT that look like “SELECT AGE, GENDER, WEIGHT, HEIGHT, LATEST CD4 FROM PATIENT WHERE DATE > 1/1/2008″. These calls could be turned into some LogicService calls on AGE, WEIGHT, LATEST CD4, etc to get the ResultSet instead of passing that (unusable) query to mysql. The LogicService’s resultSet could then be returned to BIRT for display as a data set. There isn’t any documentation on MySQL Forge on how this would work yet. Mark’s slides should show up as an option on the mysqlconf’s wiki soon though.

Side note: The jdbc driver’s version number does not match with the mysql version numbers. Version 5.1 of the driver can be used with the current 5.0 of MySQL.

OpenMRS Replication for Dummies

Ben — Tue, 15 Apr 2008 22:56:52 +0000

Patrick Galbraith presented gave an interesting presention titled “Replication for Dummies” at the MySQL Conference today. Previously, I had the notion that replicating servers and creating a master/slave MySQL setup was difficult. However, Patrick had a master/slave, then dual master, then dual master with a slave attached running in under 20 mins. He stepped through the my.cnf properties required for both the master server and the slave server — less than 10 for each!

The gist of how replication works is this:

The master server writes all insert/update statements to a log file while applying them to itself.
The slave server knows about this log file and simply applies each insert/update to itself.
The slave server keeps track of where in the binlog file it last read from, so if the slave goes offline, it can start again from its exact position to get up-to-date.

It doesn’t seem like Patrick has his notes on the MySQLConf2008 wiki yet, but Brian found a nice easy replication howto we can follow instead. I look forward to setting this up for AMPATH and pointing all of our data managers’ read requests to the slave database instead of wasting cycles on the main master database. (that is royal “I” and will actually be either Brian or Simiyu )