Learn what is a datasource and how to create and customize DataSource bean in Spring boot applications.
1. What is DataSource
PooledObjectFactory.destroyObject(org.apache.tomcat.dbcp.pool2.PooledObject) setAutoCommitOnReturn public void setAutoCommitOnReturn(boolean autoCommitOnReturn) setCacheState public void setCacheState(boolean cacheState) setClearStatementPoolOnReturn public void setClearStatementPoolOnReturn(boolean clearStatementPoolOnReturn). 13-Feb-2015 15:30:44.178 WARNING main org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.getObjectInstance WARNING! Property maxActive is not used since DBCP2, use maxTotal instead. MaxTotal default value is 8. You have set value of '20' for 'maxActive' property WARNING! Property removeAbandoned is not used since DBCP2.
A datasource is a factory for connections to any physical data source. An alternative to the DriverManager facility. It uses a URL along with some credentials to establish a database connection.
An object that implements the
javax.sql.DataSource interface will typically be registered with JNDI service and can be discovered using it’s JNDI name.
A datasource may be used to obtain :
- connection which can be used in connection pooling
- connection which can be used in distribured transactions and connection pooling
2. DataSource Configuration
Spring boot allows defining datasource configuration in both ways i.e. Java config and properties config.
DataSourceAutoConfiguration checks for DataSource.class (or EmbeddedDatabaseType.class) on the classpath and few other things before configuring a DataSource bean for us.
If not already defined, include spring-boot-starter-data-jpa to project. It brings all necessary dependencies including JDBC drivers for various databases e.g.
mysql-connector-java for connecting to mysql.
If we are planning to use embedded database at some step (e.g. testing), we can import H2 db separately.
DataSource configuration is provided by external configuration properties (
spring.datasource.* ) in
The properties configuration decouple the configuration from application code. This way, we can import the datasource configurations from even configuration provider systems.
Below given configuration shows sample properties for H2, MySQL, Oracle and SQL server databases.
We often do not need to specify the
driver-class-name, since Spring Boot can deduce it for most databases from the url.
2.3. DataSource Bean
Recommended way to create DataSource bean is using DataSourceBuilder class within a class annotated with the @Configuration annotation. The datasource uses the underlying connection pool as well.
2.4. JNDI DataSource
If we deploy your Spring Boot application to an Application Server, we might want to configure and manage the DataSource by using the Application Server’s built-in features and access it by using JNDI.
We can do this using the spring.datasource.jndi-name property. e.g.
3. Connection Pooling
3.1. HikariCP, tomcat pooling and commons DBCP2
For a pooling DataSource to be created, Spring boot verifies that a valid
Driver class is available. If we set
spring.datasource.driver-class-name property then that mentioned driver class has to be loadable.
The auto-configuration first tries to find and configure HikariCP. If HikariCP is available, it always choose it. Otherwise, if the Tomcat pooling is found, it is configured.
If neither HikariCP nor the Tomcat pooling datasource are available and if Commons DBCP2 is available, it is used.
spring-boot-starter-data-jpa starter automatically get a dependency to
3.2. Custom settings
It is also possible to fine-tune implementation-specific settings by using their respective prefix (
For example, we can use below properties to customize a DBCP2 connection pool.
4. Multiple Datasources with Spring boot
To configure multiple data sources, create as many bean definitions you want but mark one of the
DataSource instances as @Primary, because various auto-configurations down the road expect to be able to get one by type.
Remember that if we create your own DataSource, the auto-configuration backs off. So we are responsible for providing configurations for all datasource beans.
While autowiring the datasource, spring boot will prefer the primary datasource i.e. “mySqlDataSource”. To autowire another non-primary datasource, use @Qualifier annotation.
Spring boot provides very easy ways to create datasource beans – either using properties config or using java config
@Bean. Spring boot provides ready-made auto configuration to use which can be further customized with advanced options in
Spring boot tries to find and configure connection pooling first HikariCP, second Tomcat pooling and then finally Commons DBCP2.
HikariCP comes inbuilt with
We can configure multiple datasources and one of them must be marked as
@Primary. Primary datasource is autowired by default, and other datasources need to be autowired along with
Happy Learning !!
JNDI Datasource How-To
Table of Contents
- Database Connection Pool (DBCP 2) Configurations
- Oracle 8i with OCI client
- Common Problems
JNDI Datasource configuration is covered extensively in theJNDI-Resources-HOWTO. However, feedback from
tomcat-user hasshown that specifics for individual configurations can be rather tricky.
Here then are some example configurations that have been posted totomcat-user for popular databases and some general tips for db usage.
You should be aware that since these notes are derived from configurationand/or feedback posted to
tomcat-user YMMV :-). Please let usknow if you have any other tested configurations that you feel may be of useto the wider audience, or if you feel we can improve this section in anyway.
Please note that JNDI resource configuration changed somewhat betweenTomcat 7.x and Tomcat 8.x as they are using different versions ofApache Commons DBCP library. You will most likely need to modify olderJNDI resource configurations to match the syntax in the example below in orderto make them work in Tomcat 9.See Tomcat Migration Guidefor details.
Also, please note that JNDI DataSource configuration in general, and thistutorial in particular, assumes that you have read and understood theContext andHost configuration references, includingthe section about Automatic Application Deployment in the latter reference.
DriverManager, the service provider mechanism and memory leaks
java.sql.DriverManager supports theserviceprovider mechanism. This feature is that all the available JDBC driversthat announce themselves by providing a
META-INF/services/java.sql.Driverfile are automatically discovered, loaded and registered,relieving you from the need to load the database driver explicitly beforeyou create a JDBC connection.However, the implementation is fundamentally broken in all Java versions fora servlet container environment. The problem is that
java.sql.DriverManager will scan for the drivers only once.
The JRE Memory Leak Prevention Listenerthat is included with Apache Tomcat solves this by triggering the driver scanduring Tomcat startup. This is enabled by default. It means that onlylibraries visible to the common class loader and its parents will be scanned fordatabase drivers. This include drivers in
$CATALINA_BASE/lib, the class path and (where the JRE supports it)the endorsed directory. Drivers packaged in web applications (in
WEB-INF/lib) and in the shared class loader (where configured) willnot be visible and will not be loaded automatically. If you are consideringdisabling this feature, note that the scan would be triggered by the first webapplication that is using JDBC, leading to failures when this web application isreloaded and for other web applications that rely on this feature.
Thus, the web applications that have database drivers in their
WEB-INF/lib directory cannot rely on the service providermechanism and should register the drivers explicitly.
The list of drivers in
java.sql.DriverManager is alsoa known source of memory leaks. Any Drivers registeredby a web application must be deregistered when the web application stops.Tomcat will attempt to automatically discover and deregister anyJDBC drivers loaded by the web application class loader when the webapplication stops.However, it is expected that applications do this for themselves viaa
Database Connection Pool (DBCP 2) Configurations
The default database connection pool implementation in Apache Tomcatrelies on the libraries from theApache Commons project.The following libraries are used:
- Commons DBCP 2
- Commons Pool 2
These libraries are located in a single JAR at
$CATALINA_HOME/lib/tomcat-dbcp.jar. However,only the classes needed for connection pooling have been included, and thepackages have been renamed to avoid interfering with applications.
DBCP 2 provides support for JDBC 4.1.
See the DBCP 2 documentation for a complete list of configuration parameters.
Preventing database connection pool leaks
A database connection pool creates and manages a pool of connectionsto a database. Recycling and reusing already existing connectionsto a database is more efficient than opening a new connection.
There is one problem with connection pooling. A web application hasto explicitly close ResultSet's, Statement's, and Connection's.Failure of a web application to close these resources can result inthem never being available again for reuse, a database connection pool 'leak'.This can eventually result in your web application database connections failingif there are no more available connections.
There is a solution to this problem. The Apache Commons DBCP 2 can beconfigured to track and recover these abandoned database connections. Notonly can it recover them, but also generate a stack trace for the codewhich opened these resources and never closed them.
To configure a DBCP 2 DataSource so that abandoned database connections areremoved and recycled, add one or both of the following attributes to the
Resource configuration for your DBCP 2 DataSource:
The default for both of these attributes is
false. Note that
removeAbandonedOnMaintenance has no effect unless poolmaintenance is enabled by setting
timeBetweenEvictionRunsMillisto a positive value. See theDBCP 2 documentation for full documentation on these attributes.
removeAbandonedTimeout attribute to set the numberof seconds a database connection has been idle before it is considered abandoned.
The default timeout for removing abandoned connections is 300 seconds.
logAbandoned attribute can be set to
trueif you want DBCP 2 to log a stack trace of the code which abandoned thedatabase connection resources.
The default is
MySQL DBCP 2 Example
Versions of MySQL and JDBCdrivers that have been reported to work:
- MySQL 3.23.47, MySQL 3.23.47 using InnoDB, MySQL 3.23.58, MySQL 4.0.1alpha
- Connector/J 3.0.11-stable (the official JDBC Driver)
- mm.mysql 2.0.14 (an old 3rd party JDBC Driver)
Tomcat Dbcp2 Maxactive
Before you proceed, don't forget to copy the JDBC Driver's jar into
1. MySQL configuration
Ensure that you follow these instructions as variations can cause problems.
Create a new test user, a new database and a single test table.Your MySQL user must have a password assigned. The driverwill fail if you try to connect with an empty password.
Note: the above user should be removed once testing iscomplete!
Next insert some test data into the testdata table.
2. Context configuration
Configure the JNDI DataSource in Tomcat by adding a declaration for yourresource to your Context.
3. web.xml configuration
Now create a
WEB-INF/web.xml for this test application.
4. Test code
Now create a simple
test.jsp page for use later.
That JSP page makes use ofJSTL'sSQL and Core taglibs. You can get it fromApache Tomcat Taglibs - Standard Tag Libraryproject — just make sure you get a 1.1.x or later release. Once you haveJSTL, copy
standard.jar to your web app's
Finally deploy your web app into
$CATALINA_BASE/webapps eitheras a warfile called
DBTest.war or into a sub-directory called
Once deployed, point a browser at
http://localhost:8080/DBTest/test.jsp to view the fruits ofyour hard work.
Oracle 8i, 9i & 10g
Oracle requires minimal changes from the MySQL configuration except for theusual gotchas :-)
Drivers for older Oracle versions may be distributed as *.zip files ratherthan *.jar files. Tomcat will only use
*.jar files installed in
classes12.zip will need to be renamed with a
.jarextension. Since jarfiles are zipfiles, there is no need to unzip and jar thesefiles - a simple rename will suffice.
For Oracle 9i onwards you should use
oracle.jdbc.driver.OracleDriver as Oracle have statedthat
oracle.jdbc.driver.OracleDriver is deprecated and supportfor this driver class will be discontinued in the next major release.
1. Context configuration
In a similar manner to the mysql config above, you will need to define yourDatasource in your Context. Here we define aDatasource called myoracle using the thin driver to connect as user scott,password tiger to the sid called mysid. (Note: with the thin driver this sid isnot the same as the tnsname). The schema used will be the default schema for theuser scott.
Tomcat 8 Dbcp2 Configuration
Use of the OCI driver should simply involve a changing thin to oci in the URL string.
2. web.xml configuration
You should ensure that you respect the element ordering defined by the DTD when youcreate you applications web.xml file.
3. Code example
You can use the same example application as above (assuming you create the required DBinstance, tables etc.) replacing the Datasource code with something like
PostgreSQL is configured in a similar manner to Oracle.
1. Required files
Copy the Postgres JDBC jar to $CATALINA_HOME/lib. As with Oracle, thejars need to be in this directory in order for DBCP 2's Classloader to findthem. This has to be done regardless of which configuration step you take next.
2. Resource configuration
You have two choices here: define a datasource that is shared across all Tomcatapplications, or define a datasource specifically for one application.
2a. Shared resource configuration
Use this option if you wish to define a datasource that is shared acrossmultiple Tomcat applications, or if you just prefer defining your datasourcein this file.
This author has not had success here, although others have reported so.Clarification would be appreciated here.
2b. Application-specific resource configuration
Use this option if you wish to define a datasource specific to your application,not visible to other Tomcat applications. This method is less invasive to yourTomcat installation.
Create a resource definition for your Context.The Context element should look something like the following.
3. web.xml configuration
4. Accessing the datasource
When accessing the datasource programmatically, remember to prepend
java:/comp/env to your JNDI lookup, as in the following snippet ofcode. Note also that 'jdbc/postgres' can be replaced with any value you prefer, providedyou change it in the above resource definition file as well.
These solutions either utilise a single connection to the database (not recommended for anything otherthan testing!) or some other pooling technology.
Tomcat Dbcp2 Maven
Oracle 8i with OCI client
Whilst not strictly addressing the creation of a JNDI DataSource using the OCI client, these notes can be combined with theOracle and DBCP 2 solution above.
In order to use OCI driver, you should have an Oracle client installed. You should have installedOracle8i(8.1.7) client from cd, and download the suitable JDBC/OCIdriver(Oracle8i 126.96.36.199 JDBC/OCI Driver) from otn.oracle.com.
classes12.zip file to
classes12.jarfor Tomcat, copy it into
$CATALINA_HOME/lib.You may also have to remove the
javax.sql.* classesfrom this file depending upon the version of Tomcat and JDK you are using.
Putting it all together
Ensure that you have the
.so in your
LD_LIBRARY_PATH (possibly in
$ORAHOMEbin) and also confirm that the native library can be loaded by a simple test programusing
You should next create a simple test servlet or JSP that has thesecritical lines:
where database is of the form
host:port:SID Now if you try to access the URL of yourtest servlet/JSP and what you get is a
ServletException with a root cause of
UnsatisfiedLinkError indicates that you have
- a mismatch between your JDBC classes file andyour Oracle client version. The giveaway here is the message stating that a needed library file cannot befound. For example, you may be using a classes12.zip file from Oracle Version 8.1.6 with a Version 8.1.5Oracle client. The classesXXX.zip file and Oracle client software versions must match.
- It has been reported that ignoring the driver you have downloaded from otn and usingthe classes12.zip file from the directory
$ORAHOMEjdbclibwill also work.
Next you may experience the error
ORA-06401 NETCMN: invalid driver designator
The Oracle documentation says : 'Cause: The login (connect) string contains an invaliddriver designator. Action: Correct the string and re-submit.'Change the database connect string (of the form
host:port:SID) with this one:
Ed. Hmm, I don't think this is really needed if you sort out your TNSNames - but I'm not an Oracle DBA :-)
Here are some common problems encountered with a web application whichuses a database and tips for how to solve them.
Intermittent Database Connection Failures
Tomcat runs within a JVM. The JVM periodically performs garbage collection(GC) to remove java objects which are no longer being used. When the JVMperforms GC execution of code within Tomcat freezes. If the maximum timeconfigured for establishment of a database connection is less than the amountof time garbage collection took you can get a database connection failure.
To collect data on how long garbage collection is taking add the
-verbose:gc argument to your
CATALINA_OPTSenvironment variable when starting Tomcat. When verbose gc is enabledyour
$CATALINA_BASE/logs/catalina.out log file will includedata for every garbage collection including how long it took.
When your JVM is tuned correctly 99% of the time a GC will take lessthan one second. The remainder will only take a few seconds. Rarely,if ever should a GC take more than 10 seconds.
Make sure that the db connection timeout is set to 10-15 seconds.For DBCP 2 you set this using the parameter
Random Connection Closed Exceptions
These can occur when one request gets a db connection from the connectionpool and closes it twice. When using a connection pool, closing theconnection just returns it to the pool for reuse by another request,it doesn't close the connection. And Tomcat uses multiple threads tohandle concurrent requests. Here is an example of the sequenceof events which could cause this error in Tomcat:
Here is an example of properly written code to use a database connectionobtained from a connection pool:
Context versus GlobalNamingResources
Please note that although the above instructions place the JNDI declarations in a Context element, it is possible and sometimes desirable to place these declarations in the GlobalNamingResources section of the server configuration file. A resource placed in the GlobalNamingResources section will be shared among the Contexts of the server.
JNDI Resource Naming and Realm Interaction
In order to get Realms to work, the realm must refer to the datasource as defined in the <GlobalNamingResources> or <Context> section, not a datasource as renamed using <ResourceLink>.