Occasionally when programming in Java you will need a List too large to hold in memory. MapDB is an open source Java library that allows you to create a disk based List (and other Java Collections) very easily.

MapDB is a hybrid of an embedded database engine and the Java Collections framework. It provides Maps, Sets, Lists, Queues, etc. that can be stored in off-heap memory or on disk. MapDB is available via Maven central so it can be easily included in your project. Visit the MapDB home page for more information.

Creating a Disk Based List in Java

Below is an example of creating a disk based ArrayList (IndexTreeList is an ArrayList type object backed by a tree). You can see in the example that DB and DBMaker objects are used to create the List. Each instance of DB represents a single transaction session, so commit() is needed to actually save the data to disk. Note that multiple Lists, Sets, etc. can be written to the same file.

import org.mapdb.DB;
import org.mapdb.DBMaker;
import org.mapdb.Serializer;
import java.util.Map;

public class MapDbTest {

    public static void writeListToDisk() {
        //use DBMaker to create a DB object stored on disk
        //provide output location of list
        DB db = DBMaker.fileDB("/tmp/testMapDB.db").make();

        //use the DB object to create the "myList" ArrayList
        //set the specific serializer for better performance
        List<String> list = db.indexTreeList("myList", Serializer.STRING).createOrOpen();

        //populate list
        for (int i = 0; i < 1000; i++) {
            list.add("item" + i);
        }

        //persist changes on disk
        db.commit();

        //close to protect from data corruption
        db.close();
    }

}

Reading a Disk Based List in Java

Reading a List that has been previously serialized to disk is very similar to creating a List for writes. We simply have to specify file containing the List, as well as the name of the List to deserialize. Also, there is no need to commit() since we are just reading the data.

import org.mapdb.DB;
import org.mapdb.DBMaker;
import org.mapdb.Serializer;
import java.util.Map;

public class MapDbTest {

    public static void readListFromDisk() {
        //use DBMaker to create a DB object of List stored on disk
        //provide location
        DB db = DBMaker.fileDB("/tmp/testMapDB.db").make();

        //use the DB object to open the "myList" ArrayList
        List<String> list = db.indexTreeList("myList", Serializer.STRING).createOrOpen();

        //read from list
        for (int i = 0; i < 1000; i++) {
            System.out.println(list.get(i));
        }

        //close to protect from data corruption
        db.close();
    }

}

MapDB Maven Dependency

Sample MapDB Maven dependency in pom.xml:

<dependency>
  <groupId>org.mapdb</groupId>
  <artifactId>mapdb</artifactId>
  <version>3.0.5</version>
</dependency>