android - Guide to use a large sqlite propulated database

Question

I know this kind questions has been around a lot (most of them with no valid answer), and getting through the answers i didn't find what i'm searching for which is:

1. if i want to use pictures in my database (insertion with sqlite browser), is there any specification(format,size) or limite? so they would work fine when i retrieve them in android. (PS: i have almost 100 JPEG pictures, each has approximately 1-2 Mo size , which count in total 150Mo just for pictures)

2. what's the size limit of database that can be inserted in android? ( i've read about inserting database in external file or something like that(because APK shouldn't be more than 50Mo), can you show me how?

why i'm asking those questions? because i have used a JPEG picture with 7Mb in my database and the app stopped..so i need to know how to use correctly pictures and manage size of database Thanks

import com.readystatesoftware.sqliteasset.SQLiteAssetHelper;

public class Database {
    private static final int DATABASE_VERSION = 1;
    private static final String DATABASE_NAME = "URTs.db";
    private static final String DATABASE_TABLE = "OrganAnatomy";
    public static final String DATABASE_ID = "_id";
    public static final String DATABASE_GROUP_1 = "Larynx_features";
    public static final String DATABASE_CHILD_1 = "Larynx";
    public static final String DATABASE_CHILD_2 = "pictures";    

    private final Context mContext;
    private DatabaseHelper mDatabaseHelper;
    private SQLiteDatabase mDB;

    public Database(Context context) {
        mContext = context;
    }

    public void open() {
        mDatabaseHelper = new DatabaseHelper(mContext, DATABASE_NAME, null, DATABASE_VERSION);
        mDB = mDatabaseHelper.getWritableDatabase();
    }    

    public void close() {
        if (mDatabaseHelper != null) mDatabaseHelper.close();
    }

    public Cursor getDatabase() {
        String whereclause = DATABASE_CHILD_1 + " IS NOT NULL";
        return mDB.query(DATABASE_TABLE, null, whereclause, null, null, null, DATABASE_ID);
    }    

    public Cursor getID(long rowID) {
        return mDB.query(DATABASE_TABLE, null, "_id" + " = "
                + rowID , null, null, null, null);    
    }    

    public class DatabaseHelper extends SQLiteAssetHelper {    
        public DatabaseHelper(Context context, String name, SQLiteDatabase.CursorFactory factory, int version) {
            super(context, name, factory, version);    
        }
    }    
}

Answer 1

With regard to the 50Mb (now 100Mb) APK size you can find out about APK Expansion files at APK Expansion Files, which includes :-

Each time you upload an APK using the Google Play Console, you have the option to add one or two expansion files to the APK. Each file can be up to 2GB and it can be any format you choose, but we recommend you use a compressed file to conserve bandwidth during the download. Conceptually, each expansion file plays a different role:

The main expansion file is the primary expansion file for additional resources required by your application. The patch expansion file is optional and intended for small updates to the main expansion file. While you can use the two expansion files any way you wish, we recommend that the main expansion file deliver the primary assets and should rarely if ever updated; the patch expansion file should be smaller and serve as a “patch carrier,” getting updated with each major release or as necessary.

However, even if your application update requires only a new patch expansion file, you still must upload a new APK with an updated versionCode in the manifest. (The Play Console does not allow you to upload an expansion file to an existing APK.)

With regard to storing images in SQLite, for images that are larger than around 100k, you should store the images as files and store the path to the image in the database.

There is a restriction, not with SQLite (see below), but with the maximum size of an Android Cursor Window, which is 2Mb, that restricts or can have a noticeable detrimental impact when retrieving large blobs. Hence why your 7Mb image could be stored but not retrieved.

For SQLite there is a limit as per :-

Maximum length of a string or BLOB

The maximum number of bytes in a string or BLOB in SQLite is defined by the preprocessor macro SQLITE_MAX_LENGTH. The default value of this macro is 1 billion (1 thousand million or 1,000,000,000). You can raise or lower this value at compile-time using a command-line option like this:

-DSQLITE_MAX_LENGTH=123456789 The current implementation will only support a string or BLOB length up to 231-1 or 2147483647. And some built-in functions such as hex() might fail well before that point. In security-sensitive applications it is best not to try to increase the maximum string and blob length. In fact, you might do well to lower the maximum string and blob length to something more in the range of a few million if that is possible.

During part of SQLite's INSERT and SELECT processing, the complete content of each row in the database is encoded as a single BLOB. So the SQLITE_MAX_LENGTH parameter also determines the maximum number of bytes in a row.

The maximum string or BLOB length can be lowered at run-time using the sqlite3_limit(db,SQLITE_LIMIT_LENGTH,size) interface.

Limits In SQLite

Example App

This App Stores Smaller Images in DB but stores the path for larger images (size to determine which based upon public static final int MAX_FILE_SIZE = 100 * 1024;)

The DatabaseHelper DBHelper.java :-

public class DBHelper extends SQLiteOpenHelper {
    public static final String DBNAME = "images.db";
    public static final int DBVERSION = 1;

    // The maximum size of an image that should be stored 100K
    public static final int MAX_FILE_SIZE = 100 * 1024;

    public static final String TB_IMAGE = "image";
    public static final String COL_IMAGE_ID = BaseColumns._ID;
    public static final String COL_IMAGE_PATH = "image_path";
    public static final String COL_IMAGE_NAME = "image_name";
    public static final String COl_IMAGE_DESCRIPTION = "image_description";
    public static final String COL_IMAGE_SIZE = "image_size";
    public static final String COL_IMAGE_IMAGE = "image";

    SQLiteDatabase mDB;

    /**
     * Construct DBHelper, note that it will open the database and
     * thus create it if it doesn't exist
     * @param context   a context from the invoking activity
     */
    public DBHelper(Context context) {
        super(context, DBNAME, null, DBVERSION);
        mDB = this.getWritableDatabase();
    }

    /**
     * Create the table(s)
     * @param db
     */
    @Override
    public void onCreate(SQLiteDatabase db) {
        String crtsql = "CREATE TABLE IF NOT EXISTS " + TB_IMAGE +
                "(" +
                COL_IMAGE_ID + " INTEGER PRIMARY KEY, " +
                COL_IMAGE_PATH + " TEXT UNIQUE, " +
                COL_IMAGE_NAME + " TEXT, " +
                COl_IMAGE_DESCRIPTION + " TEXT, " +
                COL_IMAGE_SIZE + " INTEGER, " +
                COL_IMAGE_IMAGE + " BLOB DEFAULT x'00'" +
                ")";
        db.execSQL(crtsql);
    }

    @Override
    public void onUpgrade(SQLiteDatabase sqLiteDatabase, int i, int i1) {

    }

    /**
     * Return a Cursor with all the rows from the image table
     * @return  The Cursor
     */
    public Cursor getImageList() {
        return mDB.query(TB_IMAGE,null,null,null,null,null,null);
    }


    /**
     * Store an image row in the image table, noting that is the image
     *  size is small than the max size that the image will be stored as a blob
     *  otherwise a blob of 1 byte is stored due to the default value.
     * @param path          the path to the image
     * @param description   a description for the image
     * @return              the id (rowid) of the row
     */
    public long addImageFromPath(String path, String description) {

        ContentValues cv = new ContentValues();
        File f = new File(path);
        InputStream is;

        // If the file doesn't exist don't store a row
        if (!f.exists()) {
            return -1;
        }

        // Always store the name, description, path and size
        cv.put(COL_IMAGE_NAME,f.getName());
        cv.put(COl_IMAGE_DESCRIPTION,description);
        cv.put(COL_IMAGE_SIZE,f.length());
        cv.put(COL_IMAGE_PATH,f.getAbsolutePath());

        // If the size is less than the max then get the filestream
        // and convert to a byte[].
        // Note if larger then the max file size the default x'00' blob
        // will be applied
        if (f.length() < MAX_FILE_SIZE) {
            byte[] buffer = new byte[(int) f.length()];
            try {
                is = new FileInputStream(f);
                is.read(buffer);
            } catch (IOException e) {
                e.printStackTrace();
                return -1;
            }
            cv.put(COL_IMAGE_IMAGE,buffer);
        }
        // Do the insert
        return mDB.insert(TB_IMAGE,null,cv);
    }

    /**
     * get the image as a bitmap from the DB if stored, otherwise get it from
     * the file, according to the id.
     * @param id    the id of the row in the image table
     * @return      the bitmap to be returned (note may be empty bitmap)
     */
    public Bitmap getImage(long id) {
        byte[] ba = new byte[0];

        // If the image is stored in the DB then extract and return the bitmap
        if (isStoredAsImage(id)) {
            return getImageAsBitMap(id);
        }
        // If not then get the respective row from the DB
        Cursor csr = mDB.query(
                TB_IMAGE,
                null,
                COL_IMAGE_ID+"=?",
                new String[]{String.valueOf(id)},
                null,
                null,
                null
        );

        // Prepare to convert the path to a file
        String path = ""; //<<<< default to  empty path
        File f = new File(path); //<<< default to empty file
        // If a valid row was found get the path and File from the row
        if (csr.moveToFirst()) {
            path = csr.getString(csr.getColumnIndex(COL_IMAGE_PATH));
            f = new File(path);
        }
        // done with the cursor so close it
        csr.close();

        // If the file exists then return the Bitmap
        if (f.exists()) {
            return BitmapFactory.decodeFile(f.getAbsolutePath());
        }
        // return an empty bitmap
        return BitmapFactory.decodeByteArray(ba,0,ba.length);
    }

    /**
     * Check to see if an image is stored in the DB,
     *  note assumes anything less than 8 bytes isn't an image
     * @param id    the id of the row in the image table
     * @return      true if like an image is stored, otherwise false
     */
    private boolean isStoredAsImage(long id) {
        boolean rv = true;
        byte[] ba = new byte[0];

        // Get the respective row from the image table
        Cursor csr = mDB.query(
                TB_IMAGE,
                null,
                COL_IMAGE_ID+"=?",
                new String[]{String.valueOf(id)},
                null,
                null,
                null
        );

        // If a row was found get the blob into byte array ba
        // if not then ready to return false
        if (csr.moveToFirst()) {
            ba = csr.getBlob(csr.getColumnIndex(COL_IMAGE_IMAGE));
        } else {
            rv = false;
        }
        // If the byte array ba is less then 8 bytes then ready to return false
        if (ba == null || ba.length < 8) {
            rv =  false;
        }
        // done with the Cursor so close it
        csr.close();
        // return the result
        return rv;
    }

    /**
     * get the image (assumes isStoredAsImage is used prior to invocation)
     * @param id    the id of the respective row
     * @return      the bitmap (may be 0 length)
     */
    private Bitmap getImageAsBitMap(long id) {
        byte[] ba = new byte[0];
        Bitmap bmp;
        Cursor csr =mDB.query(
                TB_IMAGE,
                null,
                COL_IMAGE_ID+"=?",
                new String[]{String.valueOf(id)},
                null,
                null,
                null
        );
        if (csr.moveToFirst()) {
            ba = csr.getBlob(csr.getColumnIndex(COL_IMAGE_IMAGE));
        }
        csr.close();
        return BitmapFactory.decodeByteArray(ba,0,ba.length);
    }
}

The MainActivity MainActivity.java

public class MainActivity extends AppCompatActivity {

    public static String IMAGE_STORE_PATH;
    public static final String IMAGES_DIRECTORY = "images";
    private static File images_file;

    ArrayAdapter<String> mAdapter;
    ListView mListView01, mListView02;
    ArrayList<String> mImages;
    CursorAdapter mCsrAdapter;
    Cursor mCsr;
    ImageView mImageView;
    DBHelper mDBHlpr;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        // get the View/Viewgroup IDs
        mListView01 = this.findViewById(R.id.listview001); // File List
        mListView02 = this.findViewById(R.id.listview002); // DB List
        mImageView = this.findViewById(R.id.imageview001); // Image display

        // get an instance of the DBHelper
        mDBHlpr = new DBHelper(this);

        // Copy images from raw folder to data/data/<package>/Files/images
        // A