Building a camera from scratch: MT9D111, OV2640 complete working software, hardware

Mars Decent Imager (MARDI) used the KAI-2020
(Formerly from Truesense Imaging now On Semi) 2MP (1600*1200) image sensor

[Complete hardware guide to building a camera [HERE]. This post touches on software development]

Image sensors are probably the most versatile sensors out there. Whether it's on a robotic project, on a car as driving aid, on a drone for remote sensing or color tracking, multispectral imaging cameras on multi-million remote sensing satellites or even as extreme as taking an image of Mars Curiosity Rover descending on Mars' red surface (as shown above), the ubiquity of these sensors cannot be denied. 

Currently, Seoul National University's Cube Satellite, SNUSAT-1, has a secondary imaging payload to take images of the earth. The payload is not mission critical and the whole point of having the camera is for public attention. It's hard imagine a 10cm*10cm*10cm in space, but it's way easier to look at a space image and understand that the satellite is out there, tumbling and untumbling in low earth orbit. 

Let's be clear though. Although image sensors are fun to have in the projects, they are a pain to work with. Unless you have linux running on your processor with OpenCV library to capture images, making a camera work can be a monumental task. For a mechanically trained engineer like me who didn't even know what a ground was in an electrical circuit, having been tasked to build a proper functioning camera was like climbing everest without an oxygen tank. 

The way I began was by stripping down a GoPro and trying to integrate the camera on to the satellite. That was the topic of my undergraduate thesis [HERE] and [HERE] but it soon became increasingly clear that I had to build a camera from scratch. The people who had been funding us wanted to have a completely Korean based camera, which meant using a Korean sensor and having it's own dedicated processing unit. 

CP8108 USB module on mounted on a 3D printed structure

PixelPlus POA030R [HERE] and ClairPixel's CP8108 [Above] were two sensors I tried working with. POA030R had a working hardware however, I was not able to make the software work. The communication (I2C also called the Two Wire Interface (TWI, SCCB in some cases) worked well but due to lack of proper documentation, refused to give any data out. With the CP08108 USB module I tried to work with, enumeration was successful but besides that, no image was obtained. 

In other words, just pain after pain after pain with no tangible results. The good thing though was the funding stopped and I was allowed to use whatever sensor was there out in the market and not just a Korean one.

The Software Breakthrough:

OV2640 Image Sensor

Hardware

After some disastrous, time wasting failures, I looked at the market and decided to work on cameras that people had already worked with. That meant that the codes were out there. Understanding color space and how imagery data works plus SCCB on Omnivision's OV7670 [HERE] was a good starting point, however, things became only slightly clear when I started working on OV2640.

ComPortMaster allows you to receive hex values and then save them as a .dat file

Complete OV2640 Camera Working Code [Here]

Properties
Processor: STM32F429ZIT6 (Discovery board), No external memory used
Communication: SCCB with Camera, UART with Computer, DCMI with Camera
Compiler/Debugger: IAR Workbench
Library: STD Peripheral Library, CMSIS

Image Format Output: JPEG
Image Output Size: 160*120 (QQVGA)

Serial Terminal: Connected through UART->RS232->USB, Terminal used: shown as above

Results:

Opened on Paint starring my nexus 5

On Semi's 2M MT9D111 Image Sensor

I have also been tasked with working on SNUSAT-2's Wide Angle Camera which uses the same processor and has MT9P031 5MP image sensor from On Semiconductor (On Semi bought Aptina). However, a MT9P031 cheap module was not available and had to be bought directly from On Semiconductor which took time. I mean, the sensors have still not arrived yet even after 6 solid months.

Which brought me to the MT9D111. The 2M camera module is omnipresent in the market, has a lot of source code out there and even has a dedicated registry generator (anyone who has worked with cameras will understand how important this really is). 

For the past four months, I had been trying, with the subsystem designs being done in parallel, to make the MT9D111 sensor work. However, something was just not right. Data was coming out, the data clearly showed that the camera was responding to dark and bright images, but no tangible "Aha that's a picture" type result was out. 

After giving up on the semi-JPEG output, I decided on getting RAW RGB565 images out instead and after fiddling with the register settings for about a month or so, I finally got the camera to work. 

The difficulty lied in:
1) Load and load of register settings to work with

2) Register values generated from the SOC2010, which is basically the MT9D111 register generator, were weird because the images that were being produced were badly misplaced in terms of color

3) As I said again, there's ton of registers because it's a system on chip(IFP, MCU, SRAM) plus pages (0,1,2) and then they have context (A[preview] and B[capture])

Once I went through the datasheet at least a thousand times and then dreaming about it for atleast another thousand times, I finally got the idea that I should keep things simple and lo and behold the camera started spitting out good, viewable images. 

First image captured

Complete MT9D111 Camera Software [HERE]
All available source code for MT9D111 scavenged out from the internet [HERE] 

Properties

Processor: STM32F429ZIT6 (Discovery board), No external memory used
Communication: I2C with Camera, UART with Computer, DCMI with Camera
Compiler/Debugger: IAR Workbench
Library: HAL Library, CMSIS, TM Library [HERE], FatFs

Image Format Output: RGB565
Image Output Size: 160*120 (QQVGA)

MT9D111 Registery Generator: Download [HERE]

Serial Terminal: Connected through UART->RS232->USB, Terminal used: ComPortMaster
RAW Image Viewer: www.rawpixels.net[HERE]

Software Layer. Note, RTOS has not been implemented

I will have a detailed post on how to approach barebones software designing later. For now, have a look at the code, compile and enjoy.