An Overview of Activation of Android Camera

When a Android device boots up, the kernel will be loaded first. And then, CameraProvider will load CameraHAL, after which CameraService is able to communicate with CameraProvider to make use of CameraHAL.

When a APP trys taking control of the camera, Camera2 API will try getting CameraService, which will activate CameraProvider with the HIDL service. Finally, CameraHAL will call the driver.

Activation of HwServiceManager

HwServiceManager is the management center of HIDL service, which is in control of all HIDL service in the system and started by the init process. And the CameraProvider is registered to it.

The script that start HwServiceManager up and used by the init process is hwservicemanager.rc.

# system/hwservicemanager/hwservicemanager.rc

service hwservicemanager /system/bin/hwservicemanager
    user system
    disabled
    group system readproc
    critical
    onrestart setprop hwservicemanager.ready false
    onrestart class_restart main
    onrestart class_restart hal
    onrestart class_restart early_hal
    writepid /dev/cpuset/system-background/tasks
    class animation
    shutdown critical

And the source code of HwServiceManger is system/hwservicemanager/service.cpp. Let’s take a look at the main function.

/* system/hwservicemanager/service.cpp
 * main()
 */
int main() {
    // If hwservicemanager crashes, the system may be unstable and hard to debug. This is both why
    // we log this and why we care about this at all.
    setProcessHidlReturnRestriction(HidlReturnRestriction::ERROR_IF_UNCHECKED);

    sp<ServiceManager> manager = new ServiceManager();
    setRequestingSid(manager, true);

    if (!manager->add(serviceName, manager).withDefault(false)) {
        ALOGE("Failed to register hwservicemanager with itself.");
    }

    sp<TokenManager> tokenManager = new TokenManager();
    if (!manager->add(serviceName, tokenManager).withDefault(false)) {
        ALOGE("Failed to register ITokenManager with hwservicemanager.");
    }

    // Tell IPCThreadState we're the service manager
    sp<IBinder> binder = toBinder<IServiceManager>(manager);
    sp<BHwBinder> service = static_cast<BHwBinder*>(binder.get()); // local binder object
    IPCThreadState::self()->setTheContextObject(service);
    // Then tell the kernel
    ProcessState::self()->becomeContextManager(nullptr, nullptr);

    int rc = property_set("hwservicemanager.ready", "true");
    if (rc) {
        ALOGE("Failed to set \"hwservicemanager.ready\" (error %d). "\
              "HAL services will not start!\n", rc);
    }

    sp<Looper> looper = Looper::prepare(0 /* opts */);

    (void)HwBinderCallback::setupTo(looper);
    (void)ClientCallbackCallback::setupTo(looper, manager);

    ALOGI("hwservicemanager is ready now.");

    while (true) {
        looper->pollAll(-1 /* timeoutMillis */);
    }

    return 0;
}

Activation of CameraProvider

After HwServiceManager boots up, the booting of CameraProvider follows. Just like HwServiceManger, CameraProvider has its own booting script.

# hardware/interfaces/camera/provider/2.4/default/android.hardware.camera.provider@2.4-service.rc

service vendor.camera-provider-2-4 /vendor/bin/hw/android.hardware.camera.provider@2.4-service
    interface android.hardware.camera.provider@2.4::ICameraProvider legacy/0
    class hal
    user cameraserver
    group audio camera input drmrpc
    ioprio rt 4
    capabilities SYS_NICE
    task_profiles CameraServiceCapacity HighPerformance

After compiling, there will be a script under /vendor/etc/init/, which is the folder of a Android device. And that script will start up CameraProvider.

AndroidDevice:/vendor/etc/init # ls | grep "camera"
android.hardware.camera.provider@2.4-service_64.rc

AndroidDevice:/vendor/etc/init # cat android.hardware.camera.provider@2.4-service_64.rc
#! /bin/sh
#
#Copyright (c) 2019-2021 Qualcomm Technologies, Inc.
#All Rights Reserved.
#Confidential and Proprietary - Qualcomm Technologies, Inc.
#

on property:sys.boot_completed=1
    # Add a cpuset for the camera daemon
    # We want all cores for camera
    mkdir /dev/cpuset/camera-daemon
    write /dev/cpuset/camera-daemon/cpus 0-7
    write /dev/cpuset/camera-daemon/mems 0
    chown cameraserver cameraserver /dev/cpuset/camera-daemon
    chown cameraserver cameraserver /dev/cpuset/camera-daemon/tasks
    chmod 0660 /dev/cpuset/camera-daemon/tasks

service vendor.camera-provider-2-4 /vendor/bin/hw/android.hardware.camera.provider@2.4-service_64
    override
    interface android.hardware.camera.provider@2.4::ICameraProvider legacy/0
    interface vendor.qti.hardware.camera.postproc@1.0::IPostProcService camerapostprocservice
    class hal
    user cameraserver
    group audio camera input drmrpc oem_2907
    ioprio rt 4
    capabilities SYS_NICE
    writepid /dev/cpuset/camera-daemon/tasks /dev/stune/foreground/tasks

The source code about how CameraProvider starts follows.

/* hardware/interfaces/camera/provider/2.4/default/service.cpp
 * main()
 */

int main()
{
    ALOGI("CameraProvider@2.4 legacy service is starting.");
    // The camera HAL may communicate to other vendor components via
    // /dev/vndbinder
    android::ProcessState::initWithDriver("/dev/vndbinder");
    status_t status;
    if (kLazyService) {
        status = defaultLazyPassthroughServiceImplementation<ICameraProvider>("legacy/0",
                                                                              /*maxThreads*/ 6);
    } else {
        status = defaultPassthroughServiceImplementation<ICameraProvider>("legacy/0",
                                                                          /*maxThreads*/ 6);
    }
    return status;
}

As we can see above, the service will initialize the dirver and then start up CameraProvider.
The following sequence diagram found on the Internet will help understand how the service activate CameraProvider in a Passthrough way, which will register the HIDL service to HwServiceManager I mentioned earlier.

Especially, I’d like to record the source code of CameraProvider_2_4 for a long-time reference.

/* CameraProvider_2_4.h */
#ifndef ANDROID_HARDWARE_CAMERA_PROVIDER_V2_4_CAMERAPROVIDER_H
#define ANDROID_HARDWARE_CAMERA_PROVIDER_V2_4_CAMERAPROVIDER_H

#include <android/hardware/camera/provider/2.4/ICameraProvider.h>
#include <hidl/Status.h>
#include <hidl/MQDescriptor.h>

namespace android {
namespace hardware {
namespace camera {
namespace provider {
namespace V2_4 {
namespace implementation {

using ::android::hardware::camera::common::V1_0::Status;
using ::android::hardware::camera::provider::V2_4::ICameraProvider;
using ::android::hardware::camera::provider::V2_4::ICameraProviderCallback;
using ::android::hardware::Return;
using ::android::hardware::hidl_string;
using ::android::sp;

template<typename IMPL>
struct CameraProvider : public ICameraProvider {
    CameraProvider() : impl() {}
    ~CameraProvider() {}

    // Caller must use this method to check if CameraProvider ctor failed
    bool isInitFailed() { return impl.isInitFailed(); }

    // Methods from ::android::hardware::camera::provider::V2_4::ICameraProvider follow.
    Return<Status> setCallback(const sp<ICameraProviderCallback>& callback) override {
        return impl.setCallback(callback);
    }

    Return<void> getVendorTags(getVendorTags_cb _hidl_cb) override {
        return impl.getVendorTags(_hidl_cb);
    }

    Return<void> getCameraIdList(getCameraIdList_cb _hidl_cb) override {
        return impl.getCameraIdList(_hidl_cb);
    }

    Return<void> isSetTorchModeSupported(isSetTorchModeSupported_cb _hidl_cb) override {
        return impl.isSetTorchModeSupported(_hidl_cb);
    }

    Return<void> getCameraDeviceInterface_V1_x(
            const hidl_string& cameraDeviceName,
            getCameraDeviceInterface_V1_x_cb _hidl_cb) override {
        return impl.getCameraDeviceInterface_V1_x(cameraDeviceName, _hidl_cb);
    }

    Return<void> getCameraDeviceInterface_V3_x(
            const hidl_string& cameraDeviceName,
            getCameraDeviceInterface_V3_x_cb _hidl_cb) override {
        return impl.getCameraDeviceInterface_V3_x(cameraDeviceName, _hidl_cb);
    }

private:
    IMPL impl;
};

////////////////////////////////////////////////////////////////////////////////

/* CameraProvider_2_4.cpp */

#include "CameraProvider_2_4.h"
#include "LegacyCameraProviderImpl_2_4.h"
#include "ExternalCameraProviderImpl_2_4.h"

const char *kLegacyProviderName = "legacy/0";
const char *kExternalProviderName = "external/0";

namespace android {
namespace hardware {
namespace camera {
namespace provider {
namespace V2_4 {
namespace implementation {

using android::hardware::camera::provider::V2_4::ICameraProvider;

extern "C" ICameraProvider* HIDL_FETCH_ICameraProvider(const char* name);

template<typename IMPL>
CameraProvider<IMPL>* getProviderImpl() {
    CameraProvider<IMPL> *provider = new CameraProvider<IMPL>();
    if (provider == nullptr) {
        ALOGE("%s: cannot allocate camera provider!", __FUNCTION__);
        return nullptr;
    }
    if (provider->isInitFailed()) {
        ALOGE("%s: camera provider init failed!", __FUNCTION__);
        delete provider;
        return nullptr;
    }
    return provider;
}

ICameraProvider* HIDL_FETCH_ICameraProvider(const char* name) {
    using namespace android::hardware::camera::provider::V2_4::implementation;
    ICameraProvider* provider = nullptr;
    if (strcmp(name, kLegacyProviderName) == 0) {
        provider = getProviderImpl<LegacyCameraProviderImpl_2_4>();
    } else if (strcmp(name, kExternalProviderName) == 0) {
        provider = getProviderImpl<ExternalCameraProviderImpl_2_4>();
        provider = getProviderImpl<ExternalCameraProviderImpl_2_4>();
    } else {
        ALOGE("%s: unknown instance name: %s", __FUNCTION__, name);
    }

    return provider;
}

}  // namespace implementation
}  // namespace V2_4
}  // namespace provider

And the real implementaion that calls CameraHAL is in the following source files:

i@host:~/Code/android$ find hardware/interfaces/camera/provider/2.4/default/ -name "*CameraProviderImpl*"
hardware/interfaces/camera/provider/2.4/default/ExternalCameraProviderImpl_2_4.cpp
hardware/interfaces/camera/provider/2.4/default/ExternalCameraProviderImpl_2_4.h
hardware/interfaces/camera/provider/2.4/default/LegacyCameraProviderImpl_2_4.cpp
hardware/interfaces/camera/provider/2.4/default/LegacyCameraProviderImpl_2_4.h

Let’s take a look at the constructor and initializer.

//hardware/interfaces/camera/provider/2.4/default/LegacyCameraProviderImpl_2_4.h

struct LegacyCameraProviderImpl_2_4 : public camera_module_callbacks_t 
{ /* ... */ }

////////////////////////////////////////////////////////////////////////////////

//hardware/interfaces/camera/provider/2.4/default/LegacyCameraProviderImpl_2_4.cpp
LegacyCameraProviderImpl_2_4::LegacyCameraProviderImpl_2_4() :
        camera_module_callbacks_t({sCameraDeviceStatusChange,
                                   sTorchModeStatusChange}) {
    mInitFailed = initialize();
}

bool LegacyCameraProviderImpl_2_4::initialize() {
    camera_module_t *rawModule;
    int err = hw_get_module(CAMERA_HARDWARE_MODULE_ID,
            (const hw_module_t **)&rawModule);
    if (err < 0) {
        ALOGE("Could not load camera HAL module: %d (%s)", err, strerror(-err));
        return true;
    }

    mModule = new CameraModule(rawModule);
    err = mModule->init();
    if (err != OK) {
        ALOGE("Could not initialize camera HAL module: %d (%s)", err, strerror(-err));
        mModule.clear();
        return true;
    }
    ALOGI("Loaded \"%s\" camera module", mModule->getModuleName());

    // Setup vendor tags here so HAL can setup vendor keys in camera characteristics
    VendorTagDescriptor::clearGlobalVendorTagDescriptor();
    if (!setUpVendorTags()) {
        ALOGE("%s: Vendor tag setup failed, will not be available.", __FUNCTION__);
    }

    // Setup callback now because we are going to try openLegacy next
    err = mModule->setCallbacks(this);
    if (err != OK) {
        ALOGE("Could not set camera module callback: %d (%s)", err, strerror(-err));
        mModule.clear();
        return true;
    }

    mPreferredHal3MinorVersion =
        property_get_int32("ro.vendor.camera.wrapper.hal3TrebleMinorVersion", 3);
    ALOGV("Preferred HAL 3 minor version is %d", mPreferredHal3MinorVersion);
    switch(mPreferredHal3MinorVersion) {
        case 2:
        case 3:
            // OK
            break;
        default:
            ALOGW("Unknown minor camera device HAL version %d in property "
                    "'camera.wrapper.hal3TrebleMinorVersion', defaulting to 3",
                    mPreferredHal3MinorVersion);
            mPreferredHal3MinorVersion = 3;
    }

    mNumberOfLegacyCameras = mModule->getNumberOfCameras();
    for (int i = 0; i < mNumberOfLegacyCameras; i++) {
        uint32_t device_version;
        auto rc = mModule->getCameraDeviceVersion(i, &device_version);
        if (rc != NO_ERROR) {
            ALOGE("%s: Camera device version query failed!", __func__);
            mModule.clear();
            return true;
        }

        if (checkCameraVersion(i, device_version) != OK) {
            ALOGE("%s: Camera version check failed!", __func__);
            mModule.clear();
            return true;
        }

        char cameraId[kMaxCameraIdLen];
        snprintf(cameraId, sizeof(cameraId), "%d", i);
        std::string cameraIdStr(cameraId);
        mCameraStatusMap[cameraIdStr] = CAMERA_DEVICE_STATUS_PRESENT;

        addDeviceNames(i);
    }

    return false; // mInitFailed
}

And here follows related structures.

//hardware/libhardware/include/hardware/camera_common.h

typedef struct camera_module {
    /**
     * Common methods of the camera module.  This *must* be the first member of
     * camera_module as users of this structure will cast a hw_module_t to
     * camera_module pointer in contexts where it's known the hw_module_t
     * references a camera_module.
     */
    hw_module_t common;

    /**
     * get_number_of_cameras:
     *
     * Returns the number of camera devices accessible through the camera
     * module.  The camera devices are numbered 0 through N-1, where N is the
     * value returned by this call. The name of the camera device for open() is
     * simply the number converted to a string. That is, "0" for camera ID 0,
     * "1" for camera ID 1.
     */
    int (*get_number_of_cameras)(void);

    /**
     * get_camera_info:
     *
     * Return the static camera information for a given camera device. This
     * information may not change for a camera device.
     */
    int (*get_camera_info)(int camera_id, struct camera_info *info);

    /**
     * set_callbacks:
     *
     * Provide callback function pointers to the HAL module to inform framework
     * of asynchronous camera module events. The framework will call this
     * function once after initial camera HAL module load, after the
     * get_number_of_cameras() method is called for the first time, and before
     * any other calls to the module.
     */
    int (*set_callbacks)(const camera_module_callbacks_t *callbacks);

    /**
     * get_vendor_tag_ops:
     *
     * Get methods to query for vendor extension metadata tag information. The
     * HAL should fill in all the vendor tag operation methods, or leave ops
     * unchanged if no vendor tags are defined.
     */
    void (*get_vendor_tag_ops)(vendor_tag_ops_t* ops);

    /**
     * open_legacy:
     *
     * Open a specific legacy camera HAL device if multiple device HAL API
     * versions are supported by this camera HAL module. For example, if the
     * camera module supports both CAMERA_DEVICE_API_VERSION_1_0 and
     * CAMERA_DEVICE_API_VERSION_3_2 device API for the same camera id,
     * framework can call this function to open the camera device as
     * CAMERA_DEVICE_API_VERSION_1_0 device.
     */
    int (*open_legacy)(const struct hw_module_t* module, const char* id,
            uint32_t halVersion, struct hw_device_t** device);

    /**
     * set_torch_mode:
     *
     * Turn on or off the torch mode of the flash unit associated with a given
     * camera ID. If the operation is successful, HAL must notify the framework
     * torch state by invoking
     * camera_module_callbacks.torch_mode_status_change() with the new state.
     */
    int (*set_torch_mode)(const char* camera_id, bool enabled);

    /**
     * init:
     *
     * This method is called by the camera service before any other methods
     * are invoked, right after the camera HAL library has been successfully
     * loaded. It may be left as NULL by the HAL module, if no initialization
     * in needed.
     *
     * It can be used by HAL implementations to perform initialization and
     * other one-time operations.
     */
    int (*init)();

    /**
     * get_physical_camera_info:
     *
     * Return the static metadata for a physical camera as a part of a logical
     * camera device. This function is only called for those physical camera
     * ID(s) that are not exposed independently. In other words, camera_id will
     * be greater or equal to the return value of get_number_of_cameras().
     */
    int (*get_physical_camera_info)(int physical_camera_id,
            camera_metadata_t **static_metadata);

    /**
     * is_stream_combination_supported:
     *
     * Check for device support of specific camera stream combination.
     */
    int (*is_stream_combination_supported)(int camera_id,
            const camera_stream_combination_t *streams);

    /**
     * notify_device_state_change:
     *
     * Notify the camera module that the state of the overall device has
     * changed in some way that the HAL may want to know about.
     */
    void (*notify_device_state_change)(uint64_t deviceState);

    /**
     * get_camera_device_version:
     *
     * Return the device version for a given camera device. This value may not change for a camera
     * device. The version returned here must be the same as the one from get_camera_info.
     */
    int (*get_camera_device_version)(int camera_id, uint32_t *version);

    /* reserved for future use */
    void* reserved[1];
} camera_module_t;

typedef struct camera_module_callbacks {

    /**
     * camera_device_status_change:
     *
     * Callback to the framework to indicate that the state of a specific camera
     * device has changed. At module load time, the framework will assume all
     * camera devices are in the CAMERA_DEVICE_STATUS_PRESENT state. The HAL
     * must call this method to inform the framework of any initially
     * NOT_PRESENT devices.
     *
     * This callback is added for CAMERA_MODULE_API_VERSION_2_1.
     *
     * camera_module_callbacks: The instance of camera_module_callbacks_t passed
     *   to the module with set_callbacks.
     *
     * camera_id: The ID of the camera device that has a new status.
     *
     * new_status: The new status code, one of the camera_device_status_t enums,
     *   or a platform-specific status.
     *
     */
    void (*camera_device_status_change)(const struct camera_module_callbacks*,
            int camera_id,
            int new_status);

    /**
     * torch_mode_status_change:
     *
     * Callback to the framework to indicate that the state of the torch mode
     * of the flash unit associated with a specific camera device has changed.
     * At module load time, the framework will assume the torch modes are in
     * the TORCH_MODE_STATUS_AVAILABLE_OFF state if android.flash.info.available
     * is reported as true via get_camera_info() call.
     *
     * This callback is added for CAMERA_MODULE_API_VERSION_2_4.
     *
     * camera_module_callbacks: The instance of camera_module_callbacks_t
     *   passed to the module with set_callbacks.
     *
     * camera_id: The ID of camera device whose flash unit has a new torch mode
     *   status.
     *
     * new_status: The new status code, one of the torch_mode_status_t enums.
     */
    void (*torch_mode_status_change)(const struct camera_module_callbacks*,
            const char* camera_id,
            int new_status);

} camera_module_callbacks_t;

As a conclusion, CameraHAL is accessible to CameraProvider, whcih is registered to HwServiceManager after activation. Therefore, when other service needs the CameraHAL, it just needs to communicate with CameraProvider.

Activation of CameraHAL

As we can see in LegacyCameraProviderImpl_2_4::initialize(), CameraHAL is initialized with class CameraModule, which is a wrapper class for HAL camera module.

// hardware/interfaces/camera/common/1.0/default/include/CameraModule.h
class CameraModule : public RefBase {
public:
    explicit CameraModule(camera_module_t *module);
    virtual ~CameraModule();

    // Must be called after construction
    // Returns OK on success, NO_INIT on failure
    int init();

    int getCameraDeviceVersion(int cameraId, uint32_t* version);
    int getCameraInfo(int cameraId, struct camera_info *info);
    int getDeviceVersion(int cameraId);
    int getNumberOfCameras(void);
    int open(const char* id, struct hw_device_t** device);
    bool isOpenLegacyDefined() const;
    int openLegacy(const char* id, uint32_t halVersion, struct hw_device_t** device);
    int setCallbacks(const camera_module_callbacks_t *callbacks);
    bool isVendorTagDefined() const;
    void getVendorTagOps(vendor_tag_ops_t* ops);
    bool isSetTorchModeSupported() const;
    int setTorchMode(const char* camera_id, bool enable);
    uint16_t getModuleApiVersion() const;
    const char* getModuleName() const;
    uint16_t getHalApiVersion() const;
    const char* getModuleAuthor() const;
    // Only used by CameraModuleFixture native test. Do NOT use elsewhere.
    void *getDso();
    // Only used by CameraProvider
    void removeCamera(int cameraId);
    int getPhysicalCameraInfo(int physicalCameraId, camera_metadata_t **physicalInfo);
    int isStreamCombinationSupported(int cameraId, camera_stream_combination_t *streams);
    void notifyDeviceStateChange(uint64_t deviceState);

    static bool isLogicalMultiCamera(
            const common::V1_0::helper::CameraMetadata& metadata,
            std::unordered_set<std::string>* physicalCameraIds);

private:
    // Derive camera characteristics keys defined after HAL device version
    static void deriveCameraCharacteristicsKeys(uint32_t deviceVersion, CameraMetadata &chars);
    // Helper function to append available[request|result|chars]Keys
    static void appendAvailableKeys(CameraMetadata &chars,
            int32_t keyTag, const Vector<int32_t>& appendKeys);
    status_t filterOpenErrorCode(status_t err);
    camera_module_t *mModule;
    int mNumberOfCameras;
    KeyedVector<int, camera_info> mCameraInfoMap;
    KeyedVector<int, int> mDeviceVersionMap;
    KeyedVector<int, camera_metadata_t*> mPhysicalCameraInfoMap;
    Mutex mCameraInfoLock;
};

} // namespace helper
} // namespace V1_0
} // namespace common
} // namespace camera
} // namespace hardware
} // namespace android

#endif

////////////////////////////////////////////////////////////////////////////////

// hardware/interfaces/camera/common/1.0/default/CameraModule.cpp

CameraModule::CameraModule(camera_module_t *module) : mNumberOfCameras(0) {
    if (module == NULL) {
        ALOGE("%s: camera hardware module must not be null", __FUNCTION__);
        assert(0);
    }
    mModule = module;
}

int CameraModule::init() {
    ATRACE_CALL();
    int res = OK;
    if (getModuleApiVersion() >= CAMERA_MODULE_API_VERSION_2_4 &&
            mModule->init != NULL) {
        ATRACE_BEGIN("camera_module->init");
        res = mModule->init();
        ATRACE_END();
    }
    mNumberOfCameras = getNumberOfCameras();
    mCameraInfoMap.setCapacity(mNumberOfCameras);
    return res;
}

int CameraModule::setCallbacks(const camera_module_callbacks_t *callbacks) {
    int res = OK;
    ATRACE_BEGIN("camera_module->set_callbacks");
    if (getModuleApiVersion() >= CAMERA_MODULE_API_VERSION_2_1) {
        res = mModule->set_callbacks(callbacks);
    }
    ATRACE_END();
    return res;
}

int CameraModule::getNumberOfCameras() {
    int numCameras;
    ATRACE_BEGIN("camera_module->get_number_of_cameras");
    numCameras = mModule->get_number_of_cameras();
    ATRACE_END();
    return numCameras;
}

By the way, the RefBase appeared above is a class provides a lightweight reference counting mechanism to manage the lifecycle of objects, which further avoids problems such as memory leaks and dangling pointers.

To understand the CameraModule.cpp, the following source code may help.

i@host:~/Code/android$ find hardware/ -name "camera.h" -o -name "hardware.h"
hardware/libhardware/modules/camera/3_4/camera.h
hardware/libhardware/include/hardware/hardware.h
hardware/libhardware/include/hardware/camera.h

// hardware/libhardware/include/hardware/hardware.h
/**
 * Get the module info associated with a module by id.
 *
 * @return: 0 == success, <0 == error and *module == NULL
 */
int hw_get_module(const char *id, const struct hw_module_t **module);

Here follows a sequence diagram I drew to show this process more visually and clearly.