Building Android ROMs (Omni and other AOSP-based firmware)

Appears that buying a Sony Xperia phone several years ago was a good call – they appeared to support OpenSource Android very well even for devices that are no longer officially supported, and they even provide instructions on how how to unlock those devices, and build your own ROMs based on AOSP – Android Open Source Project. And they also published instructions on how to build Android 10 (Q) which was released just a few weeks ago!

But despite lots of documentation available on Internet, its still very hard to start compiling Android-based ROMs even if you have access to all the source code and have a good vendor like Sony. With this post, I want to share what I learned so far.

In no way I can pretend to be an expert in this field. If you are looking for one, I can recommend these guys:

By the way, everything I am writing about here is all described at https://source.android.com and https://github.com/omnirom/android, and I am just putting pieces from different pages together here.

Setting up your build environment

Android is based on Linux, so usually its built using Linux machines. There is no support to build ROMs at Windows, and some support at MacOS. Here, I will only be talking about Linux since never owned a Mac.

Also, please note that you need to have at least 200+Gb of disk space to successfully build Android. I personally have 2TB since I build 3 different ROMS: Omni, AOSP 9 and AOSP 10

Usually, it does not matter which distribution of Linux you use, but most build systems out there use Ubuntu 14.04 (Trusty) as a standard. I tried many combinations, and it looks like most of those recommendations are outdated. E.g. Android 9 (Pie) requires Java 8 to be built, and Ubuntu 14.04 LTS does not have it, so there’s a hack to get that working. For example, Android Project itself provides a Dockerfile that creates an image based on Ubuntu 14.04 for building Android ROMs. It worked for me for some time, but I ended up just using a virtual machine with the latest Ubuntu LTS.

Instructions below should work with most recent Ubuntu installations.

First, you need to install the required libraries and build tools:

sudo dpkg --add-architecture i386

sudo apt-get update && sudo apt-get install -y git-core gnupg flex bison gperf build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip python openjdk-8-jdk libssl-dev lzip repo

Note that I’ve added an extra library libssl-dev which is not required for AOSP builds, but is needed for OmniROM. And lzip which is needed if you want to add OpenGApps to your custom ROM.

I am also using the latest version of the repo tool that is available in Ubuntu repositories. It may be a bit out of date, but so far I did not have any issues with it. I just don’t like to add binaries to the OS manually anymore.

If you want to use the latest version of repo as everyone else recommends, just run a couple more commands:

sudo wget -P /usr/local/bin https://storage.googleapis.com/git-repo-downloads/repo

sudo chmod a+x /usr/local/bin/repo

The last step is to configure your Git user:

git config --global user.name "First Last"
git config --global user.email your@email

And that’s it! Now we are ready to checkout Android and start compiling!

Checking out Android code

Initializing repository

Getting code in Android consists of two stages: first, you init the repository, and then you synchronize (download) the code. And of course you start with creating a directory for it:

mkdir android
cd android

AOSP

Again, I am only repeating the existing documentation here, it’s all very well explained at AOSP website: https://source.android.com/setup/build/downloading#getting-the-files

repo init -u https://android.googlesource.com/platform/manifest -b <your branch>

In most cases you don’t checkout the latest master branch and instead pick a tag (or branch) corresponding to an Android version release – you can find the list of branches here: https://android.googlesource.com/platform/manifest/+refs

Here’s why: most vendors align their code, binaries, and corresponding instructions with specific Android releases. For example, in instructions for my Sony phone I can only use android-10.0.0_r1 while android-10.0.0_r3 is already available. Furthermore, at the moment of writing this, OpenGApps which I am using, is not yet available for Android 10, so I have to use Android 9 instead, and while the latest Android release was android-9.0.0_r48, I can only use r46 since that’s the latest Android 9 release my vendor provided their source code for.

Of course, probably we can try using different release versions of vendor code and AOSP, but should there be any bugs, I currently don’t have enough skills to fix them.

OmniROM

Omni is an amazing ROM based on AOSP. It has many useful features and fixes while is not bloated with vendor software. I use Omni as stable ROM for my phone between my experiments with AOSP, and the more I learn about building my own version of AOSP, the more I see how great Omni is, and how knowledgeable are people that build it.

To get Omni, the repo init command has to use a different URL:

repo init -u git://github.com/omnirom/android.git -b <branch>

With Omni, using branches is much easier – you just pick Android version. They already took care about everything else. You still get the list of branches from the Omni Github project itself, and it’s definitely worth visiting it since there they also provide very simple instructions on how to build there (I am going to duplicate those instructions below with minor changes):

https://github.com/omnirom/android

Customizing your ROM

Usually you make the necessary customizations right after you initialized your repo – this is where you add references to any components that you are adding to your ROM. For example, if you want to add OpenGApps, you just add an xml file (with any name, for example, `opengapps.xml`) to the .repo/local_manifests/ directory. Below is my version of it. Note this just copied from the OpenGapps AOSP instructions by the link above:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="opengapps" fetch="https://github.com/opengapps/"  />
<remote name="gitlab" fetch="https://gitlab.opengapps.org/opengapps/"  />

<project path="vendor/opengapps/build" name="aosp_build" revision="master" remote="opengapps" />

<project path="vendor/opengapps/sources/all" name="all" clone-depth="1" revision="master" remote="gitlab" />

<!-- arm64 depends on arm -->
<project path="vendor/opengapps/sources/arm" name="arm" clone-depth="1" revision="master" remote="gitlab" />
<project path="vendor/opengapps/sources/arm64" name="arm64" clone-depth="1" revision="master" remote="gitlab" />

<project path="vendor/opengapps/sources/x86" name="x86" clone-depth="1" revision="master" remote="gitlab" />
<project path="vendor/opengapps/sources/x86_64" name="x86_64" clone-depth="1" revision="master" remote="gitlab" />

</manifest>

This way, you can also add and customize APKs and any open source software in your build. Unfortunately, this is where I am still learning, so not much information in this section yet – I will be updating it as soon as I have more progress, and meanwhile you can find more information at the Android website, and via the links I provide in the end of this post.

Note that if you’ve read the OpenGApps AOSP instructions, they also require you to change your device.xml settings, and you don’t have the file after you just initialized the repository. Usually this is done by forking the repository that contains your device.xml and making the necessary changes in your fork. And then you just ensure that your AOSP-based ROM’s manifests point to your fork instead of the original one. You can see lots of examples of this in the OmniROM repository (and that also explains why everytime you look at someone’s custom Android ROM repos, you see dozens of forks of Android-related repositories)

Unfortunately, I am still in process of learning how to do this properly, so I can’t really point to any forks that I’ve made so far – instead, I just sync the repo and do all my customizations just before starting the build. But that’s a wrong way to do that – all my changes are lost every time I update my local repo. In the end of this post, I am sharing some links that can help you understand this process better.

Downloading code

After repository was initialized, its time to download (or synchronize – the same command is used to update) code. This is a very long (hours, sometimes even days) process, and below is an optimized command you can use that will help making it faster. By the way, if you are using a VM or server like I do, it’s good time to use something like screen so that you can disconnect from this session while its running:

repo sync --force-sync --no-clone-bundle --no-tags -j$(nproc --all)

Creating a local mirror

If you are downloading AOSP, you will notice how slow it is. AOSP Downloading page provides instructions on how to create a local mirror. Instructions below are taken from there:

# Create a directory for your mirror
mkdir -p /path/to/my/mirror
cd /path/to/my/mirror

# Mirror AOSP repository
repo init -u https://android.googlesource.com/mirror/manifest --mirror
repo sync

After that, you can init a repo by referring to your local mirror instead of slow android.googlesource.com:

repo init -u /path/to/my/mirror/platform/manifest -b <your branch>

This saves a lot of time in my case, when I have two parallel Android repos, and only spend time syncing mirror once. But note that it takes a lot of space. I had to reserve 2 Tb only for my Android directories:

Building the code

After you synchronized (and customized) the code, you can now start building it. It’s done in two simple steps:

Step 1: Initialize environment

First, you run the envsetup script that initializes your local environment, set paths to proper tools and defines required variables. This also means that you need to run it every time you re-login. And this step is always the same for all types of Androd-based ROMs:

source build/envsetup.sh

After that, you need to set a target for your build – and that’s a matter of preference how to do that. E.g. Sony’s AOSP build recommends you running the lunch command that will automatically fetch all information about available device configurations and offer you to choose one. OmniROM, on contrary, will offer you to run the brunch command that will fetch device configuration based on your input and start the build.

This is unfortunately another item that I am still learning, so I will just put both instructions:

# AOSP
lunch

# OmniROM
brunch <device name>

Step 2: Start the build

After that, you can just run the make command, and the build will be started. Note that it may take a couple of hours, and in the example command below I am adding an argument to use all available CPU cores on the system:

make -j $(nproc --all)

It’s normal to have warnings during the build, and in most cases they can be safely ignored. I so far only once heard about an Android build that has no warnings, but I never saw it with my own eyes. My plan however is to help get rid of these warnings as soon as I gain the required knowledge.

Flashing the new ROM

After the build is completed, your binaries will be available in the out/target/product/<your phone> directory.

If you were building AOSP, you’ll get images corresponding to partitions in your Android phone, e.g:

boot.img
cache.img
recovery.img
system.img
userdata.img

And its normal if there’s more img files than your phone has partitions (e.g. in my case, there are always some ramdisk partitions). You can just ignore those.

All you need now is just to flash the image files. Usually, image name will correspond to a partition it goes to, e.g.:

fastboot flash boot boot.img
fastboot flash cache cache.img
fastboot flash recovery recovery.img
fastboot flash system system.img

# note that the following command deletes all data at your phone
fastboot flash userdata userdata.img 

Flashing is a large topic by itself, and its beyond this blogpost. For example, with OmniROM, you’ll get a zip archive instead of a bunch of image files. That file can be flashed via tools like TWRP – just follow OmniROM instructions.

I personally prefer to use images, and I usually find img artifacts remained from OmniROM build and flash them manually.

Useful links

Here’s some very useful links that can help you start building your own Android-based ROMs:

https://bootlin.com/doc/legacy/android/android-slides.pdf

https://forum.xda-developers.com/general/xda-university/

https://elinux.org/Master-android

https://android.googlesource.com/platform/manifest/+refs

Mini project for my daughter

Been a while since I used my soldering iron last time. Really enjoyed prototyping simple electronic circuits with my kids tonight.

This was 2nd time this year. First time was in June:

Can continue working on my micro-controller projects now under Linux

I thought it’s impossible, and was so surprised when I tried and it just worked under Debian Sid and Virtual Box with Windows 7 Pro.

Atmel Studio was a very easy install in a VM, and it had no problems finding my USB programmers and devices attached to my host Linux machine. It’s very good news given that my new digital oscilloscope with logic analyzer arrives tomorrow :)

One of my old reported vulnerabilities was published: CVE-2014-8733

CVE-2014-8733: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-8733

It was fun to work with Hadoop security in 2014… This vuln was a tricky one because I was responsible for Hadoop managed service platform security, and our clients had SSH access to Hadoop cluster nodes in some cases.

If I remember correctly, fix wasn’t easy – required release of new CDH version which moved configuration parameters between files (world readable access was required for Hadoop client to function). And then, several months later, it reappeared again after another patch.

Shell script – waiting until specified day/time

I participated in a project in which job scheduling logic was completely implemented in shell script (it was a small project initially, and then it has grown in time). One of tasks was to enable scheduler script to wait until a specified time to run a job. Surprisingly, search in Internet did not bring anything that would look simple and elegant enough. There were bunch of quite complex solutions which did not really address my requirements and would be too hard to maintain for support teams in future… I came up with my own solution for this after reading Linux manuals, which I wanted to share.

It appears, ‘date’ utility is very flexible in terms of definition and representation of time. For example, user can supply target date/time in a string, using words like ‘Today 9am” or “Thursday 12pm” etc. In combination with ‘sleep’ command, I was able to achieve the desired functionality with required precision (seconds). I ended up writing this simple bash function:

#-[ wait_until ]------------------------------------------
# Waits until a given day/time
# $1 - target time/date in 'date' command format, e.g. "Today 13:00"
wait_until()
{
echo "Waiting until $1..."

TIME_TARGET=$( date -d "$1" +%s )
TIME_NOW=$( date +%s )

if [ $TIME_NOW -lt $TIME_TARGET ]; then

        TIME_WAIT=$( expr $TIME_TARGET - $TIME_NOW )
        echo "Waiting time: ${TIME_WAIT} seconds"
        sleep ${TIME_WAIT}s

else
        echo "Target time is already in past. No waiting required..."
fi
}

Use is very simple:

wait_until "Today 1pm"
#... some code ...
wait_until "Tomorrow 6am"
#... some code ...

etc…

Hope it helps someone who will be looking for this solution in Internet, just like I did before I had to write it :)

ESXi and Kali weekend

Installed Kali Linux in my virtual lab this weekend – just to make a snapshot of currently available packages and, as usual, steal a couple of ideas for my own pentest Linux VM. Two ideas I will never steal from Kali are Safari Icon for Firefox and use of Gnome 3.

Last weekend’s small DIY project

I really enjoyed this. I know it does not look like much, but I am very happy about this little project: I finally learned how to solder properly and lead-free, and I now understand AVR microcontrollers architecture and specifics. What’s left is to finish power optimization and start periodically add different sequences for LEDs and different reactions to button clicks.

And I also learned about powering small devices and about Joule Thief / step-up regulators, induction, surface-mount and PCB design/ordering (however, these are not used in this project)

Github:
https://github.com/samoylenko/avr_s1

schema

Whoa! Debian Linux works amazingly good at my hardware!

When I said my last goodbye to Windows, I thought it will be Ubuntu to become my new primary home operating system – I had it for a couple of years in past, with very very good experience for both me and my wife, don’t even remember why I switched back to Windows. Probably because I disliked what they did to Gnome, or the switch to Unity, or Il-2 Sturmovik did not work well with Wine, or all three together and lack of time to deal with it – so I thought I’ll pay Microsoft to make all decisions for me… I really don’t remember now. But for me, it was always clear that Ubuntu is the best Linux distribution for desktop hardware because of its superior support for drivers and firmware. I prefer Debian at my virtual desktops and Ubuntu again at servers, both hardware and virtual.

So I removed Windows 10 and installed Ubuntu, and spent a couple of days playing with it – everything worked well, but I was really missing my Debian XFCE customized desktop experience to which I am used to. And I decided to try Debian at my laptop. I was sure I won’t be able to get it work with my wifi card, or cardreader, or something else… I was so surprised that everything worked out of the box! And the install went smooth, over wifi, and everything was just great! The biggest surprise, however, was that Debian appeared to work much better with my Microsoft (hehe) Bluetooth mouse than Ubuntu. Hibernation, suspend, sensors etc work without any problems as well.

It’s been almost a week, and I am very happy with Debian as my primary OS now. I am very confident it will stay, and I am not switching back to Windows or even Ubuntu any time soon – I have an old EEE-PC with Debian installed for around 2 years already (for kids to play) which I was periodically checking and updating, and was amazed of how good and stable it is.

I only had to do two things to make it happen:

  1. I used unofficial Debian installer with non-free firmware support to install it at my laptop.
  2. I had to manually add my touchpad configuration to enable all the fancy multi-touch etc:


Here’s a screenshot of my Debian desktop:

Screenshot - 08232015 - 11:53:48 PM

Farewell, Windows

I’ve got Windows 10 at my laptop around 3 weeks ago and was testing it since. I do like the new design, and the OS itself is definitely better than any previous Windows version, including 7. And all of my life I’ve always been a Windows guy, preferring Linux only in server or virtual environments. But I just can’t leave all the privacy concerns be. I hate the idea someone is constantly watching what I am doing and where I am going. And I can’t bear the fact that Microsoft can scan my hard drive or sneak on me through my webcam anytime they want. It’s gone way too far.

When they announced the free upgrade, I knew it will be something serious and users will have still have to pay, but in some different way. Well, now we know what it is… Sorry, Microsoft, I am not buying this anymore.

Three years ago, when I wanted to refresh my software development skills and learn something new on a professional level, I was choosing between C# and Java, and I really wanted it to be the former given all of my previous experience, but the price of Visual Studio and MSDN subscription was astronomical, and everything was restricted to only one OS and one vendor while for Java everything was free and cross-platform. The choice was obvious and now I am a Java programmer in addition to my other skills.

Now it’s my home desktop operating system’s turn, and the choice is obvious as well…

Goodbye, Windows, taking your last picture for memories:

1