Posts

  • A plain conversation about application security

    This is an extract from a real conversation with a colleague at one of my previous jobs. Even though their name is not mentioned here, I asked for their permission before publishing this:

    Colleague:

    I think we talked a bit when you were in your previous role about the way traditional app sec works here. Where it’s similar to how it’s been in my other experiences. What kind of stuff have you figured out about moving that into the whole CI/CD process? Anything cool or interesting yet?

    me:

    The most important thing I know about application security, and you know that too - is that it will fail if software engineers are not interested. So if there is no buy-in from that side, another problem needs to be addressed first. If a software engineer is not interested in learning about bugs they may have in their code, it’s a problem of a different category. A culture problem.

    And then, the next thing I learned is that I can be a security/pentest expert whatsoever and know everything about software vulnerabilities, but:

    a) I am not the one who owns the risk - business does. And they are the party to make decisions on how to address it.

    b) I don’t have access to the code repository, neither I have enough skills to fix the issue myself. Only the software developer can do that. So again. I can’t really tell I own the risk or its resolution

    Colleague:

    Do you ever do code reviews with people to look into security issues? I see that code reviews are a big part of a lot of App Sec programs, but I’m only familiar with them as a team mate building something with my team.

    me:

    Of course, code reviews are an essential aspect of my daily work.

    First of all, code reviews are costly. Companies can’t just spend their entire budgets on experts who can perform code reviews. Plus, those reviews take a lot of time which is also a very limited resource. That’s actually how DAST/SAST automation came into play: we don’t want those experts who review the code manually to spend their time on items that can be detected automatically. At the same time, SAST/DAST is also expensive and takes a long time. Not as long as a human review, but still. So that’s how tools like dependency checks and linting came into play.

    So every code pipeline should have Lint (e.g. SonarQube)->Dependency check (.e.g Snyk) -> SAST (e.g. Veracode) -> DAST (Burp/OWASP ZAP/sqlmap/other tools) with just manual reviews for the most critical and risky projects

    And there’s an important comment here that OWASP and some other large companies had a couple of years of research spent to see how effective security scanning tools are, with a verdict that if you have everything set up perfectly, you only get 4-10% findings. The rest is mostly human review, and again humans are all different. So that’s the expectation that we all should be aware of.

    Colleague:

    Wow, this is really insightful. So with these automated tools that’s where you can start hooking things into the SDLC. Like how my team hooked into SonarQube in dev and found a bunch of things to fix and measured our test coverage.

    me:

    That’s why the entire CI/CD and consequential DevOps comes into play: we know that even if we invest A LOT, we still will have security issues in production. And the best thing we can do is ensure we can quickly react to any findings we have, get systems back online fast, and have patches applied in no time.

    Yeah, the first thing I would be looking to have is what you described: SonarQube, test coverage and pull requests (code review) process

    And every modern programming language will have a set of awesome linting tools. Even markdown does :)

    …But still, application security is a matter of culture. Because what is security? The best definition I ever heard was ‘secure system is the system that behaves only in the manner it is expected to behave, and nothing else.’

    So it’s not, e.g., enough to just have sonarqube. It’s about fixing everything as well. And not even waiting until you push code into the repository. You want your IDE to lint as much as it can (that’s the reason I am paying for Intellij IDEA Ultimate - it’s linters are the best in the industry, and it has Sonar integration)

    Colleague:

    So modern app sec is really just translating defense-in-depth principals to every step in the app dev process. Which makes sense but maybe wasn’t as easy before.

    me:

    Well, this is a good point. Because it’s also important to mention that security controls are no longer developers’ responsibility in most cases. Usually, modern programming frameworks, or platforms, will take care of security controls. So I’d say application security is about knowing your framework and using it properly. If a developer has to implement security controls, usually it’s already all wrong.

    So yeah, it’s about security architecture design now, and defense-in-depth is the major aspect of it. Together with threat modeling that considers the entire context of the app.

    Looking at your question again, I’ll try to rephrase. It’s the combination of two:

    1. secure design that uses common security solutions and not reinventing the wheel (i.e., not implementing security controls from scratch but rely on well known frameworks/platforms)

    2. Using automation as much as possible to prevent [security] bugs from getting into code.

    3. Would be to ensure communication and collaboration

    (sorry for the large texts, this is the passion and research of my life you just triggered ;) )

    Colleague:

    No apologies needed, this is all really great insight. You should give a talk about this at a conference.

    me

    haha, I appreciate this :) thank you for your kind words

    I didn’t invent any of this, really. It’s what I keep hearing over and over at ‘Paul Security Weekly’ and ‘The Secure Developer’ podcasts (would totally recommend)

    Colleague:

    A lot of this is stuff I’ve already heard or was vaguely aware of, but you put all the pieces together really well. I am a big podcast fan, so I’ll check those out.

    One of the things that always intimidates me with application security is that people say you need to have a development background or be really good at “code reviews,” and I’ve developed and I’ve done code reviews, but when you google to find more information, it sure seems vague and like everyone does those things differently. So I’m left wondering, when do you know if you’ve done enough development to understand what you need to, and what’s this person’s idea of a code review if everyone does them differently. Maybe just realizing that is the key haha

    me

    Excellent question :) I don’t think there’s really a way for someone to be a code review expert of sorts. Because it’s not enough to just understand vulnerabilities, but the central part of the problem is to provide the right solution for it. And that would depend on the specific language and framework used in a particular project.

    So I’d say the only item here that makes sense is reviews performed by experienced developers in their space, and not by security people.

    Are you a Java guy? I have a very simple but very arguable example :)

    <p th:utext="'Hello, ' + ${name} + '!'"/>
    

    (I also know this is very different from what security training has, but this is real life)

    Colleague:

    Looking at the code, it seems like it takes in a parameter name, and the desired output is to put a <p> tag on the HTML with “Hello, [Name]!” in it.

    Okay, so the issue I’m seeing is just that it’s taking in any string and putting it into the page. so you need input sanitization

    me:

    And that would be the 99% answer because this is what training tells us. it is entirely correct from a security perspective.

    But in reality, first of all, input sanitization is almost impossible in this case.

    And even if you figure out some perfect regex, it will slow down the application by a lot, it will be a huge performance problem - just like any regex. That’s what developers told me. And I tested it all, and they are right. This is a real-life example, and a typical Java legacy app would have 100s of these.

    Now I want to add a comment of my own before giving my version of the solution. What I also learned is that most injection vulns consist of two parts:

    ---- input ----> [routine] ----> output ---->

    So two conditions need to be met for a vulnerability to exist:

    1. Input needs to contain character sequences that are treated as control sequences by a downstream component, e.g. database, or a subprocess.

    2. And the output must contain the same chars that are interpreted as control sequences by the downstream component or system that receives that output.

    The idea is that, e.g., during HTML rendering, like in the above case, the second condition would be for output to contain characters like <> so that they are rendered into control tags like <script>

    And the problem here is the [routine] part above can have multiple inputs. E.g., it can be called by some other routine, some function, be inherited, etc. so it’s actually like this:

    ---- input ----> [routine] ----> output ----> 
            input 2 /    | input 3
    

    So the lesson I learned looking for vulns resolution is that input validation is extremely hard to do right. And still, there is output encoding that actually will prevent an injection.

    So the real-life solution here by a senior developer is just to use output encoding (escaping). Which is just by changing the view:

    Was:

    <p th:utext="'Hello, ' + ${name} + '!'"/>
    

    Fixed:

    <p th:text="'Hello, ' + ${name} + '!'"/>
    

    Changing that tag from utext to text required knowledge of that specific thymeleaf framework on top of Spring

    In the same way, SQL injection is prevented by using prepared statements that de-facto escape parameters.

    Back to the XSS example, in real life, our company app had 800 of these. And developers were initially forced to filter input. And they failed after 1.5 years doing that.

    Colleague:

    Yeah, that’s wild - I know about escaping things and I know you can do it easily in like PHP and other formats. It’s crazy to think that every framework or tool is going to have a different approach, but I’m sure as you learn them you start to see patterns.

    me:

    And you are absolutely right about patterns - the biggest one is ‘don’t disable a control that exists by default just because it’s not convenient’ :)

    Usually, frameworks will have a secure solution available by default. That utext stuff in the example above takes time to find in docs.

  • Building Android ROMs (Omni and other AOSP-based firmware)

    upd: archiving my blog in 2021 and reading this post again, I am no longer a fan of Sony phones and of Android.

    Appears that buying a Sony Xperia phone several years ago was a good call - they appeared to support OpenSource Android very well even for devices that are no longer officially supported, and they even provide instructions on how how to unlock those devices, and build your own ROMs based on AOSP - Android Open Source Project. And they also published instructions on how to build Android 10 (Q) which was released just a few weeks ago!

    But despite lots of documentation available on Internet, its still very hard to start compiling Android-based ROMs even if you have access to all the source code and have a good vendor like Sony. With this post, I want to share what I learned so far.

    In no way I can pretend to be an expert in this field. If you are looking for one, I can recommend these guys:

    By the way, everything I am writing about here is all described at https://source.android.com and https://github.com/omnirom/android, and I am just putting pieces from different pages together here.

    Setting up your build environment

    Android is based on Linux, so usually its built using Linux machines. There is no support to build ROMs at Windows, and some support at MacOS. Here, I will only be talking about Linux since never owned a Mac.

    Also, please note that you need to have at least 200+Gb of disk space to successfully build Android. I personally have 2TB since I build 3 different ROMS: Omni, AOSP 9 and AOSP 10

    Usually, it does not matter which distribution of Linux you use, but most build systems out there use Ubuntu 14.04 (Trusty) as a standard. I tried many combinations, and it looks like most of those recommendations are outdated. E.g. Android 9 (Pie) requires Java 8 to be built, and Ubuntu 14.04 LTS does not have it, so there’s a hack to get that working. For example, Android Project itself provides a Dockerfile that creates an image based on Ubuntu 14.04 for building Android ROMs. It worked for me for some time, but I ended up just using a virtual machine with the latest Ubuntu LTS.

    Instructions below should work with most recent Ubuntu installations.

    First, you need to install the required libraries and build tools:

    sudo dpkg --add-architecture i386
    
    sudo apt-get update && sudo apt-get install -y git-core gnupg flex bison gperf build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip python openjdk-8-jdk libssl-dev lzip repo
    

    Note that I’ve added an extra library libssl-dev which is not required for AOSP builds, but is needed for OmniROM. And lzip which is needed if you want to add OpenGApps to your custom ROM.

    I am also using the latest version of the repo tool that is available in Ubuntu repositories. It may be a bit out of date, but so far I did not have any issues with it. I just don’t like to add binaries to the OS manually anymore.

    If you want to use the latest version of repo as everyone else recommends, just run a couple more commands:

    sudo wget -P /usr/local/bin https://storage.googleapis.com/git-repo-downloads/repo
    
    sudo chmod a+x /usr/local/bin/repo
    

    The last step is to configure your Git user:

    git config --global user.name "First Last"
    git config --global user.email your@email
    

    And that’s it! Now we are ready to checkout Android and start compiling!

    Checking out Android code

    Initializing repository

    Getting code in Android consists of two stages: first, you init the repository, and then you synchronize (download) the code. And of course you start with creating a directory for it:

    mkdir android
    cd android
    

    AOSP

    Again, I am only repeating the existing documentation here, it’s all very well explained at AOSP website: https://source.android.com/setup/build/downloading#getting-the-files

    repo init -u https://android.googlesource.com/platform/manifest -b <your branch>
    

    In most cases you don’t checkout the latest master branch and instead pick a tag (or branch) corresponding to an Android version release - you can find the list of branches here: https://android.googlesource.com/platform/manifest/+refs

    Here’s why: most vendors align their code, binaries, and corresponding instructions with specific Android releases. For example, in instructions for my Sony phone I can only use android-10.0.0_r1 while android-10.0.0_r3 is already available. Furthermore, at the moment of writing this, OpenGApps which I am using, is not yet available for Android 10, so I have to use Android 9 instead, and while the latest Android release was android-9.0.0_r48, I can only use r46 since that’s the latest Android 9 release my vendor provided their source code for.

    Of course, probably we can try using different release versions of vendor code and AOSP, but should there be any bugs, I currently don’t have enough skills to fix them.

    OmniROM

    Omni is an amazing ROM based on AOSP. It has many useful features and fixes while is not bloated with vendor software. I use Omni as stable ROM for my phone between my experiments with AOSP, and the more I learn about building my own version of AOSP, the more I see how great Omni is, and how knowledgeable are people that build it.

    To get Omni, the repo init command has to use a different URL:

    repo init -u git://github.com/omnirom/android.git -b <branch>
    

    With Omni, using branches is much easier - you just pick Android version. They already took care about everything else. You still get the list of branches from the Omni Github project itself, and it’s definitely worth visiting it since there they also provide very simple instructions on how to build there (I am going to duplicate those instructions below with minor changes):

    https://github.com/omnirom/android

    Customizing your ROM

    Usually you make the necessary customizations right after you initialized your repo - this is where you add references to any components that you are adding to your ROM. For example, if you want to add OpenGApps, you just add an xml file (with any name, for example, `opengapps.xml`) to the .repo/local_manifests/ directory. Below is my version of it. Note this just copied from the OpenGapps AOSP instructions by the link above:

    <?xml version="1.0" encoding="UTF-8"?>
    <manifest>
    <remote name="opengapps" fetch="https://github.com/opengapps/"  />
    <remote name="gitlab" fetch="https://gitlab.opengapps.org/opengapps/"  />
    
    <project path="vendor/opengapps/build" name="aosp_build" revision="master" remote="opengapps" />
    
    <project path="vendor/opengapps/sources/all" name="all" clone-depth="1" revision="master" remote="gitlab" />
    
    <!-- arm64 depends on arm -->
    <project path="vendor/opengapps/sources/arm" name="arm" clone-depth="1" revision="master" remote="gitlab" />
    <project path="vendor/opengapps/sources/arm64" name="arm64" clone-depth="1" revision="master" remote="gitlab" />
    
    <project path="vendor/opengapps/sources/x86" name="x86" clone-depth="1" revision="master" remote="gitlab" />
    <project path="vendor/opengapps/sources/x86_64" name="x86_64" clone-depth="1" revision="master" remote="gitlab" />
    
    </manifest>
    

    This way, you can also add and customize APKs and any open source software in your build. Unfortunately, this is where I am still learning, so not much information in this section yet - I will be updating it as soon as I have more progress, and meanwhile you can find more information at the Android website, and via the links I provide in the end of this post.

    Note that if you’ve read the OpenGApps AOSP instructions, they also require you to change your device.xml settings, and you don’t have the file after you just initialized the repository. Usually this is done by forking the repository that contains your device.xml and making the necessary changes in your fork. And then you just ensure that your AOSP-based ROM’s manifests point to your fork instead of the original one. You can see lots of examples of this in the OmniROM repository (and that also explains why everytime you look at someone’s custom Android ROM repos, you see dozens of forks of Android-related repositories)

    Unfortunately, I am still in process of learning how to do this properly, so I can’t really point to any forks that I’ve made so far - instead, I just sync the repo and do all my customizations just before starting the build. But that’s a wrong way to do that - all my changes are lost every time I update my local repo. In the end of this post, I am sharing some links that can help you understand this process better.

    Downloading code

    After repository was initialized, its time to download (or synchronize - the same command is used to update) code. This is a very long (hours, sometimes even days) process, and below is an optimized command you can use that will help making it faster. By the way, if you are using a VM or server like I do, it’s good time to use something like [screen](https://linux.die.net/man/1/screen) so that you can disconnect from this session while its running:

    repo sync --force-sync --no-clone-bundle --no-tags -j$(nproc --all)
    

    Creating a local mirror

    If you are downloading AOSP, you will notice how slow it is. AOSP Downloading page provides instructions on how to create a local mirror. Instructions below are taken from there:

    # Create a directory for your mirror
    mkdir -p /path/to/my/mirror
    cd /path/to/my/mirror
    
    # Mirror AOSP repository
    repo init -u https://android.googlesource.com/mirror/manifest --mirror
    repo sync
    

    After that, you can init a repo by referring to your local mirror instead of slow android.googlesource.com:

    repo init -u /path/to/my/mirror/platform/manifest -b <your branch>
    

    This saves a lot of time in my case, when I have two parallel Android repos, and only spend time syncing mirror once. But note that it takes a lot of space. I had to reserve 2 Tb only for my Android directories:

    Building the code

    After you synchronized (and customized) the code, you can now start building it. It’s done in two simple steps:

    Step 1: Initialize environment

    First, you run the envsetup script that initializes your local environment, set paths to proper tools and defines required variables. This also means that you need to run it every time you re-login. And this step is always the same for all types of Androd-based ROMs:

    source build/envsetup.sh
    

    After that, you need to set a target for your build - and that’s a matter of preference how to do that. E.g. Sony’s AOSP build recommends you running the lunch command that will automatically fetch all information about available device configurations and offer you to choose one. OmniROM, on contrary, will offer you to run the brunch command that will fetch device configuration based on your input and start the build.

    This is unfortunately another item that I am still learning, so I will just put both instructions:

    # AOSP
    lunch
    
    # OmniROM
    brunch <device name>
    

    Step 2: Start the build

    After that, you can just run the make command, and the build will be started. Note that it may take a couple of hours, and in the example command below I am adding an argument to use all available CPU cores on the system:

    make -j $(nproc --all)
    

    It’s normal to have warnings during the build, and in most cases they can be safely ignored. I so far only once heard about an Android build that has no warnings, but I never saw it with my own eyes. My plan however is to help get rid of these warnings as soon as I gain the required knowledge.

    Flashing the new ROM

    After the build is completed, your binaries will be available in the out/target/product/<your phone> directory.

    If you were building AOSP, you’ll get images corresponding to partitions in your Android phone, e.g:

    boot.img cache.img recovery.img system.img userdata.img

    And its normal if there’s more img files than your phone has partitions (e.g. in my case, there are always some ramdisk partitions). You can just ignore those.

    All you need now is just to flash the image files. Usually, image name will correspond to a partition it goes to, e.g.:

    fastboot flash boot boot.img
    fastboot flash cache cache.img
    fastboot flash recovery recovery.img
    fastboot flash system system.img
    
    # note that the following command deletes all data at your phone
    fastboot flash userdata userdata.img 
    

    Flashing is a large topic by itself, and its beyond this blogpost. For example, with OmniROM, you’ll get a zip archive instead of a bunch of image files. That file can be flashed via tools like TWRP - just follow OmniROM instructions.

    I personally prefer to use images, and I usually find img artifacts remained from OmniROM build and flash them manually.

    Here’s some very useful links that can help you start building your own Android-based ROMs:

    https://bootlin.com/doc/legacy/android/android-slides.pdf

    https://forum.xda-developers.com/general/xda-university/

    https://elinux.org/Master-android

    https://android.googlesource.com/platform/manifest/+refs

  • A conversation about TLS certificates in application security

    A conversation about TLS certificates in application security, from a chat with a colleague while working on an actual project. Published with permission.

    Alright, so here’s that fun part most people in security don’t know about.

    For us, using TLS was sort of natural because we are a security team. But many software engineers avoid it, especially at the beginning of their career. And then when they finally got to the point when they actually want to start using TLS, that usually means that they got pretty advanced in what they do.

    Often that also means that they learned to do most things by just googling them and copy-pasting solutions from StackOverflow. Nobody ever reads manuals, so eventually dealing with certs becomes mostly ‘hacking’ certs because, you know, certs are hard. Reading takes time, so whoever comes up with a quick solution from StackOverflow, everybody is going to use it without even knowing it’s wrong. If it works - don’t touch it, right?

    Nobody will spend time learning certs properly because it’s complex and takes lots of time, which can be spent playing with stuff like React and other cool technologies.

    And you know that getting legit certs that are accepted by browsers is a long process, and most of the time, people use self-signed certs just like you and me yesterday. That’s the natural order of things. And not everyone is signing certs with a global CA root certificate included by default in browser and operating system trust stores, so things actually get more complicated for many companies.

    Many companies I worked with use self-signed certs for internal TLS, saving tons of money this way. You know, now that I am typing it, I think that maybe, the whole reason our company uses a global CA, and there are those discussions in our Slack’s #certificates channel about those humongous costs is precisely for the same reasons that people didn’t know how to tackle the self-signed problem this large text is about :)

    Every time an engineer is implementing something major in their company, they get the problem you got, “signed by unknown CA” type errors. And you know how bad this can end: you can see that -tls-skip-verify we are using in our scripts.

    So to the solutions. There are two primary solutions I know:

    1. The best and proper solution is to add the root cert to the operating system’s trust store. Every OS has a place where it stores root certificates it trusts. Everybody knows it, but because every OS does it differently, people just never get to learn how to add those certs correctly.

      In Windows, that would be right-clicking the certificate and selecting ‘add to Root CA’-something (I don’t remember that precisely). In Linux, it’s just putting your root CA public cert to a particular folder and then running a command to process it and add to the system :)

      Here’s an example for RedHat based OS (Includes CentOS, Fedora etc):

      ln -s /etc/vault.d/tls/cert.pem /etc/pki/ca-trust/source/anchors/vault-self-trust.pem
      update-ca-trust export
      

      Surprisingly, in the same way, using something like CygWin or GitBash has the same trick - those two are also like an OS inside your OS :) The only problem with that - it requires you to be an admin.

      E.g., in Debian-based Linux distros (Ubuntu, Mint etc.) that would be a bit different: the directory for new certs to place is /etc/ssl/certs. And command to run is update-ca-certificates --fresh. You can always find these commands by googling something like ‘redhat add root certificate’ :) Most people google just the error they get from curl or their browser.

      Now, why is this way the best way to do it - the crucial part here, it doesn’t just update your OS certs. The colossal problem you never want to solve, and instead, you want to delegate it to the operating system consists of the following: all programming frameworks and languages such as Java, .NET, Ruby, JavaScript, Python (all of them) do have their own, separate certificate storages :)

      Most server operating systems will have hooks triggered when you update your OS Root certs with the commands above - hooks that automatically rebuild certificate storages for the frameworks and languages. If those are correctly installed using package managers - the whole reason why you don’t want to compile stuff in the modern world and put it to /opt or /usr/local and instead only use official repositories. It also explains why major Linux distros are not up to date with recent software versions - that packaging takes time).

      And of course, in the modern corporate world, Java and Python are among the most popular languages, so those are pretty much guaranteed to be automatically updated with certs if you are using a good Linux distro and installed them correctly (using the system package manager and good repos)

    2. The second way is very hard. And you probably already guessed that it’s about adding a Root CA certificate to a specific language or framework you are using. You never want to do that, but sometimes you got to, especially when you’re not an admin at the machine you are using (no root). Again, in the corporate world, this solution is not recommended.

      For example, Java uses trust stores, encrypted files that are deployed together with your java app. You explicitly add that trust store to JVM parameters when it’s run. So things can get complicated there.

      While useful in dev, those are really bad scenarios - you can imagine nothing good happens when there’s a discrepancy between OS root CA trust and App’s root CA trust.

    That’s it :) thank you for reading :))

  • My virtual lab is back!

    It’s been a while since I worked with my home lab - and now I am unpacking it again, for a new journey. This time, I will be learning OpenStack and trying out various security and automation tools with it.

  • Mini project for my daughter

    Been a while since I used my soldering iron last time. Really enjoyed prototyping simple electronic circuits with my kids tonight.

    This was 2nd time this year. First time was in June:

  • Can continue working on my micro-controller projects now under Linux

    I thought it’s impossible, and was so surprised when I tried and it just worked under Debian Sid and Virtual Box with Windows 7 Pro.

    Atmel Studio was a very easy install in a VM, and it had no problems finding my USB programmers and devices attached to my host Linux machine. It’s very good news given that my new digital oscilloscope with logic analyzer arrives tomorrow :)

  • One of my old reported vulnerabilities was published: CVE-2014-8733

    CVE-2014-8733: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-8733

    It was fun to work with Hadoop security in 2014… This vuln was a tricky one because I was responsible for Hadoop managed service platform security, and our clients had SSH access to Hadoop cluster nodes in some cases.

    If I remember correctly, fix wasn’t easy - required release of new CDH version which moved configuration parameters between files (world readable access was required for Hadoop client to function). And then, several months later, it reappeared again after another patch.

  • Shell script - waiting until specified day/time

    I participated in a project in which job scheduling logic was completely implemented in shell script (it was a small project initially, and then it has grown in time). One of tasks was to enable scheduler script to wait until a specified time to run a job. Surprisingly, search in Internet did not bring anything that would look simple and elegant enough. There were bunch of quite complex solutions which did not really address my requirements and would be too hard to maintain for support teams in future… I came up with my own solution for this after reading Linux manuals, which I wanted to share.

    It appears, ‘date’ utility is very flexible in terms of definition and representation of time. For example, user can supply target date/time in a string, using words like ‘Today 9am” or “Thursday 12pm” etc. In combination with ‘sleep’ command, I was able to achieve the desired functionality with required precision (seconds). I ended up writing this simple bash function:

    #-[ wait_until ]------------------------------------------
    # Waits until a given day/time
    # $1 - target time/date in 'date' command format, e.g. "Today 13:00"
    wait_until()
    {
    echo "Waiting until $1..."
    
    TIME_TARGET=$( date -d "$1" +%s )
    TIME_NOW=$( date +%s )
    
    if [ $TIME_NOW -lt $TIME_TARGET ]; then
    
        TIME_WAIT=$( expr $TIME_TARGET - $TIME_NOW )
        echo "Waiting time: ${TIME_WAIT} seconds"
        sleep ${TIME_WAIT}s
    
    else
        echo "Target time is already in past. No waiting required..."
    fi
    }
    

    Use is very simple:

    wait_until "Today 1pm"
    #... some code ...
    wait_until "Tomorrow 6am"
    #... some code ...
    

    etc…

    Hope it helps someone who will be looking for this solution in Internet, just like I did before I had to write it :)

  • ESXi and Kali weekend

    Installed Kali Linux in my virtual lab this weekend - just to make a snapshot of currently available packages and, as usual, steal a couple of ideas for my own pentest Linux VM. Two ideas I will never steal from Kali are Safari Icon for Firefox and use of Gnome 3.

  • Last weekend's small DIY project

    I really enjoyed this. I know it does not look like much, but I am very happy about this little project: I finally learned how to solder properly and lead-free, and I now understand AVR microcontrollers architecture and specifics. What’s left is to finish power optimization and start periodically add different sequences for LEDs and different reactions to button clicks.

    And I also learned about powering small devices and about Joule Thief / step-up regulators, induction, surface-mount and PCB design/ordering (however, these are not used in this project)

    Github: https://github.com/samoylenko/avr_s1

    photo

    schema

  • Whoa! Debian Linux works amazingly good at my hardware!

    When I said my last goodbye to Windows, I thought it will be Ubuntu to become my new primary home operating system - I had it for a couple of years in past, with very very good experience for both me and my wife, don’t even remember why I switched back to Windows. Probably because I disliked what they did to Gnome, or the switch to Unity, or Il-2 Sturmovik did not work well with Wine, or all three together and lack of time to deal with it - so I thought I’ll pay Microsoft to make all decisions for me… I really don’t remember now. But for me, it was always clear that Ubuntu is the best Linux distribution for desktop hardware because of its superior support for drivers and firmware. I prefer Debian at my virtual desktops and Ubuntu again at servers, both hardware and virtual.

    So I removed Windows 10 and installed Ubuntu, and spent a couple of days playing with it - everything worked well, but I was really missing my Debian XFCE customized desktop experience to which I am used to. And I decided to try Debian at my laptop. I was sure I won’t be able to get it work with my wifi card, or cardreader, or something else… I was so surprised that everything worked out of the box! And the install went smooth, over wifi, and everything was just great! The biggest surprise, however, was that Debian appeared to work much better with my Microsoft (hehe) Bluetooth mouse than Ubuntu. Hibernation, suspend, sensors etc work without any problems as well.

    It’s been almost a week, and I am very happy with Debian as my primary OS now. I am very confident it will stay, and I am not switching back to Windows or even Ubuntu any time soon - I have an old EEE-PC with Debian installed for around 2 years already (for kids to play) which I was periodically checking and updating, and was amazed of how good and stable it is.

    I only had to do two things to make it happen:

    1. I used unofficial Debian installer with non-free firmware support to install it at my laptop.
    2. I had to manually add my touchpad configuration to enable all the fancy multi-touch etc:

    Here’s a screenshot of my Debian desktop:

    Screenshot - 08232015 - 11:53:48 PM

  • Farewell, Windows

    I’ve got Windows 10 at my laptop around 3 weeks ago and was testing it since. I do like the new design, and the OS itself is definitely better than any previous Windows version, including 7. And all of my life I’ve always been a Windows guy, preferring Linux only in server or virtual environments. But I just can’t leave all the privacy concerns be. I hate the idea someone is constantly watching what I am doing and where I am going. And I can’t bear the fact that Microsoft can scan my hard drive or sneak on me through my webcam anytime they want. It’s gone way too far.

    When they announced the free upgrade, I knew it will be something serious and users will have still have to pay, but in some different way. Well, now we know what it is… Sorry, Microsoft, I am not buying this anymore.

    Three years ago, when I wanted to refresh my software development skills and learn something new on a professional level, I was choosing between C# and Java, and I really wanted it to be the former given all of my previous experience, but the price of Visual Studio and MSDN subscription was astronomical, and everything was restricted to only one OS and one vendor while for Java everything was free and cross-platform. The choice was obvious and now I am a Java programmer in addition to my other skills.

    Now it’s my home desktop operating system’s turn, and the choice is obvious as well…

    Goodbye, Windows, taking your last picture for memories: 1

  • Hadoop without Kerberos - simple attack examples

    In this post, I am going to illustrate that it’s practically impossible to protect any data in Hadoop clusters without Kerberos (‘Secure mode’) enabled. I hope this will help admins and security folks see that Kerberos is the only way to make Hadoop more or less secure - without it, there is no authentication in Hadoop at all. But as you can see from my previous posts about Hadoop, even with Kerberos enabled, there are still very serious challenges, so Kerberos is just a start, not the final solution.

    At this time, I will focus on the most important component of Hadoop ecosystem - HDFS, Hadoop’s distributed file system which is used to store all data in Hadoop in most cases.

    1. Target setup.

    For this, I will create 2 Hadoop clusters in my lab:

    • Cluster 1: used by Alice and Bob
    • Cluster 2: used by Cindy and Dan

    Each user will have their own home folder in HDFS protected by access lists, and a secret file which must not be accessible to anyone but the owner:

    File Owner File Name Content Location Permissions
    Alice alice_secret.txt Alice’s secret file Cluster1: /user/alice/ rw—- (0600)
    Bob bob_secret.txt Bob’s secret file Cluster1: /user/bob/ rw—- (0600)
    Cindy cindy_secret.txt Cindy’s secret file Cluster2: /user/cindy/ rw—- (0600)
    Dan dan_secret.txt Dan’s secret file Cluster2: /user/dan/ rw—- (0600)

    Creation of secret files:

    1-2

    Checking security of secret files:

    1-3

    At this time, Kerberos is not enabled at both the clusters, so let’s see how we can attack Hadoop clusters which don’t have Kerberos enabled.

    The biggest challenge around attacking Hadoop clusters without Kerberos is descovery of IP addresses and ports of Hadoop daemons - the rest is quite easy. All my scenarios are ‘internal attacker’ only - I hope no one these days exposes their Hadoop clusters to the Internet, it’d be a huge mistake. I will leave the discovery of Hadoop services outside of the scope of this post, but you can use e.g. Cloudera’s Hadoop port reference to begin with.

    2. Attacking No-Kerberos Hadoop Clusters

    For the most Hadoop installations that I’ve seen, only a web browser and a REST client like CURL are needed to break into a Hadoop cluster :) But below, I will also use Hadoop client - just for the full picture.

    2.1. Web browser based attack

    By default, HDFS comes with Web UI enabled. And it’s usually available at port 50070 (we are connecting to Name Node Web UI):

    2-1

    The problem is, this port is quite complex to protect, and the common way is to enable Kerberos at it. Otherwise it’s accessible by anyone, or there will be a custom authentication solution (I never encountered one in my 2 years Hadoop Security journey, so probability is very low).

    It’ll not take long until you will discover your way to browse HDFS (located in “Utilities”->”Browse the file system”):

    2-2

    But when you will try to access Alice’s or Bob’s data, you’ll get an expected “Permission Denied” error:

    2-3

    This web-server runs as ‘Dr. Who’ user by default, and this user definitely has no access to Alice’s or Bob’s file. This is an expected behavior, but in Hadoop, with Kerberos disabled, unfortunately it’s too easy to circumvent this. Simply said, Hadoop will trust any username you will provide to it. Just type the following into the address line under “Browse Directory” (the one that has “Go!” button in the end), and you’ll get access to Alice’s directory:

    /user/alice?user.name=alice&op=LISTSTATUS#

    2-4

    As you can see, by default Hadoop just accepts parameter “user.name” as the current user’s name, with no authentication.

    What’s interesting - this one I’ve just encountered, this is something new that I had to put this data into the web form’s field, and suppress whatever it adds at the end of URL via the # character. In previous versions of Hadoop you could just add ?user.name=<user> parameter to the end of your browser’s current URL and get access to anything.

    As you can see, it’s very easy to get through any Hadoop ACLs (access lists) - it will always tell you the object owner’s username, and you can always add it as a user.name parameter and become this user. We’ll use this with CURL as well.

    2.2. REST API based attack

    Full information how to access HDFS via REST is available at this page: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html Very simple - just like above, we only need to add “user.name” parameter in the end. And, as always, the biggest challenge is to locate the WebHDFS IP address and port - the same as we used above. 50070 by default.

    The trick is to properly construct URL for your tool (cURL in my case) - it’s fully documented in the Apache document referred above:

    curl -i "http://192.168.1.121:50070/webhdfs/v1/?op=LISTSTATUS&user.name=alice"
             ^1                        ^2         ^3 ^4          ^5 
    
    Where: 
     1 - http://<namenode address>:<port>
     2 - Constant: "/webhdfs/v1"
     3 - Path. At this example, it's "/" - filesystem root
     4 - operation "?op=<operation>"
     5 - any user name
    

    So, we can just browse using REST API and give it an owner name when encounter an ACL:

    1. Let’s see what’s in /user: 3-1

    Alright, we see “alice”, “bob”, “hdfs” and “history”, with all the information including owners.

    2. Let’s try to open Bob’s user directory:

    3-2

    3. No access as expected. Ok, let’s try again with ‘user.name=bob’ this time:

    3-3

    4. Now we see Bob’s secret file. Let’s open it:

    4-1

    (note that I had to add the -L switch to cURL to enable it follow redirect - all data in HDFS is stored at Data Nodes, and Name Node redirects the client directly to the Data Node which has the data)

    As you can see, we have successfully opened Bob’s secret file.

    Same with Alice at the same cluster, and with Cindy and Dan at cluster 2:

    4-2

    4-3

    4-4

    As you can see, it’s still the good old user.name parameter. Let’s do something different now

    2.2. Hadoop Client based attack

    If you prefer to use Hadoop client, it’s even simplier. Let’s install it first. It’s very easy - just download, unpack and set the JAVA_HOME variable:

    5-1

    5-2

    Typical command to access HDFS in Hadoop is “hdfs dfs - ". E.g. to list '/user' directory:

    hdfs dfs -ls /user
    

    In our case, we are an attacker, and we are not part of the cluster, so we are adding another parameter to this command - -fs to point it to the NameNode we will be working with. it’s usually the same IP we used before, but a different port. Default is 8020 (Name Node port, I am using Cloudera distribution). Let’s try it:

    # Cluster 1
    hdfs dfs -fs hdfs://192.168.1.121:8020 -ls /user
    
    # Cluster 2
    hdfs dfs -fs hdfs://192.168.1.131:8020 -ls /user
    

    5-3

    Now let’s try and access Dan’s protected directory:

    hdfs dfs -fs hdfs://192.168.1.131:8020 -ls /user/dan
    

    ‘Permission denied’, as expected. Well, ‘user.name’ won’t work here, because that was a parameter for Web access. But there are two ways to become ‘Dan’ for Hadoop at client side. Both are very easy. And as you can see, I am working from a PC that does not belong to any of the targeted clusters. It’s a virtual machine that can be anywhere on the network.

    2.2.1. Environment variable

    First way is to set an environment variable called HADOOP_USER_NAME :) Yes, just like this:

    export HADOOP_USER_NAME=<user>
    

    5-4

    2.2.2. Local OS user

    Just wanted to add another way to change user name for a Hadoop cluster without Kerberos enabled. Just for the sake of completeness. It requires root privileges at the local system (scenario with e.g. developer from inside organization runs a VM just like I do).

    By default, Hadoop client takes current user name from operating system and passes it to the server. So as an admin at my VM, I can create ‘dan’, ‘alice’, ‘bob’ or ‘cindy’ local user and run Hadoop client as this user:

    5-5

    3. Hadoop Security without Kerberos - conclusion

    As you can see, there is no authentication in Hadoop without Kerberos. Any Hadoop cluster without Kerberos authentication enabled is de-facto a single-user system, where the user is identified by its ability to access Hadoop services, and this user has full admin access to the cluster (I skipped superusers impersonation for a purpose)

  • hadoop.security.auth_to_local examples

    In my previous post “An important Hadoop security configuration parameter you may have missed” I was talking about importance of the hadoop.security.auth_to_local configuration parameter and promised to provide some solutions using this parameter.

    I want to focus on a couple of practical use examples in this post, and if you want to learn more about this, here are links to the existing documentation:

    The overall idea is there are two major types of principals coming in your Hadoop ecosystems:

    1. Three-parts principals: <user>/<host/service>@<domain>, e.g. hdfs/node1.my.domain@MY.DOMAIN
    2. Two-parts principals: <user>@<DOMAIN>, e.g. bob@MY.DOMAIN

    Rules for these two types are defined separately. Hadoop needs to convert these principals into usernames, and this is where auth_to_local rules are used. By default these rules just remove everything but the part for the current (default) domain. I spoke about the negative consequences of leaving this default in my previous post. In short, they are:

    1. Users are not uniquely mapped to their UIDs (aka ‘users overloading’) - e.g. if user bob@MY.DOMAIN has a confidential data stored in Hadoop cluster with no access to anyone but Bob, still there can be another user created with principal bob/<anything>@MY.DOMAIN which will be translated by default into ‘bob’ user, too, and will get access to all Bob’s data.

    2. Superusers (hdfs, yarn, hive, mapred, hue etc.) are shared between clusters in the same domain (e.g. if at all of your clusters, namenode runs as local ‘hdfs’ user, then hdfs@<DOMAIN> user is de-facto the superuser at all your clusters). Also, e.g. datanode user at Cluster1 is still a legal user (and in Cloudera distribution, superuser) at Cluster2 and all other clusters in the same domain (and if it users hdfs/<node>@<DOMAIN> principal, then it’s also an HDFS superuser at all your clusters).

    3. Any user of type <user>@<DOMAIN> with the <user> part being equal to the OS user which has started a Hadoop process is de-facto the superuser of this process (e.g. hdfs@<DOMAIN> becomes superuser at all clusters in the same domain in most installations that I saw, same with hive@, hue@ etc.)

    To solve these issues, the following auth_to_local rules can be applied:

    1. Rule to prevent any unwanted users coming in to this cluster:

      RULE:\[2:$1/$2@$0\](^.\*$)s/^.\*$/nobody/
      

      What this rule does: it converts any Kerberos principal of type <user>/<something>@<DOMAIN> into a nobody user. It works like a firewall rule ‘Block All’ and must be placed at the end of your auth_to_local rules.

    2. Since the rule above blocks all 3-part principals, which are generally used by Hadoop service users (e.g. HDFS NN / DN or YARN RM / NM etc), we now need to whitelist these users. For each of your services, create a line(s) using the following template:

      RULE:\[2:$1/$2@$0\](^<service>/<node>@<domain>$)s/^.\*$/<service>/
      

      Example for a 8-nodes cluster with YARN and HDFS, will have the following rules:

      RULE:\[2:$1/$2@$0\](^hdfs/node\[1-8\].my.domain@MY.DOMAIN$)s/^.\*$/hdfs/
      RULE:\[2:$1/$2@$0\](^yarn/node\[1-8\].my.domain@MY.DOMAIN$)s/^.\*$/yarn/
      RULE:\[2:$1/$2@$0\](^yarn/yarnm@MY.DOMAIN$)s/^.\*$/yarn/
      RULE:\[2:$1/$2@$0\](^HTTP/node\[1-8\].my.domain@MY.DOMAIN$)s/^.\*$/HTTP/
      

      Note the yarn/yarnrm principal - it’s used by YARN RM to renew auth tokens. You also can replace the regex node[1-8] with a number of lines for each node at your cluster, this is useful if regex can’t be used to list all cluster nodes, e.g.:

      RULE:\[2:$1/$2@$0\](^hdfs/node1.my.domain@MY.DOMAIN$)s/^.\*$/hdfs/
      RULE:\[2:$1/$2@$0\](^hdfs/node2.my.domain@MY.DOMAIN$)s/^.\*$/hdfs/ ...
      RULE:\[2:$1/$2@$0\](^hdfs/node8.my.domain@MY.DOMAIN$)s/^.\*$/hdfs/
      RULE:\[2:$1/$2@$0\](^hdfs/node14.my.domain@MY.DOMAIN$)s/^.\*$/hdfs/
      RULE:\[2:$1/$2@$0\](^hdfs/node22.my.domain@MY.DOMAIN$)s/^.\*$/hdfs/
      
    3. Please also note that majority of Hadoop ecosystem daemons will accept any user with the same name as the local OS user that runs this principal. as a superuser. So users below are de-facto the default superusers for the corresponding services in most default Hadoop installs:

      • hdfs@<DOMAIN>
      • yarn@<DOMAIN>
      • hive@<DOMAIN>
      • etc.

      These users never exist in my Hadoop cluster setups: all my superusers are managed through groups. In clusters managed by me, those only will be human users, members of Admin groups (ping me if you want more information on how to setup this, but it’s widely available in the Internet and at Apache website), and some specific technical users (for example, in Cloudera CDH which I use, it is required that Hue user is part of the HDFS and Hive admin group etc). Since technical users come in as 3-part Kerberos principals in form of <user>/<node>@<DOMAIN>, they will be converted into corresponding superusers by the rules above, e.g. hdfs/node1.my.doman@MY.DOMAIN -> hdfs (same username as NN daemons runs as, and therefore is HDFS superuser). But we want to prevent a scenario when someone creates a user hdfs@MY.DOMAIN, which automatically becomes the HDFS superuser at all clusters of the same domain, so we need to block those users using the rules below:

      RULE:\[1:$1@$0\](^hue@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^sentry@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^hive@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^oozie@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^yarn@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^mapred@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^hdfs@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^zookeeper@.\*)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^httpfs@.\*$)s/^.\*$/nobody/
      RULE:\[1:$1@$0\](^HTTP@.\*$)s/^.\*$/nobody/
      etc
      

      These rules should include all the local OS users that start services at your cluster, and in my case, the example below includes the default configuration of a CDH cluster which has Hue, Sentry, Hive, Oozie, YARN, HDFS and ZooKeeper installed.

    That’s pretty much it. The 3 solutions above will help ensure that you can have any number of secure clusters in the same domain managed by different teams or one team as required.

    Since I use Cloudera distribution and direct integration of my Hadoop clusters with Active Directory (via Kerberos and LDAP group mappings), I slightly change the rules from #2 above to rename Cloudera Manager’s automatically generated default principals into usernames that are unique for each Hadoop cluster I have. I then can easily implement any kind of granular access lists and group membership by just creating these users in Active Directory and assigning them to the required groups and ACLS:

    Cluster1 (nodes 1-4):

    RULE:\[2:$1/$2@$0\](^hdfs/node\[1-4\].my.domain@MY.DOMAIN$)s/^.\*$/cl1\_hdfs/
    RULE:\[2:$1/$2@$0\](^yarn/node\[1-4\].my.domain@MY.DOMAIN$)s/^.\*$/cl2\_yarn/
    RULE:\[2:$1/$2@$0\](^yarn/yarnm@MY.DOMAIN$)s/^.\*$/cl1\_yarn/
    RULE:\[2:$1/$2@$0\](^HTTP/node\[1-4\].my.domain@MY.DOMAIN$)s/^.\*$/HTTP/
    

    Cluster2 (nodes 5-8):

    RULE:\[2:$1/$2@$0\](^hdfs/node\[5-8\].my.domain@MY.DOMAIN$)s/^.\*$/cl2\_hdfs/
    RULE:\[2:$1/$2@$0\](^yarn/node\[5-8\].my.domain@MY.DOMAIN$)s/^.\*$/cl2\_yarn/
    RULE:\[2:$1/$2@$0\](^yarn/yarnm@MY.DOMAIN$)s/^.\*$/cl2\_yarn/
    RULE:\[2:$1/$2@$0\](^HTTP/node\[5-8\].my.domain@MY.DOMAIN$)s/^.\*$/HTTP/
    

    etc.

    Please note that HTTP principal is part of Kerberos convention, and must remain HTTP at all times.

    Please feel free to ping me at any time with any questions you might have, or if you need help securing your Hadoop clusters.

  • Solution for Lenovo ThinkPad problem with sleeping mode in Windows 8.1

    upd: Even better solution is to use the latest Windows 8.1 ISO which can downloaded using the following Microsoft’s tool. It instantly detects the video adapter and installs the required driver. http://windows.microsoft.com/en-us/windows-8/create-reset-refresh-media

    This seems to be a standard problem for many Lenovo laptops: after Windows 8 enters sleeping mode, computer does not wake up - all the lights and fans are on, but display remains always black, until reset.

    Support forums did not help much - people talk about this problem for a couple of years already, some advice updating BIOS, but this did not work in my case. Only the stock image from Lenovo Recovery CDs seemed not to have this problem (but did have Superfish and stuff ;-) )

    Solution appeared to be quite easy - I just needed to install the latest Intel HD Graphics Driver :) That’s it. Here’s the location of this driver: https://downloadcenter.intel.com/download/24785/Intel-Iris-and-HD-Graphics-Driver-for-Windows-7-8-8-1-64-bit

    It just seems that Lenovo System Update software does not update this driver automatically.

    1

  • Kaspersky Antivirus appears to became just another bloatware nowadays

    So much disappointed… After all these years I finally decided to buy Kaspersky, and appeared it became just another bloatware now. Guess the folks don’t care about their firm’s karma anymore…

  • Now I'm a Certified Ethical Hacker

    Description: http://www.eccouncil.org/Certification/certified-ethical-hacker

  • Kaspersky does the same to your secure connections as what Superfish did to Lenovo users'

    I did not use Kaspersky software for over a decade now. Gave it a try today…

  • Installing XenServer at my home virtual lab. Again :)

  • Configuring Cloudera Navigator to use external authentication

    Cloudera, author of one of the most popular Hadoop distributions, has created a great tool for Hadoop security monitoring and auditing, called Cloudera Navigator. I find its initial configuration process a little bit tricky, so I wanted to document it in this post. Cloudera’s original document on how to do this is located here: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cn_sg_external_auth.html

    I currently use the latest version of Cloudera Hadoop distribution with Cloudera Manager 5.3.1 (trial enterprise license) and Navigator 2.2.1. It openly shows its full version and build in a tool-tip on its logo and in ‘About’ section right at the login page (so in case there’s a vulnerability published in future, hackers won’t need to spend time finding out target’s version ;-) ):

    1

    In my home lab’s cluster I use Active Directory authentication, and my goal is to make Navigator work with it, too. If you don’t use Active Directory or LDAP in your Hadoop installs, don’t worry about unavailability of ‘Administration’ page - it’s only about configuring groups in LDAP/Active Directory:

    2

    However, it seems that in this version of Cloudera Manager (5.3.1) ‘Navigator Administrator’ role presents but is not used, and only ‘Full Administrators’ can login to Navigator. Also, please note user names are case sensitive in Cloudera Manager and Cloudera Navigator.

    Navigator’s external authentication is configured via the following steps:

    1. Configure Navigator’s ‘Authentication Backend Order’ to enable Cloudera Manager authentication
    2. Configure LDAP parameters for Cloudera Navigator
    3. Restart Cloudera Navigator
    4. Login with Cloudera Manager’s local Full Administrator user
    5. Use ‘Administration’ page in Cloudera Navigator to configure external admins group
    6. Logout and test if you can now login with an external admin user
    7. [Optional] Configure Cloudera Navigator to use only external authentication and restart it again.

    My cluster is already configured to use LDAP authentication for Cloudera Manager, so I need to create a new local admin user to login into Navigator. You can skip this step if you are still using the default ‘admin’ user (in my installs I convert it into an ‘emergency admin’).

    First, I create a new local ‘Full Administrator’ user in Cloudera Manager (once again, as I said above, in this version of Navigator, Cloudera Manager’s role ‘Navigator Administrator’ does not seem to be used at all, and can’t login to Navigator):

    3

    Then I go to ‘Cloudera Management Service’ -> Configuration -> Navigator Metadata Server Default Group -> External Authentication and configure LDAP. My home lab uses Active Directory server with bind authentication enabled. I highly recommend you to use LDAPS over LDAP (and HTTPS over HTTP for login pages), but I don’t have that since it’s not critical for a lab environment:

    Authentication Backend Order -> Cloudera Manager then External External Authentication Type -> Active Directory (you can choose LDAP) LDAP URL -> ldap://winserver.lab.local (ldaps:// here would be better. this is address of my AD server) LDAP Bind User Distinguished Name -> hadoop-bind (if you use a bind user, don’t add a domain part the user name - Navigator will do this for you) LDAP Bind Password -> ******** (yep, eight stars it is) Active Directory NT Domain - LAB.LOCAL (my domain name) LDAP User Search Base -> DC=LAB,DC=LOCAL (OU to where to look for users) LDAP Group Search Base -> DC=LAB,DC=LOCAL (OU to where to look for groups)

    4

    Then I save configuration and restart Navigator (I usually do this by going to ‘Instances’, marking both Navigator roles and restarting just them, not the entire Cloudera Management Services):

    5

    When Navigator is back online, I login to it with my newly created Full Administator user:

    6

    ‘Administration’ page is now active:

    7

    I use ‘Find a group…’ text box to locate my lab admins group. Please note that Navigator will send a query to your LDAP server every time you press a button, and by default, a wildcard is used, which is not the recommended behavior for a LDAP client (imagine a production LDAP with thousands of groups). You can configure this query at Navigator’s External Authentication page in Cloudera Manager.

    8

    And click ‘Manage role assignment to configure it as Navigator admins:

    9

    Then I save it, logout from Navigator and test the new role assignments with my Active Directory user ‘labadmin1’ which is member of ‘LabAdmins’ group newly configured Navigator admins:

    10

    11

    Done. I then login back to Cloudera Manager, remove my local Full Admin user ‘NavAdmin’ (since I don’t need it anymore) and configure Navigator to use External authentication only. Now members of my group LabAdmins in Active Directory can login to Navigator to configure other roles as required.

  • Plans for the weekend - firmware for my OLED business card project

    Making another attempt to reactivate my OLED business card project. Due to lack of time and other priorities, I had to stop it for several months. I think that I’ve find an optimal hardware design and power source. Given limitations of ATtiny85, I started writing firmware in assembly, now I am starting it again in C.

    The board on picture was assembled to allow me work at kitchen table while watching my kids :)

    At the picture (left to right): Adafruit Monochrome 128x32 I2C OLED graphic display (SSD1306), 9V battery, DYI ATtiny85 dev board, AVRISP mkII programmer

  • An important Hadoop security configuration parameter you may have missed

    Hadoop has one security parameter, which importance I think is not stressed well enough in currently published documentation. While there are instructions on how to configure it, I did not see anyone talking about the consequences of leaving this parameter with its default value, and as far as I know, almost nobody ever changes it due to complexity. This parameter is

    hadoop.security.auth_to_local - “Maps kerberos principals to local user names”

    (description from current core-default.xml)

    It’s telling Hadoop how to translate Kerberos principals into Hadoop user names. By default, it simply translates <user>/<part2>@<DOMAIN> into <user> for default domain (ignores the 2nd part of Kerberos principal). Here’s what current Apache Hadoop documentation says about it:

    “By default, it picks the first component of principal name as a user name if the realms matches to the default_realm (usually defined in /etc/krb5.conf). For example, host/full.qualified.domain.name@REALM.TLD is mapped to host by default rule.”

    This means that for example if you have users with names hdfs, Alyce and Bob, and they use the following principals to authenticate with your cluster:

    HDFS - hdfs@YOUR.DOMAIN,
    Alyce - alyce@YOUR.DOMAIN,
    Bob - bob@YOUR.DOMAIN
    

    If auth_to_local is not configured in your cluster, those are actually not the only principals that can authenticate as your Hadoop users, because the following principals, if exist, will also become your HDFS, Alyce and Bob per the default mapping:

    hdfs/host123.your.domain@YOUR.DOMAIN => hdfs
    hdfs/clusterB@YOUR.DOMAIN => hdfs
    alyce/team2@YOUR.DOMAIN => Alyce
    alyce/something.else@YOUR.DOMAIN => Alyce
    bob/library@YOUR.DOMAIN => Bob
    bob/research@YOUR.DOMAIN => Bob
    

    … (very, very large list of possible combinations of second part of Kerberos principal and domain name) …

    hdfs/<anything>@YOUR.DOMAIN is HDFS
    alyce/<anything>@YOUR.DOMAIN is Alyce
    bob/<anything>@YOUR.DOMAIN is Bob
    

    For many regulatory bodies and auditing companies, this is a baseline security requirement for every user on the system to have only one unique identity. As we just learned, in Hadoop, by default, users de-facto can be identified with almost an infinite number of IDs. And this can be exploited by malicious users inside company to get access to sensitive data or fully take over control of the cluster.

    Let’s look at an example:

    First, user Bob with principal bob@LAB.LOCAL uploads a file secret.txt to his home directory in HDFS and ensures its protected by access lists:

    1

    Alyce can’t access Bob’s secret file, even if she knows where its located:

    2

    A new principal is created in Active Directory, bob/i.am.bad@DOMAIN.LOCAL whether by tricking AD admin, or by the admin himself:

    3

    Now the person who owns the new principal can access all Bob’s information, including his secret file. As you can see, Hadoop sees bob/i.am.bad as bob, and the newly uploaded file by this user becomes a legit Bob’s file:

    4

    The same can be done to any user of your cluster, including superusers. In a typical Active Directory setup, usernames and groups information is open for all domain users, so it will be easy for a malicious user inside the company to establish their targets and plan their actions.

    But in the end, some good news, too: HDFS Audit logs still track full principal names, so you can check if this took place in your cluster, and monitor this kind of activities until the issue is fixed.

    5

    I am still working on the best solution for this issue and a filter for auth_to_local that will work for most. I will publish it when it’s ready, but please feel free to contact me if you want to speed this up or work on this together. Currently I am at the point where I am going to try exclude use of second part of Kerberos principal completely - in modern world, and in majority of secure Hadoop clusters I am aware of, Active Directory is the provider of Kerberos, and second part of principals name is no more used by System Administrators and Identity/Access Management teams.

  • Myth about hard-coded 'hdfs' superuser in Hadoop

    I often hear about the hard-coded ‘hdfs’ superuser in Hadoop clusters, and various challenges around managing it in scenarios when there is more than one team in the same organization using Hadoop in their projects.

    I think it’s very important to mention that there is no hardcoded ‘hdfs’ superuser in Hadoop. Name Node just gives admin rights to the system user name which started its process. So if you are starting Name Node as root (please don’t do this), your superuser name will be ‘root’. If you are starting it as ‘namenode’, this will make ‘namenode’ user a superuser.

    Here’s what HDFS Permissions Guide says about this (quoting entire ‘Super-User’ section):

    The super-user is the user with the same identity as name node process itself. Loosely, if you started the name node, then you are the super-user. The super-user can do anything in that permissions checks never fail for the super-user. There is no persistent notion of who was the super-user; when the name node is started the process identity determines who is the super-user for now. The HDFS super-user does not have to be the super-user of the name node host, nor is it necessary that all clusters have the same super-user. Also, an experimenter running HDFS on a personal workstation, conveniently becomes that installation’s super-user without any configuration.

    In addition, the administrator my identify a distinguished group using a configuration parameter. If set, members of this group are also super-users.

    And that’s just HDFS admin. For other components of Hadoop ecosystem, they all have their own admin users, but some in default configurations will allow other components’ admin users manage them.

    I guess this myth exists because the default system user name used to start HDFS daemons by majority of automated Hadoop installations is ‘hdfs’.

    (and of course don’t forget about dfs.permissions.superusergroup and dfs.cluster.administrators)

  • DYI ATtiny85 dev board

  • Bought some infosec books to refresh my knowledge

  • DIY Fume Extractor

  • My next micro-controller project

  • The Real Grumpy Cat :)

  • Getting “avrdude: initialization failed, rc=-1″ with ATmega328 after pulled it from Arduino?

    I’ve spent the entire day solving this issue :) While all ATtiny85 were responsive and happy, both ATmega328 refused to work with avrdude throwing the error below while working perfectly with Arduino.

    avrdude_1

    The problem was that Arduino uses a 16Mhz external crystal as chip’s clock, and chip is pre-programmed to use it. Just add a 16 Mhz crystal with two 22 pF capacitors as at picture below:

    ATmega_crystal

    And microcontroller will respond to avrdude:

    avrdude_2

    As far as I know, microcontrollers can be instructed to use internal clock instead of external crystal, but I am yet to find out how to do this.

    In general, while crystal takes extra 2 pins off AVR, it enables microcontroller to run at a better (and more precise) frequency. In my small projects I plan to use ATtiny mostly with internal clock since it only has 8 pins. With ATmega, I probably will use a crystal more often, but I pulled it from Arduino and started all this AVR programming hobby because I wanted as much compact solutions as possible, with no to minimal extra stuff, so I still prefer more pins to a faster clock.

  • Using breadboard to program AVRs

  • DYI ICSP 6-pin to breadboard adapter

  • A nice remark on password complexity

    pwgen, my favorite Linux tool to generate random passwords that can be memorized:

    -s, –secure Generate completely random, hard-to-memorize passwords. These should only be used for machine passwords, since otherwise it’s almost guaranteed that users will simply write the password on a piece of paper taped to the monitor…

    I like to see how alternative thinking in Information Security community is emerging. Good that we started to realize people are not robots, and commands and programming will never work here. This is basic risk management, to consider ‘human elements’ in any program. And while I am sure there are people who will disagree and bring up some very good and solid arguments, I don’t understand why in world of Information Security, one of the most modern and fast-evolving professions, we are still trying to rely on ideas that are decades old, and never really worked since…

  • DYI Dual Microcontroller Programmer for AVR

  • First steps in electronics

  • Application Security without rose colored glasses

subscribe via RSS